Vanishing Gradients in Reinforcement Finetuning of Language Models

Vanishing Gradients in Reinforcement Finetuning of Language Models

31 October 2023

Preetum Nakkiran

Papers citing "Vanishing Gradients in Reinforcement Finetuning of Language Models"

13 / 13 papers shown

Title
Preference learning made easy: Everything should be understood through win rate Lily H. Zhang Rajesh Ranganath 85 0 0 14 Feb 2025
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization Noam Razin Sadhika Malladi Adithya Bhaskar Danqi Chen Sanjeev Arora Boris Hanin 91 14 0 11 Oct 2024
Imitating Language via Scalable Inverse Reinforcement Learning Markus Wulfmeier Michael Bloesch Nino Vieillard Arun Ahuja Jorg Bornschein ... Jost Tobias Springenberg Nikola Momchev Olivier Bachem Matthieu Geist Martin Riedmiller 36 9 0 02 Sep 2024
Discovering Preference Optimization Algorithms with and for Large Language Models Chris Xiaoxuan Lu Samuel Holt Claudio Fanconi Alex J. Chan Jakob Foerster M. Schaar R. T. Lange OffRL 37 15 0 12 Jun 2024
Preference Learning Algorithms Do Not Learn Preference Rankings Angelica Chen Sadhika Malladi Lily H. Zhang Xinyi Chen Qiuyi Zhang Rajesh Ranganath Kyunghyun Cho 30 23 0 29 May 2024
Dense Reward for Free in Reinforcement Learning from Human Feedback Alex J. Chan Hao Sun Samuel Holt M. Schaar 18 31 0 01 Feb 2024
WARM: On the Benefits of Weight Averaged Reward Models Alexandre Ramé Nino Vieillard Léonard Hussenot Robert Dadashi Geoffrey Cideron Olivier Bachem Johan Ferret 114 93 0 22 Jan 2024
Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization Spencer Frei Gal Vardi Peter L. Bartlett Nathan Srebro 30 22 0 02 Mar 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,953 0 04 Mar 2022
Continuous vs. Discrete Optimization of Deep Neural Networks Omer Elkabetz Nadav Cohen 68 44 0 14 Jul 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 280 1,595 0 18 Sep 2019
Deep Reinforcement Learning for Dialogue Generation Jiwei Li Will Monroe Alan Ritter Michel Galley Jianfeng Gao Dan Jurafsky 214 1,327 0 05 Jun 2016
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 175 3,510 0 10 Jun 2015