Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models

v1v2 (latest)

Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models

8 January 2025

Roberto-Rafael Maura-Rivero

Francesco Visin

ArXiv (abs)PDF HTML

Papers citing "Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models"

18 / 18 papers shown

Title
Robust Multi-Objective Controlled Decoding of Large Language Models Seongho Son William Bankes Sangwoong Yoon Shyam Sundhar Ramesh Xiaohang Tang Ilija Bogunovic 129 2 0 11 Mar 2025
Axioms for AI Alignment from Human Feedback Luise Ge Daniel Halpern Evi Micha Ariel D. Procaccia Itai Shapira Yevgeniy Vorobeychik Junlin Wu 77 24 0 23 May 2024
WARM: On the Benefits of Weight Averaged Reward Models Alexandre Ramé Nino Vieillard Léonard Hussenot Robert Dadashi Geoffrey Cideron Olivier Bachem Johan Ferret 189 104 0 22 Jan 2024
A Minimaximalist Approach to Reinforcement Learning from Human Feedback Gokul Swamy Christoph Dann Rahul Kidambi Zhiwei Steven Wu Alekh Agarwal OffRL 132 112 0 08 Jan 2024
Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking Jacob Eisenstein Chirag Nagpal Alekh Agarwal Ahmad Beirami Alex DÁmour ... Katherine Heller Stephen Pfohl Deepak Ramachandran Peter Shaw Jonathan Berant 100 100 0 14 Dec 2023
Evaluating and Mitigating Discrimination in Language Model Decisions Alex Tamkin Amanda Askell Liane Lovitt Esin Durmus Nicholas Joseph Shauna Kravec Karina Nguyen Jared Kaplan Deep Ganguli 97 76 0 06 Dec 2023
The Consensus Game: Language Model Generation via Equilibrium Search Athul Paul Jacob Songlin Yang Gabriele Farina Jacob Andreas 95 23 0 13 Oct 2023
Confronting Reward Model Overoptimization with Constrained RLHF Ted Moskovitz Aaditya K. Singh DJ Strouse Tuomas Sandholm Ruslan Salakhutdinov Anca D. Dragan Stephen Marcus McAleer 103 55 0 06 Oct 2023
A Long Way to Go: Investigating Length Correlations in RLHF Prasann Singhal Tanya Goyal Jiacheng Xu Greg Durrett 160 161 0 05 Oct 2023
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training Zeqiu Wu Yushi Hu Weijia Shi Nouha Dziri Alane Suhr Prithviraj Ammanabrolu Noah A. Smith Mari Ostendorf Hannaneh Hajishirzi ALM 168 336 0 02 Jun 2023
Improving alignment of dialogue agents via targeted human judgements Amelia Glaese Nat McAleese Maja Trkebacz John Aslanides Vlad Firoiu ... John F. J. Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks G. Irving ALM AAML 326 538 0 28 Sep 2022
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen ... Jack Clark Sam McCandlish C. Olah Benjamin Mann Jared Kaplan 304 2,632 0 12 Apr 2022
Ethical and social risks of harm from Language Models Laura Weidinger John F. J. Mellor Maribeth Rauh Conor Griffin J. Uesato ... Lisa Anne Hendricks William S. Isaac Sean Legassick G. Irving Iason Gabriel PILM 230 1,045 0 08 Dec 2021
Attention Flows are Shapley Value Explanations Kawin Ethayarajh Dan Jurafsky FAtt TDI 83 35 0 31 May 2021
Learning to summarize from human feedback Nisan Stiennon Long Ouyang Jeff Wu Daniel M. Ziegler Ryan J. Lowe Chelsea Voss Alec Radford Dario Amodei Paul Christiano ALM 314 2,195 0 02 Sep 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 1.2K 42,753 0 28 May 2020
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 709 19,377 0 20 Jul 2017
Deep reinforcement learning from human preferences Paul Christiano Jan Leike Tom B. Brown Miljan Martic Shane Legg Dario Amodei 246 3,389 0 12 Jun 2017