On The Fragility of Learned Reward Functions

9 January 2023

Adam Gleave

Papers citing "On The Fragility of Learned Reward Functions"

21 / 21 papers shown

Title
Establishing Reliability Metrics for Reward Models in Large Language Models Yizhou Chen Yawen Liu Xuesi Wang Qingtao Yu Guangda Huzhang Anxiang Zeng Han Yu Zhiming Zhou 30 0 0 21 Apr 2025
FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions Daniel Marta Simon Holk Miguel Vasco Jens Lundell Timon Homberger F. L. Busch Olov Andersson Danica Kragic Iolanda Leite 38 0 0 14 Apr 2025
Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback Wei Shen Guanlin Liu Zheng Wu Ruofei Zhu Qingping Yang Chao Xin Yu Yue Lin Yan 86 8 0 28 Mar 2025
Policy Filtration in RLHF to Fine-Tune LLM for Code Generation Wei Shen Chuheng Zhang OffRL 41 6 0 11 Sep 2024
Listwise Reward Estimation for Offline Preference-based Reinforcement Learning Heewoong Choi Sangwon Jung Hongjoon Ahn Taesup Moon OffRL 39 2 0 08 Aug 2024
HAF-RM: A Hybrid Alignment Framework for Reward Model Training Shujun Liu Xiaoyu Shen Yuhang Lai Siyuan Wang Shengbin Yue Zengfeng Huang Xuanjing Huang Zhongyu Wei 31 1 0 04 Jul 2024
Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence Junru Lu Jiazheng Li Siyu An Meng Zhao Yulan He Di Yin Xing Sun 44 14 0 16 Jun 2024
Dishonesty in Helpful and Harmless Alignment Youcheng Huang Jingkun Tang Duanyu Feng Zheng-Wei Zhang Wenqiang Lei Jiancheng Lv Anthony G. Cohn LLMSV 46 3 0 04 Jun 2024
Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents San Kim Gary Geunbae Lee AAML 40 3 0 21 May 2024
Learning to Watermark LLM-generated Text via Reinforcement Learning Xiaojun Xu Yuanshun Yao Yang Liu 21 10 0 13 Mar 2024
Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking Cassidy Laidlaw Shivam Singhal Anca Dragan AAML 24 11 0 05 Mar 2024
Generalizing Reward Modeling for Out-of-Distribution Preference Learning Chen Jia 41 2 0 22 Feb 2024
WARM: On the Benefits of Weight Averaged Reward Models Alexandre Ramé Nino Vieillard Léonard Hussenot Robert Dadashi Geoffrey Cideron Olivier Bachem Johan Ferret 116 93 0 22 Jan 2024
Secrets of RLHF in Large Language Models Part II: Reward Modeling Bing Wang Rui Zheng Luyao Chen Yan Liu Shihan Dou ... Qi Zhang Xipeng Qiu Xuanjing Huang Zuxuan Wu Yuanyuan Jiang ALM 36 93 0 11 Jan 2024
FoMo Rewards: Can we cast foundation models as reward functions? Ekdeep Singh Lubana Johann Brehmer P. D. Haan Taco S. Cohen OffRL LRM 48 2 0 06 Dec 2023
Improving Generalization of Alignment with Human Preferences through Group Invariant Learning Rui Zheng Wei Shen Yuan Hua Wenbin Lai Shihan Dou ... Xiao Wang Haoran Huang Tao Gui Qi Zhang Xuanjing Huang 56 14 0 18 Oct 2023
Compositional preference models for aligning LMs Dongyoung Go Tomasz Korbak Germán Kruszewski Jos Rozen Marc Dymetman 23 15 0 17 Oct 2023
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback Stephen Casper Xander Davies Claudia Shi T. Gilbert Jérémy Scheurer ... Erdem Biyik Anca Dragan David M. Krueger Dorsa Sadigh Dylan Hadfield-Menell ALM OffRL 47 473 0 27 Jul 2023
Defining and Characterizing Reward Hacking Joar Skalse Nikolaus H. R. Howe Dmitrii Krasheninnikov David M. Krueger 59 55 0 27 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 319 11,953 0 04 Mar 2022
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 280 1,595 0 18 Sep 2019