ODIN: Disentangled Reward Mitigates Hacking in RLHF

ODIN: Disentangled Reward Mitigates Hacking in RLHF

11 February 2024

Heng-Chiao Huang

Bryan Catanzaro

Papers citing "ODIN: Disentangled Reward Mitigates Hacking in RLHF"

17 / 17 papers shown

Title
On the Robustness of Reward Models for Language Model Alignment Jiwoo Hong Noah Lee Eunki Kim Guijin Son Woojin Chung Aman Gupta Shao Tang James Thorne 29 0 0 12 May 2025
Energy-Based Reward Models for Robust Language Model Alignment Anamika Lochab Ruqi Zhang 135 0 0 17 Apr 2025
Adversarial Training of Reward Models Alexander Bukharin Haifeng Qian Shengyang Sun Adithya Renduchintala Soumye Singhal Zhongqi Wang Oleksii Kuchaiev Olivier Delalleau T. Zhao AAML 32 0 0 08 Apr 2025
Information-Theoretic Reward Decomposition for Generalizable RLHF Liyuan Mao Haoran Xu Amy Zhang Weinan Zhang Chenjia Bai 33 0 0 08 Apr 2025
Probabilistic Uncertain Reward Model Wangtao Sun Xiang Cheng Xing Yu Haotian Xu Zhao Yang Shizhu He Jun Zhao Kang Liu 58 0 0 28 Mar 2025
The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking Yuchun Miao Sen Zhang Liang Ding Yuqi Zhang L. Zhang Dacheng Tao 81 3 0 31 Jan 2025
Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment Chaoqi Wang Zhuokai Zhao Yibo Jiang Zhaorun Chen Chen Zhu ... Jiayi Liu Lizhu Zhang Xiangjun Fan Hao Ma Sinong Wang 77 3 0 17 Jan 2025
Self-Generated Critiques Boost Reward Modeling for Language Models Yue Yu Zhengxing Chen Aston Zhang L Tan Chenguang Zhu ... Suchin Gururangan Chao-Yue Zhang Melanie Kambadur Dhruv Mahajan Rui Hou LRM ALM 94 16 0 25 Nov 2024
Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown Xingzhou Lou Dong Yan Wei Shen Yuzi Yan Jian Xie Junge Zhang 45 22 0 01 Oct 2024
RRM: Robust Reward Model Training Mitigates Reward Hacking Tianqi Liu Wei Xiong Jie Jessie Ren Lichang Chen Junru Wu ... Yuan Liu Bilal Piot Abe Ittycheriah Aviral Kumar Mohammad Saleh AAML 56 13 0 20 Sep 2024
From Lists to Emojis: How Format Bias Affects Model Alignment Xuanchang Zhang Wei Xiong Lichang Chen Tianyi Zhou Heng Huang Tong Zhang ALM 35 11 0 18 Sep 2024
On the Transformations across Reward Model, Parameter Update, and In-Context Prompt Deng Cai Huayang Li Tingchen Fu Siheng Li Weiwen Xu ... Leyang Cui Yan Wang Lemao Liu Taro Watanabe Shuming Shi KELM 30 2 0 24 Jun 2024
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators Yann Dubois Balázs Galambosi Percy Liang Tatsunori Hashimoto ALM 55 321 0 06 Apr 2024
WARM: On the Benefits of Weight Averaged Reward Models Alexandre Ramé Nino Vieillard Léonard Hussenot Robert Dadashi Geoffrey Cideron Olivier Bachem Johan Ferret 111 93 0 22 Jan 2024
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,953 0 04 Mar 2022
Unsolved Problems in ML Safety Dan Hendrycks Nicholas Carlini John Schulman Jacob Steinhardt 186 273 0 28 Sep 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 280 1,595 0 18 Sep 2019