A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future

12 April 2025

Papers citing "A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future"

47 / 97 papers shown

Title
Advancing LLM Reasoning Generalists with Preference Trees Lifan Yuan Ganqu Cui Hanbin Wang Ning Ding Xingyao Wang ... Zhenghao Liu Bowen Zhou Hao Peng Zhiyuan Liu Maosong Sun LRM 126 122 0 02 Apr 2024
Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model Qi Gou Cam-Tu Nguyen 113 13 0 28 Mar 2024
sDPO: Don't Use Your Data All at Once Dahyun Kim Yungi Kim Wonho Song Hyeonwoo Kim Yunsu Kim Sanghoon Kim Chanjun Park 73 35 0 28 Mar 2024
Disentangling Length from Quality in Direct Preference Optimization Ryan Park Rafael Rafailov Stefano Ermon Chelsea Finn ALM 98 145 0 28 Mar 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James V. Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 180 261 0 20 Mar 2024
ORPO: Monolithic Preference Optimization without Reference Model Jiwoo Hong Noah Lee James Thorne OSLM 100 266 0 12 Mar 2024
Provably Robust DPO: Aligning Language Models with Noisy Feedback Sayak Ray Chowdhury Anush Kini Nagarajan Natarajan 80 70 0 01 Mar 2024
Noise Contrastive Alignment of Language Models with Explicit Rewards Huayu Chen Guande He Lifan Yuan Ganqu Cui Hang Su Jun Zhu 103 56 0 08 Feb 2024
MusicRL: Aligning Music Generation to Human Preferences Geoffrey Cideron Sertan Girgin Mauro Verzetti Damien Vincent Matej Kastelic ... Olivier Pietquin Matthieu Geist Léonard Hussenot Neil Zeghidour A. Agostinelli 67 22 0 06 Feb 2024
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models Zhihong Shao Peiyi Wang Qihao Zhu Runxin Xu Jun-Mei Song ... Haowei Zhang Mingchuan Zhang Yiming Li Yu-Huan Wu Daya Guo ReLM LRM 169 1,288 0 05 Feb 2024
Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback Yifu Yuan Jianye Hao Yi-An Ma Zibin Dong Hebin Liang Jinyi Liu Zhixin Feng Kai-Wen Zhao Yan Zheng OffRL ALM 64 16 0 04 Feb 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 370 338 0 18 Jan 2024
LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin Shihan Dou Enyu Zhou Yan Liu Songyang Gao Jun Zhao ... Jiang Zhu Rui Zheng Tao Gui Qi Zhang Xuanjing Huang CLL MoE KELM 83 36 0 15 Dec 2023
Nash Learning from Human Feedback Rémi Munos Michal Valko Daniele Calandriello M. G. Azar Mark Rowland ... Nikola Momchev Olivier Bachem D. Mankowitz Doina Precup Bilal Piot 116 147 0 01 Dec 2023
Towards Understanding Sycophancy in Language Models Mrinank Sharma Meg Tong Tomasz Korbak David Duvenaud Amanda Askell ... Oliver Rausch Nicholas Schiefer Da Yan Miranda Zhang Ethan Perez 356 244 0 20 Oct 2023
SALMON: Self-Alignment with Instructable Reward Models Zhiqing Sun Songlin Yang Hongxin Zhang Qinhong Zhou Zhenfang Chen David D. Cox Yiming Yang Chuang Gan ALM SyDa 103 41 0 09 Oct 2023
Reward Model Ensembles Help Mitigate Overoptimization Thomas Coste Usman Anwar Robert Kirk David M. Krueger NoLa ALM 90 138 0 04 Oct 2023
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct Haipeng Luo Qingfeng Sun Can Xu Pu Zhao Jian-Guang Lou ... Xiubo Geng Qingwei Lin Shifeng Chen Yansong Tang Dongmei Zhang LRM OSLM 262 467 0 18 Aug 2023
Reinforced Self-Training (ReST) for Language Modeling Çağlar Gülçehre T. Paine S. Srinivasan Ksenia Konyushkova L. Weerts ... Chenjie Gu Wolfgang Macherey Arnaud Doucet Orhan Firat Nando de Freitas OffRL 125 309 0 17 Aug 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 455 4,444 0 09 Jun 2023
Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards Alexandre Ramé Guillaume Couairon Mustafa Shukor Corentin Dancette Jean-Baptiste Gaya Laure Soulier Matthieu Cord MoMe 103 157 0 07 Jun 2023
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion Dongfu Jiang Xiang Ren Bill Yuchen Lin ELM 86 333 0 05 Jun 2023
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training Zeqiu Wu Yushi Hu Weijia Shi Nouha Dziri Alane Suhr Prithviraj Ammanabrolu Noah A. Smith Mari Ostendorf Hannaneh Hajishirzi ALM 153 335 0 02 Jun 2023
Let's Verify Step by Step Hunter Lightman V. Kosaraju Yura Burda Harrison Edwards Bowen Baker Teddy Lee Jan Leike John Schulman Ilya Sutskever K. Cobbe ALM OffRL LRM 198 1,240 0 31 May 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 389 4,169 0 29 May 2023
HuatuoGPT, towards Taming Language Model to Be a Doctor Hongbo Zhang Junying Chen Feng Jiang Fei Yu Zhihong Chen ... Zhiyi Zhang Qingying Xiao Xiang Wan Benyou Wang Haizhou Li LM&MA AI4MH ELM 86 205 0 24 May 2023
Aligning Large Language Models through Synthetic Feedback Sungdong Kim Sanghwan Bae Jamin Shin Soyoung Kang Donghyun Kwak Kang Min Yoo Minjoon Seo ALM SyDa 135 70 0 23 May 2023
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback Yann Dubois Xuechen Li Rohan Taori Tianyi Zhang Ishaan Gulrajani Jimmy Ba Carlos Guestrin Percy Liang Tatsunori B. Hashimoto ALM 137 608 0 22 May 2023
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment Hanze Dong Wei Xiong Deepanshu Goyal Yihan Zhang Winnie Chow Boyao Wang Shizhe Diao Jipeng Zhang Kashun Shum Tong Zhang ALM 98 468 0 13 Apr 2023
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation Jiazheng Xu Xiao Liu Yuchen Wu Yuxuan Tong Qinkai Li Ming Ding Jie Tang Yuxiao Dong 145 408 0 12 Apr 2023
RRHF: Rank Responses to Align Language Models with Human Feedback without tears Zheng Yuan Hongyi Yuan Chuanqi Tan Wei Wang Songfang Huang Feiran Huang ALM 167 384 0 11 Apr 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,761 0 15 Mar 2023
Aligning Text-to-Image Models using Human Feedback Kimin Lee Hao Liu Moonkyung Ryu Olivia Watkins Yuqing Du Craig Boutilier Pieter Abbeel Mohammad Ghavamzadeh S. Gu EGVM 112 285 0 23 Feb 2023
Few-Shot Preference Learning for Human-in-the-Loop RL Joey Hejna Dorsa Sadigh OffRL 113 100 0 06 Dec 2022
Defining and Characterizing Reward Hacking Joar Skalse Nikolaus H. R. Howe Dmitrii Krasheninnikov David M. Krueger 117 61 0 27 Sep 2022
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning Antonia Creswell Murray Shanahan I. Higgins ReLM LRM 112 364 0 19 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 891 13,228 0 04 Mar 2022
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 362 4,598 0 27 Oct 2021
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training Kimin Lee Laura M. Smith Pieter Abbeel OffRL 65 289 0 09 Jun 2021
A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges Christian Arzate Cruz Takeo Igarashi OffRL 54 96 0 27 May 2021
Learning to summarize from human feedback Nisan Stiennon Long Ouyang Jeff Wu Daniel M. Ziegler Ryan J. Lowe Chelsea Voss Alec Radford Dario Amodei Paul Christiano ALM 262 2,192 0 02 Sep 2020
A Survey of Deep Active Learning Pengzhen Ren Yun Xiao Xiaojun Chang Po-Yao (Bernie) Huang Zhihui Li Brij B. Gupta Xiaojiang Chen Xin Wang 114 1,153 0 30 Aug 2020
Active Preference-Based Gaussian Process Regression for Reward Learning Erdem Biyik Nicolas Huynh Mykel J. Kochenderfer Dorsa Sadigh GP 81 109 0 06 May 2020
Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm David Silver Thomas Hubert Julian Schrittwieser Ioannis Antonoglou Matthew Lai ... D. Kumaran T. Graepel Timothy Lillicrap Karen Simonyan Demis Hassabis 159 1,785 0 05 Dec 2017
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 571 19,315 0 20 Jul 2017
Deep reinforcement learning from human preferences Paul Christiano Jan Leike Tom B. Brown Miljan Martic Shane Legg Dario Amodei 218 3,377 0 12 Jun 2017
Playing Atari with Deep Reinforcement Learning Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra Martin Riedmiller 132 12,272 0 19 Dec 2013