Accelerating RLHF Training with Reward Variance Increase

29 May 2025

Papers citing "Accelerating RLHF Training with Reward Variance Increase"

30 / 30 papers shown

Title
DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization Gang Li Ming Lin Tomer Galanti Zhengzhong Tu Tianbao Yang 64 1 0 18 May 2025
Qwen3 Technical Report An Yang A. Li Baosong Yang Beichen Zhang Binyuan Hui ... Zekun Wang Zeyu Cui Zhenru Zhang Zhenhong Zhou Zihan Qiu LLMAG OSLM LRM 81 35 0 14 May 2025
Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning Yixuan Even Xu Yash Savani Fei Fang Zico Kolter OffRL 71 4 0 18 Apr 2025
GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models Jixiao Zhang Chunsheng Zuo LRM 70 14 0 13 Apr 2025
Understanding R1-Zero-Like Training: A Critical Perspective Zichen Liu Changyu Chen Wenjun Li Penghui Qi Tianyu Pang Chao Du Wee Sun Lee Min Lin OffRL LRM 122 108 0 26 Mar 2025
What Makes a Reward Model a Good Teacher? An Optimization Perspective Noam Razin Zixuan Wang Hubert Strauss Stanley Wei Jason D. Lee Sanjeev Arora 81 9 0 19 Mar 2025
DAPO: An Open-Source LLM Reinforcement Learning System at Scale Qiying Yu Zheng Zhang Ruofei Zhu Yufeng Yuan Xiaochen Zuo ... Ya Zhang Lin Yan Mu Qiao Yonghui Wu Mingxuan Wang OffRL LRM 131 131 0 18 Mar 2025
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI Daya Guo Dejian Yang Haowei Zhang Junxiao Song ... Shiyu Wang S. Yu Shunfeng Zhou Shuting Pan S.S. Li ReLM VLM OffRL AI4TS LRM 303 1,503 0 22 Jan 2025
The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models Yanjun Chen Dawei Zhu Yirong Sun Xinghao Chen Wei Zhang Xiaoyu Shen ALM 55 2 0 09 Oct 2024
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Haoxiang Wang Wei Xiong Tengyang Xie Han Zhao Tong Zhang 75 159 0 18 Jun 2024
Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs Rui Yang Ruomeng Ding Yong Lin Huan Zhang Tong Zhang 91 48 0 14 Jun 2024
Understanding the performance gap between online and offline alignment algorithms Yunhao Tang Daniel Guo Zeyu Zheng Daniele Calandriello Yuan Cao ... Rémi Munos Bernardo Avila-Pires Michal Valko Yong Cheng Will Dabney OffRL OnRL 79 68 0 14 May 2024
Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models Yi-Lin Tuan Xilun Chen Eric Michael Smith Louis Martin Soumya Batra Asli Celikyilmaz William Yang Wang Daniel M. Bikel 50 11 0 01 Apr 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James V. Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 128 250 0 20 Mar 2024
How do Large Language Models Navigate Conflicts between Honesty and Helpfulness? Ryan Liu T. Sumers Ishita Dasgupta Thomas Griffiths LLMAG 57 15 0 11 Feb 2024
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models Zhihong Shao Peiyi Wang Qihao Zhu Runxin Xu Jun-Mei Song ... Haowei Zhang Mingchuan Zhang Yiming Li Yu-Huan Wu Daya Guo ReLM LRM 94 953 0 05 Feb 2024
Secrets of RLHF in Large Language Models Part I: PPO Rui Zheng Shihan Dou Songyang Gao Yuan Hua Wei Shen ... Hang Yan Tao Gui Qi Zhang Xipeng Qiu Xuanjing Huang ALM OffRL 70 163 0 11 Jul 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 297 3,712 0 29 May 2023
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling Stella Biderman Hailey Schoelkopf Quentin G. Anthony Herbie Bradley Kyle O'Brien ... USVSN Sai Prashanth Edward Raff Aviya Skowron Lintang Sutawika Oskar van der Wal 81 1,231 0 03 Apr 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 832 13,788 0 15 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 938 12,840 0 27 Feb 2023
Scaling Laws for Reward Model Overoptimization Leo Gao John Schulman Jacob Hilton ALM 65 516 0 19 Oct 2022
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen ... Jack Clark Sam McCandlish C. Olah Benjamin Mann Jared Kaplan 224 2,457 0 12 Apr 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 726 12,525 0 04 Mar 2022
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 544 41,106 0 28 May 2020
Sample Efficient Policy Gradient Methods with Recursive Variance Reduction Pan Xu F. Gao Quanquan Gu 59 86 0 18 Sep 2019
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 434 1,664 0 18 Sep 2019
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 285 18,685 0 20 Jul 2017
Deep reinforcement learning from human preferences Paul Christiano Jan Leike Tom B. Brown Miljan Martic Shane Legg Dario Amodei 105 3,243 0 12 Jun 2017
High-Dimensional Continuous Control Using Generalized Advantage Estimation John Schulman Philipp Moritz Sergey Levine Michael I. Jordan Pieter Abbeel OffRL 60 3,368 0 08 Jun 2015