Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs

18 March 2025

Eric Thibodeau-Laufer

Papers citing "Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs"

44 / 44 papers shown

Title
Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoning Zhaohui Yang Shilei Jiang Chen Hu Linjing Li Shihong Deng D. Jiang OffRL 75 0 0 20 May 2025
R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning Yi-Fan Zhang Xingyu Lu X. Hu Chaoyou Fu Bin Wen ... Jianfei Chen Fan Yang Zheng Zhang Yan Li Liang Wang OffRL LRM 88 6 0 05 May 2025
A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility Andreas Hochlehnert Hardik Bhatnagar Vishaal Udandarao Samuel Albanie Ameya Prabhu Matthias Bethge ReLM ALM LRM 172 19 0 09 Apr 2025
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI Daya Guo Dejian Yang Haowei Zhang Junxiao Song ... Shiyu Wang S. Yu Shunfeng Zhou Shuting Pan S.S. Li ReLM VLM OffRL AI4TS LRM 359 1,641 0 22 Jan 2025
Kimi k1.5: Scaling Reinforcement Learning with LLMs Kimi Team Angang Du Bofei Gao Bowei Xing Changjiu Jiang ... Zihao Huang Ziyao Xu Zhiyong Yang Zonghan Yang Zongyu Lin OffRL ALM AI4TS VLM LRM 233 272 0 22 Jan 2025
Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion Yannis Flet-Berliac Nathan Grinsztajn Florian Strub Bill Wu Eugene Choi ... Arash Ahmadian Yash Chandak M. G. Azar Olivier Pietquin Matthieu Geist OffRL 133 8 0 17 Jan 2025
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models Michael Noukhovitch Shengyi Huang Sophie Xhonneux Arian Hosseini Rishabh Agarwal Rameswar Panda OffRL 125 11 0 23 Oct 2024
Generative Reward Models Dakota Mahan Duy Phung Rafael Rafailov Chase Blagden Nathan Lile Louis Castricato Jan-Philipp Fränken Chelsea Finn Alon Albalak VLM SyDa OffRL 54 40 0 02 Oct 2024
Averaging log-likelihoods in direct alignment Nathan Grinsztajn Yannis Flet-Berliac M. G. Azar Florian Strub Bill Wu ... Chris Cremer Arash Ahmadian Yash Chandak Olivier Pietquin Matthieu Geist MoMe 63 6 0 27 Jun 2024
Improve Mathematical Reasoning in Language Models by Automated Process Supervision Liangchen Luo Yinxiao Liu Rosanne Liu Samrat Phatale Harsh Lara ... Lei Shu Yun Zhu Lei Meng Jiao Sun Abhinav Rastogi LRM 83 171 0 05 Jun 2024
Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data Fahim Tajwar Anika Singh Archit Sharma Rafael Rafailov Jeff Schneider Tengyang Xie Stefano Ermon Chelsea Finn Aviral Kumar 78 124 0 22 Apr 2024
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning Ruiqi Zhang Licong Lin Yu Bai Song Mei MU 112 169 0 08 Apr 2024
Human Alignment of Large Language Models through Online Preference Optimisation Daniele Calandriello Daniel Guo Rémi Munos Mark Rowland Yunhao Tang ... Michal Valko Tianqi Liu Rishabh Joshi Zeyu Zheng Bilal Piot 80 65 0 13 Mar 2024
Direct Language Model Alignment from Online AI Feedback Shangmin Guo Biao Zhang Tianlin Liu Tianqi Liu Misha Khalman ... Thomas Mesnard Yao-Min Zhao Bilal Piot Johan Ferret Mathieu Blondel ALM 63 153 0 07 Feb 2024
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models Zhihong Shao Peiyi Wang Qihao Zhu Runxin Xu Jun-Mei Song ... Haowei Zhang Mingchuan Zhang Yiming Li Yu-Huan Wu Daya Guo ReLM LRM 115 1,107 0 05 Feb 2024
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models Avi Singh John D. Co-Reyes Rishabh Agarwal Ankesh Anand Piyush Patil ... Yamini Bansal Ethan Dyer Behnam Neyshabur Jascha Narain Sohl-Dickstein Noah Fiedel ALM LRM ReLM SyDa 195 183 0 11 Dec 2023
A General Theoretical Paradigm to Understand Learning from Human Preferences M. G. Azar Mark Rowland Bilal Piot Daniel Guo Daniele Calandriello Michal Valko Rémi Munos 167 616 0 18 Oct 2023
Efficient Memory Management for Large Language Model Serving with PagedAttention Woosuk Kwon Zhuohan Li Siyuan Zhuang Ying Sheng Lianmin Zheng Cody Hao Yu Joseph E. Gonzalez Haotong Zhang Ion Stoica VLM 158 2,196 0 12 Sep 2023
Reinforced Self-Training (ReST) for Language Modeling Çağlar Gülçehre T. Paine S. Srinivasan Ksenia Konyushkova L. Weerts ... Chenjie Gu Wolfgang Macherey Arnaud Doucet Orhan Firat Nando de Freitas OffRL 113 298 0 17 Aug 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 322 4,298 0 09 Jun 2023
Let's Verify Step by Step Hunter Lightman V. Kosaraju Yura Burda Harrison Edwards Bowen Baker Teddy Lee Jan Leike John Schulman Ilya Sutskever K. Cobbe ALM OffRL LRM 173 1,140 0 31 May 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 353 3,935 0 29 May 2023
Solving Quantitative Reasoning Problems with Language Models Aitor Lewkowycz Anders Andreassen David Dohan Ethan Dyer Henryk Michalewski ... Theo Gutman-Solo Yuhuai Wu Behnam Neyshabur Guy Gur-Ari Vedant Misra ReLM ELM LRM 163 833 0 29 Jun 2022
STaR: Bootstrapping Reasoning With Reasoning E. Zelikman Yuhuai Wu Jesse Mu Noah D. Goodman ReLM LRM 123 486 0 28 Mar 2022
Self-Consistency Improves Chain of Thought Reasoning in Language Models Xuezhi Wang Jason W. Wei Dale Schuurmans Quoc Le Ed H. Chi Sharan Narang Aakanksha Chowdhery Denny Zhou ReLM BDL LRM AI4CE 511 3,618 0 21 Mar 2022
Simulating Bandit Learning from User Feedback for Extractive Question Answering Ge Gao Eunsol Choi Yoav Artzi 57 14 0 18 Mar 2022
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 267 4,397 0 27 Oct 2021
A general class of surrogate functions for stable and efficient reinforcement learning Sharan Vaswani Olivier Bachem Simone Totaro Robert Mueller Shivam Garg Matthieu Geist Marlos C. Machado Pablo Samuel Castro Nicolas Le Roux OffRL 57 16 0 12 Aug 2021
Measuring Mathematical Problem Solving With the MATH Dataset Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang D. Song Jacob Steinhardt ReLM FaML 163 2,255 0 05 Mar 2021
Text Generation by Learning from Demonstrations Richard Yuanzhe Pang He He OffRL 58 80 0 16 Sep 2020
Beyond variance reduction: Understanding the true impact of baselines on policy optimization Wesley Chung Valentin Thomas Marlos C. Machado Nicolas Le Roux OffRL 58 22 0 31 Aug 2020
An operator view of policy gradient methods Dibya Ghosh Marlos C. Machado Nicolas Le Roux OffRL 46 27 0 19 Jun 2020
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems Sergey Levine Aviral Kumar George Tucker Justin Fu OffRL GP 549 2,022 0 04 May 2020
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 458 1,725 0 18 Sep 2019
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation Dmitry Kalashnikov A. Irpan P. Pastor Julian Ibarz Alexander Herzog ... Deirdre Quillen E. Holly Mrinal Kalakrishnan Vincent Vanhoucke Sergey Levine 113 1,460 0 27 Jun 2018
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost Noam M. Shazeer Mitchell Stern ODL 74 1,047 0 11 Apr 2018
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures L. Espeholt Hubert Soyer Rémi Munos Karen Simonyan Volodymyr Mnih ... Vlad Firoiu Tim Harley Iain Dunning Shane Legg Koray Kavukcuoglu 208 1,598 0 05 Feb 2018
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 473 19,006 0 20 Jul 2017
Efficient iterative policy optimization Nicolas Le Roux 35 6 0 28 Dec 2016
Variance-based regularization with convex objectives John C. Duchi Hongseok Namkoong 68 349 0 08 Oct 2016
Tighter bounds lead to improved classifiers Nicolas Le Roux 25 8 0 29 Jun 2016
Safe and Efficient Off-Policy Reinforcement Learning Rémi Munos T. Stepleton Anna Harutyunyan Marc G. Bellemare OffRL 138 615 0 08 Jun 2016
Asynchronous Methods for Deep Reinforcement Learning Volodymyr Mnih Adria Puigdomenech Badia M. Berk Mirza Alex Graves Timothy Lillicrap Tim Harley David Silver Koray Kavukcuoglu 191 8,850 0 04 Feb 2016
The Arcade Learning Environment: An Evaluation Platform for General Agents Marc G. Bellemare Yavar Naddaf J. Veness Michael Bowling 109 3,004 0 19 Jul 2012