Policy-labeled Preference Learning: Is Preference Enough for RLHF?

v1v2 (latest)

Policy-labeled Preference Learning: Is Preference Enough for RLHF?

6 May 2025

ArXiv (abs)PDF HTML

Papers citing "Policy-labeled Preference Learning: Is Preference Enough for RLHF?"

16 / 16 papers shown

Title
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models Michael Noukhovitch Shengyi Huang Sophie Xhonneux Arian Hosseini Rishabh Agarwal Rameswar Panda OffRL 140 11 0 23 Oct 2024
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer Zhihan Liu Miao Lu Shenao Zhang Boyi Liu Hongyi Guo Yingxiang Yang Jose H. Blanchet Zhaoran Wang 117 62 0 26 May 2024
DPO Meets PPO: Reinforced Token Optimization for RLHF Han Zhong Zikang Shan Guhao Feng Wei Xiong Xinle Cheng Li Zhao Di He Jiang Bian Liwei Wang 125 72 0 29 Apr 2024
Token-level Direct Preference Optimization Yongcheng Zeng Guoqing Liu Weiyu Ma Ning Yang Haifeng Zhang Jun Wang 97 63 0 18 Apr 2024
Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences Andi Nika Debmalya Mandal Parameswaran Kamalaruban Georgios Tzannetos Goran Radanović Adish Singla 49 14 0 04 Mar 2024
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model Chen Ye Wei Xiong Yuheng Zhang Nan Jiang Tong Zhang OffRL 81 15 0 11 Feb 2024
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 389 4,139 0 29 May 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,699 0 15 Mar 2023
Dichotomy of Control: Separating What You Can Control from What You Cannot Mengjiao Yang Dale Schuurmans Pieter Abbeel Ofir Nachum OffRL 71 44 0 24 Oct 2022
Offline Reinforcement Learning with Implicit Q-Learning Ilya Kostrikov Ashvin Nair Sergey Levine OffRL 301 924 0 12 Oct 2021
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training Kimin Lee Laura M. Smith Pieter Abbeel OffRL 65 288 0 09 Jun 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 474 1,766 0 18 Sep 2019
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor Tuomas Haarnoja Aurick Zhou Pieter Abbeel Sergey Levine 317 8,406 0 04 Jan 2018
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 541 19,265 0 20 Jul 2017
Deep reinforcement learning from human preferences Paul Christiano Jan Leike Tom B. Brown Miljan Martic Shane Legg Dario Amodei 218 3,365 0 12 Jun 2017
High-Dimensional Continuous Control Using Generalized Advantage Estimation John Schulman Philipp Moritz Sergey Levine Michael I. Jordan Pieter Abbeel OffRL 129 3,438 0 08 Jun 2015