DPO Meets PPO: Reinforced Token Optimization for RLHF

29 April 2024

Papers citing "DPO Meets PPO: Reinforced Token Optimization for RLHF"

50 / 50 papers shown

Title
Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans Yansheng Qiu Li Xiao Zhaopan Xu Pengfei Zhou Zheng Wang Kaipeng Zhang ELM LRM 19 0 0 16 May 2025
Policy-labeled Preference Learning: Is Preference Enough for RLHF? Taehyun Cho Seokhun Ju Seungyub Han Dohyeong Kim Kyungjae Lee Jungwoo Lee OffRL 29 0 0 06 May 2025
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL Jiarui Yao Yifan Hao Hanning Zhang Hanze Dong Wei Xiong Nan Jiang Tong Zhang LRM 62 0 0 05 May 2025
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji Yanqiu Wu Zhibin Wu Shoujin Wang Jian Yang Mark Dras Usman Naseem 41 1 0 05 May 2025
Learning Explainable Dense Reward Shapes via Bayesian Optimization Ryan Koo Ian Yang Vipul Raheja Mingyi Hong Kwang-Sung Jun Dongyeop Kang 31 0 0 22 Apr 2025
Kongzi: A Historical Large Language Model with Fact Enhancement Jiashu Yang Ningning Wang Yian Zhao Chaoran Feng Junjia Du Hao Pang Zhirui Fang Xuxin Cheng HILM ALM LRM 41 0 0 13 Apr 2025
Supervised Optimism Correction: Be Confident When LLMs Are Sure Jingyang Zhang Rushuai Yang Shunyu Liu Ting-En Lin Fei Huang Yi Chen Yong Li Dacheng Tao OffRL 29 0 0 10 Apr 2025
Entropy-Based Adaptive Weighting for Self-Training Xiaoxuan Wang Yihe Deng Mingyu Derek Ma Wei Wang LRM 52 0 0 31 Mar 2025
Controlling Large Language Model with Latent Actions Chengxing Jia Ziniu Li Pengyuan Wang Yi-Chen Li Zhenyu Hou Yuxiao Dong Y. Yu 58 0 0 27 Mar 2025
MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modeling Zhaopeng Feng Jiahan Ren Jiayuan Su Jiamei Zheng Zhihang Tang Hongwei Wang Zuozhu Liu LRM 65 1 0 15 Mar 2025
Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment Wen Yang Junhong Wu Chen Wang Chengqing Zong J.N. Zhang 75 1 0 06 Mar 2025
DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models Ruizhe Chen Wenhao Chai Zhifei Yang Xiaotian Zhang Qiufeng Wang Tony Q.S. Quek Soujanya Poria Zuozhu Liu 50 0 0 06 Mar 2025
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation Songming Zhang Xue Zhang Tong Zhang Bojie Hu Yufeng Chen Jinan Xu 52 1 0 04 Mar 2025
A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning Shashank Gupta Chaitanya Ahuja Tsung-Yu Lin Sreya Dutta Roy Harrie Oosterhuis Maarten de Rijke Satya Narayan Shukla 59 1 0 02 Mar 2025
Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference Wenjie Qiu Yi-Chen Li Xuqin Zhang Tianyi Zhang Yiming Zhang Zongzhang Zhang Yang Yu ALM 51 0 0 01 Mar 2025
Self-rewarding correction for mathematical reasoning Wei Xiong Hanning Zhang Chenlu Ye Lichang Chen Nan Jiang Tong Zhang ReLM KELM LRM 75 10 0 26 Feb 2025
Advantage-Guided Distillation for Preference Alignment in Small Language Models Shiping Gao Fanqi Wan Jiajian Guo Xiaojun Quan Qifan Wang ALM 58 0 0 25 Feb 2025
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF Shicong Cen Jincheng Mei Katayoon Goshvadi Hanjun Dai Tong Yang Sherry Yang Dale Schuurmans Yuejie Chi Bo Dai OffRL 65 24 0 20 Feb 2025
PIPA: Preference Alignment as Prior-Informed Statistical Estimation Junbo Li Zhangyang Wang Qiang Liu OffRL 106 0 0 09 Feb 2025
Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model Yueqin Yin Shentao Yang Yujia Xie Ziyi Yang Yuting Sun Hany Awadalla Weizhu Chen Mingyuan Zhou 52 1 0 07 Jan 2025
Online Learning from Strategic Human Feedback in LLM Fine-Tuning Shugang Hao Lingjie Duan 92 3 0 22 Dec 2024
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF Heyang Zhao Chenlu Ye Quanquan Gu Tong Zhang OffRL 57 3 0 07 Nov 2024
Token-level Proximal Policy Optimization for Query Generation Yichen Ouyang Lu Wang Fangkai Yang Pu Zhao Chenghua Huang ... Saravan Rajmohan Weiwei Deng Dongmei Zhang Feng Sun Qi Zhang OffRL 205 3 0 01 Nov 2024
Fine-Tuning and Evaluating Open-Source Large Language Models for the Army Domain Daniel C. Ruiz John Sell 11 1 0 27 Oct 2024
Fast Best-of-N Decoding via Speculative Rejection Hanshi Sun Momin Haider Ruiqi Zhang Huitao Yang Jiahao Qiu Ming Yin Mengdi Wang Peter L. Bartlett Andrea Zanette BDL 45 28 0 26 Oct 2024
Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning H. Fernando Han Shen Parikshit Ram Yi Zhou Horst Samulowitz Nathalie Baracaldo Tianyi Chen CLL 62 3 0 20 Oct 2024
GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets Oh Joon Kwon Daiki E. Matsunaga Kee-Eung Kim AI4CE 31 0 0 19 Oct 2024
Process Reward Model with Q-Value Rankings W. Li Yixuan Li LRM 59 15 0 15 Oct 2024
Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review Emma Croxford Yanjun Gao Nicholas Pellegrino Karen K. Wong Graham Wills Elliot First Frank J. Liao Cherodeep Goswami Brian Patterson Majid Afshar HILM ELM LM&MA 37 1 0 26 Sep 2024
Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models Yuanzhao Zhai Tingkai Yang Kele Xu Feng Dawei Cheng Yang Bo Ding Huaimin Wang 156 9 0 14 Sep 2024
Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback Jiayi Zhou Yalan Qin Juntao Dai Yaodong Yang 44 4 0 30 Aug 2024
Selective Preference Optimization via Token-Level Reward Function Estimation Kailai Yang Zhiwei Liu Qianqian Xie Jimin Huang Erxue Min Sophia Ananiadou 33 10 0 24 Aug 2024
Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates Hui Wei Shenghua He Tian Xia Andy H. Wong Jingyang Lin Mei Han Mei Han ALM ELM 66 23 0 23 Aug 2024
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Haoxiang Wang Wei Xiong Tengyang Xie Han Zhao Tong Zhang 54 142 0 18 Jun 2024
Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs Rui Yang Ruomeng Ding Yong Lin Huan Zhang Tong Zhang 44 43 0 14 Jun 2024
Bootstrapping Language Models with DPO Implicit Rewards Changyu Chen Zichen Liu Chao Du Tianyu Pang Qian Liu Arunesh Sinha Pradeep Varakantham Min Lin SyDa ALM 62 23 0 14 Jun 2024
3D-Properties: Identifying Challenges in DPO and Charting a Path Forward Yuzi Yan Yibo Miao J. Li Yipin Zhang Jian Xie Zhijie Deng Dong Yan 57 11 0 11 Jun 2024
Group Robust Preference Optimization in Reward-free RLHF Shyam Sundhar Ramesh Yifan Hu Iason Chaimalas Viraj Mehta Pier Giuseppe Sessa Haitham Bou-Ammar Ilija Bogunovic 26 25 0 30 May 2024
Self-Exploring Language Models: Active Preference Elicitation for Online Alignment Shenao Zhang Donghan Yu Hiteshi Sharma Ziyi Yang Shuohang Wang Hany Hassan Zhaoran Wang LRM 48 28 0 29 May 2024
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer Zhihan Liu Miao Lu Shenao Zhang Boyi Liu Hongyi Guo Yingxiang Yang Jose H. Blanchet Zhaoran Wang 48 43 0 26 May 2024
The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback Ruitao Chen Liwei Wang 75 1 0 18 May 2024
RLHF Workflow: From Reward Modeling to Online RLHF Hanze Dong Wei Xiong Bo Pang Haoxiang Wang Han Zhao Yingbo Zhou Nan Jiang Doyen Sahoo Caiming Xiong Tong Zhang OffRL 29 99 0 13 May 2024
Token-level Direct Preference Optimization Yongcheng Zeng Guoqing Liu Weiyu Ma Ning Yang Haifeng Zhang Jun Wang 24 44 0 18 Apr 2024
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Hassan Awadallah Tengyang Xie 152 115 0 04 Apr 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James V. Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 85 220 0 20 Mar 2024
Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment Rui Yang Xiaoman Pan Feng Luo Shuang Qiu Han Zhong Dong Yu Jianshu Chen 103 69 0 15 Feb 2024
Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation Xiaoyu Chen Han Zhong Zhuoran Yang Zhaoran Wang Liwei Wang 128 62 0 23 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 372 12,081 0 04 Mar 2022
Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage Masatoshi Uehara Wen Sun OffRL 100 146 0 13 Jul 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 301 1,616 0 18 Sep 2019