Policy Optimization in RLHF: The Impact of Out-of-preference Data

17 December 2023

Ziniu Li

Papers citing "Policy Optimization in RLHF: The Impact of Out-of-preference Data"

26 / 26 papers shown

Title
Towards Self-Improvement of Diffusion Models via Group Preference Optimization Renjie Chen Wenfeng Lin Yichen Zhang Jiangchuan Wei Boyuan Liu Chao Feng Jiao Ran Mingyu Guo 12 0 0 16 May 2025
Improved Algorithms for Differentially Private Language Model Alignment Keyu Chen Hao Tang Qinglin Liu Yizhao Xu 31 0 0 13 May 2025
Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach Jiancong Xiao Bojian Hou Zhanliang Wang Ruochen Jin Q. Long Weijie Su Li Shen 35 0 0 04 May 2025
Controlling Large Language Model with Latent Actions Chengxing Jia Ziniu Li Pengyuan Wang Yi-Chen Li Zhenyu Hou Yuxiao Dong Y. Yu 56 0 0 27 Mar 2025
IPO: Iterative Preference Optimization for Text-to-Video Generation Xiaomeng Yang Zhiyu Tan Xuecheng Nie VGen 109 1 0 04 Feb 2025
IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment Yiming Zhang Zheng Chang Wentao Cai MengXing Ren Kang Yuan Yining Sun Zenghui Ding LM&MA 36 3 0 06 Jan 2025
Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization Xiyue Peng Hengquan Guo Jiawei Zhang Dongqing Zou Ziyu Shao Honghao Wei Xin Liu 44 0 0 25 Oct 2024
Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift Seongho Son William Bankes Sayak Ray Chowdhury Brooks Paige Ilija Bogunovic 42 4 0 26 Jul 2024
New Desiderata for Direct Preference Optimization Xiangkun Hu Tong He David Wipf 59 2 0 12 Jul 2024
Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment Qizhang Feng Siva Rajesh Kasa Santhosh Kumar Kasa Hyokun Yun C. Teo S. Bodapati 92 6 0 08 Jul 2024
ITERTL: An Iterative Framework for Fine-tuning LLMs for RTL Code Generation Peiyang Wu Nan Guo Xiao Xiao Wenming Li Mingyu Yan Xiaochun Ye 40 0 0 28 Jun 2024
Toward Optimal LLM Alignments Using Two-Player Games Rui Zheng Hongyi Guo Zhihan Liu Xiaoying Zhang Yuanshun Yao ... Tao Gui Qi Zhang Xuanjing Huang Hang Li Yang Liu 68 5 0 16 Jun 2024
Self-Improving Robust Preference Optimization Eugene Choi Arash Ahmadian Matthieu Geist Oilvier Pietquin M. G. Azar 31 8 0 03 Jun 2024
Group Robust Preference Optimization in Reward-free RLHF Shyam Sundhar Ramesh Yifan Hu Iason Chaimalas Viraj Mehta Pier Giuseppe Sessa Haitham Bou-Ammar Ilija Bogunovic 23 24 0 30 May 2024
Preference Learning Algorithms Do Not Learn Preference Rankings Angelica Chen Sadhika Malladi Lily H. Zhang Xinyi Chen Qiuyi Zhang Rajesh Ranganath Kyunghyun Cho 38 24 0 29 May 2024
On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization Jiancong Xiao Ziniu Li Xingyu Xie E. Getzen Cong Fang Qi Long Weijie J. Su 43 13 0 26 May 2024
Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation JoonHo Lee Jae Oh Woo Juree Seok Parisa Hassanzadeh Wooseok Jang ... Hankyu Moon Wenjun Hu Yeong-Dae Kwon Taehee Lee Seungjai Min 47 2 0 10 May 2024
D2PO: Discriminator-Guided DPO with Response Evaluation Models Prasann Singhal Nathan Lambert S. Niekum Tanya Goyal Greg Durrett OffRL EGVM 48 4 0 02 May 2024
From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models Qi He Jie Zeng Qianxi He Jiaqing Liang Yanghua Xiao 32 10 0 24 Apr 2024
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study Shusheng Xu Wei Fu Jiaxuan Gao Wenjie Ye Weiling Liu Zhiyu Mei Guangju Wang Chao Yu Yi Wu 45 136 0 16 Apr 2024
Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards Wei Shen Xiaoying Zhang Yuanshun Yao Rui Zheng Hongyi Guo Yang Liu ALM 40 11 0 12 Mar 2024
Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation Xiaoying Zhang Jean-François Ton Wei Shen Hongning Wang Yang Liu 39 13 0 08 Mar 2024
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles Yuanzhao Zhai Han Zhang Yu Lei Yue Yu Kele Xu Dawei Feng Bo Ding Huaimin Wang AI4CE 81 32 0 30 Dec 2023
Aligning Large Language Models with Human Preferences through Representation Engineering Wenhao Liu Xiaohua Wang Muling Wu Tianlong Li Changze Lv Zixuan Ling Jianhao Zhu Cenyuan Zhang Xiaoqing Zheng Xuanjing Huang 13 31 0 26 Dec 2023
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models Ziniu Li Tian Xu Yushun Zhang Zhihang Lin Yang Yu Ruoyu Sun Zhimin Luo 27 51 0 16 Oct 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 363 12,003 0 04 Mar 2022