Self-Play Preference Optimization for Language Model Alignment

1 May 2024

Quanquan Gu

Papers citing "Self-Play Preference Optimization for Language Model Alignment"

39 / 89 papers shown

Title
Aligning Language Models Using Follow-up Likelihood as Reward Signal Chen Zhang Dading Chong Feng Jiang Chengguang Tang Anningzhe Gao Guohua Tang Haizhou Li ALM 33 2 0 20 Sep 2024
Reward-Robust RLHF in LLMs Yuzi Yan Xingzhou Lou Jialian Li Yiping Zhang Jian Xie Chao Yu Yu Wang Dong Yan Yuan Shen 47 8 0 18 Sep 2024
Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization Jianing Wang Yang Zhou Xiaocheng Zhang Mengjiao Bao Peng Yan 30 1 0 17 Sep 2024
AIPO: Improving Training Objective for Iterative Preference Optimization Yaojie Shen Xinyao Wang Yulei Niu Ying Zhou Lexin Tang Libo Zhang Fan Chen Longyin Wen 23 2 0 13 Sep 2024
Alignment with Preference Optimization Is All You Need for LLM Safety Réda Alami Ali Khalifa Almansoori Ahmed Alzubaidi M. Seddik Mugariya Farooq Hakim Hacid 37 1 0 12 Sep 2024
Alignment of Diffusion Models: Fundamentals, Challenges, and Future Buhua Liu Shitong Shao Bao Li Lichen Bai Zhiqiang Xu Haoyi Xiong James Kwok Sumi Helal Zeke Xie 45 12 0 11 Sep 2024
Towards a Unified View of Preference Learning for Large Language Models: A Survey Bofei Gao Feifan Song Yibo Miao Zefan Cai Zhengyuan Yang ... Houfeng Wang Zhifang Sui Peiyi Wang Baobao Chang Baobao Chang 53 12 0 04 Sep 2024
FuseChat: Knowledge Fusion of Chat Models Fanqi Wan Longguang Zhong Ziyi Yang Ruijun Chen Xiaojun Quan ALM KELM MoMe 32 23 0 15 Aug 2024
Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment Karel DÓosterlinck Winnie Xu Chris Develder Thomas Demeester A. Singh Christopher Potts Douwe Kiela Shikib Mehri 38 10 0 12 Aug 2024
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge Tianhao Wu Weizhe Yuan O. Yu. Golovneva Jing Xu Yuandong Tian Jiantao Jiao Jason Weston Sainbayar Sukhbaatar ALM KELM LRM 64 73 0 28 Jul 2024
Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift Seongho Son William Bankes Sayak Ray Chowdhury Brooks Paige Ilija Bogunovic 42 4 0 26 Jul 2024
A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More Zhichao Wang Bin Bi Shiva K. Pentyala Kiran Ramnath Sougata Chaudhuri ... Z. Zhu Xiang-Bo Mao S. Asur Na Na Cheng OffRL 42 39 0 23 Jul 2024
Internal Consistency and Self-Feedback in Large Language Models: A Survey Xun Liang Shichao Song Zifan Zheng Hanyu Wang Qingchen Yu ... Rong-Hua Li Peng Cheng Zhonghao Wang Zhiyu Li Zhiyu Li HILM LRM 70 25 0 19 Jul 2024
Data-Centric Human Preference Optimization with Rationales H. Just Ming Jin Anit Kumar Sahu Huy Phan Ruoxi Jia 52 3 0 19 Jul 2024
Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning Yifang Chen Shuohang Wang Ziyi Yang Hiteshi Sharma Nikos Karampatziakis Donghan Yu Kevin G. Jamieson Simon Shaolei Du Yelong Shen OffRL 51 4 0 02 Jul 2024
Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning Yuheng Zhang Dian Yu Baolin Peng Linfeng Song Ye Tian Mingyue Huo Nan Jiang Haitao Mi Dong Yu 35 15 0 30 Jun 2024
Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning Sen Yang Leyang Cui Deng Cai Xinting Huang Shuming Shi Wai Lam 46 8 0 25 Jun 2024
Language Alignment via Nash-learning and Adaptive feedback Ari Azarafrooz Farshid Faal 29 0 0 22 Jun 2024
SAIL: Self-Improving Efficient Online Alignment of Large Language Models Mucong Ding Souradip Chakraborty Vibhu Agrawal Zora Che Alec Koppel Mengdi Wang Amrit Singh Bedi Furong Huang 47 10 0 21 Jun 2024
Finding Safety Neurons in Large Language Models Jianhui Chen Xiaozhi Wang Zijun Yao Yushi Bai Lei Hou Juanzi Li KELM LLMSV 48 11 0 20 Jun 2024
mDPO: Conditional Preference Optimization for Multimodal Large Language Models Fei Wang Wenxuan Zhou James Y. Huang Nan Xu Sheng Zhang Hoifung Poon Muhao Chen 66 15 0 17 Jun 2024
WPO: Enhancing RLHF with Weighted Preference Optimization Wenxuan Zhou Ravi Agrawal Shujian Zhang Sathish Indurthi Sanqiang Zhao Kaiqiang Song Silei Xu Chenguang Zhu 35 18 0 17 Jun 2024
Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence Junru Lu Jiazheng Li Siyu An Meng Zhao Yulan He Di Yin Xing Sun 47 14 0 16 Jun 2024
Bootstrapping Language Models with DPO Implicit Rewards Changyu Chen Zichen Liu Chao Du Tianyu Pang Qian Liu Arunesh Sinha Pradeep Varakantham Min-Bin Lin SyDa ALM 62 23 0 14 Jun 2024
Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment Yueqin Yin Zhendong Wang Yujia Xie Weizhu Chen Mingyuan Zhou 35 4 0 31 May 2024
Group Robust Preference Optimization in Reward-free RLHF Shyam Sundhar Ramesh Yifan Hu Iason Chaimalas Viraj Mehta Pier Giuseppe Sessa Haitham Bou-Ammar Ilija Bogunovic 23 24 0 30 May 2024
Self-Exploring Language Models: Active Preference Elicitation for Online Alignment Shenao Zhang Donghan Yu Hiteshi Sharma Ziyi Yang Shuohang Wang Hany Hassan Zhaoran Wang LRM 48 28 0 29 May 2024
Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment Keming Lu Bowen Yu Fei Huang Yang Fan Runji Lin Chang Zhou MoMe 32 18 0 28 May 2024
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer Zhihan Liu Miao Lu Shenao Zhang Boyi Liu Hongyi Guo Yingxiang Yang Jose H. Blanchet Zhaoran Wang 48 43 0 26 May 2024
OLAPH: Improving Factuality in Biomedical Long-form Question Answering Minbyul Jeong Hyeon Hwang Chanwoong Yoon Taewhoo Lee Jaewoo Kang MedIm HILM LM&MA 46 12 0 21 May 2024
RLHF Workflow: From Reward Modeling to Online RLHF Hanze Dong Wei Xiong Bo Pang Haoxiang Wang Han Zhao Yingbo Zhou Nan Jiang Doyen Sahoo Caiming Xiong Tong Zhang OffRL 29 98 0 13 May 2024
REBEL: Reinforcement Learning via Regressing Relative Rewards Zhaolin Gao Jonathan D. Chang Wenhao Zhan Owen Oertell Gokul Swamy Kianté Brantley Thorsten Joachims J. Andrew Bagnell Jason D. Lee Wen Sun OffRL 43 31 0 25 Apr 2024
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Hassan Awadallah Tengyang Xie 152 114 0 04 Apr 2024
Learn To be Efficient: Build Structured Sparsity in Large Language Models Haizhong Zheng Xiaoyan Bai Xueshen Liu Z. Morley Mao Beidi Chen Fan Lai Atul Prakash 51 11 0 09 Feb 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 182 456 0 02 Feb 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 242 298 0 18 Jan 2024
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models Avi Singh John D. Co-Reyes Rishabh Agarwal Ankesh Anand Piyush Patil ... Yamini Bansal Ethan Dyer Behnam Neyshabur Jascha Narain Sohl-Dickstein Noah Fiedel ALM LRM ReLM SyDa 157 146 0 11 Dec 2023
Borda Regret Minimization for Generalized Linear Dueling Bandits Yue Wu Tao Jin Hao Lou Farzad Farnoud Quanquan Gu 31 11 0 15 Mar 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 339 12,003 0 04 Mar 2022