Statistical Rejection Sampling Improves Preference Optimization

13 September 2023

Tianqi Liu

Yao-Min Zhao

Rishabh Joshi

Misha Khalman

Mohammad Saleh

Peter J. Liu

Jialu Liu

ArXiv PDF HTML

Papers citing "Statistical Rejection Sampling Improves Preference Optimization"

50 / 168 papers shown

Title
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs Rongzhi Zhang Jiaming Shen Tianqi Liu Haorui Wang Zhen Qin Feng Han Jialu Liu Simon Baumgartner Michael Bendersky Chao Zhang 37 6 0 05 Jun 2024
BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling Lin Gui Cristina Garbacea Victor Veitch BDL LM&MA 43 36 0 02 Jun 2024
Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization Yuanpu Cao Tianrong Zhang Bochuan Cao Ziyi Yin Lu Lin Fenglong Ma Jinghui Chen LLMSV 37 20 0 28 May 2024
Learning diverse attacks on large language models for robust red-teaming and safety tuning Seanie Lee Minsu Kim Lynn Cherif David Dobre Juho Lee ... Kenji Kawaguchi Gauthier Gidel Yoshua Bengio Nikolay Malkin Moksh Jain AAML 63 12 0 28 May 2024
Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models Sheng-Hsuan Peng Pin-Yu Chen Matthew Hull Duen Horng Chau 50 23 0 27 May 2024
Triple Preference Optimization: Achieving Better Alignment with Less Data in a Single Step Optimization Amir Saeidi Shivanshu Verma Aswin Rrv Chitta Baral 37 0 0 26 May 2024
On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization Jiancong Xiao Ziniu Li Xingyu Xie E. Getzen Cong Fang Qi Long Weijie J. Su 43 13 0 26 May 2024
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer Zhihan Liu Miao Lu Shenao Zhang Boyi Liu Hongyi Guo Yingxiang Yang Jose H. Blanchet Zhaoran Wang 48 43 0 26 May 2024
SimPO: Simple Preference Optimization with a Reference-Free Reward Yu Meng Mengzhou Xia Danqi Chen 68 358 0 23 May 2024
Online Self-Preferring Language Models Yuanzhao Zhai Zhuo Zhang Kele Xu Hanyang Peng Yue Yu Dawei Feng Cheng Yang Bo Ding Huaimin Wang 56 0 0 23 May 2024
Annotation-Efficient Preference Optimization for Language Model Alignment Yuu Jinnai Ukyo Honda 42 0 0 22 May 2024
LIRE: listwise reward enhancement for preference alignment Mingye Zhu Yi Liu Lei Zhang Junbo Guo Zhendong Mao 26 7 0 22 May 2024
Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process Ermo Hua Biqing Qi Kaiyan Zhang Yue Yu Ning Ding Xingtai Lv Kai Tian Bowen Zhou 43 3 0 20 May 2024
Hummer: Towards Limited Competitive Preference Dataset Li Jiang Yusen Wu Junwu Xiong Jingqing Ruan Yichuan Ding Qingpei Guo Zujie Wen Jun Zhou Xiaotie Deng 34 6 0 19 May 2024
RLHF Workflow: From Reward Modeling to Online RLHF Hanze Dong Wei Xiong Bo Pang Haoxiang Wang Han Zhao Yingbo Zhou Nan Jiang Doyen Sahoo Caiming Xiong Tong Zhang OffRL 29 98 0 13 May 2024
D2PO: Discriminator-Guided DPO with Response Evaluation Models Prasann Singhal Nathan Lambert S. Niekum Tanya Goyal Greg Durrett OffRL EGVM 43 4 0 02 May 2024
Self-Play Preference Optimization for Language Model Alignment Yue Wu Zhiqing Sun Huizhuo Yuan Kaixuan Ji Yiming Yang Quanquan Gu 39 113 0 01 May 2024
Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning Yuxi Xie Anirudh Goyal Wenyue Zheng Min-Yen Kan Timothy Lillicrap Kenji Kawaguchi Michael Shieh ReLM LRM 49 87 0 01 May 2024
Soft Preference Optimization: Aligning Language Models to Expert Distributions Arsalan Sharifnassab Sina Ghiassian Saber Salehkaleybar Surya Kanoria Dale Schuurmans 33 2 0 30 Apr 2024
Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks Amir Saeidi Shivanshu Verma Chitta Baral Chitta Baral ALM 40 23 0 23 Apr 2024
OpenELM: An Efficient Language Model Family with Open Training and Inference Framework Sachin Mehta Mohammad Hossein Sekhavat Qingqing Cao Maxwell Horton Yanzi Jin ... Iman Mirzadeh Mahyar Najibi Dmitry Belenko Peter Zatloukal Mohammad Rastegari OSLM AIFin 38 50 0 22 Apr 2024
Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data Fahim Tajwar Anika Singh Archit Sharma Rafael Rafailov Jeff Schneider Tengyang Xie Stefano Ermon Chelsea Finn Aviral Kumar 44 108 0 22 Apr 2024
Mapping Social Choice Theory to RLHF Jessica Dai Eve Fleisig 35 11 0 19 Apr 2024
Relevant or Random: Can LLMs Truly Perform Analogical Reasoning? Chengwei Qin Wenhan Xia Tan Wang Fangkai Jiao Yuchen Hu Bosheng Ding Ruirui Chen Chenyu You LRM 37 4 0 19 Apr 2024
Token-level Direct Preference Optimization Yongcheng Zeng Guoqing Liu Weiyu Ma Ning Yang Haifeng Zhang Jun Wang 24 42 0 18 Apr 2024
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study Shusheng Xu Wei Fu Jiaxuan Gao Wenjie Ye Weiling Liu Zhiyu Mei Guangju Wang Chao Yu Yi Wu 40 136 0 16 Apr 2024
Impact of Preference Noise on the Alignment Performance of Generative Language Models Yang Gao Dana Alon Donald Metzler 40 16 0 15 Apr 2024
Learn Your Reference Model for Real Good Alignment Alexey Gorbatovski Boris Shaposhnikov Alexey Malakhov Nikita Surnachev Yaroslav Aksenov Ian Maksimov Nikita Balagansky Daniil Gavrilov OffRL 54 27 0 15 Apr 2024
Binary Classifier Optimization for Large Language Model Alignment Seungjae Jung Gunsoo Han D. W. Nam Kyoung-Woon On 42 21 0 06 Apr 2024
ROPO: Robust Preference Optimization for Large Language Models Xize Liang Chao Chen Shuang Qiu Jie Wang Yue-bo Wu Zhihang Fu Zhihao Shi Feng Wu Jieping Ye 48 1 0 05 Apr 2024
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Hassan Awadallah Tengyang Xie 152 114 0 04 Apr 2024
Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization Hritik Bansal Ashima Suvarna Gantavya Bhatt Nanyun Peng Kai-Wei Chang Aditya Grover ALM 64 9 0 31 Mar 2024
Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model Qi Gou Cam-Tu Nguyen 35 8 0 28 Mar 2024
Human Alignment of Large Language Models through Online Preference Optimisation Daniele Calandriello Daniel Guo Rémi Munos Mark Rowland Yunhao Tang ... Michal Valko Tianqi Liu Rishabh Joshi Zeyu Zheng Bilal Piot 52 60 0 13 Mar 2024
Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation Xiaoying Zhang Jean-François Ton Wei Shen Hongning Wang Yang Liu 39 13 0 08 Mar 2024
CURATRON: Complete Robust Preference Data for Robust Alignment of Large Language Models S. Nguyen Uma-Naresh Niranjan Theja Tulabandhula 44 0 0 05 Mar 2024
Provably Robust DPO: Aligning Language Models with Noisy Feedback Sayak Ray Chowdhury Anush Kini Nagarajan Natarajan 33 56 0 01 Mar 2024
Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration Xin Mao Fengming Li Huimin Xu Wei Zhang A. Luu ALM 45 6 0 25 Feb 2024
Generative AI Security: Challenges and Countermeasures Banghua Zhu Norman Mu Jiantao Jiao David Wagner AAML SILM 61 8 0 20 Feb 2024
Aligning Large Language Models by On-Policy Self-Judgment Sangkyu Lee Sungdong Kim Ashkan Yousefpour Minjoon Seo Kang Min Yoo Youngjae Yu OSLM 33 11 0 17 Feb 2024
Orca-Math: Unlocking the potential of SLMs in Grade School Math Arindam Mitra Hamed Khanpour Corby Rosset Ahmed Hassan Awadallah ALM MoE LRM 37 64 0 16 Feb 2024
Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment Rui Yang Xiaoman Pan Feng Luo Shuang Qiu Han Zhong Dong Yu Jianshu Chen 103 67 0 15 Feb 2024
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models Saeed Khaki JinJin Li Lan Ma Liu Yang Prathap Ramachandra 23 19 0 15 Feb 2024
ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization Feifan Song Yuxuan Fan Xin Zhang Peiyi Wang Houfeng Wang 32 8 0 14 Feb 2024
Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs Víctor Gallego SyDa 35 6 0 12 Feb 2024
ODIN: Disentangled Reward Mitigates Hacking in RLHF Lichang Chen Chen Zhu Davit Soselia Jiuhai Chen Dinesh Manocha Tom Goldstein Heng-Chiao Huang M. Shoeybi Bryan Catanzaro AAML 50 53 0 11 Feb 2024
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model Chen Ye Wei Xiong Yuheng Zhang Nan Jiang Tong Zhang OffRL 38 9 0 11 Feb 2024
Direct Language Model Alignment from Online AI Feedback Shangmin Guo Biao Zhang Tianlin Liu Tianqi Liu Misha Khalman ... Thomas Mesnard Yao-Min Zhao Bilal Piot Johan Ferret Mathieu Blondel ALM 42 130 0 07 Feb 2024
Decoding-time Realignment of Language Models Tianlin Liu Shangmin Guo Leonardo Bianco Daniele Calandriello Quentin Berthet Felipe Llinares-López Jessica Hoffmann Lucas Dixon Michal Valko Mathieu Blondel AI4CE 54 36 0 05 Feb 2024
BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback Gaurav Pandey Yatin Nandwani Tahira Naseem Mayank Mishra Guangxuan Xu Dinesh Raghu Sachindra Joshi Asim Munawar Ramón Fernández Astudillo BDL 44 3 0 04 Feb 2024