Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling Siyu Ren Zhiyong Wu Kenny Q. Zhu 34 3 0 07 Oct 2023
Confronting Reward Model Overoptimization with Constrained RLHF Ted Moskovitz Aaditya K. Singh DJ Strouse T. Sandholm Ruslan Salakhutdinov Anca D. Dragan Stephen Marcus McAleer 50 48 0 06 Oct 2023
A Long Way to Go: Investigating Length Correlations in RLHF Prasann Singhal Tanya Goyal Jiacheng Xu Greg Durrett 44 145 0 05 Oct 2023
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization Zhanhui Zhou Jie Liu Chao Yang Jing Shao Yu Liu Xiangyu Yue Wanli Ouyang Yu Qiao 40 49 0 05 Oct 2023
$$\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis$ $\mathcal{B}$ -Coder: Value-Based Deep Reinforcement Learning for Program Synthesis Zishun Yu Yunzhe Tao Liyu Chen Tao Sun Hongxia Yang 32 9 0 04 Oct 2023
Reward Model Ensembles Help Mitigate Overoptimization Thomas Coste Usman Anwar Robert Kirk David M. Krueger NoLa ALM 28 122 0 04 Oct 2023
The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models Hannah Rose Kirk Bertie Vidgen Paul Röttger Scott A. Hale 50 2 0 03 Oct 2023
Automatic Pair Construction for Contrastive Post-training Canwen Xu Corby Rosset Ethan C. Chau Luciano Del Corro Shweti Mahajan Julian McAuley Jennifer Neville Ahmed Hassan Awadallah Nikhil Rao ALM 27 4 0 03 Oct 2023
Ask Again, Then Fail: Large Language Models' Vacillations in Judgment Qiming Xie Zengzhi Wang Yi Feng Rui Xia AAML HILM 35 9 0 03 Oct 2023
TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus Rafael Ferreira Diogo Tavares Diogo Glória-Silva Rodrigo Valerio João Bordalo Ines Simoes Vasco Ramos David Semedo João Magalhães 24 4 0 03 Oct 2023
On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused? Hangfan Zhang Zhimeng Guo Huaisheng Zhu Bochuan Cao Lu Lin Jinyuan Jia Jinghui Chen Di Wu 78 24 0 02 Oct 2023
Enabling Language Models to Implicitly Learn Self-Improvement Ziqi Wang Le Hou Tianjian Lu Yuexin Wu Yunxuan Li Hongkun Yu Heng Ji ReLM LRM 16 6 0 02 Oct 2023
Parameter-Efficient Tuning Helps Language Model Alignment Tianci Xue Ziqi Wang Heng Ji ALM 38 6 0 01 Oct 2023
Adapting LLM Agents with Universal Feedback in Communication Kuan-Chieh Jackson Wang Yadong Lu Michael Santacroce Yeyun Gong Chao Zhang Yelong Shen LLMAG 36 7 0 01 Oct 2023
It HAS to be Subjective: Human Annotator Simulation via Zero-shot Density Estimation Wen Wu Wenlin Chen C. Zhang P. Woodland 21 1 0 30 Sep 2023
Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment Tianhao Wu Banghua Zhu Ruoyu Zhang Zhaojin Wen Kannan Ramchandran Jiantao Jiao 44 55 0 30 Sep 2023
Motif: Intrinsic Motivation from Artificial Intelligence Feedback Martin Klissarov P. DÓro Shagun Sodhani Roberta Raileanu Pierre-Luc Bacon Pascal Vincent Amy Zhang Mikael Henaff LRM LLMAG 39 55 0 29 Sep 2023
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training Xidong Feng Bo Liu Muning Wen Stephen Marcus McAleer Ying Wen Weinan Zhang Jun Wang LRM AI4CE 38 161 0 29 Sep 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 108 1,622 0 28 Sep 2023
Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints Chaoqi Wang Yibo Jiang Yuguang Yang Han Liu Yuxin Chen 42 82 0 28 Sep 2023
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model Avamarie Brueggeman Andrea Madotto Zhaojiang Lin Tushar Nagarajan Matt Smith ... Peyman Heidari Yue Liu Kavya Srinet Babak Damavandi Anuj Kumar MLLM 39 93 0 27 Sep 2023
Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding Jiacheng Liu Andrew Cohen Ramakanth Pasunuru Yejin Choi Hannaneh Hajishirzi Asli Celikyilmaz 24 24 0 26 Sep 2023
Large Language Model Alignment: A Survey Tianhao Shen Renren Jin Yufei Huang Chuang Liu Weilong Dong Zishan Guo Xinwei Wu Yan Liu Deyi Xiong LM&MA 29 179 0 26 Sep 2023
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data Guan-Bo Wang Sijie Cheng Xianyuan Zhan Xiangang Li Sen Song Yang Liu ALM 27 233 0 20 Sep 2023
Investigating the Catastrophic Forgetting in Multimodal Large Language Models Yuexiang Zhai Shengbang Tong Xiao Li Mu Cai Qing Qu Yong Jae Lee Yi Ma VLM MLLM CLL 77 78 0 19 Sep 2023
Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles Can Cui Yunsheng Ma Xu Cao Wenqian Ye Ziran Wang 24 107 0 19 Sep 2023
Baichuan 2: Open Large-scale Language Models Ai Ming Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian ... Youxin Jiang Yuchen Gao Yupeng Zhang Zenan Zhou Zhiying Wu ELM LRM 77 712 0 19 Sep 2023
Stabilizing RLHF through Advantage Model and Selective Rehearsal Baolin Peng Linfeng Song Ye Tian Lifeng Jin Haitao Mi Dong Yu 40 17 0 18 Sep 2023
ICLEF: In-Context Learning with Expert Feedback for Explainable Style Transfer Arkadiy Saakyan Smaranda Muresan 26 3 0 15 Sep 2023
Reward Engineering for Generating Semi-structured Explanation Paul Burgess Wray Buntine Ehsan Shareghi LRM 30 0 0 15 Sep 2023
A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Time Yeqi Gao Zhao Song Weixin Wang Junze Yin 29 26 0 14 Sep 2023
RAIN: Your Language Models Can Align Themselves without Finetuning Yuhui Li Fangyun Wei Jinjing Zhao Chao Zhang Hongyang R. Zhang SILM 44 108 0 13 Sep 2023
Statistical Rejection Sampling Improves Preference Optimization Tianqi Liu Yao-Min Zhao Rishabh Joshi Misha Khalman Mohammad Saleh Peter J. Liu Jialu Liu 61 215 0 13 Sep 2023
Mitigating the Alignment Tax of RLHF Yong Lin Hangyu Lin Wei Xiong Shizhe Diao Zeming Zheng ... Han Zhao Nan Jiang Heng Ji Yuan Yao Tong Zhang MoMe CLL 29 69 0 12 Sep 2023
OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs Patrick Haller Ansar Aynetdinov Alan Akbik 38 24 0 07 Sep 2023
FLM-101B: An Open LLM and How to Train It with $100K Budget$ Xiang Li Yiqun Yao Xin Jiang Xuezhi Fang Xuying Meng ... Li Du Bowen Qin Zheng-Wei Zhang Aixin Sun Yequan Wang 60 22 0 07 Sep 2023
Deep Reinforcement Learning from Hierarchical Preference Design Alexander Bukharin Yixiao Li Pengcheng He Tuo Zhao 25 0 0 06 Sep 2023
Making Large Language Models Better Reasoners with Alignment Peiyi Wang Lei Li Liang Chen Feifan Song Binghuai Lin Yunbo Cao Tianyu Liu Zhifang Sui ALM LRM 50 65 0 05 Sep 2023
Efficient RLHF: Reducing the Memory Usage of PPO Michael Santacroce Yadong Lu Han Yu Yuan-Fang Li Yelong Shen 35 27 0 01 Sep 2023
Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models Hritik Bansal John Dang Aditya Grover ALM 35 20 0 30 Aug 2023
Reinforcement Learning for Generative AI: A Survey Yuanjiang Cao Quan.Z Sheng Julian McAuley Lina Yao SyDa 53 10 0 28 Aug 2023
Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities Maximilian Mozes Xuanli He Bennett Kleinberg Lewis D. Griffin 41 78 0 24 Aug 2023
How to Protect Copyright Data in Optimization of Large Language Models? T. Chu Zhao Song Chiwun Yang 45 29 0 23 Aug 2023
Instruction Position Matters in Sequence Generation with Large Language Models Yanjun Liu Xianfeng Zeng Fandong Meng Jie Zhou LRM 64 8 0 23 Aug 2023
From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models Jing Yao Xiaoyuan Yi Xiting Wang Jindong Wang Xing Xie ALM 27 42 0 23 Aug 2023
PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator Chuyi Kong Yaxin Fan Xiang Wan Feng Jiang Benyou Wang 42 8 0 21 Aug 2023
Open, Closed, or Small Language Models for Text Classification? Hao Yu Zachary Yang Kellin Pelrine Jean Francois Godbout Reihaneh Rabbany 25 30 0 19 Aug 2023
PUMGPT: A Large Vision-Language Model for Product Understanding Wei Xue Zongyi Guo Baoliang Cui Zengming Tang Weiwei Zhang Haihong Tang Shuhui Wu Weiming Lu VLM 40 2 0 18 Aug 2023
Reinforced Self-Training (ReST) for Language Modeling Çağlar Gülçehre T. Paine S. Srinivasan Ksenia Konyushkova L. Weerts ... Chenjie Gu Wolfgang Macherey Arnaud Doucet Orhan Firat Nando de Freitas OffRL 66 278 0 17 Aug 2023
Convergence of Two-Layer Regression with Nonlinear Units Yichuan Deng Zhao Song Shenghao Xie 31 7 0 16 Aug 2023