Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
Impact of Preference Noise on the Alignment Performance of Generative Language Models Yang Gao Dana Alon Donald Metzler 45 17 0 15 Apr 2024
Exploring Text-to-Motion Generation with Human Preference Jenny Sheng Matthieu Lin Andrew Zhao Kevin Pruvost Yu-Hui Wen Yangguang Li Gao Huang Yong-Jin Liu VGen 42 1 0 15 Apr 2024
Learn Your Reference Model for Real Good Alignment Alexey Gorbatovski Boris Shaposhnikov Alexey Malakhov Nikita Surnachev Yaroslav Aksenov Ian Maksimov Nikita Balagansky Daniil Gavrilov OffRL 61 28 0 15 Apr 2024
Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies Benjue Weng LM&MA 54 8 0 13 Apr 2024
CuriousLLM: Elevating Multi-Document Question Answering with LLM-Enhanced Knowledge Graph Reasoning Zukang Yang Zixuan Zhu Xuan Zhu RALM 43 0 0 13 Apr 2024
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs Shreyas Chaudhari Pranjal Aggarwal Vishvak Murahari Tanmay Rajpurohit Ashwin Kalyan Karthik Narasimhan Ameet Deshpande Bruno Castro da Silva 36 35 0 12 Apr 2024
Dataset Reset Policy Optimization for RLHF Jonathan D. Chang Wenhao Zhan Owen Oertell Kianté Brantley Dipendra Kumar Misra Jason D. Lee Wen Sun OffRL 32 21 0 12 Apr 2024
Best Practices and Lessons Learned on Synthetic Data for Language Models Ruibo Liu Jerry W. Wei Fangyu Liu Chenglei Si Yanzhe Zhang ... Steven Zheng Daiyi Peng Diyi Yang Denny Zhou Andrew M. Dai SyDa EgoV 48 88 0 11 Apr 2024
JetMoE: Reaching Llama2 Performance with 0.1M Dollars Yikang Shen Zhen Guo Tianle Cai Zengyi Qin MoE ALM 46 29 0 11 Apr 2024
High-Dimension Human Value Representation in Large Language Models Samuel Cahyawijaya Delong Chen Yejin Bang Leila Khalatbari Bryan Wilie Ziwei Ji Etsuko Ishii Pascale Fung 71 5 0 11 Apr 2024
GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications Shishir G. Patil Tianjun Zhang Vivian Fang Noppapon C Roy Huang Uc Berkeley Aaron Hao Martin Casado Joseph E. Gonzalez Raluca Ada Popa Ion Stoica ALM 36 11 0 10 Apr 2024
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies Shengding Hu Yuge Tu Xu Han Chaoqun He Ganqu Cui ... Chaochao Jia Guoyang Zeng Dahai Li Zhiyuan Liu Maosong Sun MoE 56 298 0 09 Apr 2024
Latent Distance Guided Alignment Training for Large Language Models Haotian Luo 19 0 0 09 Apr 2024
The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models Giwon Hong Aryo Pradipta Gema Rohit Saxena Xiaotang Du Ping Nie ... Laura Perez-Beltrachini Max Ryabinin Xuanli He Clémentine Fourrier Pasquale Minervini LRM HILM 43 11 0 08 Apr 2024
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning Ruiqi Zhang Licong Lin Yu Bai Song Mei MU 74 139 0 08 Apr 2024
SpeechAlign: Aligning Speech Generation to Human Preferences Dong Zhang Zhaowei Li Shimin Li Xin Zhang Pengyu Wang Yaqian Zhou Xipeng Qiu ALM AuLLM 36 15 0 08 Apr 2024
Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data Tim Baumgärtner Yang Gao Dana Alon Donald Metzler AAML 43 18 0 08 Apr 2024
Towards Understanding the Influence of Reward Margin on Preference Model Performance Bowen Qin Duanyu Feng Xi Yang 42 3 0 07 Apr 2024
Regularized Conditional Diffusion Model for Multi-Task Preference Alignment Xudong Yu Chenjia Bai Haoran He Changhong Wang Xuelong Li 45 6 0 07 Apr 2024
Binary Classifier Optimization for Large Language Model Alignment Seungjae Jung Gunsoo Han D. W. Nam Kyoung-Woon On 42 21 0 06 Apr 2024
ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming Simone Tedeschi Felix Friedrich P. Schramowski Kristian Kersting Roberto Navigli Huu Nguyen Bo Li ELM 45 46 0 06 Apr 2024
Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective Duanyu Feng Bowen Qin Chen Huang Zheng Zhang Wenqiang Lei 36 31 0 06 Apr 2024
Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model Zhonghan Zhao Ke Ma Wenhao Chai Xuan Wang Kewei Chen Dongxu Guo Yanting Zhang Hongwei Wang Gaoang Wang 50 16 0 06 Apr 2024
Aligning Diffusion Models by Optimizing Human Utility Shufan Li Konstantinos Kallidromitis Akash Gokul Yusuke Kato Kazuki Kozuka 109 30 0 06 Apr 2024
Exploring Autonomous Agents through the Lens of Large Language Models: A Review Saikat Barua LM&MA LLMAG 35 15 0 05 Apr 2024
Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models Sang-Sub Jang Jaehyeong Jo Kimin Lee Sung Ju Hwang 29 15 0 05 Apr 2024
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model Xinrun Du Zhouliang Yu Songyang Gao Ding Pan Yuyang Cheng ... Tianyu Zheng Xinchen Luo Guorui Zhou Wenhu Chen Ge Zhang 48 17 0 05 Apr 2024
ROPO: Robust Preference Optimization for Large Language Models Xize Liang Chao Chen Shuang Qiu Jie Wang Yue-bo Wu Zhihang Fu Zhihao Shi Feng Wu Jieping Ye 53 2 0 05 Apr 2024
Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data Jingyu Zhang Marc Marone Tianjian Li Benjamin Van Durme Daniel Khashabi 93 9 0 05 Apr 2024
AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web Navigating Agent Hanyu Lai Xiao Liu Iat Long Iong Shuntian Yao Yuxuan Chen ... Hao Yu Hanchen Zhang Xiaohan Zhang Yuxiao Dong Jie Tang LM&Ro LLMAG 41 45 0 04 Apr 2024
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Hassan Awadallah Tengyang Xie 152 116 0 04 Apr 2024
Sailor: Open Language Models for South-East Asia Longxu Dou Qian Liu Guangtao Zeng Jia Guo Jiahui Zhou Wei Lu Min Lin LRM 45 9 0 04 Apr 2024
Investigating Regularization of Self-Play Language Models Réda Alami Abdalgader Abubaker Mastane Achab M. Seddik Salem Lahlou 38 3 0 04 Apr 2024
The Impact of Unstated Norms in Bias Analysis of Language Models Farnaz Kohankhaki D. B. Emerson David B. Emerson Laleh Seyyed-Kalantari Faiza Khan Khattak 62 1 0 04 Apr 2024
ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline Yifan Xu Xiao Liu Xinghan Liu Zhenyu Hou Yueyan Li ... Aohan Zeng Zhengxiao Du Wenyi Zhao Jie Tang Yuxiao Dong LRM 49 36 0 03 Apr 2024
Empowering Biomedical Discovery with AI Agents Shanghua Gao Ada Fang Yepeng Huang Valentina Giunchiglia Ayush Noori Jonathan Richard Schwarz Yasha Ektefaie Jovana Kondic Marinka Zitnik LLMAG AI4CE 54 70 0 03 Apr 2024
Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Models Haoran Sun Lixin Liu Junjie Li Fengyu Wang Baohua Dong Ran Lin Ruohui Huang 33 16 0 03 Apr 2024
uTeBC-NLP at SemEval-2024 Task 9: Can LLMs be Lateral Thinkers? Pouya Sadeghi Amirhossein Abaskohi Yadollah Yaghoobzadeh LRM ReLM 41 1 0 03 Apr 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 64 76 0 03 Apr 2024
Advancing LLM Reasoning Generalists with Preference Trees Lifan Yuan Ganqu Cui Hanbin Wang Ning Ding Xingyao Wang ... Zhenghao Liu Bowen Zhou Hao Peng Zhiyuan Liu Maosong Sun LRM 50 101 0 02 Apr 2024
Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation Declan Grabb Max Lamparth N. Vasan 50 15 0 02 Apr 2024
Asymptotics of Language Model Alignment Joy Qiping Yang Salman Salamatian Ziteng Sun A. Suresh Ahmad Beirami 69 23 0 02 Apr 2024
A Survey on Large Language Model-Based Game Agents Sihao Hu Tiansheng Huang Gaowen Liu Ramana Rao Kompella Gaowen Liu Selim Furkan Tekin Yichang Xu Zachary Yahn Ling Liu LLMAG LM&Ro AI4CE LM&MA 71 52 0 02 Apr 2024
Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models Yi-Lin Tuan Xilun Chen Eric Michael Smith Louis Martin Soumya Batra Asli Celikyilmaz William Yang Wang Daniel M. Bikel 35 8 0 01 Apr 2024
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward Ruohong Zhang Liangke Gui Zhiqing Sun Yihao Feng Keyang Xu ... Di Fu Chunyuan Li Alexander G. Hauptmann Yonatan Bisk Yiming Yang MLLM 58 63 0 01 Apr 2024
Stable Code Technical Report Nikhil Pinnaparaju Reshinth Adithyan Duy Phung J. Tow James Baicoianu ... Maksym Zhuravinskyi Dakota Mahan Marco Bellagente Carlos Riquelme Nathan Cooper LRM ALM 25 13 0 01 Apr 2024
Efficient Prompting Methods for Large Language Models: A Survey Kaiyan Chang Songcheng Xu Chenglong Wang Yingfeng Luo Tong Xiao Jingbo Zhu LRM 47 32 0 01 Apr 2024
Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment Yuu Jinnai Tetsuro Morimura Kaito Ariu Kenshi Abe 71 7 0 01 Apr 2024
Prior Constraints-based Reward Model Training for Aligning Large Language Models Hang Zhou Chenglong Wang Yimin Hu Tong Xiao Chunliang Zhang Jingbo Zhu ALM 51 2 0 01 Apr 2024
ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback Zhenyu Hou Yiin Niu Zhengxiao Du Xiaohan Zhang Xiao Liu ... Qinkai Zheng Minlie Huang Hongning Wang Jie Tang Yuxiao Dong ALM 44 18 0 01 Apr 2024