Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
Crowd-PrefRL: Preference-Based Reward Learning from Crowds David Chhan Ellen R. Novoseller Vernon J. Lawhern 42 5 0 17 Jan 2024
Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models T. Klein Moin Nabi 26 1 0 16 Jan 2024
Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation Haoran Xu Amr Sharaf Yunmo Chen Weiting Tan Lingfeng Shen Benjamin Van Durme Kenton W. Murray Young Jin Kim ALM 64 212 0 16 Jan 2024
MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization Shuaijie She Wei Zou Shujian Huang Wenhao Zhu Xiang Liu Xiang Geng Jiajun Chen LRM 75 34 0 12 Jan 2024
Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty Kaitlyn Zhou Jena D. Hwang Xiang Ren Maarten Sap 36 54 0 12 Jan 2024
TOFU: A Task of Fictitious Unlearning for LLMs Pratyush Maini Zhili Feng Avi Schwarzschild Zachary Chase Lipton J. Zico Kolter MU CLL 46 146 0 11 Jan 2024
Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint Zhipeng Chen Kun Zhou Wayne Xin Zhao Junchen Wan Fuzheng Zhang Di Zhang Ji-Rong Wen KELM 39 33 0 11 Jan 2024
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems Tianyu Cui Yanling Wang Chuanpu Fu Yong Xiao Sijia Li ... Junwu Xiong Xinyu Kong Zujie Wen Ke Xu Qi Li 63 57 0 11 Jan 2024
Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback Chengfeng Dou Zhi Jin Wenpin Jiao Haiyan Zhao Yongqiang Zhao Zhenwei Tao LM&MA 82 5 0 11 Jan 2024
Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning Yiqi Wang Wentao Chen Xiaotian Han Xudong Lin Haiteng Zhao Yongfei Liu Bohan Zhai Jianbo Yuan Quanzeng You Hongxia Yang LRM 47 71 0 10 Jan 2024
Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk Dennis Ulmer Elman Mansimov Kaixiang Lin Justin Sun Xibin Gao Yi Zhang LLMAG 35 27 0 10 Jan 2024
Agent Alignment in Evolving Social Norms Shimin Li Tianxiang Sun Qinyuan Cheng Xipeng Qiu LLMAG 43 8 0 09 Jan 2024
Mixtral of Experts Albert Q. Jiang Alexandre Sablayrolles Antoine Roux A. Mensch Blanche Savary ... Théophile Gervet Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LLMAG 42 1,000 0 08 Jan 2024
A Minimaximalist Approach to Reinforcement Learning from Human Feedback Gokul Swamy Christoph Dann Rahul Kidambi Zhiwei Steven Wu Alekh Agarwal OffRL 51 96 0 08 Jan 2024
LightHouse: A Survey of AGI Hallucination Feng Wang LRM HILM VLM 34 3 0 08 Jan 2024
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism DeepSeek-AI Xiao Bi : Xiao Bi Deli Chen Guanting Chen ... Yao Zhao Shangyan Zhou Shunfeng Zhou Qihao Zhu Yuheng Zou LRM ALM 139 316 0 05 Jan 2024
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance Renjie Pi Tianyang Han Jianshu Zhang Yueqi Xie Rui Pan Qing Lian Hanze Dong Jipeng Zhang Tong Zhang AAML 31 61 0 05 Jan 2024
Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decoding Yuu Jinnai Kaito Ariu 31 8 0 05 Jan 2024
Large Language Models for Social Networks: Applications, Challenges, and Solutions Jingying Zeng Richard Huang Waleed Malik Langxuan Yin Bojan Babic Danny Shacham Xiao Yan Jaewon Yang Qi He 22 7 0 04 Jan 2024
A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity Andrew Lee Xiaoyan Bai Itamar Pres Martin Wattenberg Jonathan K. Kummerfeld Rada Mihalcea 77 104 0 03 Jan 2024
Theoretical guarantees on the best-of-n alignment policy Ahmad Beirami Alekh Agarwal Jonathan Berant Alex DÁmour Jacob Eisenstein Chirag Nagpal A. Suresh 50 44 0 03 Jan 2024
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models Zixiang Chen Yihe Deng Huizhuo Yuan Kaixuan Ji Quanquan Gu SyDa 48 285 0 02 Jan 2024
A Reliable Knowledge Processing Framework for Combustion Science using Foundation Models Vansh Sharma Venkat Raman 21 7 0 31 Dec 2023
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles Yuanzhao Zhai Han Zhang Yu Lei Yue Yu Kele Xu Dawei Feng Bo Ding Huaimin Wang AI4CE 81 33 0 30 Dec 2023
ConfusionPrompt: Practical Private Inference for Online Large Language Models Peihua Mai Ran Yan Rui Ye Youjia Yang Yinchuan Li Yan Pang 24 1 0 30 Dec 2023
Improving In-context Learning via Bidirectional Alignment Chengwei Qin Wenhan Xia Fangkai Jiao Chen Chen Yuchen Hu Bosheng Ding Chenyu You 43 7 0 28 Dec 2023
Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss Jing Xu Andrew Lee Sainbayar Sukhbaatar Jason Weston 31 86 0 27 Dec 2023
Preference as Reward, Maximum Preference Optimization with Importance Sampling Zaifan Jiang Xing Huang Chao Wei 36 2 0 27 Dec 2023
Aligning Large Language Models with Human Preferences through Representation Engineering Wenhao Liu Xiaohua Wang Muling Wu Tianlong Li Changze Lv Zixuan Ling Jianhao Zhu Cenyuan Zhang Xiaoqing Zheng Xuanjing Huang 16 33 0 26 Dec 2023
Align on the Fly: Adapting Chatbot Behavior to Established Norms Chunpu Xu Steffi Chern Ethan Chern Ge Zhang Zekun Wang Ruibo Liu Jing Li Jie Fu Pengfei Liu 24 20 0 26 Dec 2023
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning Wei Liu Weihao Zeng Keqing He Yong Jiang Junxian He ALM 44 219 0 25 Dec 2023
SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling Dahyun Kim Chanjun Park Sanghoon Kim Wonsung Lee Wonho Song ... Hyunbyung Park Gyoungjin Gim Mikyoung Cha Hwalsuk Lee Sunghun Kim ALM ELM 35 136 0 23 Dec 2023
Reasons to Reject? Aligning Language Models with Judgments Weiwen Xu Deng Cai Zhisong Zhang Wai Lam Shuming Shi ALM 21 14 0 22 Dec 2023
Typhoon: Thai Large Language Models Kunat Pipatanakul Phatrasek Jirabovonvisut Potsawee Manakul Sittipong Sripaisarnmongkol Ruangsak Patomwong Pathomporn Chokchainant Kasima Tharnpipitchai 50 16 0 21 Dec 2023
Learning and Forgetting Unsafe Examples in Large Language Models Jiachen Zhao Zhun Deng David Madras James Zou Mengye Ren MU KELM CLL 94 17 0 20 Dec 2023
Climate Change from Large Language Models Hongyin Zhu Prayag Tiwari ELM 41 7 0 19 Dec 2023
Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environment Fengli Xu Jun Zhang Chen Gao J. Feng Yong Li AI4CE LLMAG 26 29 0 19 Dec 2023
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint Wei Xiong Hanze Dong Chen Ye Ziqi Wang Han Zhong Heng Ji Nan Jiang Tong Zhang OffRL 38 164 0 18 Dec 2023
Social Learning: Towards Collaborative Learning with Large Language Models Amirkeivan Mohtashami Florian Hartmann Sian Gooding Lukás Zilka Matt Sharifi Blaise Agüera y Arcas 16 10 0 18 Dec 2023
A Survey of Reasoning with Foundation Models Jiankai Sun Chuanyang Zheng Enze Xie Zhengying Liu Ruihang Chu ... Xipeng Qiu Yi-Chen Guo Hui Xiong Qun Liu Zhenguo Li ReLM LRM AI4CE 32 79 0 17 Dec 2023
Silkie: Preference Distillation for Large Visual Language Models Lei Li Zhihui Xie Mukai Li Shunian Chen Peiyi Wang Liang Chen Yazheng Yang Benyou Wang Lingpeng Kong MLLM 117 69 0 17 Dec 2023
Policy Optimization in RLHF: The Impact of Out-of-preference Data Ziniu Li Tian Xu Yang Yu 34 30 0 17 Dec 2023
Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling Jingying Zeng Jaewon Yang Waleed Malik Xiao Yan Richard Huang Qi He 30 1 0 16 Dec 2023
Distilling Large Language Models for Matching Patients to Clinical Trials Mauro Nievas Aditya Basu Yanshan Wang Hrituraj Singh ELM LM&MA 28 30 0 15 Dec 2023
Self-Evaluation Improves Selective Generation in Large Language Models Jie Jessie Ren Yao-Min Zhao Tu Vu Peter J. Liu Balaji Lakshminarayanan ELM 36 34 0 14 Dec 2023
Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking Jacob Eisenstein Chirag Nagpal Alekh Agarwal Ahmad Beirami Alex DÁmour ... Katherine Heller Stephen R. Pfohl Deepak Ramachandran Peter Shaw Jonathan Berant 32 85 0 14 Dec 2023
TigerBot: An Open Multilingual Multitask LLM Ye Chen Wei Cai Liangming Wu Xiaowei Li Zhanxuan Xin Cong Fu 153 11 0 14 Dec 2023
An Invitation to Deep Reinforcement Learning Bernhard Jaeger Andreas Geiger OffRL OOD 80 5 0 13 Dec 2023
On Diversified Preferences of Large Language Model Alignment Dun Zeng Yong Dai Pengyu Cheng Longyue Wang Tianhao Hu Wanshun Chen Nan Du Zenglin Xu ALM 40 16 0 12 Dec 2023
A dynamical clipping approach with task feedback for Proximal Policy Optimization Ziqi Zhang Jingzehua Xu Zifeng Zhuang Jinxin Liu Donglin Wang Shuai Zhang 26 1 0 12 Dec 2023