Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization Beitao Chen Xinyu Lyu Lianli Gao Jingkuan Song Hengtao Shen MLLM 58 10 0 24 May 2024
Bayesian WeakS-to-Strong from Text Classification to Generation Ziyun Cui Ziyang Zhang Wen Wu Wen Wu Chao Zhang 44 2 0 24 May 2024
Direct Preference Optimization With Unobserved Preference Heterogeneity Keertana Chidambaram Karthik Vinay Seetharaman Vasilis Syrgkanis 49 7 0 23 May 2024
Aya 23: Open Weight Releases to Further Multilingual Progress Viraat Aryabumi John Dang Dwarak Talupuru Saurabh Dash David Cairuz ... Aidan Gomez Phil Blunsom Marzieh Fadaee Ahmet Üstün Sara Hooker OSLM 60 78 0 23 May 2024
SimPO: Simple Preference Optimization with a Reference-Free Reward Yu Meng Mengzhou Xia Danqi Chen 68 372 0 23 May 2024
A Declarative System for Optimizing AI Workloads Chunwei Liu Matthew Russo Michael Cafarella Lei Cao Peter Baille Chen Zui Chen Michael Franklin Tim Kraska Samuel Madden Gerardo Vitagliano 52 22 0 23 May 2024
Multi-turn Reinforcement Learning from Preference Human Feedback Lior Shani Aviv Rosenberg Asaf B. Cassel Oran Lang Daniele Calandriello ... Bilal Piot Idan Szpektor Avinatan Hassidim Yossi Matias Rémi Munos 49 26 0 23 May 2024
Calibrated Self-Rewarding Vision Language Models Yiyang Zhou Zhiyuan Fan Dongjie Cheng Sihan Yang Zhaorun Chen Chenhang Cui Xiyao Wang Yun Li Linjun Zhang Huaxiu Yao VLM 79 27 0 23 May 2024
Reinforcing Language Agents via Policy Optimization with Action Decomposition Muning Wen Bo Liu Weinan Zhang Jun Wang Ying Wen 54 8 0 23 May 2024
Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast Chufan Shi Cheng Yang Xinyu Zhu Jiahao Wang Taiqiang Wu Siheng Li Deng Cai Yujiu Yang Yu Meng MoE 58 9 0 23 May 2024
Impact of Non-Standard Unicode Characters on Security and Comprehension in Large Language Models Johan S Daniel Anand Pal 38 0 0 23 May 2024
RaFe: Ranking Feedback Improves Query Rewriting for RAG Shengyu Mao Yong-jia Jiang Boli Chen Xiao Li Peng Wang Xinyu Wang Pengjun Xie Fei Huang Huajun Chen Ningyu Zhang RALM 50 20 0 23 May 2024
Agent Planning with World Knowledge Model Shuofei Qiao Runnan Fang Ningyu Zhang Yuqi Zhu Xiang Chen Shumin Deng Yong-jia Jiang Pengjun Xie Fei Huang Huajun Chen LLMAG LM&Ro 97 15 0 23 May 2024
Online Self-Preferring Language Models Yuanzhao Zhai Zhuo Zhang Kele Xu Hanyang Peng Yue Yu Dawei Feng Cheng Yang Bo Ding Huaimin Wang 56 0 0 23 May 2024
Your Large Language Models Are Leaving Fingerprints Hope McGovern Rickard Stureborg Yoshi Suhara Dimitris Alikaniotis DeLMO 51 11 0 22 May 2024
WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response Tianrong Zhang Bochuan Cao Yuanpu Cao Lu Lin Prasenjit Mitra Jinghui Chen AAML 45 10 0 22 May 2024
Annotation-Efficient Preference Optimization for Language Model Alignment Yuu Jinnai Ukyo Honda 47 0 0 22 May 2024
LIRE: listwise reward enhancement for preference alignment Mingye Zhu Yi Liu Lei Zhang Junbo Guo Zhendong Mao 26 7 0 22 May 2024
Class-Conditional self-reward mechanism for improved Text-to-Image models Safouane El Ghazouali Arnaud Gucciardi Umberto Michelucci EGVM 37 0 0 22 May 2024
Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction Tingchen Fu Deng Cai Lemao Liu Shuming Shi Rui Yan MoMe 73 13 0 22 May 2024
Curriculum Direct Preference Optimization for Diffusion and Consistency Models Florinel-Alin Croitoru Vlad Hondru Radu Tudor Ionescu N. Sebe Mubarak Shah EGVM 89 7 0 22 May 2024
Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity Rheeya Uppaal Apratim De Yiting He Yiquao Zhong Junjie Hu 48 9 0 22 May 2024
Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents San Kim Gary Geunbae Lee AAML 43 3 0 21 May 2024
SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling Xingzhou Lou Junge Zhang Jian Xie Lifeng Liu Dong Yan Kaiqi Huang 45 11 0 21 May 2024
OLAPH: Improving Factuality in Biomedical Long-form Question Answering Minbyul Jeong Hyeon Hwang Chanwoong Yoon Taewhoo Lee Jaewoo Kang MedIm HILM LM&MA 56 12 0 21 May 2024
Tagengo: A Multilingual Chat Dataset P. Devine 44 3 0 21 May 2024
Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teaming Jiaxu Liu Xiangyu Yin Sihao Wu Jianhong Wang Meng Fang Xinping Yi Xiaowei Huang 39 5 0 21 May 2024
Mining the Explainability and Generalization: Fact Verification Based on Self-Instruction Guangyao Lu Yulin Liu 51 0 0 21 May 2024
Aligning Transformers with Continuous Feedback via Energy Rank Alignment Shriram Chennakesavalu Frank Hu Sebastian Ibarraran Grant M. Rotskoff 41 3 0 21 May 2024
Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process Ermo Hua Biqing Qi Kaiyan Zhang Yue Yu Ning Ding Xingtai Lv Kai Tian Bowen Zhou 43 3 0 20 May 2024
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework Jian Hu Xibin Wu Weixun Wang OpenLLMAI Team Dehao Zhang Yu Cao AI4CE VLM 33 98 0 20 May 2024
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts Minghao Wu Jiahao Xu Yulin Yuan Gholamreza Haffari Longyue Wang Weihua Luo Kaifu Zhang LLMAG 119 24 0 20 May 2024
Hummer: Towards Limited Competitive Preference Dataset Li Jiang Yusen Wu Junwu Xiong Jingqing Ruan Yichuan Ding Qingpei Guo ZuJie Wen Jun Zhou Xiaotie Deng 39 6 0 19 May 2024
A Method on Searching Better Activation Functions Haoyuan Sun Zihao Wu Bo Xia Pu Chang Zibin Dong Yifu Yuan Yongzhe Chang Xueqian Wang 30 3 0 19 May 2024
Large Language Models are Biased Reinforcement Learners William M. Hayes Nicolas Yax Stefano Palminteri OffRL 45 1 0 19 May 2024
Automated Multi-level Preference for MLLMs Mengxi Zhang Wenhao Wu Yu Lu Yuxin Song Kang Rong ... Jianbo Zhao Fanglong Liu Yifan Sun Haocheng Feng Jingdong Wang MLLM 83 10 0 18 May 2024
The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback Ruitao Chen Liwei Wang 75 1 0 18 May 2024
Prompt Exploration with Prompt Regression Michael Feffer Ronald Xu Yuekai Sun Mikhail Yurochkin 43 0 0 17 May 2024
Tailoring Vaccine Messaging with Common-Ground Opinions Rickard Stureborg Sanxing Chen Ruoyu Xie Aayushi Patel Christopher Li Chloe Qinyu Zhu Tingnan Hu Jun Yang Bhuwan Dhingra 47 0 0 17 May 2024
Towards Better Question Generation in QA-based Event Extraction Zijin Hong Jian Liu 53 8 0 17 May 2024
TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction Yunfan Jiang Chen Wang Ruohan Zhang Jiajun Wu Fei-Fei Li OnRL 44 26 0 16 May 2024
Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees Yu Gui Ying Jin Zhimei Ren MedIm 48 18 0 16 May 2024
Human-AI Safety: A Descendant of Generative AI and Control Systems Safety Andrea V. Bajcsy J. F. Fisac 45 7 0 16 May 2024
Leveraging Human Revisions for Improving Text-to-Layout Models Amber Xie Chin-Yi Cheng Forrest Huang Yang Li 37 1 0 16 May 2024
NIFTY Financial News Headlines Dataset Raeid Saqur Ken Kato Nicholas Vinden Frank Rudzicz AIFin 44 1 0 16 May 2024
Harmonizing Human Insights and AI Precision: Hand in Hand for Advancing Knowledge Graph Task Shurong Wang Yufei Zhang Xuliang Huang Hongwei Wang 26 0 0 15 May 2024
Enhancing Maritime Trajectory Forecasting via H3 Index and Causal Language Modelling (CLM) Nicolas Drapier Aladine Chetouani A. Chateigner 40 2 0 15 May 2024
LLMs can learn self-restraint through iterative self-reflection Alexandre Piché Aristides Milios Dzmitry Bahdanau Chris Pal 48 5 0 15 May 2024
Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection Dylan Phelps Thomas Pickard Maggie Mi Edward Gow-Smith Aline Villavicencio 55 4 0 15 May 2024
Word Alignment as Preference for Machine Translation Qiyu Wu Masaaki Nagata Zhongtao Miao Yoshimasa Tsuruoka 62 5 0 15 May 2024