Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,645 papers shown

Title
Characterizing Similarities and Divergences in Conversational Tones in Humans and LLMs by Sampling with People Dun-Ming Huang Pol van Rijn Ilia Sucholutsky Raja Marjieh Nori Jacoby 53 2 0 06 Jun 2024
Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models Xiang Ji Sanjeev Kulkarni Mengdi Wang Tengyang Xie OffRL 72 4 0 06 Jun 2024
Aligning Agents like Large Language Models Adam Jelley Yuhan Cao Dave Bignell Sam Devlin Tabish Rashid LM&Ro 63 1 0 06 Jun 2024
AgentGym: Evolving Large Language Model-based Agents across Diverse Environments Zhiheng Xi Yiwen Ding Wenxiang Chen Boyang Hong Honglin Guo ... Qi Zhang Xipeng Qiu Xuanjing Huang Zuxuan Wu Yu-Gang Jiang LLMAG LM&Ro 43 29 0 06 Jun 2024
Uncovering Limitations of Large Language Models in Information Seeking from Tables Chaoxu Pang Yixuan Cao Chunhao Yang Ping Luo RALM LMTD 44 3 0 06 Jun 2024
UltraMedical: Building Specialized Generalists in Biomedicine Kaiyan Zhang Sihang Zeng Ermo Hua Ning Ding Zhang-Ren Chen ... Xuekai Zhu Xingtai Lv Hu Jinfang Zhiyuan Liu Bowen Zhou LM&MA 58 23 0 06 Jun 2024
Efficient Knowledge Infusion via KG-LLM Alignment Zhouyu Jiang Ling Zhong Mengshu Sun Jun Xu Rui Sun Hui Cai Shuhan Luo Qing Cui 42 9 0 06 Jun 2024
Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art Chen Cecilia Liu Iryna Gurevych Anna Korhonen 51 6 0 06 Jun 2024
VideoPhy: Evaluating Physical Commonsense for Video Generation Hritik Bansal Zongyu Lin Tianyi Xie Zeshun Zong Michal Yarom Yonatan Bitton Chenfanfu Jiang Ningyu Zhang Kai-Wei Chang Aditya Grover EGVM VGen 52 39 0 05 Jun 2024
Training of Physical Neural Networks Ali Momeni Babak Rahmani B. Scellier Logan G. Wright Peter L. McMahon ... Julie Grollier Andrea J. Liu D. Psaltis Andrea Alù Romain Fleury PINN AI4CE 62 10 0 05 Jun 2024
Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms Rafael Rafailov Yaswanth Chittepu Ryan Park Harshit S. Sikchi Joey Hejna Bradley Knox Chelsea Finn S. Niekum 89 53 0 05 Jun 2024
HYDRA: Model Factorization Framework for Black-Box LLM Personalization Yuchen Zhuang Haotian Sun Yue Yu Rushi Qiang Qifan Wang Chao Zhang Bo Dai AAML 68 16 0 05 Jun 2024
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs Rongzhi Zhang Jiaming Shen Tianqi Liu Haorui Wang Zhen Qin Feng Han Jialu Liu Simon Baumgartner Michael Bendersky Chao Zhang 45 6 0 05 Jun 2024
Adaptive Preference Scaling for Reinforcement Learning with Human Feedback Ilgee Hong Zichong Li Alexander Bukharin Yixiao Li Haoming Jiang Tianbao Yang Tuo Zhao 45 4 0 04 Jun 2024
Aligning Large Language Models via Fine-grained Supervision Dehong Xu Liang Qiu Minseok Kim Faisal Ladhak Jaeyoung Do 50 2 0 04 Jun 2024
Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller Min Cai Yuchen Zhang Shichang Zhang Fan Yin Difan Zou Yisong Yue Ziniu Hu 58 0 0 04 Jun 2024
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models Philip Anastassiou Jiawei Chen Jingshu Chen Yuanzhe Chen Zhuo Chen ... Wenjie Zhang Yanzhe Zhang Zilin Zhao Dejian Zhong Xiaobin Zhuang 65 86 0 04 Jun 2024
On the Intrinsic Self-Correction Capability of LLMs: Uncertainty and Latent Concept Guangliang Liu Haitao Mao Bochuan Cao Zhiyu Xue K. Johnson Jiliang Tang Rongrong Wang LRM 51 10 0 04 Jun 2024
Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generation Clement Chadebec O. Tasar Eyal Benaroche Benjamin Aubin VLM 65 9 0 04 Jun 2024
Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data Haolong Li Yu Ma Yinqi Zhang Chen Ye Jie Chen ReLM LRM 40 3 0 04 Jun 2024
RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Models Bichen Wang Yuzhe Zi Yixin Sun Yanyan Zhao Bing Qin MU 77 9 0 04 Jun 2024
Dishonesty in Helpful and Harmless Alignment Youcheng Huang Jingkun Tang Duanyu Feng Zheng Zhang Wenqiang Lei Jiancheng Lv Anthony G. Cohn LLMSV 51 4 0 04 Jun 2024
LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback Wen Lai Mohsen Mesgar Alexander Fraser LRM ALM 61 19 0 03 Jun 2024
Towards Transparency: Exploring LLM Trainings Datasets through Visual Topic Modeling and Semantic Frame Charles de Dampierre Andrei Mogoutov Nicolas Baumard 61 1 0 03 Jun 2024
The Life Cycle of Large Language Models: A Review of Biases in Education Jinsook Lee Yann Hicke Renzhe Yu Christopher A. Brooks René F. Kizilcec AI4Ed 47 1 0 03 Jun 2024
LoFiT: Localized Fine-tuning on LLM Representations Fangcong Yin Xi Ye Greg Durrett 45 12 0 03 Jun 2024
An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation Kun Zhu Xiaocheng Feng Xiyuan Du Yuxuan Gu Weijiang Yu Haotian Wang Qianglong Chen Zheng Chu Jingchang Chen Bing Qin 56 5 0 03 Jun 2024
Decoupled Alignment for Robust Plug-and-Play Adaptation Haozheng Luo Jiahao Yu Wenxin Zhang Jialong Li Jerry Yao-Chieh Hu Xingyu Xing Han Liu 60 11 0 03 Jun 2024
Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors Mengge Xue Zhenyu Hu Liqun Liu Kuo Liao Shuang Li Honglin Han Meng Zhao Chengguo Yin 59 5 0 03 Jun 2024
MiniGPT-Reverse-Designing: Predicting Image Adjustments Utilizing MiniGPT-4 Vahid Azizi Fatemeh Koochaki VLM 56 0 0 03 Jun 2024
Self-Improving Robust Preference Optimization Eugene Choi Arash Ahmadian Matthieu Geist Oilvier Pietquin M. G. Azar 33 8 0 03 Jun 2024
REvolve: Reward Evolution with Large Language Models using Human Feedback Rishi Hazra Alkis Sygkounas Andreas Persson Amy Loutfi Pedro Zuidberg Dos Martires 57 2 0 03 Jun 2024
Re-ReST: Reflection-Reinforced Self-Training for Language Agents Zi-Yi Dou Cheng-Fu Yang Xueqing Wu Kai-Wei Chang Nanyun Peng LRM 88 9 0 03 Jun 2024
Unlocking Guidance for Discrete State-Space Diffusion and Flow Models Hunter Nisonoff Junhao Xiong Stephan Allenspach Jennifer Listgarten 68 32 0 03 Jun 2024
BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling Lin Gui Cristina Garbacea Victor Veitch BDL LM&MA 48 37 0 02 Jun 2024
Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback Chen Chen Yuchen Hu Wen Wu Helin Wang Chng Eng Siong Chao Zhang 51 11 0 02 Jun 2024
LLMs Could Autonomously Learn Without External Supervision Ke Ji Junying Chen Anningzhe Gao Wenya Xie Xiang Wan Benyou Wang 50 4 0 02 Jun 2024
Inverse Constitutional AI: Compressing Preferences into Principles Arduin Findeis Timo Kaufmann Eyke Hüllermeier Samuel Albanie Robert Mullins SyDa 63 11 0 02 Jun 2024
Aligning Language Models with Demonstrated Feedback Omar Shaikh Michelle S. Lam Joey Hejna Yijia Shao Michael S. Bernstein Michael S. Bernstein Diyi Yang ALM 65 24 0 02 Jun 2024
A Survey on Large Language Models for Code Generation Juyong Jiang Fan Wang Jiasi Shen Sungju Kim Sunghun Kim 78 170 0 01 Jun 2024
On Overcoming Miscalibrated Conversational Priors in LLM-based Chatbots Christine Herlihy Jennifer Neville Tobias Schnabel Adith Swaminathan 65 3 0 01 Jun 2024
Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models Meftahul Ferdaus Mahdi Abdelguerfi Elias Ioup Kendall N. Niles Ken Pathak Steve Sloan 65 12 0 01 Jun 2024
Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training Maximillian Chen Ruoxi Sun Sercan O. Arik Tomas Pfister LLMAG 68 6 0 31 May 2024
Code Pretraining Improves Entity Tracking Abilities of Language Models Najoung Kim Sebastian Schuster Shubham Toshniwal 45 14 0 31 May 2024
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF Tengyang Xie Dylan J. Foster Akshay Krishnamurthy Corby Rosset Ahmed Hassan Awadallah Alexander Rakhlin 56 35 0 31 May 2024
Direct Alignment of Language Models via Quality-Aware Self-Refinement Runsheng Yu Yong Wang Xiaoqi Jiao Youzhi Zhang James T. Kwok 63 7 0 31 May 2024
LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models Elias Stengel-Eskin Peter Hase Mohit Bansal 52 5 0 31 May 2024
Improving Reward Models with Synthetic Critiques Zihuiwen Ye Fraser Greenlee-Scott Max Bartolo Phil Blunsom Jon Ander Campos Matthias Gallé ALM SyDa LRM 45 23 0 31 May 2024
Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment Yueqin Yin Zhendong Wang Yujia Xie Weizhu Chen Mingyuan Zhou 43 4 0 31 May 2024
OR-Bench: An Over-Refusal Benchmark for Large Language Models Justin Cui Wei-Lin Chiang Ion Stoica Cho-Jui Hsieh ALM 55 39 0 31 May 2024