Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
Personalized Large Language Models Stanislaw Wo'zniak Bartlomiej Koptyra Arkadiusz Janz P. Kazienko Jan Kocoñ 30 19 0 14 Feb 2024
Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation Xiaoying Zhang Baolin Peng Ye Tian Jingyan Zhou Lifeng Jin Linfeng Song Haitao Mi Helen Meng HILM 45 45 0 14 Feb 2024
Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models Goutham Rajendran Simon Buchholz Bryon Aragam Bernhard Schölkopf Pradeep Ravikumar AI4CE 96 21 0 14 Feb 2024
Into the Unknown: Self-Learning Large Language Models Teddy Ferdinan Jan Kocoñ P. Kazienko 33 2 0 14 Feb 2024
MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences Souradip Chakraborty Jiahao Qiu Hui Yuan Alec Koppel Furong Huang Dinesh Manocha Amrit Singh Bedi Mengdi Wang ALM 51 48 0 14 Feb 2024
Reinforcement Learning from Human Feedback with Active Queries Kaixuan Ji Jiafan He Quanquan Gu 29 17 0 14 Feb 2024
Rethinking Machine Unlearning for Large Language Models Sijia Liu Yuanshun Yao Jinghan Jia Stephen Casper Nathalie Baracaldo ... Hang Li Kush R. Varshney Mohit Bansal Sanmi Koyejo Yang Liu AILaw MU 82 86 0 13 Feb 2024
InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignment Jianing Wang Junda Wu Yupeng Hou Yao Liu Ming Gao Julian McAuley 35 32 0 13 Feb 2024
PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models Fei Deng Qifei Wang Wei Wei Matthias Grundmann Tingbo Hou EGVM 27 15 0 13 Feb 2024
A Dense Reward View on Aligning Text-to-Image Diffusion with Preference Shentao Yang Tianqi Chen Mingyuan Zhou EGVM 34 23 0 13 Feb 2024
Active Preference Learning for Large Language Models William Muldrew Peter Hayes Mingtian Zhang David Barber 39 16 0 12 Feb 2024
Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts Yueqin Yin Zhendong Wang Yi Gu Hai Huang Weizhu Chen Mingyuan Zhou 19 19 0 12 Feb 2024
Large Language Models as Agents in Two-Player Games Yang Liu Peng Sun Hang Li LLMAG 45 3 0 12 Feb 2024
Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs Víctor Gallego SyDa 35 6 0 12 Feb 2024
Suppressing Pink Elephants with Direct Principle Feedback Louis Castricato Nathan Lile Suraj Anand Hailey Schoelkopf Siddharth Verma Stella Biderman 71 10 0 12 Feb 2024
Mercury: A Code Efficiency Benchmark for Code Large Language Models Mingzhe Du Anh Tuan Luu Bin Ji Qian Liu See-Kiong Ng ALM ELM OffRL 24 7 0 12 Feb 2024
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model Ahmet Üstün Viraat Aryabumi Zheng-Xin Yong Wei-Yin Ko Daniel D'souza ... Shayne Longpre Niklas Muennighoff Marzieh Fadaee Julia Kreutzer Sara Hooker ALM ELM SyDa LRM 40 200 0 12 Feb 2024
TELLER: A Trustworthy Framework for Explainable, Generalizable and Controllable Fake News Detection Hui Liu Wenya Wang Haoru Li Haoliang Li 47 3 0 12 Feb 2024
Secret Collusion among Generative AI Agents: Multi-Agent Deception via Steganography S. Motwani Mikhail Baranchuk Martin Strohmeier Vijay Bolina Philip Torr Lewis Hammond Christian Schroeder de Witt 50 4 0 12 Feb 2024
ODIN: Disentangled Reward Mitigates Hacking in RLHF Lichang Chen Chen Zhu Davit Soselia Jiuhai Chen Dinesh Manocha Tom Goldstein Heng-Chiao Huang Mohammad Shoeybi Bryan Catanzaro AAML 55 54 0 11 Feb 2024
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model Chen Ye Wei Xiong Yuheng Zhang Nan Jiang Tong Zhang OffRL 40 9 0 11 Feb 2024
Using Large Language Models to Automate and Expedite Reinforcement Learning with Reward Machine Shayan Meshkat Alsadat Jean-Raphael Gaglione Daniel Neider Ufuk Topcu Zhe Xu 32 6 0 11 Feb 2024
OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning Rui Ye Wenhao Wang Jingyi Chai Dihan Li Zexi Li Yinda Xu Yaxin Du Yanfeng Wang Siheng Chen ALM FedML AIFin 13 79 0 10 Feb 2024
Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF Han Shen Zhuoran Yang Tianyi Chen OffRL 45 14 0 10 Feb 2024
V-STaR: Training Verifiers for Self-Taught Reasoners Arian Hosseini Xingdi Yuan Nikolay Malkin Rameswar Panda Alessandro Sordoni Rishabh Agarwal ReLM LRM 54 106 0 09 Feb 2024
Fight Back Against Jailbreaking via Prompt Adversarial Tuning Yichuan Mo Yuji Wang Zeming Wei Yisen Wang AAML SILM 49 25 0 09 Feb 2024
Large Language Models: A Survey Shervin Minaee Tomas Mikolov Narjes Nikzad M. Asgari-Chenaghlu R. Socher Xavier Amatriain Jianfeng Gao ALM LM&MA ELM 134 377 0 09 Feb 2024
OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models Hainiu Xu Runcong Zhao Lixing Zhu Bin Liang Yulan He 84 21 0 08 Feb 2024
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue Xing Han Lù Zdeněk Kasner Siva Reddy 39 60 0 08 Feb 2024
Limitations of Agents Simulated by Predictive Models Raymond Douglas Jacek Karwowski Chan Bae Andis Draguns Victoria Krakovna 25 0 0 08 Feb 2024
Generalized Preference Optimization: A Unified Approach to Offline Alignment Yunhao Tang Z. Guo Zeyu Zheng Daniele Calandriello Rémi Munos Mark Rowland Pierre Harvey Richemond Michal Valko Bernardo Avila-Pires Bilal Piot 34 93 0 08 Feb 2024
In-Context Learning Can Re-learn Forbidden Tasks Sophie Xhonneux David Dobre Jian Tang Gauthier Gidel Dhanya Sridhar 27 3 0 08 Feb 2024
Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation Xianghe Pang Shuo Tang Rui Ye Yuxin Xiong Bolun Zhang Yanfeng Wang Siheng Chen 122 29 0 08 Feb 2024
Noise Contrastive Alignment of Language Models with Explicit Rewards Huayu Chen Guande He Lifan Yuan Ganqu Cui Hang Su Jun Zhu 63 44 0 08 Feb 2024
A Survey on Safe Multi-Modal Learning System Tianyi Zhao Liangliang Zhang Yao Ma Lu Cheng 65 10 0 08 Feb 2024
Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications Boyi Wei Kaixuan Huang Yangsibo Huang Tinghao Xie Xiangyu Qi Mengzhou Xia Prateek Mittal Mengdi Wang Peter Henderson AAML 63 87 0 07 Feb 2024
Pedagogical Alignment of Large Language Models Shashank Sonkar Kangqi Ni Sapana Chaudhary Richard G. Baraniuk AI4Ed 18 7 0 07 Feb 2024
Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning Hao Zhao Maksym Andriushchenko Francesco Croce Nicolas Flammarion ALM 97 44 0 07 Feb 2024
Direct Language Model Alignment from Online AI Feedback Shangmin Guo Biao Zhang Tianlin Liu Tianqi Liu Misha Khalman ... Thomas Mesnard Yao-Min Zhao Bilal Piot Johan Ferret Mathieu Blondel ALM 42 134 0 07 Feb 2024
AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls Yu Du Fangyun Wei Hongyang R. Zhang LLMAG 40 38 0 06 Feb 2024
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal Mantas Mazeika Long Phan Xuwang Yin Andy Zou Zifan Wang ... Nathaniel Li Steven Basart Bo Li David A. Forsyth Dan Hendrycks AAML 35 335 0 06 Feb 2024
MusicRL: Aligning Music Generation to Human Preferences Geoffrey Cideron Sertan Girgin Mauro Verzetti Damien Vincent Matej Kastelic ... Olivier Pietquin Matthieu Geist Léonard Hussenot Neil Zeghidour A. Agostinelli 45 17 0 06 Feb 2024
Systematic Biases in LLM Simulations of Debates Amir Taubenfeld Yaniv Dover Roi Reichart Ariel Goldstein 36 52 0 06 Feb 2024
Personalized Language Modeling from Personalized Human Feedback Xinyu Li Zachary C. Lipton Liu Leqi ALM 73 48 0 06 Feb 2024
Toward Human-AI Alignment in Large-Scale Multi-Player Games Sugandha Sharma Guy Davidson Khimya Khetarpal Anssi Kanervisto Udit Arora Katja Hofmann Ida Momennejad 35 0 0 05 Feb 2024
SWAG: Storytelling With Action Guidance Zeeshan Patel Karim El-Refai Jonathan Pei Tianle Li LLMAG 26 4 0 05 Feb 2024
Psychological Assessments with Large Language Models: A Privacy-Focused and Cost-Effective Approach Sergi Blanco-Cuaresma 34 1 0 05 Feb 2024
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models Zhihong Shao Peiyi Wang Qihao Zhu Runxin Xu Jun-Mei Song ... Haowei Zhang Mingchuan Zhang Y. K. Li Yu-Huan Wu Daya Guo ReLM LRM 51 746 0 05 Feb 2024
MobilityGPT: Enhanced Human Mobility Modeling with a GPT model Ammar Haydari Dongjie Chen Zhengfeng Lai Michael Zhang Chen-Nee Chuah 72 8 0 05 Feb 2024
Decoding-time Realignment of Language Models Tianlin Liu Shangmin Guo Leonardo Bianco Daniele Calandriello Quentin Berthet Felipe Llinares-López Jessica Hoffmann Lucas Dixon Michal Valko Mathieu Blondel AI4CE 54 37 0 05 Feb 2024