Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,684 papers shown

Title
Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain Davide Mazzaccara A. Testoni Raffaella Bernardi 54 2 0 25 Jun 2024
Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning Sen Yang Leyang Cui Deng Cai Xinting Huang Shuming Shi Wai Lam 51 8 0 25 Jun 2024
Predicting the Big Five Personality Traits in Chinese Counselling Dialogues Using Large Language Models Yang Yan Lizhi Ma Anqi Li Jingsong Ma Zhenzhong Lan 34 2 0 25 Jun 2024
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment Thom Lake Eunsol Choi Greg Durrett 63 9 0 25 Jun 2024
Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models Nisarg Patel Mohith Kulkarni Mihir Parmar Aashna Budhiraja Mutsumi Nakamura Neeraj Varshney Chitta Baral ELM LRM 50 7 0 24 Jun 2024
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Shengbang Tong Ellis L Brown Penghao Wu Sanghyun Woo Manoj Middepogu ... Xichen Pan Austin Wang Rob Fergus Yann LeCun Saining Xie 3DV MLLM 62 311 0 24 Jun 2024
From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models Sean Welleck Amanda Bertsch Matthew Finlayson Hailey Schoelkopf Alex Xie Graham Neubig Ilia Kulikov Zaid Harchaoui 80 59 0 24 Jun 2024
Modulating Language Model Experiences through Frictions Katherine M. Collins Valerie Chen Ilia Sucholutsky Hannah Rose Kirk Malak Sadek Holli Sargeant Ameet Talwalkar Adrian Weller Umang Bhatt KELM 74 5 0 24 Jun 2024
WARP: On the Benefits of Weight Averaged Rewarded Policies Alexandre Ramé Johan Ferret Nino Vieillard Robert Dadashi Léonard Hussenot Pierre-Louis Cedoz Pier Giuseppe Sessa Sertan Girgin Arthur Douillard Olivier Bachem 62 15 0 24 Jun 2024
Adversarial Contrastive Decoding: Boosting Safety Alignment of Large Language Models via Opposite Prompt Optimization Zhengyue Zhao Xiaoyun Zhang Kaidi Xu Xing Hu Rui Zhang Zidong Du Qi Guo Yunji Chen 60 6 0 24 Jun 2024
On the Transformations across Reward Model, Parameter Update, and In-Context Prompt Deng Cai Huayang Li Tingchen Fu Siheng Li Weiwen Xu ... Leyang Cui Yan Wang Lemao Liu Taro Watanabe Shuming Shi KELM 37 2 0 24 Jun 2024
Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models? Yuu Jinnai 72 1 0 24 Jun 2024
MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention Yuxin Chen Chen Tang Chenran Li Ran Tian Peter Stone Masayoshi Tomizuka Wei Zhan 31 1 0 24 Jun 2024
Large Language Models Assume People are More Rational than We Really are Ryan Liu Jiayi Geng Joshua C. Peterson Ilia Sucholutsky Thomas Griffiths 76 19 0 24 Jun 2024
Statistical ranking with dynamic covariates Pinjun Dong Ruijian Han Binyan Jiang Yiming Xu 57 0 0 24 Jun 2024
Cascade Reward Sampling for Efficient Decoding-Time Alignment Bolian Li Yifan Wang A. Grama Ruqi Zhang Ruqi Zhang AI4TS 66 10 0 24 Jun 2024
Preference Tuning For Toxicity Mitigation Generalizes Across Languages Xiaochen Li Zheng-Xin Yong Stephen H. Bach CLL 41 15 0 23 Jun 2024
Can LLM Graph Reasoning Generalize beyond Pattern Memorization? Yizhuo Zhang Heng Wang Shangbin Feng Zhaoxuan Tan Xiaochuang Han Tianxing He Yulia Tsvetkov LRM 55 16 0 23 Jun 2024
PORT: Preference Optimization on Reasoning Traces Salem Lahlou Abdalgader Abubaker Hakim Hacid LRM 53 3 0 23 Jun 2024
Language Alignment via Nash-learning and Adaptive feedback Ari Azarafrooz Farshid Faal 37 0 0 22 Jun 2024
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph Roman Vashurin Ekaterina Fadeeva Artem Vazhentsev Akim Tsvigun Daniil Vasilev ... Timothy Baldwin Timothy Baldwin Maxim Panov Artem Shelmanov Artem Shelmanov HILM 70 19 0 21 Jun 2024
Robust Reinforcement Learning from Corrupted Human Feedback Alexander Bukharin Ilgee Hong Haoming Jiang Zichong Li Qingru Zhang Zixuan Zhang Tuo Zhao 60 7 0 21 Jun 2024
SAIL: Self-Improving Efficient Online Alignment of Large Language Models Mucong Ding Souradip Chakraborty Vibhu Agrawal Zora Che Alec Koppel Mengdi Wang Amrit Singh Bedi Furong Huang 56 11 0 21 Jun 2024
A SMART Mnemonic Sounds like "Glue Tonic": Mixing LLMs with Student Feedback to Make Mnemonic Learning Stick Nishant Balepur Matthew Shu Alexander Hoyle Alison Robey Shi Feng Seraphina Goldfarb-Tarrant Jordan Boyd-Graber 58 2 0 21 Jun 2024
Hybrid Alignment Training for Large Language Models Chenglong Wang Hang Zhou Kaiyan Chang Bei Li Yongyu Mu Tong Xiao Tongran Liu Jingbo Zhu 59 4 0 21 Jun 2024
A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems Florin Cuconasu Giovanni Trappolini Nicola Tonellotto Fabrizio Silvestri 60 2 0 21 Jun 2024
Steering Without Side Effects: Improving Post-Deployment Control of Language Models Asa Cooper Stickland Alexander Lyzhov Jacob Pfau Salsabila Mahdi Samuel R. Bowman LLMSV AAML 65 20 0 21 Jun 2024
Pareto-Optimal Learning from Preferences with Hidden Context Ryan Boldi Li Ding Lee Spector S. Niekum 75 6 0 21 Jun 2024
Relation Extraction with Fine-Tuned Large Language Models in Retrieval Augmented Generation Frameworks Sefika Efeoglu Adrian Paschke 47 2 0 20 Jun 2024
Preferential Multi-Objective Bayesian Optimization Raul Astudillo Kejun Li Maegan Tucker Chu Xin Cheng Aaron D. Ames Yisong Yue 59 2 0 20 Jun 2024
PKU-SafeRLHF: A Safety Alignment Preference Dataset for Llama Family Models Yalan Qin Chongye Guo Borong Zhang Boyuan Chen Josef Dai Boren Zheng Tianyi Qiu Boxun Li Yaodong Yang 47 36 0 20 Jun 2024
Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation Chunyuan Deng Yilun Zhao Yuzhao Heng Yitong Li Jiannan Cao Xiangru Tang Arman Cohan 48 13 0 20 Jun 2024
RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold Amrith Rajagopal Setlur Saurabh Garg Xinyang Geng Naman Garg Virginia Smith Aviral Kumar 59 49 0 20 Jun 2024
Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning Chaojie Wang Yanchen Deng Zhiyi Lyu Liang Zeng Jujie He Shuicheng Yan Bo An LRM ReLM 47 54 0 20 Jun 2024
Timo: Towards Better Temporal Reasoning for Language Models Zhaochen Su Jun Zhang Tong Zhu Xiaoye Qu Juntao Li Min Zhang Yu Cheng LRM 54 20 0 20 Jun 2024
Aligning Large Language Models with Diverse Political Viewpoints Dominik Stammbach Philine Widmer Eunjung Cho Çağlar Gülçehre Elliott Ash 66 4 0 20 Jun 2024
Finding Safety Neurons in Large Language Models Jianhui Chen Xiaozhi Wang Zijun Yao Yushi Bai Lei Hou Juanzi Li KELM LLMSV 50 15 0 20 Jun 2024
MACAROON: Training Vision-Language Models To Be Your Engaged Partners Shujin Wu Yi R. Fung Sha Li Yixin Wan Kai-Wei Chang Heng Ji 54 6 0 20 Jun 2024
How Many Parameters Does it Take to Change a Light Bulb? Evaluating Performance in Self-Play of Conversational Games as a Function of Model Characteristics Nidhir Bhavsar Jonathan Jordan Sherzod Hakimov David Schlangen 47 0 0 20 Jun 2024
LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback Bofei Gao Zefan Cai Runxin Xu Peiyi Wang Ce Zheng ... Chang Zhou Wen Xiao Junjie Hu Tianyu Liu Baobao Chang LRM 53 17 0 20 Jun 2024
GenderAlign: An Alignment Dataset for Mitigating Gender Bias in Large Language Models Tao Zhang Huiping Zhuang Yuxiang Xiao Huiping Zhuang Cen Chen James R. Foulds Shimei Pan CVBM 59 3 0 20 Jun 2024
What Teaches Robots to Walk, Teaches Them to Trade too -- Regime Adaptive Execution using Informed Data and LLMs Raeid Saqur 54 3 0 20 Jun 2024
APEER: Automatic Prompt Engineering Enhances Large Language Model Reranking Can Jin Hongwu Peng Shiyu Zhao Zhenting Wang Wujiang Xu Ligong Han Jiahui Zhao Kai Zhong Sanguthevar Rajasekaran Dimitris N. Metaxas KELM 57 33 0 20 Jun 2024
SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Tinghao Xie Xiangyu Qi Yi Zeng Yangsibo Huang Udari Madhushani Sehwag ... Bo Li Kai Li Danqi Chen Peter Henderson Prateek Mittal ALM ELM 60 61 0 20 Jun 2024
Adaptable Logical Control for Large Language Models Honghua Zhang Po-Nien Kung Masahiro Yoshida Guy Van den Broeck Nanyun Peng 42 8 0 19 Jun 2024
StableSemantics: A Synthetic Language-Vision Dataset of Semantic Representations in Naturalistic Images Rushikesh Zawar Shaurya Dewan Andrew F. Luo Margaret M. Henderson Michael J. Tarr Leila Wehbe VGen CoGe 49 1 0 19 Jun 2024
Towards Minimal Targeted Updates of Language Models with Targeted Negative Training Lily H. Zhang Rajesh Ranganath Arya Tafvizi 52 1 0 19 Jun 2024
Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models Guanting Dong Keming Lu Chengpeng Li Tingyu Xia Bowen Yu Chang Zhou Jingren Zhou SyDa ALM LRM 61 18 0 19 Jun 2024
Learning to Generate Answers with Citations via Factual Consistency Models Rami Aly Zhiqiang Tang Samson Tan George Karypis HILM 54 5 0 19 Jun 2024
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Haoxiang Wang Wei Xiong Tengyang Xie Han Zhao Tong Zhang 62 151 0 18 Jun 2024