Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
Similarity is Not All You Need: Endowing Retrieval Augmented Generation with Multi Layered Thoughts Chunjing Gan Dan Yang Binbin Hu Hanxiao Zhang Siyuan Li ... Lin Ju Qing Cui Jinjie Gu Lei Liang Jun Zhou 63 9 0 30 May 2024
Preference Alignment with Flow Matching Minu Kim Yongsik Lee Sehyeok Kang Jihwan Oh Song Chong Seyoung Yun 42 1 0 30 May 2024
Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding Kuo Liao Shuang Li Meng Zhao Liqun Liu Mengge Xue Zhenyu Hu Honglin Han Chengguo Yin 47 1 0 30 May 2024
Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation Chengwei Dai Kun Li Wei Zhou Song Hu LRM 48 6 0 30 May 2024
Enhancing Large Vision Language Models with Self-Training on Image Comprehension Yihe Deng Pan Lu Fan Yin Ziniu Hu Sheng Shen James Zou Kai-Wei Chang Wei Wang SyDa VLM LRM 49 37 0 30 May 2024
Is In-Context Learning Sufficient for Instruction Following in LLMs? Hao Zhao Maksym Andriushchenko Francesco Croce Nicolas Flammarion 67 12 0 30 May 2024
Stress-Testing Capability Elicitation With Password-Locked Models Ryan Greenblatt Fabien Roger Dmitrii Krasheninnikov David M. Krueger 58 14 0 29 May 2024
One-Shot Safety Alignment for Large Language Models via Optimal Dualization Xinmeng Huang Shuo Li Yan Sun Osbert Bastani Hamed Hassani Dongsheng Ding 49 4 0 29 May 2024
Preference Learning Algorithms Do Not Learn Preference Rankings Angelica Chen Sadhika Malladi Lily H. Zhang Xinyi Chen Qiuyi Zhang Rajesh Ranganath Kyunghyun Cho 40 24 0 29 May 2024
AI Risk Management Should Incorporate Both Safety and Security Xiangyu Qi Yangsibo Huang Yi Zeng Edoardo Debenedetti Jonas Geiping ... Chaowei Xiao Yue Liu Dawn Song Peter Henderson Prateek Mittal AAML 56 11 0 29 May 2024
Crowdsourcing with Difficulty: A Bayesian Rating Model for Heterogeneous Items Seong Woo Han Ozan Adigüzel Bob Carpenter 57 0 0 29 May 2024
Self-Exploring Language Models: Active Preference Elicitation for Online Alignment Shenao Zhang Donghan Yu Hiteshi Sharma Ziyi Yang Shuohang Wang Hany Hassan Zhaoran Wang LRM 53 28 0 29 May 2024
MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series Ge Zhang Scott Qu Jiaheng Liu Chenchen Zhang Chenghua Lin ... Zi-Kai Zhao Jiajun Zhang Wanli Ouyang Wenhao Huang Wenhu Chen ELM 43 44 0 29 May 2024
PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications Dingkang Yang Jinjie Wei Dongling Xiao Shunli Wang Tong Wu ... Yue Jiang Qingyao Xu Ke Li Peng Zhai Lihua Zhang LM&MA 48 8 0 29 May 2024
Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models Zhanhui Zhou Zhixuan Liu Jie Liu Zhichen Dong Chao Yang Yu Qiao ALM 52 20 0 29 May 2024
Offline Regularised Reinforcement Learning for Large Language Models Alignment Pierre Harvey Richemond Yunhao Tang Daniel Guo Daniele Calandriello M. G. Azar ... Gil Shamir Rishabh Joshi Tianqi Liu Rémi Munos Bilal Piot OffRL 46 24 0 29 May 2024
Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets Peter Devine ALM 19 3 0 29 May 2024
T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback Jiachen Li Weixi Feng Tsu-Jui Fu Xinyi Wang Sugato Basu Wenhu Chen William Y. Wang VGen 43 27 0 29 May 2024
Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning Tianle Zhang Jiayi Guan Lin Zhao Yihang Li Dongjiang Li ... Lei Sun Yue Chen Xuelong Wei Lusong Li Xiaodong He 50 1 0 29 May 2024
Efficient Model-agnostic Alignment via Bayesian Persuasion Fengshuo Bai Mingzhi Wang Zhaowei Zhang Boyuan Chen Yinda Xu Ying Wen Yaodong Yang 58 3 0 29 May 2024
Robust Preference Optimization through Reward Model Distillation Adam Fisch Jacob Eisenstein Vicky Zayats Alekh Agarwal Ahmad Beirami Chirag Nagpal Peter Shaw Jonathan Berant 86 22 0 29 May 2024
Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice Jian-Qiao Zhu Haijiang Yan Thomas Griffiths 84 2 0 29 May 2024
Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Tiansheng Huang Sihao Hu Fatih Ilhan Selim Furkan Tekin Ling Liu 59 26 0 28 May 2024
A Theoretical Understanding of Self-Correction through In-context Alignment Yifei Wang Yuyang Wu Zeming Wei Stefanie Jegelka Yisen Wang LRM 55 17 0 28 May 2024
ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generator Junda Zhu Lingyong Yan Haibo Shi Dawei Yin Lei Sha RALM 44 6 0 28 May 2024
LLM experiments with simulation: Large Language Model Multi-Agent System for Simulation Model Parametrization in Digital Twins Yuchen Xia Daniel Dittler N. Jazdi Haonan Chen M. Weyrich LM&Ro LLMAG 47 4 0 28 May 2024
Aligning to Thousands of Preferences via System Message Generalization Seongyun Lee Sue Hyun Park Seungone Kim Minjoon Seo ALM 52 38 0 28 May 2024
Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment Keming Lu Bowen Yu Fei Huang Yang Fan Runji Lin Chang Zhou MoMe 32 18 0 28 May 2024
The Evolution of Multimodal Model Architectures S. Wadekar Abhishek Chaurasia Aman Chadha Eugenio Culurciello 54 15 0 28 May 2024
Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment Jiaxiang Li Siliang Zeng Hoi-To Wai Chenliang Li Alfredo García Mingyi Hong 71 15 0 28 May 2024
Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization Yuanpu Cao Tianrong Zhang Bochuan Cao Ziyi Yin Lu Lin Fenglong Ma Jinghui Chen LLMSV 37 20 0 28 May 2024
Linguistic Collapse: Neural Collapse in (Large) Language Models Robert Wu Vardan Papyan 57 13 0 28 May 2024
Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales Ju-Seung Byun Andrew Perrault 34 1 0 27 May 2024
RAGSys: Item-Cold-Start Recommender as RAG System Emile Contal Garrin McGoldrick RALM 32 5 0 27 May 2024
Revision Matters: Generative Design Guided by Revision Edits Tao Li Chin-Yi Cheng Amber Xie Gang Li Yang Li 49 1 0 27 May 2024
ReMoDetect: Reward Models Recognize Aligned LLM's Generations Hyunseok Lee Jihoon Tack Jinwoo Shin DeLMO 42 0 0 27 May 2024
Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models Sheng-Hsuan Peng Pin-Yu Chen Matthew Hull Duen Horng Chau 55 24 0 27 May 2024
Prompt Optimization with Human Feedback Xiaoqiang Lin Zhongxiang Dai Arun Verma See-Kiong Ng Patrick Jaillet K. H. Low AAML 36 8 0 27 May 2024
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering Hongyu Yang Liyang He Min Hou Shuanghong Shen Rui Li Jiahui Hou Jianhui Ma Junda Zhao 35 4 0 27 May 2024
Position: Foundation Agents as the Paradigm Shift for Decision Making Xiaoqian Liu Xingzhou Lou Jianbin Jiao Junge Zhang OffRL LLMAG 50 6 0 27 May 2024
Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models Chia-Yi Hsu Yu-Lin Tsai Chih-Hsun Lin Pin-Yu Chen Chia-Mu Yu Chun-ying Huang 61 39 0 27 May 2024
Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings Robert Wolfe Isaac Slaughter Bin Han Bingbing Wen Yiwei Yang ... Bernease Herman E. Brown Zening Qu Nicholas Weber Bill Howe 48 4 0 27 May 2024
Triple Preference Optimization: Achieving Better Alignment with Less Data in a Single Step Optimization Amir Saeidi Shivanshu Verma Aswin Rrv Chitta Baral 40 0 0 26 May 2024
On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization Jiancong Xiao Ziniu Li Xingyu Xie E. Getzen Cong Fang Qi Long Weijie J. Su 51 13 0 26 May 2024
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer Zhihan Liu Miao Lu Shenao Zhang Boyi Liu Hongyi Guo Yingxiang Yang Jose H. Blanchet Zhaoran Wang 58 43 0 26 May 2024
Multi-Reference Preference Optimization for Large Language Models Hung Le Quan Tran D. Nguyen Kien Do Saloni Mittal Kelechi Ogueji Svetha Venkatesh 65 0 0 26 May 2024
Multi-Player Approaches for Dueling Bandits Or Raveh Junya Honda Masashi Sugiyama 54 1 0 25 May 2024
Efficient Adversarial Training in LLMs with Continuous Attacks Sophie Xhonneux Alessandro Sordoni Stephan Günnemann Gauthier Gidel Leo Schwinn AAML 56 47 0 24 May 2024
Oil & Water? Diffusion of AI Within and Across Scientific Fields Eamon Duede William Dolan André Bauer Ian Foster Karim Lakhani AI4CE 26 4 0 24 May 2024
Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement Xiyao Wang Jiuhai Chen Zhaoyang Wang Yuhang Zhou Yiyang Zhou ... Dinesh Manocha Tom Goldstein Parminder Bhatia Furong Huang Cao Xiao 74 35 0 24 May 2024