Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,707 papers shown

Title
Smaller Large Language Models Can Do Moral Self-Correction Guangliang Liu Zhiyu Xue Rongrong Wang K. Johnson Kristen Marie Johnson LRM 42 0 0 30 Oct 2024
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following Gabrielle Kaili-May Liu Bowen Shi Avi Caciularu Idan Szpektor Arman Cohan 72 4 0 30 Oct 2024
Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning Yihe Deng Paul Mineiro LRM 31 3 0 29 Oct 2024
DISCERN: Decoding Systematic Errors in Natural Language for Text Classifiers Rakesh R Menon Shashank Srivastava 31 2 0 29 Oct 2024
AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attempts Vishal Kumar Zeyi Liao Jaylen Jones Huan Sun AAML 46 2 0 29 Oct 2024
Sing it, Narrate it: Quality Musical Lyrics Translation Zhuorui Ye Jiajun Li Rongwu Xu 50 1 0 29 Oct 2024
PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference Kendong Liu Zhiyu Zhu Chuanhao Li Hui Liu H. Zeng Junhui Hou EGVM 51 2 0 29 Oct 2024
SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types Yutao Mou Shikun Zhang Wei Ye ELM 57 12 0 29 Oct 2024
A Hierarchical Language Model For Interpretable Graph Reasoning Sambhav Khurana Xiner Li Shurui Gui Shuiwang Ji LRM 52 0 0 29 Oct 2024
Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate Zhiqi Bu Xiaomeng Jin Bhanukiran Vinzamuri Anil Ramakrishna Kai-Wei Chang Volkan Cevher Mingyi Hong MU 91 7 0 29 Oct 2024
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence Minimization Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 64 4 0 29 Oct 2024
Transferable Post-training via Inverse Value Learning Xinyu Lu Xueru Wen Yaojie Lu Bowen Yu Hongyu Lin Haiyang Yu Le Sun Xianpei Han Yongbin Li 28 1 0 28 Oct 2024
Diff-Instruct*: Towards Human-Preferred One-step Text-to-image Generative Models Weijian Luo C. Zhang Debing Zhang Zhengyang Geng 35 4 0 28 Oct 2024
Matryoshka: Learning to Drive Black-Box LLMs with LLMs Changhao Li Yuchen Zhuang Rushi Qiang Haotian Sun H. Dai Chao Zhang Bo Dai LRM 33 4 0 28 Oct 2024
L3Ms -- Lagrange Large Language Models Guneet S. Dhillon Xingjian Shi Yee Whye Teh Alex Smola 306 0 0 28 Oct 2024
Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation Jaechang Kim Jinmin Goh Inseok Hwang Jaewoong Cho Jungseul Ok ELM 38 1 0 28 Oct 2024
UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function Zhichao Wang Bin Bi Z. Zhu Xiangbo Mao Jun Wang Shiyu Wang CLL 33 1 0 28 Oct 2024
Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring Honglin Mu Han He Yuxin Zhou Yunlong Feng Yang Xu ... Zeming Liu Xudong Han Qi Shi Qingfu Zhu Wanxiang Che AAML 55 1 0 28 Oct 2024
Rethinking Data Synthesis: A Teacher Model Training Recipe with Interpretation Yifang Chen David Zhu SyDa 46 0 0 27 Oct 2024
Accelerating Direct Preference Optimization with Prefix Sharing Franklin Wang Sumanth Hegde 41 0 0 27 Oct 2024
Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data Xinhong Xie Tao Li Quanyan Zhu 32 3 0 27 Oct 2024
Fine-Tuning and Evaluating Open-Source Large Language Models for the Army Domain Daniel C. Ruiz John Sell 25 1 0 27 Oct 2024
Guiding Through Complexity: What Makes Good Supervision for Hard Math Reasoning Tasks? Xuan He Da Yin Nanyun Peng LRM 44 0 0 27 Oct 2024
Fast Best-of-N Decoding via Speculative Rejection Hanshi Sun Momin Haider Ruiqi Zhang Huitao Yang Jiahao Qiu Ming Yin Mengdi Wang Peter L. Bartlett Andrea Zanette BDL 50 35 0 26 Oct 2024
Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Models Mohammad Beigi Sijia Wang Ying Shen Zihao Lin Adithya Kulkarni ... Ming Jin Jin-Hee Cho Dawei Zhou Chang-Tien Lu Lifu Huang 46 1 0 26 Oct 2024
Uncertainty-Penalized Direct Preference Optimization Sam Houliston Alizée Pace Alexander Immer Gunnar Rätsch 39 0 0 26 Oct 2024
GFlowNet Fine-tuning for Diverse Correct Solutions in Mathematical Reasoning Tasks Ryoichi Takase Masaya Tsunokake Yuta Tsuchiya Shota Inuzuka LRM 56 3 0 26 Oct 2024
Vulnerability of LLMs to Vertically Aligned Text Manipulations Zhecheng Li Yijiao Wang Bryan Hooi Yujun Cai Zhen Xiong Nanyun Peng Kai-Wei Chang 71 1 0 26 Oct 2024
2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision Shilong Li Yancheng He Hui Huang Xingyuan Bu Qingbin Liu Hangyu Guo Weixun Wang Jihao Gu Wenbo Su Bo Zheng 43 5 0 25 Oct 2024
OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization Hongliang He Wenlin Yao Kaixin Ma Wenhao Yu Han Zhang Tianqing Fang Zhenzhong Lan Dong Yu LM&Ro LLMAG 48 11 0 25 Oct 2024
Improving Inverse Folding for Peptide Design with Diversity-regularized Direct Preference Optimization Ryan Park Darren J. Hsu C. Brian Roland Maria Korshunova Chen Tessler Shie Mannor Olivia Viessmann Bruno Trentini 42 1 0 25 Oct 2024
Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learning Yujian Liu Shiyu Chang Tommi Jaakkola Yang Zhang 44 0 0 25 Oct 2024
Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization Xiyue Peng Hengquan Guo Jiawei Zhang Dongqing Zou Ziyu Shao Honghao Wei Xin Liu 49 0 0 25 Oct 2024
Inference time LLM alignment in single and multidomain preference spectrum Siyang Song Zheng Qi Nikolaos Pappas Srikanth Doss Kadarundalagi Raghuram Doss Monica Sunkara Kishaloy Halder Manuel Mager Yassine Benajiba 42 0 0 24 Oct 2024
MAP: Multi-Human-Value Alignment Palette Xinran Wang Qi Le A. N. Ahmed Enmao Diao Yi Zhou Nathalie Baracaldo Jie Ding Ali Anwar 37 2 0 24 Oct 2024
Improving Small-Scale Large Language Models Function Calling for Reasoning Tasks Graziano A. Manduzio Federico A. Galatolo M. G. Cimino Enzo Pasquale Scilingo Lorenzo Cominelli LRM 34 1 0 24 Oct 2024
Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences Weijian Luo EGVM 48 6 0 24 Oct 2024
Little Giants: Synthesizing High-Quality Embedding Data at Scale Haonan Chen Liang Wang Nan Yang Yinlin Zhu Ziliang Zhao Furu Wei Zhicheng Dou SyDa 50 1 0 24 Oct 2024
Aligning CodeLLMs with Direct Preference Optimization Yibo Miao Bofei Gao Shanghaoran Quan Junyang Lin Daoguang Zan Qingbin Liu Jian Yang Tianyu Liu Zhijie Deng 66 5 0 24 Oct 2024
LOGO -- Long cOntext aliGnment via efficient preference Optimization Zecheng Tang Zechen Sun Juntao Li Qiaoming Zhu Min Zhang 42 2 0 24 Oct 2024
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs Chris Yuhao Liu Liang Zeng Qingbin Liu Rui Yan Jujie He Chaojie Wang Shuicheng Yan Yang Liu Yahui Zhou AI4TS 54 79 0 24 Oct 2024
The Nature of Mathematical Modeling and Probabilistic Optimization Engineering in Generative AI Fulu Li 27 0 0 24 Oct 2024
Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch Yuyang Ding Xinyu Shi Xiaobo Liang Juntao Li Zhaopeng Tu Qiaoming Zhu Min Zhang LRM ELM AIMat SyDa 66 10 0 24 Oct 2024
Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model Wenhong Zhu Zhiwei He Xiaofeng Wang Pengfei Liu Rui Wang OSLM 67 4 0 24 Oct 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies Liwen Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 54 5 0 24 Oct 2024
Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback Lester James V. Miranda Yizhong Wang Yanai Elazar Sachin Kumar Valentina Pyatkin Faeze Brahman Noah A. Smith Hannaneh Hajishirzi Pradeep Dasigi 57 8 0 24 Oct 2024
Improving Model Factuality with Fine-grained Critique-based Evaluator Yiqing Xie Wenxuan Zhou Pradyot Prakash Di Jin Yuning Mao ... Sinong Wang Han Fang Carolyn Rose Daniel Fried Hejia Zhang HILM 60 6 0 24 Oct 2024
End-to-end Training for Recommendation with Language-based User Profiles Zhaolin Gao Joyce Zhou Yijia Dai Thorsten Joachims AI4Ed 72 3 0 24 Oct 2024
Cross-lingual Transfer of Reward Models in Multilingual Alignment Jiwoo Hong Noah Lee Rodrigo Martínez-Castaño César Rodríguez James Thorne 51 5 0 23 Oct 2024
Scalable Ranked Preference Optimization for Text-to-Image Generation Shyamgopal Karthik Huseyin Coskun Zeynep Akata Sergey Tulyakov J. Ren Anil Kag EGVM 57 6 0 23 Oct 2024