Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
STAIR: Improving Safety Alignment with Introspective Reasoning Yuanhang Zhang Siyuan Zhang Yao Huang Zeyu Xia Zhengwei Fang Xiao Yang Ranjie Duan Dong Yan Yinpeng Dong Jun Zhu LRM LLMSV 63 3 0 04 Feb 2025
IPO: Iterative Preference Optimization for Text-to-Video Generation Xiaomeng Yang Zhiyu Tan Xuecheng Nie VGen 114 1 0 04 Feb 2025
Process Reinforcement through Implicit Rewards Ganqu Cui Lifan Yuan Junyao Xing Hanbin Wang Wendi Li ... Yu Cheng Zhiyuan Liu Maosong Sun Bowen Zhou Ning Ding OffRL LRM 77 65 0 03 Feb 2025
Understanding the Capabilities and Limitations of Weak-to-Strong Generalization Wei Yao Wenkai Yang Junyao Xing Yankai Lin Yong Liu ELM 120 2 0 03 Feb 2025
Breaking Focus: Contextual Distraction Curse in Large Language Models Yue Huang Yanbo Wang Zixiang Xu Chujie Gao Siyuan Wu Jiayi Ye Preslav Nakov Pin-Yu Chen Xuzhi Zhang AAML 53 2 0 03 Feb 2025
Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning Udita Ghosh Dripta S. Raychaudhuri Jiachen Li Konstantinos Karydis Amit K. Roy-Chowdhury VLM 63 0 0 03 Feb 2025
Classic4Children: Adapting Chinese Literary Classics for Children with Large Language Model Jiali Chen Xusen Hei Yuqi Xue Zihan Wu Jiayuan Xie Yi Cai AI4Ed 103 1 0 03 Feb 2025
The Battling Influencers Game: Nash Equilibria Structure of a Potential Game and Implications to Value Alignment Young Wu Yancheng Zhu Jin-Yi Cai Xiaojin Zhu 123 0 0 03 Feb 2025
Preference Leakage: A Contamination Problem in LLM-as-a-judge Dawei Li Renliang Sun Yue Huang Ming Zhong Bohan Jiang Jiawei Han Xuzhi Zhang Wei Wang Huan Liu 72 15 0 03 Feb 2025
HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignment Lifan Jiang Boxi Wu Jiahui Zhang Xiaotong Guan Shuang Chen VGen 71 1 0 02 Feb 2025
Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking Jie Ren Yuhang Zhang Dongrui Liu Xiaopeng Zhang Qi Tian 70 0 0 01 Feb 2025
A Three-Branch Checks-and-Balances Frameworkfor Context-Aware Ethical Alignment of Large Language Models Edward Y. Chang AILaw 68 0 0 31 Jan 2025
GuardReasoner: Towards Reasoning-based LLM Safeguards Yue Liu Hongcheng Gao Shengfang Zhai Jun Xia Tianyi Wu Zhiwei Xue Yuxiao Chen Kenji Kawaguchi Jiaheng Zhang Bryan Hooi AI4TS LRM 139 16 0 30 Jan 2025
Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation Yun Wang Tiansheng Huang Li Shen Huanjin Yao Haotian Luo Rui Liu Naiqiang Tan Jiaxing Huang Dacheng Tao AAML MoMe CLL 124 2 0 30 Jan 2025
Diverse Preference Optimization Jack Lanchantin Angelica Chen Shehzaad Dhuliawala Ping Yu Jason Weston Sainbayar Sukhbaatar Ilia Kulikov 118 4 0 30 Jan 2025
CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answering Yumeng Wang Zhiyuan Fan Q. Wang May Fung Heng Ji 87 2 0 30 Jan 2025
Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization Zishun Yu Tengyu Xu Di Jin Karthik Abinav Sankararaman Yun He ... Eryk Helenowski Chen Zhu Sinong Wang Hao Ma Han Fang LRM 59 5 0 29 Jan 2025
COS(M+O)S: Curiosity and RL-Enhanced MCTS for Exploring Story Space via Language Models Tobias Materzok LRM 75 0 0 28 Jan 2025
TimeHF: Billion-Scale Time Series Models Guided by Human Feedback Yongzhi Qi Hao Hu Dazhou Lei Jianshen Zhang Zhengxin Shi Yulin Huang Zhengyu Chen Xiaoming Lin Zuo-jun Shen AI4TS AI4CE 59 2 0 28 Jan 2025
DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data Ke-Han Lu Zhehuai Chen Szu-Wei Fu Chao-Han Huck Yang Jagadeesh Balam Boris Ginsburg Yu-Te Wang Hung-yi Lee AuLLM SyDa 123 8 0 28 Jan 2025
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking Benjamin Feuer Micah Goldblum Teresa Datta Sanjana Nambiar Raz Besaleli Samuel Dooley Max Cembalest John P. Dickerson ALM 47 8 0 28 Jan 2025
Learning to Summarize from LLM-generated Feedback Hwanjun Song Taewon Yun Yuho Lee Jihwan Oh Gihun Lee Jason (Jinglun) Cai Hang Su 78 4 0 28 Jan 2025
Feasible Learning Juan Ramirez Ignacio Hounie Juan Elenter Jose Gallego-Posada Meraj Hashemizadeh Alejandro Ribeiro Simon Lacoste-Julien 39 0 0 28 Jan 2025
Qwen2.5-1M Technical Report An Yang Bowen Yu Chong Li Dayiheng Liu Fei Huang ... Xingzhang Ren Xinlong Yang Yongbin Li Zhiying Xu Zizhuo Zhang 79 13 0 28 Jan 2025
LiPO: Listwise Preference Optimization through Learning-to-Rank Tianqi Liu Zhen Qin Junru Wu Jiaming Shen Misha Khalman ... Mohammad Saleh Simon Baumgartner Jialu Liu Peter J. Liu Xuanhui Wang 141 51 0 28 Jan 2025
Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning Hao Sun M. Schaar 94 14 0 28 Jan 2025
Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation Takyoung Kim Kyungjae Lee Y. Jang Ji Yong Cho Gangwoo Kim Minseok Cho Moontae Lee 159 0 0 28 Jan 2025
Controllable Protein Sequence Generation with LLM Preference Optimization Xiangyu Liu Yi Liu Silei Chen Wei Hu 46 0 0 28 Jan 2025
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics Kai He Rui Mao Qika Lin Yucheng Ruan Xiang Lan Mengling Feng Min Zhang LM&MA AILaw 107 157 0 28 Jan 2025
BoKDiff: Best-of-K Diffusion Alignment for Target-Specific 3D Molecule Generation Ali Khodabandeh Yalabadi Mehdi Yazdani-Jahromi O. Garibay 51 0 0 28 Jan 2025
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning Zehan Qi Xiao-Chang Liu Iat Long Iong Hanyu Lai Xingwu Sun ... Shuntian Yao Tianjie Zhang Wei Xu J. Tang Yuxiao Dong 111 18 0 28 Jan 2025
Visual Generation Without Guidance Huayu Chen Kai Jiang Kaiwen Zheng Jianfei Chen Hang Su Jun Zhu 69 0 0 28 Jan 2025
Training Dialogue Systems by AI Feedback for Improving Overall Dialogue Impression Kai Yoshida M. Mizukami Seiya Kawano Canasai Kruengkrai Hiroaki Sugiyama Koichiro Yoshino ALM OffRL 86 1 0 28 Jan 2025
Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparators Dingkang Yang Dongling Xiao Jinjie Wei Mingcheng Li Zhaoyu Chen Ke Li Li Zhang HILM 103 4 0 28 Jan 2025
Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning Nirav Diwan Tolga Ergen Dongsub Shim Honglak Lee 36 0 0 28 Jan 2025
360Brew: A Decoder-only Foundation Model for Personalized Ranking and Recommendation Hamed Firooz Maziar Sanjabi Adrian Englhardt Aman Gupta Ben Levine ... Xiaoling Zhai Ya Xu Yu Wang Yun Dai Yun Dai ALM 64 3 0 27 Jan 2025
Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning: An Offline Training and Online Fine-Tuning Reinforcement Learning Framework Yulong Hu Tingting Dong Sen Li OffRL OnRL 67 0 0 24 Jan 2025
Improving Video Generation with Human Feedback Jie Liu Gongye Liu Jiajun Liang Ziyang Yuan Xiaokun Liu ... Pengfei Wan Di Zhang Kun Gai Yujiu Yang Wanli Ouyang VGen EGVM 69 13 0 23 Jan 2025
HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humor Zihui Wu Haichang Gao Jiacheng Luo Zhaoxiang Liu 51 0 0 23 Jan 2025
Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment Melissa Kazemi Rad Huy Nghiem Andy Luo Sahil Wadhwa Mohammad Sorower Stephen Rawls AAML 96 2 0 22 Jan 2025
Evolution and The Knightian Blindspot of Machine Learning Joel Lehman Elliot Meyerson Tarek El-Gaaly Kenneth O. Stanley Tarin Ziyaee 102 2 0 22 Jan 2025
Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback Yafu Li Xuyang Hu Xiaoye Qu Linjie Li Yu Cheng 58 3 0 22 Jan 2025
O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning Haotian Luo Li Shen Haiying He Yun Wang Shiwei Liu Wei Li Naiqiang Tan Xiaochun Cao Dacheng Tao VLM LRM 92 53 0 22 Jan 2025
Kimi k1.5: Scaling Reinforcement Learning with LLMs Kimi Team Angang Du Bofei Gao Bowei Xing Changjiu Jiang ... Zhilin Yang Zhiqi Huang Zihao Huang Ziyao Xu Zheng Yang VLM ALM OffRL AI4TS LRM 120 167 0 22 Jan 2025
InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling Yi Wang Xinhao Li Ziang Yan Yinan He Jiashuo Yu ... Kai Chen Wenhai Wang Yu Qiao Yali Wang Limin Wang 93 26 0 21 Jan 2025
From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning Yafu Li Zhilin Wang Tingchen Fu Ganqu Cui Sen Yang Yu Cheng 54 3 0 21 Jan 2025
Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction Yooseop Lee Suin Kim Yohan Jo AI4Ed 68 2 0 21 Jan 2025
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model Yuhang Zang Xiaoyi Dong Pan Zhang Yuhang Cao Ziyu Liu ... Haodong Duan Wenbin Zhang Kai Chen Dahua Lin Jiaqi Wang VLM 97 18 0 21 Jan 2025
DiffDoctor: Diagnosing Image Diffusion Models Before Treating Yiyang Wang Xi Chen Xiaogang Xu S. Ji Yongxu Liu Yujun Shen Hengshuang Zhao DiffM 69 0 0 21 Jan 2025
Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling Zhenyu Hou Xin Lv Rui Lu Jing Zhang Yongqian Li Zijun Yao Juanzi Li J. Tang Yuxiao Dong OffRL LRM ReLM 69 21 0 20 Jan 2025