Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,611 papers shown

Title
JULI: Jailbreak Large Language Models by Self-Introspection Jesson Wang Zhanhao Hu David Wagner 4 0 0 17 May 2025
Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equity Qi Zhou Jie Zhang Dongxia Wang Qiang Liu Tianlin Li Jin Song Dong Wenhai Wang Qing Guo SyDa 4 0 0 17 May 2025
SafeVid: Toward Safety Aligned Video Large Multimodal Models Yixu Wang Jiaxin Song Yifeng Gao Xin Wang Yang Yao Yan Teng Xingjun Ma Yingchun Wang Yu-Gang Jiang 7 0 0 17 May 2025
Telco-oRAG: Optimizing Retrieval-augmented Generation for Telecom Queries via Hybrid Retrieval and Neural Routing Andrei-Laurentiu Bornea Fadhel Ayed Antonio De Domenico Nicola Piovesan Tareq Si Salem Ali Maatouk 12 0 0 17 May 2025
Enhancing Complex Instruction Following for Large Language Models with Mixture-of-Contexts Fine-tuning Yuheng Lu ZiMeng Bai Caixia Yuan Huixing Jiang Xiaojie Wang LRM 7 0 0 17 May 2025
Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets Ning Lu Shengcai Liu Jiahao Wu Weiyu Chen Zhirui Zhang Yew-Soon Ong Qi Wang Ke Tang 2 0 0 17 May 2025
Can Global XAI Methods Reveal Injected Bias in LLMs? SHAP vs Rule Extraction vs RuleSHAP Francesco Sovrano 22 0 0 16 May 2025
Towards Self-Improvement of Diffusion Models via Group Preference Optimization Renjie Chen Wenfeng Lin Yichen Zhang Jiangchuan Wei Boyuan Liu Chao Feng Jiao Ran Mingyu Guo 12 0 0 16 May 2025
SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization Huashan Sun Shengyi Liao Yansen Han Yu Bai Yang Gao ... Weizhou Shen Fanqi Wan Ming Yan J.N. Zhang Fei Huang 22 0 0 16 May 2025
ShiQ: Bringing back Bellman to LLMs Pierre Clavier Nathan Grinsztajn Raphaël Avalos Yannis Flet-Berliac Irem Ergun ... Eugene Tarassov Olivier Pietquin Pierre Harvey Richemond Florian Strub Matthieu Geist OffRL 12 0 0 16 May 2025
A Systematic Analysis of Base Model Choice for Reward Modeling Kian Ahrabian Pegah Jandaghi Negar Mokhberian Sai Praneeth Karimireddy Jay Pujara 27 0 0 16 May 2025
InfiJanice: Joint Analysis and In-situ Correction Engine for Quantization-Induced Math Degradation in Large Language Models Zhen Li Yupeng Su Songmiao Wang Runming Yang C. Xie ... Ming Li Jiannong Cao Yuan Xie Ngai Wong Hongxia Yang MQ 7 0 0 16 May 2025
GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning Yong-Jin Liu Shengfang Zhai Mingzhe Du Yulin Chen Tri Cao ... Xuzhao Li Kun Wang Junfeng Fang Jiaheng Zhang Bryan Hooi OffRL LRM 12 0 0 16 May 2025
HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages Zhigang Wang Jiaqi Zeng Olivier Delalleau Hoo-Chang Shin Felipe Soares Alexander Bukharin Ellie Evans Yi Dong Oleksii Kuchaiev 22 0 0 16 May 2025
Unifying Segment Anything in Microscopy with Multimodal Large Language Model Manyu Li Ruian He Zixian Zhang Weimin Tan Bo Yan VLM 12 0 0 16 May 2025
Spectral Policy Optimization: Coloring your Incorrect Reasoning in GRPO Peter Chen Xiaopeng Li Zhiyu Li Xi Chen Tianyi Lin 9 0 0 16 May 2025
Token-Level Uncertainty Estimation for Large Language Model Reasoning Tunyu Zhang Haizhou Shi Yibin Wang Hengyi Wang Xiaoxiao He ... Ligong Han Kai Xu Huan Zhang Dimitris N. Metaxas Hao Wang LRM 9 0 0 16 May 2025
Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models Fu-Yun Wang Yunhao Shui Jingtan Piao Keqiang Sun Hongsheng Li 25 0 0 16 May 2025
Reinforcement Learning Finetunes Small Subnetworks in Large Language Models Sagnik Mukherjee Lifan Yuan Dilek Hakkani-Tur Hao Peng 7 0 0 16 May 2025
Time-R1: Towards Comprehensive Temporal Reasoning in LLMs Zijia Liu Peixuan Han Haofei Yu Haoru Li Jiaxuan You AI4TS LRM 4 0 0 16 May 2025
Rethinking Prompt Optimizers: From Prompt Merits to Optimization Zixiao Zhu Hanzhang Zhou Zijian Feng Tianjiao Li Chua Jia Jim Deryl Mak Lee Onn Gee Wah Ng Kezhi Mao LRM 34 0 0 15 May 2025
Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback Yutao Yang Jie Zhou Junsong Li Qianjun Pan Bihao Zhan Qin Chen Xipeng Qiu Liang He CLL 29 0 0 15 May 2025
ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization Wenhao Shen Wanqi Yin Xiaofeng Yang Cheng Chen Chaoyue Song Zhongang Cai Lei Yang Hao Wang Guosheng Lin 36 0 0 15 May 2025
InvDesFlow-AL: Active Learning-based Workflow for Inverse Design of Functional Materials Xiao-Qi Han Peng-Jie Guo Ze-Feng Gao Hao Sun Zhong-Yi Lu AI4CE 28 0 0 14 May 2025
WavReward: Spoken Dialogue Models With Generalist Reward Evaluators Shengpeng Ji Tianle Liang Yong Li Jialong Zuo Minghui Fang ... Xize Cheng Siqi Zheng Jin Xu Junyang Lin Zhou Zhao AuLLM ALM 33 0 0 14 May 2025
Atomic Consistency Preference Optimization for Long-Form Question Answering Jingfeng Chen Raghuveer Thirukovalluru Junlin Wang Kaiwei Luo Bhuwan Dhingra KELM HILM 22 0 0 14 May 2025
VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation Chaofan Zhang Peng Hao Xiaoge Cao Xiaoshuai Hao Shaowei Cui Shuo Wang 32 0 0 14 May 2025
Flash-VL 2B: Optimizing Vision-Language Model Performance for Ultra-Low Latency and High Throughput Bo Zhang Shuo Li Runhe Tian Yang Yang Jixin Tang Jinhao Zhou Lin Ma VLM 31 0 0 14 May 2025
Large Language Models Meet Stance Detection: A Survey of Tasks, Methods, Applications, Challenges and Future Directions Lata Pangtey Anukriti Bhatnagar Shubhi Bansal Shahid Shafi Dar Nagendra Kumar 34 0 0 13 May 2025
Preference Optimization for Combinatorial Optimization Problems Mingjun Pan Guanquan Lin You-Wei Luo Bin Zhu Zhien Dai Lijun Sun Chun Yuan 28 0 0 13 May 2025
InfoPO: On Mutual Information Maximization for Large Language Model Alignment Teng Xiao Zhen Ge Sujay Sanghavi Tian Wang Julian Katz-Samuels Marc Versage Qingjun Cui Trishul Chilimbi 31 0 0 13 May 2025
Detecting Prefix Bias in LLM-based Reward Models Ashwin Kumar Yuzi He Aram H. Markosyan Bobbie Chern Imanol Arrieta Ibarra 2 0 0 13 May 2025
Fast Text-to-Audio Generation with Adversarial Post-Training Zachary Novack Zach Evans Zack Zukowski Josiah Taylor CJ Carr ... Adnan Al-Sinan Gian Marco Iodice Julian McAuley Taylor Berg-Kirkpatrick Jordi Pons 30 0 0 13 May 2025
Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation Enci Zhang Xingang Yan Wei Lin Tianxiang Zhang Qianchun Lu LRM 33 0 0 13 May 2025
DanceGRPO: Unleashing GRPO on Visual Generation Zeyue Xue Jie Wu Yu Gao Fangyuan Kong Lingting Zhu ... Zhiheng Liu Wei Liu Qiushan Guo Weilin Huang Ping Luo EGVM VGen 52 0 0 12 May 2025
Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models Rei Higuchi Taiji Suzuki 36 0 0 12 May 2025
Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Models Weiyi Wu Xinwen Xu Chongyang Gao Xingjian Diao Siting Li Lucas A. Salas Jiang Gui 26 0 0 12 May 2025
One Trigger Token Is Enough: A Defense Strategy for Balancing Safety and Usability in Large Language Models Haoran Gu Handing Wang Yi Mei Mengjie Zhang Yaochu Jin 27 0 0 12 May 2025
DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation Jiashuo Sun Xianrui Zhong Sizhe Zhou Jiawei Han RALM 31 0 0 12 May 2025
On the Robustness of Reward Models for Language Model Alignment Jiwoo Hong Noah Lee Eunki Kim Guijin Son Woojin Chung Aman Gupta Shao Tang James Thorne 29 0 0 12 May 2025
Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent Ziyang Huang Xiaowei Yuan Yiming Ju Jun Zhao Kang Liu RALM KELM 26 1 0 12 May 2025
SEM: Reinforcement Learning for Search-Efficient Large Language Models Zeyang Sha Shiwen Cui Weiqiang Wang KELM OffRL LRM 31 0 0 12 May 2025
Critique Before Thinking: Mitigating Hallucination through Rationale-Augmented Instruction Tuning Zexian Yang Dian Li Dayan Wu Gang Liu Weiping Wang MLLM LRM 41 0 0 12 May 2025
Must Read: A Systematic Survey of Computational Persuasion Nimet Beyza Bozdag Shuhaib Mehri Xiaocheng Yang Hyeonjeong Ha Zirui Cheng Esin Durmus Jiaxuan You Heng Ji Gokhan Tur Dilek Hakkani-Tur 46 0 0 12 May 2025
A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models Junjie Ye Caishuang Huang Zhaoyu Chen Wenjie Fu Chenyuan Yang ... Tao Gui Qi Zhang Zhongchao Shi Jianping Fan Xuanjing Huang ALM 46 0 0 12 May 2025
ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolution X. Huang Weiwen Liu Xingshan Zeng Y. Huang Xinlong Hao ... Yirong Zeng Chuhan Wu Yishuo Wang R. Tang Defu Lian KELM 36 0 0 12 May 2025
Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning Xiaokun Wang Chris Jiangbo Pei Wei Shen Yi Peng ... Ai Jian Tianyidan Xie Xuchen Song Yang Liu Yahui Zhou OffRL LRM 28 0 0 12 May 2025
Technical Report: Quantifying and Analyzing the Generalization Power of a DNN Yuxuan He Junpeng Zhang Lei Cheng Hongyuan Zhang Quanshi Zhang AI4CE 26 0 0 11 May 2025
Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety Zihan Guan Mengxuan Hu Ronghang Zhu Sheng Li Anil Vullikanti AAML 31 0 0 11 May 2025
References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulation Doyoung Kim Youngjun Lee Joeun Kim Jihwan Bang Hwanjun Song Susik Yoon Jae-Gil Lee 31 0 0 10 May 2025