Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,576 papers shown

Title
R1dacted: Investigating Local Censorship in DeepSeek's R1 Language Model Ali Naseh Harsh Chaudhari Jaechul Roh Mingshi Wu Alina Oprea Amir Houmansadr AAML ELM 5 0 0 19 May 2025
ReEx-SQL: Reasoning with Execution-Aware Reinforcement Learning for Text-to-SQL Yaxun Dai Wenxuan Xie Xialie Zhuang Tianyu Yang Yiying Yang Haiqin Yang Yuhang Zhao Pingfu Chao Wenhao Jiang ReLM LRM 12 0 0 19 May 2025
On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding Haoyuan Wu Rui Ming Jilong Gao Hangyu Zhao Xueyi Chen Yikai Yang Haisheng Zheng Zhuolun He Bei Yu 9 0 0 19 May 2025
Shadow-FT: Tuning Instruct via Base Taiqiang Wu Runming Yang Jiayi Li Pengfei Hu Ngai Wong Yujiu Yang 5 0 0 19 May 2025
ProDS: Preference-oriented Data Selection for Instruction Tuning Wenya Guo Zhengkun Zhang Xumeng Liu Ying Zhang Ziyu Lu Haoze Zhu Xubo Liu Ruxue Yan 5 0 0 19 May 2025
DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization Gang Li Ming Lin Tomer Galanti Zhengzhong Tu Tianbao Yang 0 0 0 18 May 2025
Reward Inside the Model: A Lightweight Hidden-State Reward Model for LLM's Best-of-N sampling Jizhou Guo Zhaomin Wu Philip S. Yu 2 0 0 18 May 2025
MARGE: Improving Math Reasoning for LLMs with Guided Exploration Jingyue Gao Runji Lin Keming Lu Bowen Yu Junyang Lin Jianyu Chen LRM 0 0 0 18 May 2025
LAMeTA: Intent-Aware Agentic Network Optimization via a Large AI Model-Empowered Two-Stage Approach Yinqiu Liu Guangyuan Liu Jiacheng Wang Ruichen Zhang Dusit Niyato Geng Sun Zehui Xiong Zhu Han 2 0 0 18 May 2025
Enriching Patent Claim Generation with European Patent Dataset Lekang Jiang Chengzu Li Stephan Goetz 0 0 0 18 May 2025
AbFlowNet: Optimizing Antibody-Antigen Binding Energy via Diffusion-GFlowNet Fusion Abrar Rahman Abir Haz Sameen Shahgir Md Rownok Zahan Ratul Md Toki Tahmid Greg Ver Steeg Yue Dong 0 0 0 18 May 2025
Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning Xinbin Yuan Jian Zhang K. Li Zhuoxuan Cai Lujian Yao ... Enguang Wang Qibin Hou Jinwei Chen Peng-Tao Jiang Bo Li 0 0 0 18 May 2025
SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment Wenqiao Zhu Ji Liu Lulu Wang Jun Wu Yulun Zhang 0 0 0 18 May 2025
RLAP: A Reinforcement Learning Enhanced Adaptive Planning Framework for Multi-step NLP Task Solving Zepeng Ding Dixuan Wang Ziqin Luo Guochao Jiang Deqing Yang Jiaqing Liang 2 0 0 17 May 2025
SafeVid: Toward Safety Aligned Video Large Multimodal Models Yixu Wang Jiaxin Song Yifeng Gao Xin Wang Yang Yao Yan Teng Xingjun Ma Yingchun Wang Yu-Gang Jiang 2 0 0 17 May 2025
Retrospex: Language Agent Meets Offline Reinforcement Learning Critic Yufei Xiang Yiqun Shen Yeqin Zhang Cam-Tu Nguyen OffRL LLMAG KELM LRM 16 0 0 17 May 2025
Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equity Qi Zhou Jie Zhang Dongxia Wang Qiang Liu Tianlin Li Jin Song Dong Wenhai Wang Qing Guo SyDa 2 0 0 17 May 2025
JULI: Jailbreak Large Language Models by Self-Introspection Jesson Wang Zhanhao Hu David Wagner 2 0 0 17 May 2025
Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning Puning Yang Qizhou Wang Zhuo Huang Tongliang Liu Chengqi Zhang Bo Han MU 18 0 0 17 May 2025
Counterspeech the ultimate shield! Multi-Conditioned Counterspeech Generation through Attributed Prefix Learning Aswini Kumar Padhi Anil Bandhakavi Tanmoy Chakraborty 2 0 0 17 May 2025
Enhancing Complex Instruction Following for Large Language Models with Mixture-of-Contexts Fine-tuning Yuheng Lu ZiMeng Bai Caixia Yuan Huixing Jiang Xiaojie Wang LRM 2 0 0 17 May 2025
Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets Ning Lu Shengcai Liu Jiahao Wu Weiyu Chen Zhirui Zhang Y. Ong Qi Wang Ke Tang 2 0 0 17 May 2025
Telco-oRAG: Optimizing Retrieval-augmented Generation for Telecom Queries via Hybrid Retrieval and Neural Routing Andrei-Laurentiu Bornea Fadhel Ayed Antonio De Domenico Nicola Piovesan Tareq Si Salem Ali Maatouk 2 0 0 17 May 2025
VisionReasoner: Unified Visual Perception and Reasoning via Reinforcement Learning Yuqi Liu Tianyuan Qu Zhisheng Zhong Bohao Peng Shu Liu Bei Yu Jiaya Jia VLM LRM 23 0 0 17 May 2025
ShiQ: Bringing back Bellman to LLMs Pierre Clavier Nathan Grinsztajn Raphaël Avalos Yannis Flet-Berliac Irem Ergun ... Eugene Tarassov Olivier Pietquin Pierre Harvey Richemond Florian Strub Matthieu Geist OffRL 12 0 0 16 May 2025
Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models Fu-Yun Wang Yunhao Shui Jingtan Piao Keqiang Sun Hongsheng Li 25 0 0 16 May 2025
Towards Self-Improvement of Diffusion Models via Group Preference Optimization Renjie Chen Wenfeng Lin Yichen Zhang Jiangchuan Wei Boyuan Liu Chao Feng Jiao Ran Mingyu Guo 12 0 0 16 May 2025
InfiJanice: Joint Analysis and In-situ Correction Engine for Quantization-Induced Math Degradation in Large Language Models Zhen Li Yupeng Su Songmiao Wang Runming Yang C. Xie ... Ming Li Jiannong Cao Yuan Xie Ngai Wong Hongxia Yang MQ 2 0 0 16 May 2025
Token-Level Uncertainty Estimation for Large Language Model Reasoning Tunyu Zhang Haizhou Shi Yibin Wang Hengyi Wang X. He ... Ligong Han Kai Xu Huatian Zhang Dimitris N. Metaxas Hao Wang LRM 2 0 0 16 May 2025
Spectral Policy Optimization: Coloring your Incorrect Reasoning in GRPO Peter Chen Xiaopeng Li Zeju Li Xi Chen Tianyi Lin 2 0 0 16 May 2025
A Systematic Analysis of Base Model Choice for Reward Modeling Kian Ahrabian Pegah Jandaghi Negar Mokhberian Sai Praneeth Karimireddy Jay Pujara 24 0 0 16 May 2025
HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages ziqi wang Jiaqi Zeng Olivier Delalleau Hoo-Chang Shin Felipe Soares Alexander Bukharin Ellie Evans Yi Dong Oleksii Kuchaiev 22 0 0 16 May 2025
GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning Yong Liu Shengfang Zhai Mingzhe Du Yulin Chen Tri Cao ... Zhaoxin Fan Kun Wang Junfeng Fang Jiaheng Zhang Bryan Hooi OffRL LRM 7 0 0 16 May 2025
SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization Huashan Sun Shengyi Liao Yansen Han Yu Bai Chong Feng ... Weizhou Shen Fanqi Wan Ming Yan Junzhe Zhang Fei Huang 20 0 0 16 May 2025
Unifying Segment Anything in Microscopy with Multimodal Large Language Model Manyu Li Ruian He Zixian Zhang Weimin Tan Bo Yan VLM 12 0 0 16 May 2025
Can Global XAI Methods Reveal Injected Bias in LLMs? SHAP vs Rule Extraction vs RuleSHAP Francesco Sovrano 14 0 0 16 May 2025
Reinforcement Learning Finetunes Small Subnetworks in Large Language Models Sagnik Mukherjee Lifan Yuan Dilek Hakkani-Tur Hao Peng 2 0 0 16 May 2025
Rethinking Prompt Optimizers: From Prompt Merits to Optimization Zixiao Zhu Hanzhang Zhou Zijian Feng Tianjiao Li Chua Jia Jim Deryl Mak Lee Onn Gee Wah Ng Kezhi Mao LRM 31 0 0 15 May 2025
ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization Wenhao Shen Wanqi Yin Xiaofeng Yang Cheng Chen Chaoyue Song Zhongang Cai Lei Yang Hao Wang Guosheng Lin 33 0 0 15 May 2025
Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback Yutao Yang Jie Zhou Junsong Li Qianjun Pan Bihao Zhan Qin Chen Xipeng Qiu Liang He CLL 29 0 0 15 May 2025
Atomic Consistency Preference Optimization for Long-Form Question Answering Jingfeng Chen Raghuveer Thirukovalluru Junlin Wang Kaiwei Luo Bhuwan Dhingra KELM HILM 20 0 0 14 May 2025
WavReward: Spoken Dialogue Models With Generalist Reward Evaluators Shengpeng Ji Tianle Liang Yongqian Li Jialong Zuo Minghui Fang ... Xize Cheng Siqi Zheng Jin Xu Junyang Lin Zhou Zhao AuLLM ALM 33 0 0 14 May 2025
InvDesFlow-AL: Active Learning-based Workflow for Inverse Design of Functional Materials Xiao-Qi Han Peng-Jie Guo Ze-Feng Gao Hao Sun Zhong-Yi Lu AI4CE 28 0 0 14 May 2025
Flash-VL 2B: Optimizing Vision-Language Model Performance for Ultra-Low Latency and High Throughput Bo Zhang Shuo Li Runhe Tian Yang Yang Jixin Tang Jinhao Zhou Lin Ma VLM 30 0 0 14 May 2025
VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation Chaofan Zhang Peng Hao Xiaoge Cao Xiaoshuai Hao Shaowei Cui Shuo Wang 32 0 0 14 May 2025
Preference Optimization for Combinatorial Optimization Problems Mingjun Pan Guanquan Lin You-Wei Luo Bin Zhu Zhien Dai Lijun Sun Chun Yuan 23 0 0 13 May 2025
Large Language Models Meet Stance Detection: A Survey of Tasks, Methods, Applications, Challenges and Future Directions Lata Pangtey Anukriti Bhatnagar Shubhi Bansal Shahid Shafi Dar Nagendra Kumar 34 0 0 13 May 2025
Fast Text-to-Audio Generation with Adversarial Post-Training Zachary Novack Zach Evans Zack Zukowski Josiah Taylor CJ Carr ... Adnan Al-Sinan Gian Marco Iodice Julian McAuley Taylor Berg-Kirkpatrick Jordi Pons 30 0 0 13 May 2025
Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation Enci Zhang Xingang Yan Wei Lin Tianxiang Zhang Qianchun Lu LRM 30 0 0 13 May 2025
InfoPO: On Mutual Information Maximization for Large Language Model Alignment Teng Xiao Zhen Ge Sujay Sanghavi Tian Wang Julian Katz-Samuels Marc Versage Qingjun Cui Trishul Chilimbi 31 0 0 13 May 2025