Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,611 papers shown

Title
Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks Vishnu Sarukkai Zhiqiang Xie Kayvon Fatahalian LLMAG 75 0 0 01 May 2025
Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning Lang Feng Weihao Tan Zhiyi Lyu Longtao Zheng Haiyang Xu Ming Yan Fei Huang Jingyi Wang 29 0 0 01 May 2025
Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math Haoran Xu Baolin Peng Hany Awadalla Dongdong Chen Yen-Chun Chen ... Yelong Shen S. Wang Weijian Xu Jianfeng Gao Weizhu Chen ReLM LRM 77 3 0 30 Apr 2025
Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models Guanghao Zhou Panjia Qiu Chong Chen Jie Wang Zheming Yang Jian Xu Minghui Qiu OffRL LRM 58 1 0 30 Apr 2025
BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models Zhiting Fan Ruizhe Chen Zuozhu Liu 44 0 0 30 Apr 2025
WebThinker: Empowering Large Reasoning Models with Deep Research Capability X. Li Jiajie Jin Guanting Dong Hongjin Qian Yutao Zhu Yongkang Wu Ji-Rong Wen Zhicheng Dou LLMAG LRM 103 2 0 30 Apr 2025
Ada-R1: Hybrid-CoT via Bi-Level Adaptive Reasoning Optimization Hanjun Luo Haiying He Yucheng Wang Jinluan Yang Rui Liu Naiqiang Tan Xiaochun Cao Dacheng Tao Li Shen LRM 31 1 0 30 Apr 2025
CachePrune: Neural-Based Attribution Defense Against Indirect Prompt Injection Attacks Rui Wang Junda Wu Yu Xia Tong Yu R. Zhang Ryan Rossi Lina Yao Julian McAuley AAML SILM 51 0 0 29 Apr 2025
A Domain-Agnostic Scalable AI Safety Ensuring Framework Beomjun Kim Kangyeon Kim Sunwoo Kim Heejin Ahn 57 0 0 29 Apr 2025
Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts Hanhua Hong Chenghao Xiao Yang Wang Y. Liu Wenge Rong Chenghua Lin 31 0 0 29 Apr 2025
Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Think Hasan Hammoud Hani Itani Guohao Li ReLM LRM 80 1 0 29 Apr 2025
HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation Cristina Garbacea Chenhao Tan 55 0 0 29 Apr 2025
Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception Yuanchen Wu Lu Zhang Hang Yao Junlong Du Ke Yan Shouhong Ding Yunsheng Wu Xuzhao Li MLLM 71 0 0 29 Apr 2025
Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning Joykirat Singh Raghav Magazine Yash Pandya A. Nambi LLMAG KELM OffRL LRM 175 2 0 28 Apr 2025
GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasets Mingqian He Fei Zhao Chonggang Lu Ziqiang Liu Yishuo Wang Haofu Qian OffRL AI4TS VLM 72 0 0 28 Apr 2025
Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving J. Zhang Flood Sung Zhiyong Yang Yang Gao Chongjie Zhang LLMAG 44 0 0 28 Apr 2025
m-KAILIN: Knowledge-Driven Agentic Scientific Corpus Distillation Framework for Biomedical Large Language Models Training Meng Xiao Xunxin Cai Chengrui Wang Yuanchun Zhou 50 0 0 28 Apr 2025
Contextual Online Uncertainty-Aware Preference Learning for Human Feedback Nan Lu Ethan X. Fang Junwei Lu 197 0 0 27 Apr 2025
Anyprefer: An Agentic Framework for Preference Data Synthesis Yiyang Zhou Zhaoxiang Wang Tianle Wang Shangyu Xing Peng Xia ... Chetan Bansal Weitong Zhang Ying Wei Joey Tianyi Zhou Huaxiu Yao 71 1 0 27 Apr 2025
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors Ren-Wei Liang Chin-Ting Hsu Chan-Hung Yu Saransh Agrawal Shih-Cheng Huang Shang-Tse Chen Kuan-Hao Huang Shao-Hua Sun 81 0 0 27 Apr 2025
Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting Y. Hong Xiaofei Yin Xinzhong Wang Yi Tu Ya Guo Sufeng Duan Weiqiang Wang Lingyong Fang Depeng Wang Huijia Zhu CLL 96 0 0 27 Apr 2025
Calibrating Translation Decoding with Quality Estimation on LLMs Di Wu Yibin Lei Christof Monz 75 0 0 26 Apr 2025
Stabilizing Reasoning in Medical LLMs with Continued Pretraining and Reasoning Preference Optimization Wataru Kawakami Keita Suzuki Junichiro Iwasawa LRM 75 0 0 25 Apr 2025
Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review Toghrul Abbasli Kentaroh Toyoda Yuan Wang Leon Witt Muhammad Asif Ali Yukai Miao Dan Li Qingsong Wei UQCV 94 0 0 25 Apr 2025
DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models Jun Liu Hangyu Guo Ranjie Duan Xingyuan Bu Yancheng He ... Yingshui Tan Yanan Wu Jihao Gu Heng Chang Jun Zhu MLLM 202 0 0 25 Apr 2025
Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization Emiliano Penaloza Tianyue H. Zhan Laurent Charlin Mateo Espinosa Zarlenga 51 0 0 25 Apr 2025
Fast-Slow Thinking for Large Vision-Language Model Reasoning W. L. Xiao Leilei Gan Weilong Dai Wanggui He Ziwei Huang ... Fangxun Shu Zhelun Yu Peng Zhang Hao Jiang Fei Wu ReLM LRM AI4CE 212 1 0 25 Apr 2025
Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization Kesen Zhao B. Zhu Qianru Sun Hanwang Zhang MLLM LRM 86 0 0 25 Apr 2025
TRACE Back from the Future: A Probabilistic Reasoning Approach to Controllable Language Generation Gwen Yidou Weng Benjie Wang Mathias Niepert BDL 182 0 0 25 Apr 2025
Aligning Language Models for Icelandic Legal Text Summarization Þórir Hrafn Harðarson Hrafn Loftsson Stefán Ólafsson AILaw AI4TS ELM 82 0 0 25 Apr 2025
Safety in Large Reasoning Models: A Survey Cheng Wang Yong-Jin Liu Yangqiu Song Duzhen Zhang ZeLin Li Junfeng Fang Bryan Hooi LRM 218 1 0 24 Apr 2025
DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training Xiaoyu Tian Sitong Zhao Haotian Wang Shuaiting Chen Yiping Peng Yunjie Ji Han Zhao Xiangang Li LRM 59 3 0 24 Apr 2025
Do Words Reflect Beliefs? Evaluating Belief Depth in Large Language Models Shariar Kabir Kevin Esterling Yue Dong 32 0 0 23 Apr 2025
SMART: Tuning a symbolic music generation system with an audio domain aesthetic reward Nicolas Jonason Luca Casini Bob L. T. Sturm 29 1 0 23 Apr 2025
Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion Ruixiang Zhang Shuangfei Zhai Yizhe Zhang James Thornton Zijing Ou Joshua M. Susskind Navdeep Jaitly DiffM 35 0 0 23 Apr 2025
Param $Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost Sheng Cao Mingrui Wu Karthik Prasad Yuandong Tian Zechun Liu MoMe 85 0 0 23 Apr 2025
Private Federated Learning using Preference-Optimized Synthetic Data Charlie Hou Mei-Yu Wang Yige Zhu Daniel Lazar Giulia Fanti FedML Presented at ResearchTrend Connect \| FedML on 07 May 2025 59 1 0 23 Apr 2025
GreenMind: A Next-Generation Vietnamese Large Language Model for Structured and Logical Reasoning Luu Quy Tung Hoang Quoc Viet Vo Trong Thu LRM 32 0 0 23 Apr 2025
Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Control Hannah Cyberey David Evans LLMSV 78 1 0 23 Apr 2025
Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoning Chris Yichen Wei Yi Peng Xuben Wang Weijie Qiu ... Jianhao Zhang Y. Hao Xuchen Song Yang Liu Yahui Zhou OffRL AI4TS SyDa LRM VLM 79 0 0 23 Apr 2025
Safety Pretraining: Toward the Next Generation of Safe AI Pratyush Maini Sachin Goyal Dylan Sam Alex Robey Yash Savani Yiding Jiang Andy Zou Zacharcy C. Lipton J. Zico Kolter 63 0 0 23 Apr 2025
Unified Molecule Generation and Property Prediction Adam Izdebski Jan Olszewski Pankhil Gawade Krzysztof Koras Serra Korkmaz Valentin Rauscher Jakub M. Tomczak E. Szczurek 36 0 0 23 Apr 2025
ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data Haoran Gu Handing Wang Yi Mei Mengjie Zhang Yaochu Jin 27 1 0 23 Apr 2025
Neural Theorem Proving: Generating and Structuring Proofs for Formal Verification Balaji Rao William Eiers Carlo Lipizzi 37 0 0 23 Apr 2025
Compass-V2 Technical Report Sophia Maria MoE LRM 41 0 0 22 Apr 2025
Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model Junshu Pan Wei Shen Shulin Huang Qiji Zhou Yue Zhang 74 0 0 22 Apr 2025
WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents Siyu Zhou Tianyi Zhou Yijun Yang Guodong Long Deheng Ye Jing Jiang Chengqi Zhang LM&Ro 32 0 0 22 Apr 2025
AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization Jinda Lu Jinghan Li Yuan Gao Junkang Wu Jiancan Wu Xuben Wang Xiangnan He 174 0 0 22 Apr 2025
Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedback N. Wang Bingkun Yao Jie Zhou Yuchen Hu Xi Wang Nan Guan Zhe Jiang OffRL 34 1 0 22 Apr 2025
Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning Jie Cheng Ruixi Qiao Lijun Li Chao Guo Jianmin Wang Gang Xiong Yisheng Lv Fei-Yue Wang LRM 175 3 0 21 Apr 2025