Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,611 papers shown

Title
Mavors: Multi-granularity Video Representation for Multimodal Large Language Model Yang Shi Jiaheng Liu Yushuo Guan Zhikai Wu Yujie Zhang ... Bohan Zeng Wei Zhang Fuzheng Zhang Wenjing Yang Di Zhang VGen VLM 73 0 0 14 Apr 2025
InstructEngine: Instruction-driven Text-to-Image Alignment Xingyu Lu Yihan Hu Yang Zhang Kaiyu Jiang Changyi Liu ... Bin Wen C. Yuan Fan Yang Tingting Gao Di Zhang 48 0 0 14 Apr 2025
Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? Yanbo Wang Jiyang Guan Jian Liang Ran He 56 0 0 14 Apr 2025
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models Jinguo Zhu Weiyun Wang Zhe Chen Z. Liu Shenglong Ye ... Dahua Lin Yu Qiao Jifeng Dai Wenhai Wang Wei Wang MLLM VLM 70 19 1 14 Apr 2025
Improving In-Context Learning with Reasoning Distillation Nafis Sadeq Xin Xu Zhouhang Xie Julian McAuley Byungkyu Kang Prarit Lamba Xiang Gao RALM ReLM LRM 40 0 0 14 Apr 2025
Training Small Reasoning LLMs with Cognitive Preference Alignment Wenrui Cai Chengyu Wang Junbing Yan Jun Huang Xiangzhong Fang LRM 26 1 0 14 Apr 2025
Aligning Anime Video Generation with Human Feedback Bingwen Zhu Yudong Jiang Baohan Xu Siqian Yang Mingyu Yin Yidi Wu Huyang Sun Zuxuan Wu EGVM VGen 57 0 0 14 Apr 2025
QM-ToT: A Medical Tree of Thoughts Reasoning Framework for Quantized Model Zongxian Yang Jiayu Qian Z. Huang Kay Chen Tan LM&MA LRM 31 0 0 13 Apr 2025
Alleviating the Fear of Losing Alignment in LLM Fine-tuning Kang Yang Guanhong Tao X. Chen Jun Xu 36 0 0 13 Apr 2025
AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defender Weixiang Zhao Jiahe Guo Yulin Hu Yang Deng An Zhang ... Xinyang Han Yanyan Zhao Bing Qin Tat-Seng Chua Ting Liu AAML LLMSV 43 1 0 13 Apr 2025
DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training Zhenting Wang Guofeng Cui Kun Wan Wentian Zhao 35 1 0 13 Apr 2025
Kongzi: A Historical Large Language Model with Fact Enhancement Jiashu Yang Ningning Wang Yian Zhao Chaoran Feng Junjia Du Hao Pang Zhirui Fang Xuxin Cheng HILM ALM LRM 41 0 0 13 Apr 2025
Slow Thinking for Sequential Recommendation Junjie Zhang Beichen Zhang Wenqi Sun Hongyu Lu Wayne Xin Zhao Yu Chen Zhicheng Dou OffRL LRM 39 0 0 13 Apr 2025
GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models Jixiao Zhang Chunsheng Zuo LRM 48 7 0 13 Apr 2025
SaRO: Enhancing LLM Safety through Reasoning-based Alignment Yutao Mou Yuxiao Luo Shikun Zhang Wei Ye LLMSV LRM 36 0 0 13 Apr 2025
A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future Jialun Zhong Wei Shen Yanzeng Li Songyang Gao Hua Lu Yicheng Chen Yang Zhang Wei Zhou Jinjie Gu Lei Zou LRM 45 3 0 12 Apr 2025
A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directions Chengyu Wang Taolin Zhang Richang Hong Jun Huang ReLM LRM 45 1 0 12 Apr 2025
Feature-Aware Malicious Output Detection and Mitigation Weilong Dong Peiguang Li Yu Tian Xinyi Zeng Fengdi Li Sirui Wang AAML 24 0 0 12 Apr 2025
Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Time Wang Yang Xiang Yue V. Chaudhary Xiaotian Han ReLM LRM 78 2 0 12 Apr 2025
Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model Team Seawead Ceyuan Yang Zhijie Lin Yang Zhao Shanchuan Lin ... Zuquan Song Zhenheng Yang Jiashi Feng Jianchao Yang Lu Jiang DiffM 96 2 0 11 Apr 2025
Generating Fine Details of Entity Interactions Xinyi Gu Jiayuan Mao 32 0 0 11 Apr 2025
Playpen: An Environment for Exploring Learning Through Conversational Interaction Nicola Horst Davide Mazzaccara Antonia Schmidt Michael Sullivan Filippo Momentè ... Alexander Koller Oliver Lemon David Schlangen Mario Giulianelli Alessandro Suglia OffRL 39 0 0 11 Apr 2025
DocAgent: A Multi-Agent System for Automated Code Documentation Generation Dayu Yang Antoine Simoulin Xin Qian Xiaoyi Liu Yuwei Cao Zhaopu Teng Grey Yang LLMAG 59 0 0 11 Apr 2025
Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning FangZhi Xu Hang Yan Chang Ma Haiteng Zhao Qiushi Sun Kanzhi Cheng Junxian He Jun Liu Zhiyong Wu LRM 34 2 0 11 Apr 2025
On The Landscape of Spoken Language Models: A Comprehensive Survey Siddhant Arora Kai-Wei Chang Chung-Ming Chien Yifan Peng Haibin Wu Yossi Adi Emmanuel Dupoux Hung-yi Lee Karen Livescu Shinji Watanabe 52 3 0 11 Apr 2025
Discriminator-Free Direct Preference Optimization for Video Diffusion Haoran Cheng Qide Dong Liang Peng Zhizhou Sha Weiguo Feng Jinghui Xie Zhao Song Shilei Wen Xiaofei He Boxi Wu VGen 173 0 0 11 Apr 2025
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories Xing Han Lù Amirhossein Kazemnejad Nicholas Meade Arkil Patel Dongchan Shin Alejandra Zambrano Karolina Stañczak Peter Shaw Christopher Pal Siva Reddy LLMAG 42 1 0 11 Apr 2025
$SAEs $\textit{Can}$ Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs$ SAEs $\textit{Can}$ Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs Aashiq Muhamed Jacopo Bonato Mona Diab Virginia Smith MU 66 1 0 11 Apr 2025
VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning Haozhe Wang C. Qu Zuming Huang Wei Chu Fangzhen Lin Wenhu Chen OffRL ReLM SyDa LRM VLM 80 1 0 10 Apr 2025
MM-IFEngine: Towards Multimodal Instruction Following Shengyuan Ding Shenxi Wu Xiangyu Zhao Yuhang Zang Haodong Duan Xiaoyi Dong Pan Zhang Yuhang Cao Dahua Lin Jiaqi Wang OffRL 60 2 0 10 Apr 2025
Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining Rosie Zhao Alexandru Meterez Sham Kakade Cengiz Pehlevan Samy Jelassi Eran Malach ReLM LRM 159 2 0 10 Apr 2025
Talking Point based Ideological Discourse Analysis in News Events Nishanth Nakshatri Nikhil Mehta Siyi Liu Sihao Chen Daniel J. Hopkins Dan Roth Dan Goldwasser 34 0 0 10 Apr 2025
Supervised Optimism Correction: Be Confident When LLMs Are Sure Jingyang Zhang Rushuai Yang Shunyu Liu Ting-En Lin Fei Huang Yi Chen Yong Li Dacheng Tao OffRL 29 0 0 10 Apr 2025
Perception-R1: Pioneering Perception Policy with Reinforcement Learning En Yu Kangheng Lin Liang Zhao Jisheng Yin Yana Wei ... Zheng Ge Xiangyu Zhang Daxin Jiang Jingyu Wang Wenbing Tao VLM OffRL LRM 40 3 0 10 Apr 2025
2D-Curri-DPO: Two-Dimensional Curriculum Learning for Direct Preference Optimization Mengyang Li Zhong Zhang 27 0 0 10 Apr 2025
Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions Angela Lopez-Cardona Sebastian Idesis Ioannis Arapakis 31 0 0 09 Apr 2025
CAReDiO: Cultural Alignment of LLM via Representativeness and Distinctiveness Guided Data Optimization Jing Yao Xiaoyuan Yi Jindong Wang Zhicheng Dou Xing Xie 31 0 0 09 Apr 2025
Perception in Reflection Yana Wei Liang Zhao Kangheng Lin En Yu Yuang Peng ... Jianjian Sun Haoran Wei Zheng Ge Xiangyu Zhang Vishal M. Patel 31 0 0 09 Apr 2025
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion Longguang Zhong Fanqi Wan Ziyi Yang Guosheng Liang Tianyuan Shi Xiaojun Quan MoMe 57 0 0 09 Apr 2025
SemEval-2025 Task 5: LLMs4Subjects -- LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog Jennifer D’Souza Sameer Sadruddin Holger Israel Mathias Begoin Diana Slawig 65 5 0 09 Apr 2025
Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models Wei Chen Xin Yan Bin Wen Fan Yang Tingting Gao Di Zhang Long Chen MLLM 97 0 0 09 Apr 2025
Holistic Capability Preservation: Towards Compact Yet Comprehensive Reasoning Models Ling Team Caizhi Tang Chilin Fu Chunwei Wu Jia Guo ... Shuaicheng Li Wenjie Qu Yingting Wu Y. Liu Zhenyu Huang LRM 33 0 0 09 Apr 2025
Bridging the Gap Between Preference Alignment and Machine Unlearning Xiaohua Feng Yuyuan Li Huwei Ji Jiaming Zhang L. Zhang Tianyu Du Chaochao Chen MU 43 0 0 09 Apr 2025
Leanabell-Prover: Posttraining Scaling in Formal Reasoning Jingyuan Zhang Qi Wang Xingguang Ji Yong-Jin Liu Yang Yue Fuzheng Zhang Di Zhang Guorui Zhou Kun Gai LRM 44 4 0 08 Apr 2025
PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learning Xinpeng Ding Kaipeng Zhang Jinahua Han Lanqing Hong Hang Xu Xuelong Li MLLM VLM 236 0 0 08 Apr 2025
Stratified Expert Cloning with Adaptive Selection for User Retention in Large-Scale Recommender Systems Chengzhi Lin Annan Xie Shuchang Liu Wuhong Wang Chuyuan Wang Yongqi Liu OffRL 30 0 0 08 Apr 2025
Information-Theoretic Reward Decomposition for Generalizable RLHF Liyuan Mao Haoran Xu Amy Zhang Weinan Zhang Chenjia Bai 40 0 0 08 Apr 2025
Understanding Machine Unlearning Through the Lens of Mode Connectivity Jiali Cheng Hadi Amiri MU 200 0 0 08 Apr 2025
Sharpness-Aware Parameter Selection for Machine Unlearning Saber Malekmohammadi Hong kyu Lee Li Xiong MU 208 0 0 08 Apr 2025
Can you Finetune your Binoculars? Embedding Text Watermarks into the Weights of Large Language Models Fay Elhassan Niccolò Ajroldi Antonio Orvieto Jonas Geiping 151 0 0 08 Apr 2025