Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,611 papers shown

Title
Learning Guarantee of Reward Modeling Using Deep Neural Networks Yuanhang Luo Yeheng Ge Ruijian Han Guohao Shen 36 0 0 10 May 2025
xGen-small Technical Report Erik Nijkamp Bo Pang Egor Pakhomov Akash Gokul Jin Qu Silvio Savarese Yingbo Zhou Caiming Xiong LLMAG 58 0 0 10 May 2025
Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation Stefan Vasilev Christian Herold Baohao Liao Seyyed Hadi Hashemi Shahram Khadivi Christof Monz MU 203 0 0 09 May 2025
Assessing Robustness to Spurious Correlations in Post-Training Language Models Julia Shuieh Prasann Singhal Apaar Shanker John Heyer George Pu Samuel Denton LRM 34 0 0 09 May 2025
Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes Zhuocheng Gong Jian Guan Wei Wu Huishuai Zhang Dongyan Zhao 69 1 0 08 May 2025
ReAlign: Bilingual Text-to-Motion Generation via Step-Aware Reward-Guided Alignment Wanjiang Weng Xiaofeng Tan Hongsong Wang Pan Zhou VGen 51 0 0 08 May 2025
G-FOCUS: Towards a Robust Method for Assessing UI Design Persuasiveness Jaehyun Jeon Janghan Yoon Minsoo Kim Sumin Shim Yejin Choi Hanbin Kim Youngjae Yu AAML 47 0 0 08 May 2025
ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoning Ziqing Qiao Yongheng Deng Jiali Zeng Dong Wang Lai Wei Fandong Meng Jie Zhou Ju Ren Yaoxue Zhang LRM 54 0 0 08 May 2025
Flow-GRPO: Training Flow Matching Models via Online RL Jie Liu Gongye Liu Jiajun Liang Yong Li Jiaheng Liu Xueliang Wang Pengfei Wan Di Zhang Wanli Ouyang AI4CE 70 0 0 08 May 2025
LSRP: A Leader-Subordinate Retrieval Framework for Privacy-Preserving Cloud-Device Collaboration Wenjie Qu Pengyue Jia Xin Li Derong Xu Maolin Wang ... Zhaocheng Du Huifeng Guo Y. Liu Ruiming Tang Xiangyu Zhao 49 0 0 08 May 2025
Scalable Chain of Thoughts via Elastic Reasoning Yuhui Xu Hanze Dong Lei Wang Doyen Sahoo Junnan Li Caiming Xiong OffRL LRM 51 2 0 08 May 2025
Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization Wenjun Cao AAML 44 0 0 07 May 2025
Guide your favorite protein sequence generative model Junhao Xiong Hunter Nisonoff Ishan Gaur Jennifer Listgarten DiffM 56 0 0 07 May 2025
Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment Xueyao Zhang Yufei Wang Chaoren Wang Zehan Li Zhuo Chen Zhizheng Wu 155 0 0 07 May 2025
EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning Zhenghao Xing Xiaowei Hu Chi-Wing Fu Wei Wang Jifeng Dai Pheng-Ann Heng MLLM OffRL VLM LRM 55 0 0 07 May 2025
GASCADE: Grouped Summarization of Adverse Drug Event for Enhanced Cancer Pharmacovigilance Sofia Jamil Aryan Dabad Bollampalli Areen Reddy S. Saha Rajiv Misra Adil A. Shakur 56 0 0 07 May 2025
On-Device LLM for Context-Aware Wi-Fi Roaming Ju-Hyung Lee Yanqing Lu Klaus Doppler 30 0 0 07 May 2025
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $α$ - $β$ -Divergence Guanghui Wang Zhiyong Yang Zihan Wang Shi Wang Qianqian Xu Qingming Huang 42 0 0 07 May 2025
LLAMAPIE: Proactive In-Ear Conversation Assistants Tuochao Chen Nicholas Batchelder Alisa Liu Noah A. Smith Shyamnath Gollakota 172 0 0 07 May 2025
OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models Xiaoyu Xu Minxin Du Qingqing Ye Haibo Hu MU 57 0 0 07 May 2025
Policy-labeled Preference Learning: Is Preference Enough for RLHF? Taehyun Cho Seokhun Ju Seungyub Han Dohyeong Kim Kyungjae Lee Jungwoo Lee OffRL 29 0 0 06 May 2025
RIFT: Closed-Loop RL Fine-Tuning for Realistic and Controllable Traffic Simulation Keyu Chen Wenchao Sun Hao Cheng Sifa Zheng 52 0 0 06 May 2025
PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model Baijiong Lin Weisen Jiang Yuancheng Xu Hao Chen Ying-Cong Chen 28 0 0 06 May 2025
Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning Yibin Wang Zhimin Li Yuhang Zang Chunyu Wang Qinglin Lu Cheng Jin Jize Wang LRM 48 0 0 06 May 2025
Soft Best-of-n Sampling for Model Alignment C. M. Verdun Alex Oesterling Himabindu Lakkaraju Flavio du Pin Calmon BDL 201 0 0 06 May 2025
VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making Jake Grigsby Yuke Zhu Michael S Ryoo Juan Carlos Niebles OffRL VLM 41 0 0 06 May 2025
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch Zimu Lu Yiran Yang Houxing Ren Haotian Hou Han Xiao Ke Wang Weikang Shi Aojun Zhou Mingjie Zhan Yiming Li LLMAG 47 0 0 06 May 2025
FairPO: Robust Preference Optimization for Fair Multi-Label Learning Soumen Kumar Mondal Akshit Varmora Prateek Chanda Ganesh Ramakrishnan 45 0 0 05 May 2025
Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models Zhengliang Shi Lingyong Yan Weiwei Sun Yue Feng Pengjie Ren Xinyu Ma Shuaiqiang Wang D. Yin Maarten de Rijke Z. Ren RALM 48 0 0 05 May 2025
R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning Yi-Fan Zhang Xingyu Lu X. Hu Chaoyou Fu Bin Wen ... Jianfei Chen Fan Yang Z. Zhang Tingting Gao Liang Wang OffRL LRM 48 0 0 05 May 2025
What Is AI Safety? What Do We Want It to Be? Jacqueline Harding Cameron Domenico Kirk-Giannini 78 0 0 05 May 2025
Bielik 11B v2 Technical Report Krzysztof Ociepa Łukasz Flis Krzysztof Wróbel Adrian Gwoździej Remigiusz Kinas 34 0 0 05 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 74 1 0 05 May 2025
SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing Ming Li Xin Gu Fan Chen X. Xing Longyin Wen Chong Chen Sijie Zhu DiffM 83 1 0 05 May 2025
Improving Model Alignment Through Collective Intelligence of Open-Source LLMS Junlin Wang Roy Xie Shang Zhu Jue Wang Ben Athiwaratkun Bhuwan Dhingra Shuaiwen Leon Song Ce Zhang James Zou ALM 38 0 0 05 May 2025
RM-R1: Reward Modeling as Reasoning Xiusi Chen Gaotang Li Zehua Wang Bowen Jin Cheng Qian ... Y. Zhang D. Zhang Tong Zhang Hanghang Tong Heng Ji ReLM OffRL LRM 185 1 0 05 May 2025
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL Jiarui Yao Yifan Hao Hanning Zhang Hanze Dong Wei Xiong Nan Jiang Tong Zhang LRM 62 0 0 05 May 2025
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning Tianjian Li Daniel Khashabi 60 0 0 05 May 2025
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji Yanqiu Wu Zhibin Wu Shoujin Wang Jian Yang Mark Dras Usman Naseem 41 0 0 05 May 2025
AKD : Adversarial Knowledge Distillation For Large Language Models Alignment on Coding tasks Ilyas Oulkadda Julien Perez ALM 47 0 0 05 May 2025
Demystifying optimized prompts in language models Rimon Melamed Lucas H. McCabe H. H. Huang 44 0 0 04 May 2025
Semantic Probabilistic Control of Language Models Kareem Ahmed Catarina G Belém Padhraic Smyth Sameer Singh 44 0 0 04 May 2025
Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach Jiancong Xiao Bojian Hou Zhanliang Wang Ruochen Jin Q. Long Weijie Su Li Shen 35 0 0 04 May 2025
Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study Xiaoyu Tian Sitong Zhao Haotian Wang Shuaiting Chen Yiping Peng Yunjie Ji Han Zhao Xiangang Li OffRL LRM 37 0 0 04 May 2025
R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation Meng-Hao Guo Jiajun Xu Yi Zhang Jiaxi Song Haoyang Peng ... Yongming Rao Houwen Peng Han Hu Gordon Wetzstein Shi-Min Hu ELM LRM 60 2 0 04 May 2025
LookAlike: Consistent Distractor Generation in Math MCQs Nisarg Parikh Nigel Fernandez Alexander Scarlatos Simon Woodhead Andrew S. Lan 53 0 0 03 May 2025
Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm Sarvesh Shashidhar Ritik Nachiketa Patil Suraj Racha Ganesh Ramakrishnan 24 0 0 03 May 2025
On the Limitations of Steering in Language Model Alignment Chebrolu Niranjan Kokil Jaidka G. Yeo LLMSV 43 0 0 02 May 2025
Optimal Interactive Learning on the Job via Facility Location Planning Shivam Vats Michelle Zhao Patrick Callaghan Mingxi Jia Maxim Likhachev Oliver Kroemer George Konidaris 34 0 0 01 May 2025
Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks Vishnu Sarukkai Zhiqiang Xie Kayvon Fatahalian LLMAG 75 0 0 01 May 2025