Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization Yunan Wang Jijie Li Bo Zhang Liangdong Wang Guang Liu 63 0 0 20 Mar 2025
Tuning LLMs by RAG Principles: Towards LLM-native Memory Jiale Wei Shuchi Wu Ruochen Liu Xiang Ying Jingbo Shang Fangbo Tao RALM 77 0 0 20 Mar 2025
Disentangling Uncertainties by Learning Compressed Data Representation Zhiyu An Zhibo Hou Wan Du UQCV UD 76 0 0 20 Mar 2025
Towards Automatic Continual Learning: A Self-Adaptive Framework for Continual Instruction Tuning Peiyi Lin Fukai Zhang Kai Niu Hao Fu CLL 76 0 0 20 Mar 2025
Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning Chen Li Nazhou Liu Kai Yang 46 3 0 20 Mar 2025
Cultural Alignment in Large Language Models Using Soft Prompt Tuning Reem I. Masoud Martin Ferianc Philip C. Treleaven Miguel R. D. Rodrigues ALM 49 0 0 20 Mar 2025
Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Models M. Wong C. Tan ALM 83 4 0 19 Mar 2025
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings Austin Xu Srijan Bansal Yifei Ming Semih Yavuz Shafiq Joty ELM 103 3 0 19 Mar 2025
Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systems Sejong Kim Hyunseo Song Hyunwoo Seo Hyunjun Kim RALM 87 2 0 19 Mar 2025
Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context Junyi Ao Dekun Chen Xiaohai Tian Wenjie Feng Jingyang Zhang Lu Lu Yansen Wang Haizhou Li Zhizheng Wu AuLLM 71 0 0 19 Mar 2025
From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment Jia-Nan Li Jian Guan Songhao Wu Wei Wu Rui Yan 75 1 0 19 Mar 2025
DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning R. Zhao Junliang Ye Ziyi Wang Guangce Liu Yiwen Chen Yikai Wang Jun Zhu AI4CE 50 0 0 19 Mar 2025
SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks Yifei Zhou Song Jiang Yuandong Tian Jason Weston Sergey Levine Sainbayar Sukhbaatar Xian Li LLMAG LRM 62 5 0 19 Mar 2025
How much do LLMs learn from negative examples? Shadi S. Hamdan Deniz Yuret 58 0 0 18 Mar 2025
Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs Nicolas Le Roux Marc G. Bellemare Jonathan Lebensold Arnaud Bergeron Joshua Greaves Alex Fréchette Carolyne Pelletier Eric Thibodeau-Laufer Sándor Toth Sam Work OffRL 91 4 0 18 Mar 2025
DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal Vaibhav Aggarwal Ojasv Kamal Abhinav Japesh Zhijing Jin Bernhard Schölkopf 52 1 0 18 Mar 2025
Inference-Time Intervention in Large Language Models for Reliable Requirement Verification Paul Darm James Xie A. Riccardi 46 0 0 18 Mar 2025
EvolvingGrasp: Evolutionary Grasp Generation via Efficient Preference Alignment Yufei Zhu Yiming Zhong Zemin Yang Peishan Cong Jingyi Yu X. Zhu Y. Ma 61 1 0 18 Mar 2025
Survey of Adversarial Robustness in Multimodal Large Language Models Chengze Jiang Zhuangzhuang Wang Minjing Dong Jie Gui AAML 68 0 0 18 Mar 2025
Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM Yazeed Alnumay Alexandre Barbet Anna Bialas William Darling Shaan Desai ... Stephanie Howe Olivia Lasche Justin Lee Anirudh Shrinivason Jennifer Tracey 94 0 0 18 Mar 2025
Aligning Multimodal LLM with Human Preference: A Survey Tao Yu Yuyao Zhang Chaoyou Fu Junkang Wu Jinda Lu ... Qingsong Wen Z. Zhang Yan Huang Liang Wang Tieniu Tan 230 2 0 18 Mar 2025
Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation Songjun Tu Jiahao Lin Xiangyu Tian Qichao Zhang Linjing Li ... Nan Xu Wei He Xiangyuan Lan D. Jiang Dongbin Zhao LRM 62 4 0 17 Mar 2025
A Survey on Transformer Context Extension: Approaches and Evaluation Yijun Liu Jinzheng Yu Yang Xu Zhongyang Li Qingfu Zhu LLMAG 85 1 0 17 Mar 2025
MAP: Multi-user Personalization with Collaborative LLM-powered Agents Christine P. Lee Jihye Choi Bilge Mutlu LLMAG 75 0 1 17 Mar 2025
Rewards Are Enough for Fast Photo-Realistic Text-to-image Generation Yihong Luo Tianyang Hu Weijian Luo Kenji Kawaguchi Jing Tang EGVM 258 0 0 17 Mar 2025
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization Jingyi Zhang Jiaxing Huang Huanjin Yao Shunyu Liu Xikun Zhang Shijian Lu Dacheng Tao LRM 65 25 0 17 Mar 2025
Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptions Wan Ju Kang Eunki Kim Na Min An Sangryul Kim Haemin Choi Ki Hoon Kwak James Thorne 54 0 0 17 Mar 2025
Aligned Probing: Relating Toxic Behavior and Model Internals Andreas Waldis Vagrant Gautam Anne Lauscher Dietrich Klakow Iryna Gurevych 50 0 0 17 Mar 2025
MagicID: Hybrid Preference Optimization for ID-Consistent and Dynamic-Preserved Video Customization Hengjia Li Lifan Jiang Xi Xiao Tianyang Wang Hongwei Yi Boxi Wu D. Cai VGen 58 0 0 16 Mar 2025
A Survey on the Optimization of Large Language Model-based Agents Shangheng Du Jiabao Zhao Jinxin Shi Zhentao Xie Xin Jiang Yanhong Bai Liang He LLMAG LM&Ro LM&MA 316 1 0 16 Mar 2025
BalancedDPO: Adaptive Multi-Metric Alignment Dipesh Tamboli Souradip Chakraborty Aditya Malusare B. Banerjee Amrit Singh Bedi Vaneet Aggarwal EGVM 67 0 0 16 Mar 2025
Augmented Adversarial Trigger Learning Zhe Wang Yanjun Qi 63 0 0 16 Mar 2025
PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing Cheng Deng Luoyang Sun Jiwen Jiang Yongcheng Zeng Xinjian Wu ... Haoyang Li Lei Chen Lionel M. Ni Jun Wang Jun Wang 222 0 0 15 Mar 2025
MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modeling Zhaopeng Feng Jiahan Ren Jiayuan Su Jiamei Zheng Zhihang Tang Hongwei Wang Zuozhu Liu LRM 70 1 0 15 Mar 2025
From Demonstrations to Rewards: Alignment Without Explicit Human Preferences Siliang Zeng Yao Liu Huzefa Rangwala George Karypis Mingyi Hong Rasool Fakoor 57 2 0 15 Mar 2025
reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs Zhaofeng Wu Michihiro Yasunaga Andrew Cohen Yoon Kim Asli Celikyilmaz Marjan Ghazvininejad 50 2 0 14 Mar 2025
SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networks Hyunwoo Park Baekryun Seong Sang-Ki Ko GNN 65 0 0 14 Mar 2025
Preference Elicitation for Multi-objective Combinatorial Optimization with Active Learning and Maximum Likelihood Estimation Marianne Defresne Jayanta Mandi Tias Guns 55 0 0 14 Mar 2025
Broaden your SCOPE! Efficient Multi-turn Conversation Planning for LLMs using Semantic Space Zhiliang Chen Xinyuan Niu Chuan-Sheng Foo Bryan Kian Hsiang Low 55 1 0 14 Mar 2025
Residual Policy Gradient: A Reward View of KL-regularized Objective Pengcheng Wang Xinghao Zhu Yuxin Chen Chenfeng Xu Masayoshi Tomizuka Chenran Li 45 0 0 14 Mar 2025
D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuning Jia Zhang Chen-Xi Zhang Yong-Jin Liu Yi-Xuan Jin Xiao-Wen Yang Bo Zheng Yi Liu Lan-Zhe Guo 54 2 0 14 Mar 2025
Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards Zijing Hu Fengda Zhang Long Chen Kun Kuang Jiahui Li Kaifeng Gao Jun Xiao X. Wang Wenwu Zhu EGVM 61 0 0 14 Mar 2025
Implicit Bias-Like Patterns in Reasoning Models Messi H.J. Lee Calvin K. Lai LRM 61 0 0 14 Mar 2025
Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-tuning Yiwei Chen Yuguang Yao Yihua Zhang Bingquan Shen Gaowen Liu Sijia Liu AAML MU 66 1 0 14 Mar 2025
Align in Depth: Defending Jailbreak Attacks via Progressive Answer Detoxification Yingjie Zhang Tong Liu Zhe Zhao Guozhu Meng Kai Chen AAML 60 1 0 14 Mar 2025
Variational Bayesian Personalized Ranking Bin Liu Xiaohong Liu Q. Luo Ziqiao Shang Jielei Chu Lin Ma Zhaoyu Li Fei Teng Guangtao Zhai Tianrui Li 55 0 0 14 Mar 2025
Fine-Tuning Diffusion Generative Models via Rich Preference Optimization Hanyang Zhao Haoxian Chen Yucheng Guo Genta Indra Winata Tingting Ou Ziyu Huang D. Yao Wenpin Tang 61 0 0 13 Mar 2025
Take Off the Training Wheels Progressive In-Context Learning for Effective Alignment Zhenyu Liu Dongfang Li Xinshuo Hu X. Zhao Yibin Chen Baotian Hu Min-Ling Zhang 56 1 0 13 Mar 2025
MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance Jia Xu Tianyi Wei Bojian Hou Patryk Orzechowski Shu Yang Ruochen Jin Rachael Paulbeck Joost B. Wagenaar George Demiris Li Shen AI4MH 49 0 0 13 Mar 2025
Representation-based Reward Modeling for Efficient Safety Alignment of Large Language Model Qiyuan Deng X. Bai Kehai Chen Yaowei Wang Liqiang Nie Min Zhang OffRL 71 0 0 13 Mar 2025