Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance Jia Xu Tianyi Wei Bojian Hou Patryk Orzechowski Shu Yang Ruochen Jin Rachael Paulbeck Joost B. Wagenaar George Demiris Li Shen AI4MH 49 0 0 13 Mar 2025
RankPO: Preference Optimization for Job-Talent Matching Yuyao Zhang Ming Wang Yu Wang Xiaohui Wang 51 0 0 13 Mar 2025
World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning Siyin Wang Zhaoye Fei Qinyuan Cheng Shanghang Zhang Panpan Cai Jinlan Fu Xipeng Qiu 61 1 0 13 Mar 2025
OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses Angela Lopez-Cardona Sebastian Idesis Miguel Barreda-Ángeles Sergi Abadal Ioannis Arapakis 56 0 0 13 Mar 2025
Memory-enhanced Retrieval Augmentation for Long Video Understanding Huaying Yuan Zhengyang Liang Minhao Qin Hongjin Qian Yan Shu Zhicheng Dou Zhicheng Dou VOS RALM VLM 89 1 0 12 Mar 2025
LREF: A Novel LLM-based Relevance Framework for E-commerce Tian Tang Zhixing Tian Zhenyu Zhu Chenyang Wang Haiqing Hu Guoyu Tang Lin Liu Sulong Xu 65 0 0 12 Mar 2025
Local Look-Ahead Guidance via Verifier-in-the-Loop for Automated Theorem Proving Sara Rajaee Kumar Pratik Gabriele Cesa Arash Behboodi OffRL LRM 63 0 0 12 Mar 2025
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning Bowen Jin Hansi Zeng Zhenrui Yue Dong Wang Sercan O. Arik Dong Wang Hamed Zamani Jiawei Han RALM ReLM KELM OffRL AI4TS LRM 89 38 0 12 Mar 2025
Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks Lutfi Eren Erdogan Nicholas Lee Sehoon Kim Suhong Moon Hiroki Furuta Gopala Anumanchipalli Kemal Kurniawan Amir Gholami LLMAG LM&Ro AIFin 80 2 0 12 Mar 2025
Got Compute, but No Data: Lessons From Post-training a Finnish LLM Elaine Zosa Ville Komulainen S. Pyysalo 70 0 0 12 Mar 2025
ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning Bo Liu Yunxiang Li Yangqiu Song Hanjing Wang Linyi Yang Mark Schmidt Jun Wang Weinan Zhang Shuyue Hu Ying Wen LLMAG KELM LRM AI4CE 94 6 0 12 Mar 2025
Reinforcement Learning is all You Need Yongsheng Lian ReLM OffRL LRM 70 0 0 12 Mar 2025
HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding Rui Yang Lin Song Yicheng Xiao Runhui Huang Yixiao Ge Ying Shan Hengshuang Zhao MLLM 62 0 0 12 Mar 2025
RewardSDS: Aligning Score Distillation via Reward-Weighted Sampling Itay Chachy Guy Yariv Sagie Benaim 252 0 0 12 Mar 2025
Teaching LLMs How to Learn with Contextual Fine-Tuning Younwoo Choi Muhammad Adil Asif Ziwen Han John Willes Rahul G. Krishnan LRM 41 0 0 12 Mar 2025
Aligning to What? Limits to RLHF Based Alignment Logan Barnhart Reza Akbarian Bafghi Stephen Becker M. Raissi 47 1 0 12 Mar 2025
Backtracking for Safety Bilgehan Sel Dingcheng Li Phillip Wallis Vaishakh Keshava Ming Jin Siddhartha Reddy Jonnalagadda KELM 55 0 0 11 Mar 2025
GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Training Tong Wei Yijun Yang Junliang Xing Yuanchun Shi Zongqing Lu Deheng Ye OffRL LRM 49 1 0 11 Mar 2025
Preference-Based Alignment of Discrete Diffusion Models Umberto Borso Davide Paglieri Jude Wells Tim Rocktaschel 72 2 0 11 Mar 2025
Robust Multi-Objective Controlled Decoding of Large Language Models Seongho Son William Bankes Sangwoong Yoon Shyam Sundhar Ramesh Xiaohang Tang Ilija Bogunovic 49 0 0 11 Mar 2025
EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments Dongping Li Tielong Cai Tianci Tang Wenhao Chai Katherine Rose Driggs-Campbell Gaoang Wang LM&Ro 71 0 0 11 Mar 2025
Generating Robot Constitutions & Benchmarks for Semantic Safety P. Sermanet Anirudha Majumdar A. Irpan Dmitry Kalashnikov Vikas Sindhwani LM&Ro 71 1 0 11 Mar 2025
Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation Wenlong Meng Fan Zhang Wendao Yao Zhenyuan Guo Yongqian Li Chengkun Wei Wenzhi Chen AAML 42 2 0 11 Mar 2025
AI-native Memory 2.0: Second Me Jiale Wei Xiang Ying Tao Gao Fangyi Bao Felix Tao Jingbo Shang 64 1 0 11 Mar 2025
RigoChat 2: an adapted language model to Spanish using a bounded dataset and reduced hardware Gonzalo Santamaría Gómez Guillem García Subies Pablo Gutiérrez Ruiz Mario González Valero Natàlia Fuertes ... Nuria Aldama García David Betancur Sánchez Kateryna Sushkova Marta Guerrero Nieto Á. Jiménez 56 0 0 11 Mar 2025
AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning Bo Jiang Shaoyu Chen Qian Zhang Wenyu Liu Xinggang Wang OffRL LRM VLM 84 5 0 10 Mar 2025
Mitigating Preference Hacking in Policy Optimization with Pessimism Dhawal Gupta Adam Fisch Christoph Dann Alekh Agarwal 76 0 0 10 Mar 2025
Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning Qinghao Ye Xianhan Zeng Fu Li Chong Li Haoqi Fan CoGe 88 2 0 10 Mar 2025
RePO: ReLU-based Preference Optimization Junkang Wu Kexin Huang Xue Wang Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He Xuben Wang 76 0 0 10 Mar 2025
VisRL: Intention-Driven Visual Perception via Reinforced Reasoning Zhangquan Chen Xufang Luo Dongsheng Li OffRL LRM 75 3 0 10 Mar 2025
VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models Jiacheng Ruan Wenzhen Yuan Xian Gao Ye Guo Daoxin Zhang Zhe Xu Yao Hu Ting Liu Yuzhuo Fu LRM VLM 75 4 0 10 Mar 2025
DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs Jongwoo Ko Tianyi Chen Sungnyun Kim Tianyu Ding Luming Liang Ilya Zharkov Se-Young Yun VLM 249 0 0 10 Mar 2025
Large Language Models Often Say One Thing and Do Another Ruoxi Xu Hongyu Lin Xianpei Han Jia Zheng Weixiang Zhou Le Sun Yingfei Sun 55 1 0 10 Mar 2025
Process-Supervised LLM Recommenders via Flow-guided Tuning Chongming Gao Mengyao Gao Chenxiao Fan Shuai Yuan Wentao Shi Xiangnan He 76 2 0 10 Mar 2025
Safety Guardrails for LLM-Enabled Robots Zachary Ravichandran Alexander Robey Vijay Kumar George Pappas Hamed Hassani 66 2 0 10 Mar 2025
UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality Zelei Cheng Xin-Qiang Cai Yuting Tang Pushi Zhang Boming Yang Masashi Sugiyama Xinyu Xing 54 0 0 10 Mar 2025
Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues Alexander Scarlatos Naiming Liu Jaewook Lee Richard Baraniuk Andrew Lan 60 2 0 09 Mar 2025
Can Small Language Models Reliably Resist Jailbreak Attacks? A Comprehensive Evaluation Wenhui Zhang Huiyu Xu Peng Kuang Zeqing He Ziqi Zhu Kui Ren AAML PILM 74 0 0 09 Mar 2025
Dr Genre: Reinforcement Learning from Decoupled LLM Feedback for Generic Text Rewriting Yufei Li John Nham Ganesh Jawahar Lei Shu David C. Uthus Yun-hsuan Sung Chengrun Yang Itai Rolnick Yi Qiao Cong Liu OffRL 70 0 0 09 Mar 2025
PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training Cong Chen Mingyu Liu Chenchen Jing Y. Zhou Fengyun Rao Hao Chen Bo Zhang Chunhua Shen MLLM AAML VLM 67 5 0 09 Mar 2025
Alignment for Efficient Tool Calling of Large Language Models Hongshen Xu Zihan Wang Zichen Zhu Lei Pan Xingyu Chen Lu Chen Kai Yu 51 0 0 09 Mar 2025
Evaluating and Aligning Human Economic Risk Preferences in LLMs Jiaheng Liu Yi Yang Kar Yan Tam 73 0 0 09 Mar 2025
GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devices Xudong Lu Yinghao Chen Renshou Wu Haohao Gao Xi Chen ... Fangyuan Li Yafei Wen Xiaoxin Chen Shuai Ren Hongsheng Li 89 0 0 08 Mar 2025
Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models Thomas Winninger Boussad Addad Katarzyna Kapusta AAML 68 0 0 08 Mar 2025
From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models Muzhi Dai Jiashuo Sun Zhiyuan Zhao Shixuan Liu Rui Li Junyu Gao Xuelong Li VLM 58 1 0 08 Mar 2025
ROCM: RLHF on consistency models Shivanshu Shekhar Tong Zhang 52 0 0 08 Mar 2025
Research on Superalignment Should Advance Now with Parallel Optimization of Competence and Conformity HyunJin Kim Xiaoyuan Yi Jing Yao Muhua Huang Jinyeong Bak James Evans Xing Xie 49 0 0 08 Mar 2025
Personalized Text Generation with Contrastive Activation Steering Jinghao Zhang Yi Liu Wenjie Wang Qiang Liu Shu Wu Liang Wang Tat-Seng Chua LLMSV 46 0 0 07 Mar 2025
Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs Ling Team B. Zeng Chenyu Huang Chao Zhang Changxin Tian ... Zhaoxin Huan Zujie Wen Zhenhang Sun Zhuoxuan Du Z. He MoE ALM 111 2 0 07 Mar 2025
Soft Policy Optimization: Online Off-Policy RL for Sequence Models Taco Cohen David W. Zhang Kunhao Zheng Yunhao Tang Rémi Munos Gabriel Synnaeve OffRL 88 0 0 07 Mar 2025