Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
Towards User-level Private Reinforcement Learning with Human Feedback Jingyang Zhang Mingxi Lei Meng Ding Mengdi Li Zihang Xiang Difei Xu Jinhui Xu Di Wang 52 0 0 22 Feb 2025
Fine-Tuning Qwen 2.5 3B for Realistic Movie Dialogue Generation Kartik Gupta VGen 54 0 0 22 Feb 2025
A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models Jiahui Geng Qing Li Herbert Woisetschlaeger Zongxiong Chen Yansen Wang Preslav Nakov Hans-Arno Jacobsen Fakhri Karray MU 46 1 0 22 Feb 2025
C-3DPO: Constrained Controlled Classification for Direct Preference Optimization Kavosh Asadi Julien Han Xingzi Xu Dominique Perrault-Joncas Shoham Sabach Karim Bouyarmane Mohammad Ghavamzadeh 34 0 0 22 Feb 2025
Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging Lin Lu Zhigang Zuo Ziji Sheng Pan Zhou MoMe 62 0 0 22 Feb 2025
IPO: Your Language Model is Secretly a Preference Classifier Shivank Garg Ayush Singh Shweta Singh Paras Chopra 244 1 0 22 Feb 2025
A generative approach to LLM harmfulness detection with special red flag tokens Sophie Xhonneux David Dobre Mehrnaz Mohfakhami Leo Schwinn Gauthier Gidel 58 1 0 22 Feb 2025
A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics Ting-Ruen Wei Haowei Liu Xuyang Wu Yi Fang LRM AI4CE ReLM KELM 256 2 0 21 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Ziyang Chen Mingxiao Li Shangsong Liang Zhaochun Ren V. Honavar 110 6 0 21 Feb 2025
Self-Taught Agentic Long Context Understanding Yufan Zhuang Xiaodong Yu Jialian Wu Xingwu Sun Zihan Wang Jiang Liu Yusheng Su Jingbo Shang Zicheng Liu Emad Barsoum LRM 41 0 0 21 Feb 2025
Eeyore: Realistic Depression Simulation via Supervised and Preference Optimization Siyang Liu Bianca Brie Wenda Li Laura Biester Andrew Lee J. Pennebaker Rada Mihalcea 45 0 0 21 Feb 2025
Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation Shuo Tang Xianghe Pang Zexi Liu Bohan Tang Guangyi Liu Xiaowen Dong Yunhong Wang Yanfeng Wang Tian Jin SyDa LLMAG 135 4 0 21 Feb 2025
C3AI: Crafting and Evaluating Constitutions for Constitutional AI Yara Kyrychenko Ke Zhou Edyta Bogucka Daniele Quercia ELM 55 3 0 21 Feb 2025
BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment Sizhe Wang Yongqi Tong Hengyuan Zhang Dawei Li Xin Zhang Tianlong Chen 95 5 0 21 Feb 2025
Faster WIND: Accelerating Iterative Best-of- $N$ Distillation for LLM Alignment Tong Yang Jincheng Mei H. Dai Zixin Wen Shicong Cen Dale Schuurmans Yuejie Chi Bo Dai 50 4 0 20 Feb 2025
Drift: Decoding-time Personalized Alignments with Implicit User Preferences Minbeom Kim Kang-il Lee Seongho Joo Hwaran Lee Thibaut Thonet Kyomin Jung AI4TS 121 1 0 20 Feb 2025
Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images Shengguang Wu Fan-Yun Sun Kaiyue Wen Nick Haber VLM 80 0 0 20 Feb 2025
UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning Vaidehi Patil Elias Stengel-Eskin Joey Tianyi Zhou MU CLL 81 2 0 20 Feb 2025
Simplify RLHF as Reward-Weighted SFT: A Variational Method Yuhao Du Zehan Li Pengyu Cheng Zhihong Chen Yuejiao Xie Xiang Wan Anningzhe Gao 48 1 0 20 Feb 2025
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF Shicong Cen Jincheng Mei Katayoon Goshvadi Hanjun Dai Tong Yang Sherry Yang Dale Schuurmans Yuejie Chi Bo Dai OffRL 70 24 0 20 Feb 2025
Soft Token Attacks Cannot Reliably Audit Unlearning in Large Language Models Haokun Chen Sebastian Szyller Weilin Xu N. Himayat MU AAML 48 0 0 20 Feb 2025
Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment Zhili Liu Yunhao Gou Kai Chen Lanqing Hong Jiahui Gao ... Yu Zhang Zhenguo Li Xin Jiang Qiang Liu James T. Kwok MoE 116 9 0 20 Feb 2025
SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generation Song Duong Florian Le Bronnec Alexandre Allauzen Vincent Guigue Alberto Lumbreras Laure Soulier Patrick Gallinari HILM 52 0 0 20 Feb 2025
Diffusion Models in Recommendation Systems: A Survey Ting-Ruen Wei Yi Fang 92 2 0 20 Feb 2025
Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model Emre Can Acikgoz Jeremiah Greer Akul Datta Ze Yang William Zeng Oussama Elachqar Emmanouil Koukoumidis Dilek Hakkani-Tur Gokhan Tur LLMAG 111 3 0 20 Feb 2025
ChatWise: AI-Powered Engaging Conversations for Enhancing Senior Cognitive Wellbeing Zhengbang Yang Zhuangdi Zhu AI4MH 31 0 0 19 Feb 2025
Slamming: Training a Speech Language Model on One GPU in a Day Gallil Maimon Avishai Elmakies Yossi Adi 40 3 0 19 Feb 2025
UniKnow: A Unified Framework for Reliable Language Model Behavior across Parametric and External Knowledge Youna Kim Hyuhng Joon Kim Minjoon Choi Sungmin Cho Hyunsoo Cho Sang-goo Lee Taeuk Kim KELM 68 0 0 19 Feb 2025
Savaal: Scalable Concept-Driven Question Generation to Enhance Human Learning Kimia Noorbakhsh Joseph Chandler Pantea Karimi M. Alizadeh H. Balakrishnan LRM 51 1 0 18 Feb 2025
Multi-Attribute Steering of Language Models via Targeted Intervention Duy Nguyen Archiki Prasad Elias Stengel-Eskin Joey Tianyi Zhou LLMSV 110 0 0 18 Feb 2025
Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees Yongtao Wu Luca Viano Yihang Chen Zhenyu Zhu Kimon Antonakopoulos Quanquan Gu V. Cevher 69 0 0 18 Feb 2025
Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models Artyom Kharinaev Viktor Moskvoretskii Egor Shvetsov Kseniia Studenikina Bykov Mikhail Evgeny Burnaev MQ 55 0 0 18 Feb 2025
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization Shuo Xing Yuping Wang Peiran Li Ruizheng Bai Yansen Wang Chan-wei Hu Chengxuan Qian Huaxiu Yao Zhengzhong Tu 99 6 0 18 Feb 2025
Oreo: A Plug-in Context Reconstructor to Enhance Retrieval-Augmented Generation Sha Li Naren Ramakrishnan RALM KELM 159 1 0 18 Feb 2025
Policy-to-Language: Train LLMs to Explain Decisions with Flow-Matching Generated Rewards Xinyi Yang Liang Zeng Heng Dong Chao Yu X. Wu H. Yang Yu Wang Milind Tambe Tonghan Wang 83 2 0 18 Feb 2025
Personalized Top-k Set Queries Over Predicted Scores Sohrab Namazi Nia Subhodeep Ghosh Senjuti Basu Roy S. Amer-Yahia 76 0 0 18 Feb 2025
SEA: Low-Resource Safety Alignment for Multimodal Large Language Models via Synthetic Embeddings Weikai Lu Hao Peng Huiping Zhuang Cen Chen Ziqian Zeng 20 0 0 18 Feb 2025
Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models Yingqian Cui Pengfei He Jingying Zeng Hui Liu Xianfeng Tang ... Zhen Li Suhang Wang Yue Xing Jiliang Tang Qi He LRM 52 9 0 18 Feb 2025
Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models Daiki Chijiwa Taku Hasegawa Kyosuke Nishida Kuniko Saito Susumu Takeuchi 52 0 0 18 Feb 2025
Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs Longxu Dou Qian Liu Fan Zhou Changyu Chen Zili Wang ... Tianyu Pang Chao Du Xinyi Wan Wei Lu Min Lin 137 2 0 18 Feb 2025
Computational Safety for Generative AI: A Signal Processing Perspective Pin-Yu Chen 83 1 0 18 Feb 2025
HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation L. Yang Xinchen Zhang Ye Tian Chenming Shang Minghao Xu Wentao Zhang Bin Cui 102 1 0 17 Feb 2025
Preference Optimization for Reasoning with Pseudo Feedback Fangkai Jiao Geyang Guo Xingxing Zhang Nancy F. Chen Chenyu You Furu Wei LRM 111 9 0 17 Feb 2025
Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment Jingcheng Deng Zhongtao Jiang Liang Pang Liwei Chen Kun Xu Zihao Wei Huawei Shen Xueqi Cheng 62 1 0 17 Feb 2025
Unhackable Temporal Rewarding for Scalable Video MLLMs En Yu Kangheng Lin Liang Zhao Yana Wei Zining Zhu ... Jianjian Sun Zheng Ge Xinsong Zhang Jingyu Wang Wenbing Tao 69 4 0 17 Feb 2025
Diversity-Oriented Data Augmentation with Large Language Models Zaitian Wang Jinghan Zhang Xinhao Zhang Kunpeng Liu Pengfei Wang Yuanchun Zhou 80 1 0 17 Feb 2025
LeDex: Training LLMs to Better Self-Debug and Explain Code Nan Jiang Xiaopeng Li Shiqi Wang Qiang Zhou Soneya Binta Hossain Baishakhi Ray Varun Kumar Xiaofei Ma Anoop Deoras LRM 92 12 0 17 Feb 2025
A Survey of Personalized Large Language Models: Progress and Future Directions Jiahong Liu Zexuan Qiu Zhongyang Li Quanyu Dai Jieming Zhu Minda Hu Menglin Yang Irwin King LM&MA 63 4 0 17 Feb 2025
STRIVE: Structured Reasoning for Self-Improvement in Claim Verification Haisong Gong Jing Li Junfei Wu Qiang Liu Shu Wu Liang Wang LRM 46 0 0 17 Feb 2025
Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding Kung-Hsiang Huang Can Qin Haoyi Qiu Philippe Laban Chenyu You Caiming Xiong C. Wu VLM 150 3 0 17 Feb 2025