Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
A General Framework to Enhance Fine-tuning-based LLM Unlearning J. Ren Zhenwei Dai Xianfeng Tang Hui Liu Jingying Zeng ... R. Goutam Suhang Wang Yue Xing Qi He Hui Liu MU 163 1 0 25 Feb 2025
Debt Collection Negotiations with Large Language Models: An Evaluation System and Optimizing Decision Making with Multi-Agent Xiaofeng Wang Zizhuo Zhang Jinguang Zheng Yiming Ai Rui Wang 45 1 0 25 Feb 2025
CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimization Shuming Shi Ruobing Zuo Gaolei He Jianlin Wang Chenyang Xu Zhengfeng Yang 74 0 0 25 Feb 2025
RankCoT: Refining Knowledge for Retrieval-Augmented Generation through Ranking Chain-of-Thoughts Mingyan Wu Zhenghao Liu Yukun Yan Xinze Li S. Yu Zheni Zeng Yu Gu Ge Yu RALM AI4TS LRM 57 1 0 25 Feb 2025
Can LLMs Explain Themselves Counterfactually? Zahra Dehghanighobadi Asja Fischer Muhammad Bilal Zafar LRM 49 0 0 25 Feb 2025
Advantage-Guided Distillation for Preference Alignment in Small Language Models Shiping Gao Fanqi Wan Jiajian Guo Xiaojun Quan Qifan Wang ALM 60 0 0 25 Feb 2025
On Synthetic Data Strategies for Domain-Specific Generative Retrieval Haoyang Wen Jiang Guo Yi Zhang Jiarong Jiang Zhendong Wang SyDa 79 0 0 25 Feb 2025
AgentRM: Enhancing Agent Generalization with Reward Modeling Yu Xia Jingru Fan Weize Chen Siyu Yan Xin Cong Zhong Zhang Yaojie Lu Yankai Lin Zhiyuan Liu Maosong Sun 61 1 0 25 Feb 2025
AMPO: Active Multi-Preference Optimization Taneesh Gupta Rahul Madhavan Xuchao Zhang Chetan Bansal Saravan Rajmohan 55 0 0 25 Feb 2025
Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data Siqi Guo Ilgee Hong Vicente Balmaseda Changlong Yu Liang Qiu Xin Liu Haoming Jiang Tuo Zhao Tianbao Yang 52 0 0 25 Feb 2025
Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems Matthew Barker Andrew Bell Evan Thomas James Carr Thomas Andrews Umang Bhatt 87 1 0 25 Feb 2025
NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigms Yashan Wang Shangda Wu Jianhuai Hu Xingjian Du Yueqi Peng Yongxin Huang Shuai Fan Xiaobing Li Feng Yu Maosong Sun 111 2 0 25 Feb 2025
OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference Xiangyu Zhao Shengyuan Ding Zicheng Zhang Haian Huang Maosong Cao ... Wenhai Wang Guangtao Zhai Haodong Duan Hua Yang Kai Chen 126 7 0 25 Feb 2025
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment Tianze Wang Dongnan Gui Yifan Hu Shuhang Lin Linjun Zhang 42 0 0 25 Feb 2025
Language Models' Factuality Depends on the Language of Inquiry Tushar Aggarwal Kumar Tanmay Ayush Agrawal Kumar Ayush Hamid Palangi Paul Pu Liang HILM KELM 76 2 0 25 Feb 2025
Stackelberg Game Preference Optimization for Data-Efficient Alignment of Language Models Xu Chu Zhixin Zhang Tianyu Jia Yujie Jin 85 0 0 25 Feb 2025
Single- vs. Dual-Prompt Dialogue Generation with LLMs for Job Interviews in Human Resources Joachim De Baer A. Seza Doğruöz T. Demeester Chris Develder 52 0 0 25 Feb 2025
FairGen: Controlling Sensitive Attributes for Fair Generations in Diffusion Models via Adaptive Latent Guidance Mintong Kang Vinayshekhar Bannihatti Kumar Shamik Roy Abhishek Kumar Sopan Khosla Balakrishnan Narayanaswamy Rashmi Gangadharaiah 53 0 0 25 Feb 2025
Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning Wenkai Yang Shuming Ma Yankai Lin Furu Wei LRM 53 25 0 25 Feb 2025
Larger or Smaller Reward Margins to Select Preferences for Alignment? Kexin Huang Junkang Wu Ziqian Chen Xue Wang Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He Xuben Wang 55 0 0 25 Feb 2025
Spontaneous Giving and Calculated Greed in Language Models Yuxuan Li Hirokazu Shirado ReLM LRM AI4CE 48 1 0 24 Feb 2025
RLTHF: Targeted Human Feedback for LLM Alignment Yifei Xu Tusher Chakraborty Emre Kıcıman Bibek Aryal Eduardo Rodrigues ... Rafael Padilha Leonardo Nunes Shobana Balakrishnan Songwu Lu Ranveer Chandra 121 1 0 24 Feb 2025
Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions Joseph Suh Erfan Jahanparast Suhong Moon Minwoo Kang Serina Chang ALM LM&MA 62 1 0 24 Feb 2025
NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions Weizhe Yuan Jane Dwivedi-Yu Song Jiang Karthik Padthe Yang Li ... Ilia Kulikov Kyunghyun Cho Yuandong Tian Jason Weston Xian Li ReLM LRM 71 14 0 24 Feb 2025
Streaming Looking Ahead with Token-level Self-reward Han Zhang Ruixin Hong Dong Yu 49 1 0 24 Feb 2025
AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models Qin Zhu Fei Huang Runyu Peng Keming Lu Bowen Yu Qinyuan Cheng Xipeng Qiu Xuanjing Huang Junyang Lin ReLM ELM LRM 59 2 0 24 Feb 2025
LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjoint Qianli Ma Dongrui Liu Qian Chen Linfeng Zhang Jing Shao MoMe 246 0 0 24 Feb 2025
Unveiling Reasoning Thresholds in Language Models: Scaling, Fine-Tuning, and Interpretability through Attention Maps Yen-Che Hsiao Abhishek Dutta LRM ReLM ELM 71 0 0 24 Feb 2025
Is Free Self-Alignment Possible? Dyah Adila Changho Shin Yijing Zhang Frederic Sala MoMe 118 2 0 24 Feb 2025
Training a Generally Curious Agent Fahim Tajwar Yiding Jiang Abitha Thankaraj Sumaita Sadia Rahman J. Zico Kolter Jeff Schneider Ruslan Salakhutdinov 126 1 0 24 Feb 2025
BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models Yupeng Chang Yi-Ju Chang Yuan Wu AI4CE ALM 100 0 0 24 Feb 2025
HIPPO: Enhancing the Table Understanding Capability of Large Language Models through Hybrid-Modal Preference Optimization Zhenghao Liu Han Wang Xinze Li Qiushi Xiong Xiaocui Yang ... Yukun Yan Qi Shi Fangfang Li Ge Yu Maosong Sun LMTD 61 0 0 24 Feb 2025
PIP-KAG: Mitigating Knowledge Conflicts in Knowledge-Augmented Generation via Parametric Pruning Pengcheng Huang Zhenghao Liu Yukun Yan Xiaoyuan Yi Hao Chen Zhiyuan Liu Maosong Sun Tong Xiao Ge Yu Chenyan Xiong 104 1 0 24 Feb 2025
LLM-QE: Improving Query Expansion by Aligning Large Language Models with Ranking Preferences Sijia Yao Pengcheng Huang Zhenghao Liu Yu Gu Yukun Yan S. Yu Ge Yu RALM 41 0 0 24 Feb 2025
Shakti-VLMs: Scalable Vision-Language Models for Enterprise AI Syed Abdul Gaffar Shakhadri Kruthika KR Kartik Basavaraj Angadi VLM 65 0 0 24 Feb 2025
Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding Tian Jin Ellie Y. Cheng Zack Ankner Nikunj Saunshi Blake M. Elias Amir Yazdanbakhsh Jonathan Ragan-Kelley Suvinay Subramanian Michael Carbin 69 3 0 24 Feb 2025
Post-edits Are Preferences Too Nathaniel Berger Stefan Riezler M. Exel Matthias Huck 53 0 0 24 Feb 2025
Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance Chenghua Huang Lu Wang Fangkai Yang Pu Zhao Zechao Li Qingwei Lin Dongmei Zhang Saravan Rajmohan Qi Zhang OffRL 57 1 0 24 Feb 2025
Improving LLM General Preference Alignment via Optimistic Online Mirror Descent Yuheng Zhang Dian Yu Tao Ge Linfeng Song Zhichen Zeng Haitao Mi Nan Jiang Dong Yu 68 1 0 24 Feb 2025
Aligning Compound AI Systems via System-level DPO Xiangwen Wang Yibo Jacky Zhang Zhoujie Ding Katherine Tsai Sanmi Koyejo 43 0 0 24 Feb 2025
Extreme Speech Classification in the Era of LLMs: Exploring Open-Source and Proprietary Models Sarthak Mahajan Nimmi Rangaswamy 60 0 0 24 Feb 2025
SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation Intervention Jiaqi Wu Chen Chen Chunyan Hou Xiaojie Yuan AAML 59 0 0 24 Feb 2025
Sequence-level Large Language Model Training with Contrastive Preference Optimization Zhili Feng Dhananjay Ram Cole Hawkins Aditya Rawal Jinman Zhao Sheng Zha 64 0 0 23 Feb 2025
DISC: Dynamic Decomposition Improves LLM Inference Scaling Jonathan Light Wei Cheng Wu Yue Masafumi Oyamada Mengdi Wang Santiago Paternain Haifeng Chen ReLM LRM 69 2 0 23 Feb 2025
RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis Jianwei Wang Junyao Yang Haoran Li Huiping Zhuang Cen Chen Ziqian Zeng SyDa 54 0 0 23 Feb 2025
Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines Xinwei Long Zhiyuan Ma Ermo Hua Kaiyan Zhang Biqing Qi Bowen Zhou RALM 53 0 0 23 Feb 2025
Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs Jonathan Rystrøm Hannah Rose Kirk Scott A. Hale 51 4 0 23 Feb 2025
Retrieval-Augmented Fine-Tuning With Preference Optimization For Visual Program Generation Deokhyung Kang Jeonghun Cho Yejin Jeon Sunbin Jang Minsub Lee Jawoon Cho Gary Geunbae Lee 46 0 0 23 Feb 2025
CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale Chenlong Wang Zhaoyang Chu Zhengxiang Cheng Xuyi Yang Kaiyue Qiu Yao Wan Zhou Zhao Xuanhua Shi Danny Chen ALM SyDa 50 0 0 23 Feb 2025
Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review Pei Fu Tongkun Guan Zining Wang Zhentao Guo Chen Duan ... Boming Chen Jiayao Ma Qianyi Jiang Kai Zhou Junfeng Luo VLM 69 0 0 23 Feb 2025