Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,663 papers shown

Title
Metacognition for Unknown Situations and Environments (MUSE) Rodolfo Valiente Praveen K. Pilly 89 0 0 20 Nov 2024
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations Gaurav Verma Rachneet Kaur Nishan Srishankar Zhen Zeng T. Balch Manuela Veloso LLMAG 79 5 0 20 Nov 2024
Patience Is The Key to Large Language Model Reasoning Yijiong Yu LRM 75 0 0 20 Nov 2024
DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs Zhihan Liu Shenao Zhang Yongfei Liu Boyi Liu Yingxiang Yang Zhaoran Wang 115 4 0 20 Nov 2024
A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection Gabriel Chua Shing Yee Chan Shaun Khoo 120 1 0 20 Nov 2024
Reward Modeling with Ordinal Feedback: Wisdom of the Crowd Shang Liu Yu Pan Guanting Chen Xiaocheng Li 90 2 0 19 Nov 2024
Aligning Few-Step Diffusion Models with Dense Reward Difference Learning Ziyi Zhang Li Shen Sen Zhang Deheng Ye Yong Luo Miaojing Shi Di Lin Dacheng Tao 103 1 0 18 Nov 2024
Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering Xinyan Guan Yanjiang Liu Xinyu Lu Boxi Cao Xianpei Han ... Le Sun Jie Lou Bowen Yu Yaojie Lu Hongyu Lin ALM 105 3 0 18 Nov 2024
TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models Tingyu Qu Mingxiao Li Tinne Tuytelaars Marie-Francine Moens VLM 56 2 0 17 Nov 2024
SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization Hongrui Jia Chaoya Jiang Haiyang Xu Wei Ye Mengfan Dong Ming Yan Ji Zhang Fei Huang Shikun Zhang MLLM 103 2 0 17 Nov 2024
Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment Andrew Konya Aviv Ovadya K. J. Kevin Feng Quan Ze Chen Lisa Schirch Colin Irwin Amy X. Zhang ALM 69 2 0 15 Nov 2024
Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization Yuhan Fu Ruobing Xie Xingwu Sun Zhanhui Kang Xirong Li MLLM 60 4 0 15 Nov 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 76 56 1 15 Nov 2024
Efficient Alignment of Large Language Models via Data Sampling Amrit Khera Rajat Ghosh Debojyoti Dutta 56 1 0 15 Nov 2024
Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment Yuang Cai Yuyu Yuan Jinsheng Shi Qinhong Lin 51 0 0 14 Nov 2024
Evaluating the Generation of Spatial Relations in Text and Image Generative Models Shang Hong Sim Clarence Lee A. Tan Cheston Tan EGVM 41 2 0 12 Nov 2024
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset Khaoula Chehbouni Jonathan Colaço-Carr Yash More Jackie CK Cheung G. Farnadi 80 1 0 12 Nov 2024
SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMs Ruben Härle Felix Friedrich Manuel Brack Bjorn Deiseroth P. Schramowski Kristian Kersting 56 0 0 11 Nov 2024
Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMs Megh Thakkar Yash More Quentin Fournier Matthew D Riemer Pin-Yu Chen Amal Zouaq Payel Das Sarath Chandar ALM LLMSV 33 4 0 11 Nov 2024
AssistRAG: Boosting the Potential of Large Language Models with an Intelligent Information Assistant Yujia Zhou Zheng Liu Zhicheng Dou AIFin LRM RALM 41 2 0 11 Nov 2024
Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching A. Jain Harley Wiltzer Jesse Farebrother Irina Rish Glen Berseth Sanjiban Choudhury 67 1 0 11 Nov 2024
Beyond Toxic Neurons: A Mechanistic Analysis of DPO for Toxicity Reduction Yushi Yang Filip Sondej Harry Mayne Adam Mahdi 31 2 0 10 Nov 2024
Clustering Algorithms and RAG Enhancing Semi-Supervised Text Classification with Large LLMs Shan Zhong Jiahao Zeng Yongxin Yu Bohong Lin 50 1 0 09 Nov 2024
Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings Miguel Moura Ramos Tomás Almeida Daniel Vareta Filipe Azevedo Sweta Agrawal Patrick Fernandes André F. T. Martins 58 2 0 08 Nov 2024
Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization Zhuotong Chen Fang Liu Jennifer Zhu Wanyu Du Yanjun Qi 52 0 0 07 Nov 2024
One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity Sonia K. Murthy Tomer Ullman Jennifer Hu ALM 48 12 0 07 Nov 2024
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF Heyang Zhao Chenlu Ye Quanquan Gu Tong Zhang OffRL 71 4 0 07 Nov 2024
RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation Ian Poey Jiajun Liu Qishuai Zhong Adrien Chenailler 71 0 0 06 Nov 2024
From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning Zhirui Deng Zhicheng Dou Yinlin Zhu Ji-Rong Wen Ruibin Xiong Mang Wang Xin Wu 54 6 0 06 Nov 2024
SEE-DPO: Self Entropy Enhanced Direct Preference Optimization Shivanshu Shekhar Shreyas Singh Tong Zhang 54 4 0 06 Nov 2024
Mitigating Metric Bias in Minimum Bayes Risk Decoding Geza Kovacs Daniel Deutsch Markus Freitag 47 6 0 05 Nov 2024
Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment Jason Vega Junsheng Huang Gaokai Zhang Hangoo Kang Minjia Zhang Gagandeep Singh 44 0 0 05 Nov 2024
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization Yuxi Xie Guanzhen Li Xiao Xu Min-Yen Kan MLLM VLM 65 17 0 05 Nov 2024
Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent Xingwu Sun Yanfeng Chen Yanwen Huang Ruobing Xie Jiaqi Zhu ... Zhanhui Kang Yong Yang Yuhong Liu Di Wang Jie Jiang MoE ALM ELM 81 27 0 04 Nov 2024
Culinary Class Wars: Evaluating LLMs using ASH in Cuisine Transfer Task Hoonick Lee Mogan Gim Donghyeon Park Donghee Choi Jaewoo Kang 41 0 0 04 Nov 2024
Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback Guan-Ting Lin Prashanth Gurunath Shivakumar Aditya Gourav Yile Gu Ankur Gandhe Hung-yi Lee I. Bulyko 59 9 0 04 Nov 2024
Sample-Efficient Alignment for LLMs Zichen Liu Changyu Chen Chao Du Wee Sun Lee Min Lin 41 4 0 03 Nov 2024
PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment Dongxu Liu Bing Xu Yinzhuo Chen Bufan Xu Wenpeng Lu Muyun Yang Tiejun Zhao MoE 49 1 0 02 Nov 2024
TODO: Enhancing LLM Alignment with Ternary Preferences Yuxiang Guo Lu Yin Bo Jiang Jiaqi Zhang 67 1 0 02 Nov 2024
SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models Jianyi Zhang Da-Cheng Juan Cyrus Rashtchian Chun-Sung Ferng Heinrich Jiang Yiran Chen 50 4 0 01 Nov 2024
Token-level Proximal Policy Optimization for Query Generation Yichen Ouyang Lu Wang Fangkai Yang Pu Zhao Chenghua Huang ... Saravan Rajmohan Weiwei Deng Dongmei Zhang Feng Sun Qi Zhang OffRL 296 3 0 01 Nov 2024
Active Preference-based Learning for Multi-dimensional Personalization Minhyeon Oh Seungjoon Lee Jungseul Ok 36 1 0 01 Nov 2024
MoD: A Distribution-Based Approach for Merging Large Language Models Quy-Anh Dang Chris Ngo MoMe VLM 41 0 0 01 Nov 2024
Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction Utsav Singh Souradip Chakraborty Wesley A Suttle Brian M. Sadler Anit Kumar Sahu Mubarak Shah Vinay P. Namboodiri Amrit Singh Bedi 73 1 0 01 Nov 2024
Enhancing the Traditional Chinese Medicine Capabilities of Large Language Model through Reinforcement Learning from AI Feedback Song Yu Xiaofei Xu Fangfei Xu Li Li LM&MA 48 1 0 01 Nov 2024
Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation Bohan Lyu Yadi Cao Duncan Watson-Parris Leon Bergen Taylor Berg-Kirkpatrick Rose Yu 67 3 0 01 Nov 2024
LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation Bowen Li Zhaoyu Li Qiwei Du Jinqi Luo Wenshan Wang ... Katia Sycara Pradeep Kumar Ravikumar Alexander G. Gray X. Si Sebastian A. Scherer AI4CE LRM 91 3 0 01 Nov 2024
Desert Camels and Oil Sheikhs: Arab-Centric Red Teaming of Frontier LLMs Muhammed Saeed Elgizouli Mohamed Mukhtar Mohamed Shaina Raza Muhammad Abdul-Mageed Shady Shehata 58 0 0 31 Oct 2024
The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge Dake Guo Jixun Yao Xinfa Zhu Kangxiang Xia Zhao Guo Ziyu Zhang Yun Wang Jie Liu Lei Xie 39 1 0 31 Oct 2024
OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models Junda Wu Xintong Li Ruoyu Wang Yu Xia Yuxin Xiong ... Xiang Chen Branislav Kveton Lina Yao Jingbo Shang Julian McAuley OffRL LRM 34 1 0 31 Oct 2024