Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,645 papers shown

Title
MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization Kangyu Zhu Peng Xia Yun Li Hongtu Zhu Sheng Wang Huaxiu Yao 111 1 0 09 Dec 2024
Reinforcement Learning Enhanced LLMs: A Survey Shuhe Wang Shengyu Zhang Jing Zhang Runyi Hu Xiaoya Li Tianwei Zhang Jiwei Li Fei Wu G. Wang Eduard H. Hovy OffRL 138 7 0 05 Dec 2024
Progress-Aware Video Frame Captioning Zihui Xue Joungbin An Xitong Yang Kristen Grauman 129 1 0 03 Dec 2024
Time-Reversal Provides Unsupervised Feedback to LLMs Yerram Varun Rahul Madhavan Sravanti Addepalli A. Suggala Karthikeyan Shanmugam Prateek Jain LRM SyDa 79 0 0 03 Dec 2024
PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos Meng Cao Haoran Tang Haoze Zhao Hangyu Guo Jing Liu Ge Zhang Ruyang Liu Qiang Sun Ian Reid Xiaodan Liang 117 2 0 02 Dec 2024
Harnessing Preference Optimisation in Protein LMs for Hit Maturation in Cell Therapy Katarzyna Janocha Annabel Ling Alice Godson Yulia Lampi Simon Bornschein Nils Y. Hammerla 87 2 0 02 Dec 2024
Yi-Lightning Technical Report 01. AI : Alan Wake Albert Wang Bei Chen ... Yuxuan Sha Zhaodong Yan Zhiyuan Liu Zirui Zhang Zonghong Dai OSLM 104 3 0 02 Dec 2024
Towards Adaptive Mechanism Activation in Language Agent Ziyang Huang Jun Zhao Kang Liu LLMAG AI4CE 90 0 0 01 Dec 2024
ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning Yang Wu Huayi Zhang Yizheng Jiao Lin Ma Xiaozhong Liu Jinhong Yu Dongyu Zhang Dezhi Yu Wei Xu 100 1 0 01 Dec 2024
DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling Xin Xie Dong Gong 95 1 0 01 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 126 2 0 01 Dec 2024
o1-Coder: an o1 Replication for Coding Yuxiang Zhang Shangxi Wu Yuqi Yang Jiangming Shu Jinlin Xiao Chao Kong Jitao Sang LRM 71 35 0 29 Nov 2024
Simple and Provable Scaling Laws for the Test-Time Compute of Large Language Models Yanxi Chen Xuchen Pan Yaliang Li Bolin Ding Jingren Zhou LRM 99 7 0 29 Nov 2024
Sparse Attention Vectors: Generative Multimodal Model Features Are Discriminative Vision-Language Classifiers Chancharik Mitra Brandon Huang Tianning Chai Zhiqiu Lin Assaf Arbelle Rogerio Feris Leonid Karlinsky Trevor Darrell Deva Ramanan Roei Herzig VLM 151 4 0 28 Nov 2024
How far can bias go? -- Tracing bias from pretraining data to alignment Marion Thaler Abdullatif Köksal Alina Leidinger Anna Korhonen Hinrich Schutze 100 0 0 28 Nov 2024
Mars-PO: Multi-Agent Reasoning System Preference Optimization Xiaoxuan Lou Chaojie Wang Bo An LLMAG LRM 76 0 0 28 Nov 2024
Challenges in Human-Agent Communication Gagan Bansal J. W. Vaughan Saleema Amershi Eric Horvitz Adam Fourney Hussein Mozannar Victor C. Dibia Daniel S. Weld LLMAG AAML AI4CE 96 5 0 28 Nov 2024
MATATA: Weakly Supervised End-to-End MAthematical Tool-Augmented Reasoning for Tabular Applications Vishnou Vinayagame Gregory Senay Luis Martí LRM ReLM 80 0 0 28 Nov 2024
Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning Di Zhang Jingdi Lei Junxian Li Xunzhi Wang Yong Liu ... Steve Yang Jianbo Wu Peng Ye Wanli Ouyang Dongzhan Zhou OffRL LRM 107 6 0 27 Nov 2024
PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuning Zhen Sun Tianshuo Cong Yule Liu Chenhao Lin Xinlei He Rongmao Chen Xingshuo Han Xinyi Huang AAML 98 4 0 26 Nov 2024
A Topic-level Self-Correctional Approach to Mitigate Hallucinations in MLLMs Lehan He Zeren Chen Zhelun Shi Tianyu Yu Jing Shao Lu Sheng MLLM 118 1 0 26 Nov 2024
Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness Avinash Amballa Durga Sandeep Saluru Gayathri Akkinapalli Abhishek Sureddy Akshay Kumar Sureddy ALM 97 0 0 26 Nov 2024
Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach Shijian Deng Wentian Zhao Yu-Jhe Li Kun Wan Daniel Miranda Ajinkya Kale Yapeng Tian LRM 93 6 0 26 Nov 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 131 74 0 25 Nov 2024
Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systems Magdalena Kaiser P. Ernst György Szarvas 82 1 0 25 Nov 2024
Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Models Hao Yi Qingyang Li Yihan Hu Fuzheng Zhang Di Zhang Yong Liu VGen 88 0 0 25 Nov 2024
All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages Ashmal Vayani Dinura Dissanayake Hasindri Watawana Noor Ahsan Nevasini Sasikumar ... Monojit Choudhury Ivan Laptev Mubarak Shah Salman Khan Fahad A Khan 128 9 0 25 Nov 2024
Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward Zhiwei Jia Yuesong Nan Huixi Zhao Gengdai Liu EGVM 96 1 0 22 Nov 2024
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning Elita Lobo Chirag Agarwal Himabindu Lakkaraju LRM 92 6 0 22 Nov 2024
Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning Song Jiang Da JU Andrew Cohen Sasha Mitts Aaron Foss Justine T Kao Xian Li Yuandong Tian 86 3 0 21 Nov 2024
Hymba: A Hybrid-head Architecture for Small Language Models Xin Dong Y. Fu Shizhe Diao Wonmin Byeon Zijia Chen ... Min-Hung Chen Yoshi Suhara Y. Lin Jan Kautz Pavlo Molchanov Mamba 107 21 0 20 Nov 2024
SpecTool: A Benchmark for Characterizing Errors in Tool-Use LLMs Shirley Kokane Ming Zhu Tulika Awalgaonkar Jianguo Zhang Thai Hoang ... Juan Carlos Niebles Huan Wang Shelby Heinecke Caiming Xiong Silivo Savarese LLMAG 111 1 0 20 Nov 2024
Metacognition for Unknown Situations and Environments (MUSE) Rodolfo Valiente Praveen K. Pilly 85 0 0 20 Nov 2024
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations Gaurav Verma Rachneet Kaur Nishan Srishankar Zhen Zeng T. Balch Manuela Veloso LLMAG 77 5 0 20 Nov 2024
Patience Is The Key to Large Language Model Reasoning Yijiong Yu LRM 75 0 0 20 Nov 2024
DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs Zhihan Liu Shenao Zhang Yongfei Liu Boyi Liu Yingxiang Yang Zhaoran Wang 115 3 0 20 Nov 2024
A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection Gabriel Chua Shing Yee Chan Shaun Khoo 117 1 0 20 Nov 2024
Reward Modeling with Ordinal Feedback: Wisdom of the Crowd Shang Liu Yu Pan Guanting Chen Xiaocheng Li 87 2 0 19 Nov 2024
Aligning Few-Step Diffusion Models with Dense Reward Difference Learning Ziyi Zhang Li Shen Sen Zhang Deheng Ye Yong Luo Miaojing Shi Di Lin Dacheng Tao 103 1 0 18 Nov 2024
Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering Xinyan Guan Yanjiang Liu Xinyu Lu Boxi Cao Xianpei Han ... Le Sun Jie Lou Bowen Yu Yaojie Lu Hongyu Lin ALM 96 2 0 18 Nov 2024
TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models Tingyu Qu Mingxiao Li Tinne Tuytelaars Marie-Francine Moens VLM 52 2 0 17 Nov 2024
SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization Hongrui Jia Chaoya Jiang Haiyang Xu Wei Ye Mengfan Dong Ming Yan Ji Zhang Fei Huang Shikun Zhang MLLM 103 2 0 17 Nov 2024
Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment Andrew Konya Aviv Ovadya K. J. Kevin Feng Quan Ze Chen Lisa Schirch Colin Irwin Amy X. Zhang ALM 69 2 0 15 Nov 2024
Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization Yuhan Fu Ruobing Xie Xingwu Sun Zhanhui Kang Xirong Li MLLM 60 4 0 15 Nov 2024
Efficient Alignment of Large Language Models via Data Sampling Amrit Khera Rajat Ghosh Debojyoti Dutta 56 1 0 15 Nov 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 76 51 1 15 Nov 2024
Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment Yuang Cai Yuyu Yuan Jinsheng Shi Qinhong Lin 51 0 0 14 Nov 2024
Evaluating the Generation of Spatial Relations in Text and Image Generative Models Shang Hong Sim Clarence Lee A. Tan Cheston Tan EGVM 41 2 0 12 Nov 2024
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset Khaoula Chehbouni Jonathan Colaço-Carr Yash More Jackie CK Cheung G. Farnadi 80 0 0 12 Nov 2024
SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMs Ruben Härle Felix Friedrich Manuel Brack Bjorn Deiseroth P. Schramowski Kristian Kersting 53 0 0 11 Nov 2024