Robust Preference Optimization through Reward Model Distillation

Robust Preference Optimization through Reward Model Distillation

29 May 2024

Jacob Eisenstein

Jonathan Berant

Papers citing "Robust Preference Optimization through Reward Model Distillation"

19 / 19 papers shown

Title
Mitigating Preference Hacking in Policy Optimization with Pessimism Dhawal Gupta Adam Fisch Christoph Dann Alekh Agarwal 73 0 0 10 Mar 2025
All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning Gokul Swamy Sanjiban Choudhury Wen Sun Zhiwei Steven Wu J. Andrew Bagnell OffRL 42 7 0 03 Mar 2025
Distributionally Robust Reinforcement Learning with Human Feedback Debmalya Mandal Paulius Sasnauskas Goran Radanović 39 1 0 01 Mar 2025
Advantage-Guided Distillation for Preference Alignment in Small Language Models Shiping Gao Fanqi Wan Jiajian Guo Xiaojun Quan Qifan Wang ALM 58 0 0 25 Feb 2025
C-3DPO: Constrained Controlled Classification for Direct Preference Optimization Kavosh Asadi Julien Han Xingzi Xu Dominique Perrault-Joncas Shoham Sabach Karim Bouyarmane Mohammad Ghavamzadeh 34 0 0 22 Feb 2025
Design Considerations in Offline Preference-based RL Alekh Agarwal Christoph Dann T. V. Marinov OffRL 51 0 0 08 Feb 2025
An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems Hashmath Shaik Alex Doboli OffRL ELM 149 0 0 31 Dec 2024
CREAM: Consistency Regularized Self-Rewarding Language Models Z. Wang Weilei He Zhiyuan Liang Xuchao Zhang Chetan Bansal Ying Wei Weitong Zhang Huaxiu Yao ALM 101 7 0 16 Oct 2024
Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both Abhijnan Nath Changsoo Jung Ethan Seefried Nikhil Krishnaswamy 134 1 0 11 Oct 2024
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization Noam Razin Sadhika Malladi Adithya Bhaskar Danqi Chen Sanjeev Arora Boris Hanin 91 14 0 11 Oct 2024
Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization Jianing Wang Yang Zhou Xiaocheng Zhang Mengjiao Bao Peng Yan 30 1 0 17 Sep 2024
Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level Jie Liu Zhanhui Zhou Jiaheng Liu Xingyuan Bu Chao Yang Han-Sen Zhong Wanli Ouyang 31 16 0 17 Jun 2024
Nemotron-4 340B Technical Report Nvidia : Bo Adler Niket Agarwal Ashwath Aithal ... Jimmy Zhang Jing Zhang Vivienne Zhang Yian Zhang Chen Zhu 41 56 0 17 Jun 2024
Direct Preference Optimization with an Offset Afra Amini Tim Vieira Ryan Cotterell 71 55 0 16 Feb 2024
WARM: On the Benefits of Weight Averaged Reward Models Alexandre Ramé Nino Vieillard Léonard Hussenot Robert Dadashi Geoffrey Cideron Olivier Bachem Johan Ferret 106 93 0 22 Jan 2024
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles Yuanzhao Zhai Han Zhang Yu Lei Yue Yu Kele Xu Dawei Feng Bo Ding Huaimin Wang AI4CE 68 32 0 30 Dec 2023
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization Zhanhui Zhou Jie Liu Chao Yang Jing Shao Yu Liu Xiangyu Yue Wanli Ouyang Yu Qiao 27 48 0 05 Oct 2023
Let Me Teach You: Pedagogical Foundations of Feedback for Language Models Beatriz Borges Niket Tandon Tanja Kaser Antoine Bosselut 22 3 0 01 Jul 2023
COMBO: Conservative Offline Model-Based Policy Optimization Tianhe Yu Aviral Kumar Rafael Rafailov Aravind Rajeswaran Sergey Levine Chelsea Finn OffRL 219 413 0 16 Feb 2021