v1v2 (latest)

Advantage-Guided Distillation for Preference Alignment in Small Language Models

25 February 2025

Papers citing "Advantage-Guided Distillation for Preference Alignment in Small Language Models"

40 / 40 papers shown

Title
TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback Eunseop Yoon Hee Suk Yoon Soohwan Eom Gunsoo Han D. W. Nam DaeJin Jo Kyoung-Woon On M. Hasegawa-Johnson Sungwoong Kim C. Yoo ALM 85 21 0 23 Jul 2024
H2O-Danube3 Technical Report Pascal Pfeiffer Philipp Singer Yauhen Babakhin Gabor Fodor Nischay Dhankhar Sri Satish Ambati 29 4 0 12 Jul 2024
Direct Preference Knowledge Distillation for Large Language Models Yixing Li Yuxian Gu Li Dong Dequan Wang Yu Cheng Furu Wei 96 6 0 28 Jun 2024
WPO: Enhancing RLHF with Weighted Preference Optimization Wenxuan Zhou Ravi Agrawal Shujian Zhang Sathish Indurthi Sanqiang Zhao Kaiqiang Song Silei Xu Chenguang Zhu 89 20 0 17 Jun 2024
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs Rongzhi Zhang Jiaming Shen Tianqi Liu Haorui Wang Zhen Qin Feng Han Jialu Liu Simon Baumgartner Michael Bendersky Chao Zhang 54 8 0 05 Jun 2024
Robust Preference Optimization through Reward Model Distillation Adam Fisch Jacob Eisenstein Vicky Zayats Alekh Agarwal Ahmad Beirami Chirag Nagpal Peter Shaw Jonathan Berant 126 34 0 29 May 2024
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer Zhihan Liu Miao Lu Shenao Zhang Boyi Liu Hongyi Guo Yingxiang Yang Jose H. Blanchet Zhaoran Wang 108 58 0 26 May 2024
SimPO: Simple Preference Optimization with a Reference-Free Reward Yu Meng Mengzhou Xia Danqi Chen 122 480 0 23 May 2024
DPO Meets PPO: Reinforced Token Optimization for RLHF Han Zhong Zikang Shan Guhao Feng Wei Xiong Xinle Cheng Li Zhao Di He Jiang Bian Liwei Wang 121 67 0 29 Apr 2024
Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models Taiqiang Wu Chaofan Tao Jiahao Wang Zhe Zhao Ngai Wong ALM 79 17 0 03 Apr 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James V. Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 154 257 0 20 Mar 2024
Revisiting Knowledge Distillation for Autoregressive Language Models Qihuang Zhong Liang Ding Li Shen Juhua Liu Bo Du Dacheng Tao KELM 76 19 0 19 Feb 2024
DistiLLM: Towards Streamlined Distillation for Large Language Models Jongwoo Ko Sungnyun Kim Tianyi Chen SeYoung Yun 107 34 0 06 Feb 2024
Dense Reward for Free in Reinforcement Learning from Human Feedback Alex J. Chan Hao Sun Samuel Holt M. Schaar 64 42 0 01 Feb 2024
H2O-Danube-1.8B Technical Report Philipp Singer Pascal Pfeiffer Yauhen Babakhin Maximilian Jeblick Nischay Dhankhar Gabor Fodor SriSatish Ambati VLM 36 8 0 30 Jan 2024
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning Wei Liu Weihao Zeng Keqing He Yong Jiang Junxian He ALM 94 235 0 25 Dec 2023
Zephyr: Direct Distillation of LM Alignment Lewis Tunstall E. Beeching Nathan Lambert Nazneen Rajani Kashif Rasul ... Nathan Habib Nathan Sarrazin Omar Sanseviero Alexander M. Rush Thomas Wolf ALM 100 390 0 25 Oct 2023
BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories? Xingmeng Zhao Tongnian Wang Sheri Osborn Anthony Rios 40 6 0 25 Oct 2023
MCC-KD: Multi-CoT Consistent Knowledge Distillation Hongzhan Chen Siyue Wu Xiaojun Quan Rui Wang Ming Yan Ji Zhang LRM 51 17 0 23 Oct 2023
Mistral 7B Albert Q. Jiang Alexandre Sablayrolles A. Mensch Chris Bamford Devendra Singh Chaplot ... Teven Le Scao Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LRM 79 2,204 0 10 Oct 2023
Understanding the Effects of RLHF on LLM Generalisation and Diversity Robert Kirk Ishita Mediratta Christoforos Nalmpantis Jelena Luketina Eric Hambro Edward Grefenstette Roberta Raileanu AI4CE ALM 163 148 0 10 Oct 2023
Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond Hao Sun OffRL 73 22 0 09 Oct 2023
f-Divergence Minimization for Sequence-Level Knowledge Distillation Yuqiao Wen Zichao Li Wenyu Du Lili Mou 59 59 0 27 Jul 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 302 11,894 0 18 Jul 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 359 4,388 0 09 Jun 2023
Preference-grounded Token-level Guidance for Language Model Fine-tuning Shentao Yang Shujian Zhang Congying Xia Yihao Feng Caiming Xiong Mi Zhou 108 26 0 01 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 385 3,981 0 29 May 2023
SLiC-HF: Sequence Likelihood Calibration with Human Feedback Yao-Min Zhao Rishabh Joshi Tianqi Liu Misha Khalman Mohammad Saleh Peter J. Liu 72 297 0 17 May 2023
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression Siyue Wu Hongzhan Chen Xiaojun Quan Qifan Wang Rui Wang VLM 61 19 0 17 May 2023
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen ... Jack Clark Sam McCandlish C. Olah Benjamin Mann Jared Kaplan 249 2,561 0 12 Apr 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 877 12,973 0 04 Mar 2022
A General Language Assistant as a Laboratory for Alignment Amanda Askell Yuntao Bai Anna Chen Dawn Drain Deep Ganguli ... Tom B. Brown Jack Clark Sam McCandlish C. Olah Jared Kaplan ALM 118 779 0 01 Dec 2021
MergeDistill: Merging Pre-trained Language Models using Distillation Simran Khanuja Melvin Johnson Partha P. Talukdar 67 16 0 05 Jun 2021
Adaptive Multi-Teacher Multi-level Knowledge Distillation Yuang Liu Wei Zhang Jun Wang 70 157 0 06 Mar 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 605 4,822 0 23 Jan 2020
Distilling Policy Distillation Wojciech M. Czarnecki Razvan Pascanu Simon Osindero Siddhant M. Jayakumar G. Swirszcz Max Jaderberg 62 133 0 06 Feb 2019
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 499 19,065 0 20 Jul 2017
Sequence-Level Knowledge Distillation Yoon Kim Alexander M. Rush 119 1,115 0 25 Jun 2016
Policy Distillation Andrei A. Rusu Sergio Gomez Colmenarejo Çağlar Gülçehre Guillaume Desjardins J. Kirkpatrick Razvan Pascanu Volodymyr Mnih Koray Kavukcuoglu R. Hadsell 89 692 0 19 Nov 2015
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 362 19,660 0 09 Mar 2015