DistiLLM: Towards Streamlined Distillation for Large Language Models

DistiLLM: Towards Streamlined Distillation for Large Language Models

6 February 2024

Tianyi Chen

Papers citing "DistiLLM: Towards Streamlined Distillation for Large Language Models"

18 / 18 papers shown

Title
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $α$ - $β$ -Divergence Guanghui Wang Zhiyong Yang Ziyi Wang Shi Wang Qianqian Xu Qingming Huang 208 0 0 07 May 2025
Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques Sanjay Surendranath Girija Shashank Kapoor Lakshit Arora Dipen Pradhan Aman Raj Ankit Shetgaonkar 102 0 0 05 May 2025
EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models Xingrun Xing Zheng Liu Shitao Xiao Boyan Gao Yiming Liang Wanpeng Zhang Haokun Lin Guoqi Li Jiajun Zhang LRM 153 1 0 10 Feb 2025
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 131 5 0 22 Oct 2024
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling Wenyuan Xu Rujun Han Zhenting Wang L. Le Dhruv Madeka Lei Li Wenjie Wang Rishabh Agarwal Chen-Yu Lee Tomas Pfister 126 11 0 15 Oct 2024
Small Language Models: Survey, Measurements, and Insights Zhenyan Lu Xiang Li Dongqi Cai Rongjie Yi Fangming Liu Xiwen Zhang Nicholas D. Lane Mengwei Xu ObjD LRM 96 51 0 24 Sep 2024
HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning Tianyi Chen Xiaoyi Qu David Aponte Colby R. Banbury Jongwoo Ko Tianyu Ding Yong Ma Vladimir Lyapunov Ilya Zharkov Luming Liang 124 2 0 11 Sep 2024
f-Divergence Minimization for Sequence-Level Knowledge Distillation Yuqiao Wen Zichao Li Wenyu Du Lili Mou 54 58 0 27 Jul 2023
PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning Hojoon Lee Hanseul Cho Hyunseung Kim Daehoon Gwak Joonkee Kim Jaegul Choo Se-Young Yun Chulhee Yun OffRL 107 29 0 19 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 316 4,288 0 09 Jun 2023
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor Or Honovich Thomas Scialom Omer Levy Timo Schick ALM 102 372 0 19 Dec 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 806 12,893 0 04 Mar 2022
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization Bogdan Gliwa Iwona Mochol M. Biesek A. Wawer 111 631 0 27 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 399 20,114 0 23 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 216 7,481 0 02 Oct 2019
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction Aviral Kumar Justin Fu George Tucker Sergey Levine OffRL OnRL 109 1,055 0 03 Jun 2019
Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization Shashi Narayan Shay B. Cohen Mirella Lapata AILaw 119 1,674 0 27 Aug 2018
Get To The Point: Summarization with Pointer-Generator Networks A. See Peter J. Liu Christopher D. Manning 3DPC 276 4,019 0 14 Apr 2017

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.