A Survey on Knowledge Distillation of Large Language Models

20 February 2024

Papers citing "A Survey on Knowledge Distillation of Large Language Models"

50 / 104 papers shown

Title
VERDI: VLM-Embedded Reasoning for Autonomous Driving Bowen Feng Zhiting Mei Baiang Li Julian Ost Roger Girgis Anirudha Majumdar Felix Heide VLM LRM 149 0 0 21 May 2025
Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization Yamato Arai Yuma Ichikawa MQ 60 0 0 13 Apr 2025
Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions Yichong Zhao Susumu Goto 78 0 0 05 Mar 2025
Who Taught You That? Tracing Teachers in Model Distillation Somin Wadhwa Chantal Shaib Silvio Amir Byron C. Wallace 148 2 0 10 Feb 2025
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Nicolas Boizard Kevin El Haddad C´eline Hudelot Pierre Colombo 104 16 0 28 Jan 2025
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents Weiwei Sun Lingyong Yan Xinyu Ma Shuaiqiang Wang Pengjie Ren Zhumin Chen Dawei Yin Zhaochun Ren RALM ALM ELM LRM LM&MA 130 304 0 31 Dec 2024
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 122 5 0 22 Oct 2024
Future-Guided Learning: A Predictive Approach To Enhance Time-Series Forecasting Skye Gunasekaran Assel Kembay Hugo J. Ladret Rui-Jie Zhu Laurent Udo Perrinet Omid Kavehei Jason K. Eshraghian AI4TS 53 0 0 19 Oct 2024
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling Wenyuan Xu Rujun Han Zhenting Wang L. Le Dhruv Madeka Lei Li Wenjie Wang Rishabh Agarwal Chen-Yu Lee Tomas Pfister 104 9 0 15 Oct 2024
LUK: Empowering Log Understanding with Expert Knowledge from Large Language Models Lipeng Ma Weidong Yang Sihang Jiang Ben Fei Mingjie Zhou Shuhao Li Bo Xu Bo Xu Yanghua Xiao 94 0 0 03 Sep 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 72 49 0 09 Jul 2024
V-STaR: Training Verifiers for Self-Taught Reasoners Arian Hosseini Xingdi Yuan Nikolay Malkin Rameswar Panda Alessandro Sordoni Rishabh Agarwal ReLM LRM 66 119 0 09 Feb 2024
APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference Bowen Zhao Hannaneh Hajishirzi Qingqing Cao 47 17 0 22 Jan 2024
MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning Chenyu Wang Weixin Luo Qianyu Chen Haonan Mai Jindi Guo Sixun Dong Xiaohua Xuan MLLM LLMAG 94 18 0 19 Jan 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 285 312 0 18 Jan 2024
AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets Ernest Perkowski Rui Pan Tuan Dung Nguyen Yuan-Sen Ting Sandor Kruk ... Michael J. Smith Huiling Liu Kevin Schawinski K. Iyer I. Ciucă AI4MH 54 12 0 03 Jan 2024
GeoGalactica: A Scientific Large Language Model in Geoscience Zhouhan Lin Cheng Deng Le Zhou Tianhang Zhang Yi Xu ... Weinan Zhang Junxian He Yunqiang Zhu Xinbing Wang Cheng Zhou 20 27 0 31 Dec 2023
Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning Chengwei Qin Wenhan Xia Fangkai Jiao Chen Chen Yuchen Hu Bosheng Ding R. Chen Shafiq Joty 64 7 0 28 Dec 2023
Instruction Fusion: Advancing Prompt Evolution through Hybridization Weidong Guo Jiuding Yang Kaitong Yang Xiangyang Li Zhuwei Rao Yu-Syuan Xu Di Niu 27 5 0 25 Dec 2023
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning Wei Liu Weihao Zeng Keqing He Yong Jiang Junxian He ALM 69 231 0 25 Dec 2023
G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model Jiahui Gao Renjie Pi Jipeng Zhang Jiacheng Ye Wanjun Zhong ... Lanqing Hong Jianhua Han Hang Xu Zhenguo Li Lingpeng Kong SyDa ReLM LRM 76 105 0 18 Dec 2023
MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Following Renze Lou Kai Zhang Jian Xie Yuxuan Sun Janice Ahn Hanzi Xu Yu Su Wenpeng Yin 55 29 0 05 Dec 2023
Near-real-time Earthquake-induced Fatality Estimation using Crowdsourced Data and Large-Language Models Chenguang Wang Davis Engler Xuechun Li James Hou David J. Wald Kishor Jaiswal Susu Xu 33 3 0 04 Dec 2023
MoDS: Model-oriented Data Selection for Instruction Tuning Qianlong Du Chengqing Zong Jiajun Zhang ALM 62 83 0 27 Nov 2023
Large Language Models in Law: A Survey Jinqi Lai Wensheng Gan Jiayang Wu Zhenlian Qi Philip S. Yu ELM AILaw 76 82 0 26 Nov 2023
HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs Junying Chen Xidong Wang Anningzhe Gao Feng Jiang Shunian Chen ... Chuyi Kong Jianquan Li Xiang Wan Haizhou Li Benyou Wang LM&MA 50 63 0 16 Nov 2023
Tailoring Self-Rationalizers with Multi-Reward Distillation Sahana Ramnath Brihi Joshi Skyler Hallinan Ximing Lu Liunian Harold Li Aaron Chan Jack Hessel Yejin Choi Xiang Ren LRM ReLM 37 16 0 06 Nov 2023
MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning Bingchang Liu Chaoyu Chen Cong Liao Zi Gong Huan Wang ... Dajun Chen Min Shen Hailian Zhou Hang Yu Jianguo Li MoMe 22 27 0 04 Nov 2023
Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey Xinyu She Yue Liu Yanjie Zhao Yiling He Li Li Chakkrit Tantithamthavorn Zhan Qin Haoyu Wang ELM 52 13 0 27 Oct 2023
Zephyr: Direct Distillation of LM Alignment Lewis Tunstall E. Beeching Nathan Lambert Nazneen Rajani Kashif Rasul ... Nathan Habib Nathan Sarrazin Omar Sanseviero Alexander M. Rush Thomas Wolf ALM 74 382 0 25 Oct 2023
AgentTuning: Enabling Generalized Agent Abilities for LLMs Aohan Zeng Mingdao Liu Rui Lu Bowen Wang Xiao Liu Yuxiao Dong Jie Tang LM&MA ALM LLMAG 57 172 0 19 Oct 2023
TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks Dongfu Jiang Yishan Li Ge Zhang Wenhao Huang Bill Yuchen Lin Wenhu Chen ALM 65 64 0 01 Oct 2023
CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets Lifan Yuan Yangyi Chen Xingyao Wang Yi R. Fung Hao Peng Heng Ji LLMAG KELM 74 65 0 29 Sep 2023
Continual Learning with Dirichlet Generative-based Rehearsal Min Zeng Wei Xue Qi-fei Liu Yi-Ting Guo CLL BDL 34 5 0 13 Sep 2023
Measuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies Boshko Koloski Blaž Škrlj Marko Robnik-Šikonja Senja Pollak CLL 66 2 0 12 Sep 2023
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning Xiang Yue Xingwei Qu Ge Zhang Yao Fu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen AIMat LRM 103 391 0 11 Sep 2023
Making Large Language Models Better Reasoners with Alignment Peiyi Wang Lei Li Liang Chen Feifan Song Binghuai Lin Yunbo Cao Tianyu Liu Zhifang Sui ALM LRM 67 68 0 05 Sep 2023
StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data Yanda Li Chi Zhang Gang Yu Zhibin Wang Bin-Bin Fu Guosheng Lin Chunhua Shen Ling Chen Yunchao Wei MLLM 29 30 0 20 Aug 2023
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct Haipeng Luo Qingfeng Sun Can Xu Pu Zhao Jian-Guang Lou ... Xiubo Geng Qingwei Lin Shifeng Chen Yansong Tang Dongmei Zhang LRM OSLM 155 439 0 18 Aug 2023
Reinforced Self-Training (ReST) for Language Modeling Çağlar Gülçehre T. Paine S. Srinivasan Ksenia Konyushkova L. Weerts ... Chenjie Gu Wolfgang Macherey Arnaud Doucet Orhan Firat Nando de Freitas OffRL 98 293 0 17 Aug 2023
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning Yun Luo Zhen Yang Fandong Meng Yafu Li Jie Zhou Yue Zhang CLL KELM 92 297 0 17 Aug 2023
f-Divergence Minimization for Sequence-Level Knowledge Distillation Yuqiao Wen Zichao Li Wenyu Du Lili Mou 52 56 0 27 Jul 2023
ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning Liang Zhao En Yu Zheng Ge Jinrong Yang Hao-Ran Wei ... Jian‐Yuan Sun Yuang Peng Runpei Dong Chunrui Han Xiangyu Zhang MLLM LRM 48 54 0 18 Jul 2023
Neural Machine Translation Data Generation and Augmentation using ChatGPT Wayne Yang Garrett Nicolai 81 7 0 11 Jul 2023
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest Shilong Zhang Pei Sun Shoufa Chen Min Xiao Wenqi Shao Wenwei Zhang Yu Liu Kai-xiang Chen Ping Luo VLM MLLM 122 231 0 07 Jul 2023
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting Zhen Qin R. Jagerman Kai Hui Honglei Zhuang Junru Wu ... Tianqi Liu Jialu Liu Donald Metzler Xuanhui Wang Michael Bendersky ALM RALM 71 235 0 30 Jun 2023
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding Yanzhe Zhang Ruiyi Zhang Jiuxiang Gu Yufan Zhou Nedim Lipka Diyi Yang Tongfei Sun VLM MLLM 46 227 0 29 Jun 2023
UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations? Junda Wang Zonghai Yao Avijit Mitra Samuel Osebe Zhichao Yang Hongfeng Yu LM&MA MedIm 78 14 0 29 Jun 2023
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation Yixiao Li Yifan Yu Qingru Zhang Chen Liang Pengcheng He Weizhu Chen Tuo Zhao 85 71 0 20 Jun 2023
WizardCoder: Empowering Code Large Language Models with Evol-Instruct Ziyang Luo Can Xu Pu Zhao Qingfeng Sun Xiubo Geng Wenxiang Hu Chongyang Tao Jing Ma Qingwei Lin Daxin Jiang ELM SyDa ALM 68 665 0 14 Jun 2023