A Survey on Knowledge Distillation of Large Language Models

20 February 2024

Papers citing "A Survey on Knowledge Distillation of Large Language Models"

50 / 104 papers shown

Title
Valley: Video Assistant with Large Language model Enhanced abilitY Ruipu Luo Ziwang Zhao Min Yang Junwei Dong Da Li Pengcheng Lu Tao Wang Linmei Hu Ming-Hui Qiu MLLM 93 195 0 12 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 262 4,186 0 09 Jun 2023
Fine-Tuning Language Models with Just Forward Passes Sadhika Malladi Tianyu Gao Eshaan Nichani Alexandru Damian Jason D. Lee Danqi Chen Sanjeev Arora 103 186 0 27 May 2023
The False Promise of Imitating Proprietary LLMs Arnav Gudibande Eric Wallace Charles Burton Snell Xinyang Geng Hao Liu Pieter Abbeel Sergey Levine Dawn Song ALM 93 202 0 25 May 2023
Gorilla: Large Language Model Connected with Massive APIs Shishir G. Patil Tianjun Zhang Xin Wang Joseph E. Gonzalez ELM CLL ALM SyDa 69 544 0 24 May 2023
HuatuoGPT, towards Taming Language Model to Be a Doctor Hongbo Zhang Junying Chen Feng Jiang Fei Yu Zhihong Chen ... Zhiyi Zhang Qingying Xiao Xiang Wan Benyou Wang Haizhou Li LM&MA AI4MH ELM 71 198 0 24 May 2023
Lawyer LLaMA Technical Report Quzhe Huang Mingxu Tao Chen Zhang Zhenwei An Cong Jiang Zhibin Chen Zirui Wu Yansong Feng ELM ALM AILaw 74 51 0 24 May 2023
Reasoning with Language Model is Planning with World Model Shibo Hao Yi Gu Haodi Ma Joshua Jiahua Hong Zhen Wang D. Wang Zhiting Hu ReLM LRM LLMAG 102 551 0 24 May 2023
ExpertPrompting: Instructing Large Language Models to be Distinguished Experts Benfeng Xu An Yang Junyang Lin Quang Wang Chang Zhou Yongdong Zhang Zhendong Mao ALM 81 136 0 24 May 2023
Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization Jeonghoon Kim J. H. Lee Sungdong Kim Joonsuk Park Kang Min Yoo S. Kwon Dongsoo Lee MQ 76 102 0 23 May 2023
InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT Yichong Xu Ruochen Xu Dan Iter Yang Liu Shuohang Wang Chenguang Zhu Michael Zeng 33 10 0 22 May 2023
Distilling ChatGPT for Explainable Automated Student Answer Assessment Jiazheng Li Lin Gui Yuxiang Zhou David West Cesare Aloisi Yulan He 54 28 0 22 May 2023
Improving Small Language Models on PubMedQA via Generative Data Augmentation Zhen Guo Peiqi Wang Yanwei Wang Shangdi Yu LM&MA MedIm 28 12 0 12 May 2023
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision Zhiqing Sun Songlin Yang Qinhong Zhou Hongxin Zhang Zhenfang Chen David D. Cox Yiming Yang Chuang Gan SyDa ALM 60 329 0 04 May 2023
Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT Tong Xie Yuwei Wan Wei-Ping Huang Yufei Zhou Yixuan Liu ... Shaozhou Wang Chunyu Kit Clara Grazian Weinan Zhang Hoex 45 52 0 05 Apr 2023
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models Zhiqiang Hu Lei Wang Yihuai Lan Wanyu Xu Ee-Peng Lim Lidong Bing Xing Xu Soujanya Poria Roy Ka-wei Lee ALM 78 249 0 04 Apr 2023
A Short Survey of Viewing Large Language Models in Legal Aspect Zhongxiang Sun AILaw ELM 72 67 0 16 Mar 2023
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers Jon Saad-Falcon Omar Khattab Keshav Santhanam Radu Florian M. Franz Salim Roukos Avirup Sil Md Arafat Sultan Christopher Potts 58 43 0 01 Mar 2023
Distilling Internet-Scale Vision-Language Models into Embodied Agents T. Sumers Kenneth Marino Arun Ahuja Rob Fergus Ishita Dasgupta LM&Ro 52 25 0 29 Jan 2023
Adam: Dense Retrieval Distillation with Adaptive Dark Examples Chongyang Tao Chang Liu Tao Shen Can Xu Xiubo Geng Binxing Jiao Daxin Jiang 61 4 0 20 Dec 2022
Large Language Models Are Reasoning Teachers Namgyu Ho Laura Schmid Se-Young Yun ReLM ELM LRM 81 339 0 20 Dec 2022
Constitutional AI: Harmlessness from AI Feedback Yuntao Bai Saurav Kadavath Sandipan Kundu Amanda Askell John Kernion ... Dario Amodei Nicholas Joseph Sam McCandlish Tom B. Brown Jared Kaplan SyDa MoMe 156 1,583 0 15 Dec 2022
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models Chan Hee Song Jiaman Wu Clay Washington Brian M Sadler Wei-Lun Chao Yu-Chuan Su LLMAG LM&Ro 89 393 0 08 Dec 2022
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning Yaqing Wang Sahaj Agarwal Subhabrata Mukherjee Xiaodong Liu Jing Gao Ahmed Hassan Awadallah Jianfeng Gao MoE 52 126 0 31 Oct 2022
Large Language Models Can Self-Improve Jiaxin Huang S. Gu Le Hou Yuexin Wu Xuezhi Wang Hongkun Yu Jiawei Han ReLM AI4MH LRM 129 594 0 20 Oct 2022
On Grounded Planning for Embodied Tasks with Language Models Bill Yuchen Lin Chengsong Huang Qian Liu Wenda Gu Sam Sommerer Xiang Ren LM&Ro 55 39 0 29 Aug 2022
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning Elias Frantar Sidak Pal Singh Dan Alistarh MQ 64 226 0 24 Aug 2022
Questions Are All You Need to Train a Dense Passage Retriever Devendra Singh Sachan M. Lewis Dani Yogatama Luke Zettlemoyer J. Pineau Manzil Zaheer RALM 54 53 0 21 Jun 2022
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation Simiao Zuo Qingru Zhang Chen Liang Pengcheng He T. Zhao Weizhu Chen MoE 111 40 0 15 Apr 2022
Compression of Generative Pre-trained Language Models via Quantization Chaofan Tao Lu Hou Wei Zhang Lifeng Shang Xin Jiang Qun Liu Ping Luo Ngai Wong MQ 60 104 0 21 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 598 9,009 0 28 Jan 2022
Learning to Prompt for Continual Learning Zifeng Wang Zizhao Zhang Chen-Yu Lee Han Zhang Ruoxi Sun Xiaoqi Ren Guolong Su Vincent Perot Jennifer Dy Tomas Pfister CLL VPVLM KELM VLM 67 757 0 16 Dec 2021
ValueNet: A New Dataset for Human Value Driven Dialogue System Liang Qiu Yizhou Zhao Jinchao Li Pan Lu Baolin Peng Jianfeng Gao Song-Chun Zhu 42 35 0 12 Dec 2021
Ethical and social risks of harm from Language Models Laura Weidinger John F. J. Mellor Maribeth Rauh Conor Griffin J. Uesato ... Lisa Anne Hendricks William S. Isaac Sean Legassick G. Irving Iason Gabriel PILM 66 1,009 0 08 Dec 2021
A General Language Assistant as a Laboratory for Alignment Amanda Askell Yuntao Bai Anna Chen Dawn Drain Deep Ganguli ... Tom B. Brown Jack Clark Sam McCandlish C. Olah Jared Kaplan ALM 98 762 0 01 Dec 2021
Recursively Summarizing Books with Human Feedback Jeff Wu Long Ouyang Daniel M. Ziegler Nissan Stiennon Ryan J. Lowe Jan Leike Paul Christiano ALM 91 301 0 22 Sep 2021
Want To Reduce Labeling Cost? GPT-3 Can Help Shuohang Wang Yang Liu Yichong Xu Chenguang Zhu Michael Zeng 47 251 0 30 Aug 2021
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets Irene Solaiman Christy Dennison 63 224 0 18 Jun 2021
Generating Datasets with Pretrained Language Models Timo Schick Hinrich Schütze 126 235 0 15 Apr 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 182 4,209 0 01 Jan 2021
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning Zeyuan Allen-Zhu Yuanzhi Li FedML 108 362 0 17 Dec 2020
DynaBERT: Dynamic BERT with Adaptive Width and Depth Lu Hou Zhiqi Huang Lifeng Shang Xin Jiang Xiao Chen Qun Liu MQ 68 322 0 08 Apr 2020
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices Zhiqing Sun Hongkun Yu Xiaodan Song Renjie Liu Yiming Yang Denny Zhou MQ 93 807 0 06 Apr 2020
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering Shayne Longpre Yi Lu Zhucheng Tu Christopher DuBois 47 70 0 04 Dec 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 157 7,437 0 02 Oct 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 73 1,847 0 23 Sep 2019
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 429 1,664 0 18 Sep 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 428 24,160 0 26 Jul 2019
Experience Replay for Continual Learning David Rolnick Arun Ahuja Jonathan Richard Schwarz Timothy Lillicrap Greg Wayne CLL 92 1,139 0 28 Nov 2018
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 248 18,685 0 20 Jul 2017