v1v2 (latest)

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models

2 July 2024

Zihan Wang

Deli Chen

Damai Dai

Runxin Xu

Zhuoshu Li

Y. Wu

MoE

ALM

ArXiv (abs)PDF HTML Github (616★)

Papers citing "Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models"

21 / 21 papers shown

Title
LoRA Dropout as a Sparsity Regularizer for Overfitting Control Yang Lin Xinyu Ma Xu Chu Yujie Jin Zhibang Yang Yasha Wang Hong-yan Mei 82 26 0 15 Apr 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 251 398 0 21 Mar 2024
WizardCoder: Empowering Code Large Language Models with Evol-Instruct Ziyang Luo Can Xu Pu Zhao Qingfeng Sun Xiubo Geng Wenxiang Hu Chongyang Tao Jing Ma Qingwei Lin Daxin Jiang ELM SyDa ALM 125 690 0 14 Jun 2023
Parameter-Efficient Fine-Tuning without Introducing New Latency Baohao Liao Yan Meng Christof Monz 51 56 0 26 May 2023
Efficient Fine-Tuning of BERT Models on the Edge Danilo Vucetic Mohammadreza Tayaranian M. Ziaeefard J. Clark B. Meyer W. Gross 81 35 0 03 May 2022
Training Neural Networks with Fixed Sparse Masks Yi-Lin Sung Varun Nair Colin Raffel FedML 99 208 0 18 Nov 2021
Composable Sparse Fine-Tuning for Cross-Lingual Transfer Alan Ansell Edoardo Ponti Anna Korhonen Ivan Vulić CLL MoE 134 143 0 14 Oct 2021
Towards a Unified View of Parameter-Efficient Transfer Learning Junxian He Chunting Zhou Xuezhe Ma Taylor Berg-Kirkpatrick Graham Neubig AAML 137 953 0 08 Oct 2021
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning Runxin Xu Fuli Luo Zhiyuan Zhang Chuanqi Tan Baobao Chang Songfang Huang Fei Huang LRM 183 190 0 13 Sep 2021
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 216 2,009 0 16 Aug 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 236 5,665 0 07 Jul 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 502 10,526 0 17 Jun 2021
Hash Layers For Large Sparse Models Stephen Roller Sainbayar Sukhbaatar Arthur Szlam Jason Weston MoE 181 214 0 08 Jun 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 589 4,093 0 18 Apr 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 252 4,305 0 01 Jan 2021
Parameter-Efficient Transfer Learning with Diff Pruning Demi Guo Alexander M. Rush Yoon Kim 84 406 0 14 Dec 2020
ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization Shiyue Zhang B. Frey Joey Tianyi Zhou 73 31 0 09 Oct 2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang M. Krikun Noam M. Shazeer Zhiwen Chen MoE 127 1,191 0 30 Jun 2020
AdapterFusion: Non-Destructive Task Composition for Transfer Learning Jonas Pfeiffer Aishwarya Kamath Andreas Rucklé Kyunghyun Cho Iryna Gurevych CLL MoMe 155 859 0 01 May 2020
CLUE: A Chinese Language Understanding Evaluation Benchmark Liang Xu Hai Hu Xuanwei Zhang Lu Li Chenjie Cao ... Cong Yue Xinrui Zhang Zhen-Yi Yang Kyle Richardson Zhenzhong Lan ELM 95 387 0 13 Apr 2020
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 237 2,692 0 09 May 2017