ConPET: Continual Parameter-Efficient Tuning for Large Language Models

26 September 2023

Xu Han

Zhiyuan Liu

Maosong Sun

Papers citing "ConPET: Continual Parameter-Efficient Tuning for Large Language Models"

22 / 22 papers shown

Title
Progressive Prompts: Continual Learning for Language Models Anastasia Razdaibiedina Yuning Mao Rui Hou Madian Khabsa M. Lewis Amjad Almahairi VLM KELM CLL 92 135 0 29 Jan 2023
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models Ning Ding Yujia Qin Guang Yang Fu Wei Zonghan Yang ... Jianfei Chen Yang Liu Jie Tang Juan Li Maosong Sun 55 200 0 14 Mar 2022
ELLE: Efficient Lifelong Pre-training for Emerging Data Yujia Qin Jiajie Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou 76 72 0 12 Mar 2022
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora Xisen Jin Dejiao Zhang Henghui Zhu Wei Xiao Shang-Wen Li Xiaokai Wei Andrew O. Arnold Xiang Ren KELM CLL 83 116 0 16 Oct 2021
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts Zhengyan Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou MoE 76 123 0 05 Oct 2021
Finetuned Language Models Are Zero-Shot Learners Jason W. Wei Maarten Bosma Vincent Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai Quoc V. Le ALM UQCV 116 3,723 0 03 Sep 2021
DEMix Layers: Disentangling Domains for Modular Language Modeling Suchin Gururangan Michael Lewis Ari Holtzman Noah A. Smith Luke Zettlemoyer KELM MoE 89 134 0 11 Aug 2021
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models Elad Ben-Zaken Shauli Ravfogel Yoav Goldberg 158 1,216 0 18 Jun 2021
Hash Layers For Large Sparse Models Stephen Roller Sainbayar Sukhbaatar Arthur Szlam Jason Weston MoE 158 210 0 08 Jun 2021
Few-NERD: A Few-Shot Named Entity Recognition Dataset Ning Ding Guangwei Xu Yulin Chen Xiaobin Wang Xu Han Pengjun Xie Haitao Zheng Zhiyuan Liu 68 234 0 16 May 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 509 4,021 0 18 Apr 2021
BASE Layers: Simplifying Training of Large, Sparse Models M. Lewis Shruti Bhosale Tim Dettmers Naman Goyal Luke Zettlemoyer MoE 171 277 0 30 Mar 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 213 4,244 0 01 Jan 2021
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang M. Krikun Noam M. Shazeer Zhiwen Chen MoE 86 1,156 0 30 Jun 2020
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 518 24,351 0 26 Jul 2019
Matching the Blanks: Distributional Similarity for Relation Learning Livio Baldini Soares Nicholas FitzGerald Jeffrey Ling Tom Kwiatkowski 56 771 0 07 Jun 2019
Class-incremental Learning via Deep Model Consolidation Junting Zhang Jie Zhang Shalini Ghosh Dawei Li Serafettin Tasci Larry Heck Heming Zhang C.-C. Jay Kuo CLL 55 338 0 19 Mar 2019
Experience Replay for Continual Learning David Rolnick Arun Ahuja Jonathan Richard Schwarz Timothy Lillicrap Greg Wayne CLL 112 1,156 0 28 Nov 2018
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence Arslan Chaudhry P. Dokania Thalaiyasingam Ajanthan Philip Torr CLL 89 1,137 0 30 Jan 2018
Overcoming Catastrophic Forgetting by Incremental Moment Matching Sang-Woo Lee Jin-Hwa Kim Jaehyun Jun Jung-Woo Ha Byoung-Tak Zhang CLL 68 674 0 24 Mar 2017
Overcoming catastrophic forgetting in neural networks J. Kirkpatrick Razvan Pascanu Neil C. Rabinowitz J. Veness Guillaume Desjardins ... A. Grabska-Barwinska Demis Hassabis Claudia Clopath D. Kumaran R. Hadsell CLL 317 7,478 0 02 Dec 2016
Learning without Forgetting Zhizhong Li Derek Hoiem CLL OOD SSL 282 4,391 0 29 Jun 2016