Progressive Prompts: Continual Learning for Language Models

29 January 2023

Anastasia Razdaibiedina

Madian Khabsa

Papers citing "Progressive Prompts: Continual Learning for Language Models"

41 / 41 papers shown

Title
GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling Siqi Li Yufan Shen Xiangnan Chen Jiayi Chen Hengwei Ju ... Botian Shi Y. Liu Xinyu Cai Yu Qiao Yu Qiao VLM ELM 124 1 0 30 Apr 2025
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining Jeffrey Li Mohammadreza Armandpour Iman Mirzadeh Sachin Mehta Vaishaal Shankar ... Samy Bengio Oncel Tuzel Mehrdad Farajtabar Hadi Pouransari Fartash Faghri CLL KELM 114 0 0 02 Apr 2025
Achieving Upper Bound Accuracy of Joint Training in Continual Learning Saleh Momeni Bing Liu CLL 131 1 0 17 Feb 2025
Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning Gangwei Jiang Caigao Jiang Zhaoyi Li Siqiao Xue Jun-ping Zhou Linqi Song Defu Lian Yin Wei CLL MU 95 1 0 16 Feb 2025
Empowering Large Language Model for Continual Video Question Answering with Collaborative Prompting Chen Cai Zheng Wang J. Gao Wenyang Liu Ye Lu Runzhong Zhang Kim-Hui Yap CLL 103 1 0 20 Jan 2025
Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models Yuheng Lu Bingshuo Qian Caixia Yuan Huixing Jiang Xiaojie Wang CLL 67 0 0 22 Oct 2024
Self-Data Distillation for Recovering Quality in Pruned Large Language Models Vithursan Thangarasa Ganesh Venkatesh Mike Lasby Nish Sinnadurai Sean Lie SyDa 79 2 0 13 Oct 2024
MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning Yupeng Chen Senmiao Wang Zhihang Lin Zhihang Lin Yushun Zhang Tian Ding Ruoyu Sun Ruoyu Sun CLL 129 3 0 30 Jul 2024
Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neurons Yongqi Leng Deyi Xiong 67 7 0 09 Jul 2024
Few-shot Personalization of LLMs with Mis-aligned Responses Jaehyung Kim Yiming Yang 96 9 0 26 Jun 2024
From Matching to Generation: A Survey on Generative Information Retrieval Xiaoxi Li Jiajie Jin Yujia Zhou Yuyao Zhang Peitian Zhang Yutao Zhu Zhicheng Dou 3DV 135 50 0 23 Apr 2024
Investigating Continual Pretraining in Large Language Models: Insights and Implications cCaugatay Yildiz Nishaanth Kanna Ravichandran Prishruit Punia Matthias Bethge Beyza Ermis CLL KELM LRM 86 26 0 27 Feb 2024
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer Tu Vu Brian Lester Noah Constant Rami Al-Rfou Daniel Cer VLM LRM 183 284 0 15 Oct 2021
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks Xiao Liu Kaixuan Ji Yicheng Fu Weng Lam Tam Zhengxiao Du Zhilin Yang Jie Tang VLM 285 829 0 14 Oct 2021
LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 Chengwei Qin Shafiq Joty CLL 199 101 0 14 Oct 2021
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers Rabeeh Karimi Mahabadi James Henderson Sebastian Ruder MoE 90 481 0 08 Jun 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 480 3,952 0 18 Apr 2021
Continual Learning for Text Classification with Information Disentanglement Based Regularization Yufan Huang Yanzhe Zhang Jiaao Chen Xuezhi Wang Diyi Yang CLL 57 109 0 12 Apr 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 184 4,209 0 01 Jan 2021
Continual Learning in Task-Oriented Dialogue Systems Andrea Madotto Zhaojiang Lin Zhenpeng Zhou Seungwhan Moon Paul A. Crook Bing-Quan Liu Zhou Yu Eunjoon Cho Zhiguang Wang CLL 102 130 0 31 Dec 2020
Efficient Meta Lifelong-Learning with Limited Memory Zirui Wang Sanket Vaibhav Mehta Barnabás Póczós J. Carbonell CLL KELM 62 76 0 06 Oct 2020
Revisiting Few-sample BERT Fine-tuning Tianyi Zhang Felix Wu Arzoo Katiyar Kilian Q. Weinberger Yoav Artzi 125 445 0 10 Jun 2020
AdapterFusion: Non-Destructive Task Composition for Transfer Learning Jonas Pfeiffer Aishwarya Kamath Andreas Rucklé Kyunghyun Cho Iryna Gurevych CLL MoMe 125 837 0 01 May 2020
Pretrained Transformers Improve Out-of-Distribution Robustness Dan Hendrycks Xiaoyuan Liu Eric Wallace Adam Dziedzic R. Krishnan D. Song OOD 137 430 0 13 Apr 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 312 42,038 0 03 Dec 2019
Continual Unsupervised Representation Learning Dushyant Rao Francesco Visin Andrei A. Rusu Yee Whye Teh Razvan Pascanu R. Hadsell BDL CLL SSL DRL 34 259 0 31 Oct 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 335 19,824 0 23 Oct 2019
LAMOL: LAnguage MOdeling for Lifelong Language Learning Fan-Keng Sun Cheng-Hao Ho Hung-yi Lee CLL KELM 71 208 0 07 Sep 2019
Episodic Memory in Lifelong Language Learning Cyprien de Masson dÁutume Sebastian Ruder Lingpeng Kong Dani Yogatama CLL KELM 120 285 0 03 Jun 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems Alex Jinpeng Wang Yada Pruksachatkun Nikita Nangia Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 200 2,296 0 02 May 2019
Parameter-Efficient Transfer Learning for NLP N. Houlsby A. Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin de Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly 204 4,368 0 02 Feb 2019
Efficient Lifelong Learning with A-GEM Arslan Chaudhry MarcÁurelio Ranzato Marcus Rohrbach Mohamed Elhoseiny CLL 185 1,438 0 02 Dec 2018
Experience Replay for Continual Learning David Rolnick Arun Ahuja Jonathan Richard Schwarz Timothy Lillicrap Greg Wayne CLL 108 1,139 0 28 Nov 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.2K 93,936 0 11 Oct 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 744 7,080 0 20 Apr 2018
Gradient Episodic Memory for Continual Learning David Lopez-Paz MarcÁurelio Ranzato VLM CLL 95 2,684 0 26 Jun 2017
Overcoming catastrophic forgetting in neural networks J. Kirkpatrick Razvan Pascanu Neil C. Rabinowitz J. Veness Guillaume Desjardins ... A. Grabska-Barwinska Demis Hassabis Claudia Clopath D. Kumaran R. Hadsell CLL 298 7,410 0 02 Dec 2016
Learning without Forgetting Zhizhong Li Derek Hoiem CLL OOD SSL 269 4,357 0 29 Jun 2016
Progressive Neural Networks Andrei A. Rusu Neil C. Rabinowitz Guillaume Desjardins Hubert Soyer J. Kirkpatrick Koray Kavukcuoglu Razvan Pascanu R. Hadsell CLL AI4CE 72 2,427 0 15 Jun 2016
Character-level Convolutional Networks for Text Classification Xiang Zhang Jiaqi Zhao Yann LeCun 217 6,077 0 04 Sep 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 1.1K 149,474 0 22 Dec 2014