ELLE: Efficient Lifelong Pre-training for Emerging Data

12 March 2022

Yujia Qin

Jiajie Zhang

Yankai Lin

Zhiyuan Liu

Peng Li

Maosong Sun

Jie Zhou

ArXiv PDF HTML

Papers citing "ELLE: Efficient Lifelong Pre-training for Emerging Data"

50 / 57 papers shown

Title
Memorization vs. Reasoning: Updating LLMs with New Knowledge Aochong Oliver Li Tanya Goyal KELM 50 0 0 16 Apr 2025
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining Jeffrey Li Mohammadreza Armandpour Iman Mirzadeh Sachin Mehta Vaishaal Shankar ... Samy Bengio Oncel Tuzel Mehrdad Farajtabar Hadi Pouransari Fartash Faghri CLL KELM 61 0 0 02 Apr 2025
Continual Learning Using a Kernel-Based Method Over Foundation Models Saleh Momeni Sahisnu Mazumder Bing-Quan Liu CLL 67 1 0 20 Dec 2024
In-context Continual Learning Assisted by an External Continual Learner Saleh Momeni Sahisnu Mazumder Zixuan Ke Bing Liu CLL 88 0 0 20 Dec 2024
Exploring Forgetting in Large Language Model Pre-Training Chonghua Liao Ruobing Xie X. Sun Haowen Sun Zhanhui Kang CLL 33 0 0 22 Oct 2024
A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Models Zhihao Wang Shiyu Liu Jianheng Huang Zheng Wang Yixuan Liao Xiaoxin Chen Junfeng Yao Jinsong Su 24 1 0 05 Oct 2024
Continual Learning for Temporal-Sensitive Question Answering Wanqi Yang Yunqiu Xu Yanda Li Kunze Wang Binbin Huang Ling-Hao Chen CLL 27 3 0 17 Jul 2024
MUSCLE: A Model Update Strategy for Compatible LLM Evolution Jessica Echterhoff Fartash Faghri Raviteja Vemulapalli Ting-Yao Hu Chun-Liang Li Oncel Tuzel Hadi Pouransari KELM 58 2 0 12 Jul 2024
Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models Jupinder Parmar Sanjev Satheesh M. Patwary M. Shoeybi Bryan Catanzaro 48 28 0 09 Jul 2024
Unlocking Continual Learning Abilities in Language Models Wenyu Du Shuang Cheng Tongxu Luo Zihan Qiu Zeyu Huang Ka Chun Cheung Reynold Cheng Jie Fu KELM CLL 43 6 0 25 Jun 2024
Towards Lifelong Learning of Large Language Models: A Survey Junhao Zheng Shengjie Qiu Chengming Shi Qianli Ma KELM CLL 30 14 0 10 Jun 2024
ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks Santosh T.Y.S.S Tuan-Quang Vuong Matthias Grabmair AILaw CLL 53 4 0 23 May 2024
LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language Cagri Toraman VLM 38 5 0 13 May 2024
Continual Learning of Large Language Models: A Comprehensive Survey Haizhou Shi Zihao Xu Hengyi Wang Weiyi Qin Wenyuan Wang Yibin Wang Zifeng Wang Sayna Ebrahimi Hao Wang CLL KELM LRM 46 63 0 25 Apr 2024
Checkpoint Merging via Bayesian Optimization in LLM Pretraining Deyuan Liu Zecheng Wang Bingning Wang Weipeng Chen Chunshan Li Zhiying Tu Dianhui Chu Bo Li Dianbo Sui MoMe 44 15 0 28 Mar 2024
InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions Yifan Wang Yafei Liu Chufan Shi Haoling Li Chen Chen H. Lu Yujiu Yang CLL 39 25 0 18 Mar 2024
Investigating Continual Pretraining in Large Language Models: Insights and Implications cCaugatay Yildiz Nishaanth Kanna Ravichandran Prishruit Punia Matthias Bethge B. Ermiş CLL KELM LRM 55 25 0 27 Feb 2024
Can Similarity-Based Domain-Ordering Reduce Catastrophic Forgetting for Intent Recognition? Amogh Mannekote Xiaoyi Tian K. Boyer Bonnie J. Dorr CLL 31 1 0 21 Feb 2024
Examining Forgetting in Continual Pre-training of Aligned Large Language Models Chen An Li Hung-Yi Lee CLL KELM 23 8 0 06 Jan 2024
AcademicGPT: Empowering Academic Research Shufa Wei Xiaolong Xu Xianbiao Qi Xi Yin Jun Xia ... Chihao Dai Lihua Wang Xiaohui Liu Lei Zhang Yutao Xie LM&MA 39 3 0 21 Nov 2023
A Self-enhancement Approach for Domain-specific Chatbot Training via Knowledge Mining and Digest Ruohong Zhang Luyu Gao Chen Zheng Zhen Fan Guokun Lai Zheng Zhang Fangzhou Ai Yiming Yang Hongxia Yang 43 2 0 17 Nov 2023
Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models Yujin Kim Jaehong Yoon Seonghyeon Ye Sangmin Bae Namgyu Ho Sung Ju Hwang Se-Young Yun KELM 32 9 0 14 Nov 2023
Continual Learning Under Language Shift Evangelia Gogoulou Timothée Lesort Magnus Boman Joakim Nivre KELM CLL 27 3 0 02 Nov 2023
Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompt Gangwei Jiang Caigao Jiang Siqiao Xue James Y. Zhang Junqing Zhou Defu Lian Ying Wei VLM 32 7 0 19 Oct 2023
LEMON: Lossless model expansion Yite Wang Jiahao Su Hanlin Lu Cong Xie Tianyi Liu Jianbo Yuan Haibin Lin Ruoyu Sun Hongxia Yang 17 12 0 12 Oct 2023
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models Luiza Amador Pozzobon B. Ermiş Patrick Lewis Sara Hooker 28 20 0 11 Oct 2023
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances Zihan Zhang Meng Fang Lingxi Chen Mohammad-Reza Namazi-Rad Jun Wang KELM 19 21 0 11 Oct 2023
ConPET: Continual Parameter-Efficient Tuning for Large Language Models Chenyan Song Xu Han Zheni Zeng Kuai Li Chen Chen Zhiyuan Liu Maosong Sun Taojiannan Yang CLL KELM 19 10 0 26 Sep 2023
Create and Find Flatness: Building Flat Training Spaces in Advance for Continual Learning Wenhang Shi Yiren Chen Zhe Zhao Wei Lu Kimmo Yan Xiaoyong Du CLL 30 5 0 20 Sep 2023
Mitigating the Alignment Tax of RLHF Yong Lin Hangyu Lin Wei Xiong Shizhe Diao Zeming Zheng ... Han Zhao Nan Jiang Heng Ji Yuan Yao Tong Zhang MoMe CLL 29 65 0 12 Sep 2023
FLM-101B: An Open LLM and How to Train It with $100K Budget$ Xiang Li Yiqun Yao Xin Jiang Xuezhi Fang Xuying Meng ... LI DU Bowen Qin Zheng-Wei Zhang Aixin Sun Yequan Wang 60 21 0 07 Sep 2023
Continual Pre-Training of Large Language Models: How to (re)warm your model? Kshitij Gupta Benjamin Thérien Adam Ibrahim Mats L. Richter Quentin G. Anthony Eugene Belilovsky Irina Rish Timothée Lesort KELM 24 99 0 08 Aug 2023
Harnessing Scalable Transactional Stream Processing for Managing Large Language Models [Vision] Shuhao Zhang Xianzhi Zeng Yuhao Wu Zhonghao Yang 21 0 0 17 Jul 2023
Towards Robust and Efficient Continual Language Learning Adam Fisch Amal Rannen-Triki Razvan Pascanu J. Bornschein Angeliki Lazaridou E. Gribovskaya MarcÁurelio Ranzato CLL 24 1 0 11 Jul 2023
SciMON: Scientific Inspiration Machines Optimized for Novelty Qingyun Wang Doug Downey Heng Ji Tom Hope LLMAG 26 61 0 23 May 2023
Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models Shangbin Feng Weijia Shi Yuyang Bai Vidhisha Balachandran Tianxing He Yulia Tsvetkov KELM 45 28 0 17 May 2023
Recyclable Tuning for Continual Pre-training Yujia Qin Cheng Qian Xu Han Yankai Lin Huadong Wang Ruobing Xie Zhiyuan Liu Maosong Sun Jie Zhou CLL 26 11 0 15 May 2023
Masked Structural Growth for 2x Faster Language Model Pre-training Yiqun Yao Zheng-Wei Zhang Jing Li Yequan Wang OffRL AI4CE LRM 40 15 0 04 May 2023
Semiparametric Language Models Are Scalable Continual Learners Guangyue Peng Tao Ge Si-Qing Chen Furu Wei Houfeng Wang KELM 39 10 0 02 Mar 2023
Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study Mingxu Tao Yansong Feng Dongyan Zhao CLL KELM 29 10 0 02 Mar 2023
Continual Pre-training of Language Models Zixuan Ke Yijia Shao Haowei Lin Tatsuya Konishi Gyuhak Kim Bin Liu CLL KELM 22 123 0 07 Feb 2023
A Comprehensive Survey of Continual Learning: Theory, Method and Application Liyuan Wang Xingxing Zhang Hang Su Jun Zhu KELM CLL 36 601 0 31 Jan 2023
PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English Jianfeng Chi Wasi Uddin Ahmad Yuan Tian Kai-Wei Chang AILaw ELM 13 10 0 20 Dec 2022
Continual Learning of Natural Language Processing Tasks: A Survey Zixuan Ke Bin Liu KELM CLL VLM 23 68 0 23 Nov 2022
FPT: Improving Prompt Tuning Efficiency via Progressive Training Yufei Huang Yujia Qin Huadong Wang Yichun Yin Maosong Sun Zhiyuan Liu Qun Liu VLM LRM 27 6 0 13 Nov 2022
A Survey of Knowledge Enhanced Pre-trained Language Models Linmei Hu Zeyi Liu Ziwang Zhao Lei Hou Liqiang Nie Juanzi Li KELM VLM 24 121 0 11 Nov 2022
VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding Dou Hu Xiaolong Hou Xiyang Du Mengyuan Zhou Lian-Xin Jiang Yang Mo Xiaofeng Shi 25 12 0 01 Nov 2022
Exploring Mode Connectivity for Pre-trained Language Models Yujia Qin Cheng Qian Jing Yi Weize Chen Yankai Lin Xu Han Zhiyuan Liu Maosong Sun Jie Zhou 29 20 0 25 Oct 2022
Continual Training of Language Models for Few-Shot Learning Zixuan Ke Haowei Lin Yijia Shao Hu Xu Lei Shu Bin Liu KELM BDL CLL 87 34 0 11 Oct 2022
Prompt-based Conservation Learning for Multi-hop Question Answering Zhenyun Deng Yonghua Zhu Yang Chen Qianqian Qi Michael Witbrock Patricia J. Riddle RALM LRM 32 4 0 14 Sep 2022