ELLE: Efficient Lifelong Pre-training for Emerging Data

ELLE: Efficient Lifelong Pre-training for Emerging Data

12 March 2022

Jiajie Zhang

Yankai Lin

Zhiyuan Liu

Peng Li

Maosong Sun

Jie Zhou

Papers citing "ELLE: Efficient Lifelong Pre-training for Emerging Data"

7 / 57 papers shown

Title
Continual Pre-Training Mitigates Forgetting in Language and Vision Andrea Cossu Tinne Tuytelaars Antonio Carta Lucia C. Passaro Vincenzo Lomonaco D. Bacciu KELM VLM CLL 14 68 0 19 May 2022
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models Ning Ding Yujia Qin Guang Yang Fu Wei Zonghan Yang ... Jianfei Chen Yang Liu Jie Tang Juan Li Maosong Sun 32 196 0 14 Mar 2022
Towards Continual Knowledge Learning of Language Models Joel Jang Seonghyeon Ye Sohee Yang Joongbo Shin Janghoon Han Gyeonghun Kim Stanley Jungkyu Choi Minjoon Seo CLL KELM 230 151 0 07 Oct 2021
Knowledge Inheritance for Pre-trained Language Models Yujia Qin Yankai Lin Jing Yi Jiajie Zhang Xu Han ... Yusheng Su Zhiyuan Liu Peng Li Maosong Sun Jie Zhou VLM 19 49 0 28 May 2021
On the Transformer Growth for Progressive BERT Training Xiaotao Gu Liyuan Liu Hongkun Yu Jing Li Cheng Chen Jiawei Han VLM 69 51 0 23 Oct 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 246 4,489 0 23 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019