Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

16 October 2021

Xiang Ren

Papers citing "Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora"

28 / 28 papers shown

Title
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining Jeffrey Li Mohammadreza Armandpour Iman Mirzadeh Sachin Mehta Vaishaal Shankar ... Samy Bengio Oncel Tuzel Mehrdad Farajtabar Hadi Pouransari Fartash Faghri CLL KELM 61 0 0 02 Apr 2025
Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training Jaydeep Borkar Matthew Jagielski Katherine Lee Niloofar Mireshghallah David A. Smith Christopher A. Choquette-Choo PILM 83 1 0 24 Feb 2025
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 90 12 0 31 Dec 2024
Gradient Localization Improves Lifelong Pretraining of Language Models Jared Fernandez Yonatan Bisk Emma Strubell KELM 36 1 0 07 Nov 2024
An Investigation of Warning Erroneous Chat Translations in Cross-lingual Communication Yunmeng Li Jun Suzuki Makoto Morishita Kaori Abe Kentaro Inui 65 1 0 28 Aug 2024
Language Modeling with Editable External Knowledge Belinda Z. Li Emmy Liu Alexis Ross Abbas Zeitoun Graham Neubig Jacob Andreas KELM 32 4 0 17 Jun 2024
Investigating Continual Pretraining in Large Language Models: Insights and Implications cCaugatay Yildiz Nishaanth Kanna Ravichandran Prishruit Punia Matthias Bethge B. Ermiş CLL KELM LRM 55 25 0 27 Feb 2024
Online Continual Knowledge Learning for Language Models Yuhao Wu Tongjun Shi Karthick Sharma Chun Seah Shuhao Zhang CLL KELM 28 4 0 16 Nov 2023
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models Luiza Amador Pozzobon B. Ermiş Patrick Lewis Sara Hooker 30 20 0 11 Oct 2023
Continual Pre-Training of Large Language Models: How to (re)warm your model? Kshitij Gupta Benjamin Thérien Adam Ibrahim Mats L. Richter Quentin G. Anthony Eugene Belilovsky Irina Rish Timothée Lesort KELM 24 99 0 08 Aug 2023
PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models Leonardo Ranaldi Elena Sofia Ruzzetti Fabio Massimo Zanzotto 31 6 0 08 May 2023
An Overview on Language Models: Recent Developments and Outlook Chengwei Wei Yun Cheng Wang Bin Wang C.-C. Jay Kuo 25 42 0 10 Mar 2023
Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study Mingxu Tao Yansong Feng Dongyan Zhao CLL KELM 29 10 0 02 Mar 2023
Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views Katerina Margatina Shuai Wang Yogarshi Vyas Neha Ann John Yassine Benajiba Miguel Ballesteros 17 15 0 23 Feb 2023
Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Tasks Sudipta Kar Giuseppe Castellucci Simone Filice S. Malmasi Oleg Rokhlenko CLL KELM 46 6 0 22 Feb 2023
Addressing Distribution Shift at Test Time in Pre-trained Language Models Ayush Singh J. Ortega VLM 24 4 0 05 Dec 2022
He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues Amanda Bertsch Graham Neubig Matthew R. Gormley 45 5 0 27 Oct 2022
TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media Daniel Loureiro Aminette D'Souza Areej Muhajab Isabella A. White Gabriel Wong Luis Espinosa Anke Leonardo Neves Francesco Barbieri Jose Camacho-Collados 29 25 0 15 Sep 2022
Fine-tuned Language Models are Continual Learners Thomas Scialom Tuhin Chakrabarty Smaranda Muresan CLL LRM 145 117 0 24 May 2022
Cross-lingual Lifelong Learning Meryem M'hamdi Xiang Ren Jonathan May CLL 37 8 0 23 May 2022
Can Foundation Models Wrangle Your Data? A. Narayan Ines Chami Laurel J. Orr Simran Arora Christopher Ré LMTD AI4CE 181 214 0 20 May 2022
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models Joel Jang Seonghyeon Ye Changho Lee Sohee Yang Joongbo Shin Janghoon Han Gyeonghun Kim Minjoon Seo CLL KELM 27 91 0 29 Apr 2022
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models Ning Ding Yujia Qin Guang Yang Fu Wei Zonghan Yang ... Jianfei Chen Yang Liu Jie Tang Juan Li Maosong Sun 32 196 0 14 Mar 2022
ELLE: Efficient Lifelong Pre-training for Emerging Data Yujia Qin Jiajie Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou 24 67 0 12 Mar 2022
TimeLMs: Diachronic Language Models from Twitter Daniel Loureiro Francesco Barbieri Leonardo Neves Luis Espinosa Anke Jose Camacho-Collados 33 247 0 08 Feb 2022
Towards Interactive Language Modeling Maartje ter Hoeve Evgeny Kharitonov Dieuwke Hupkes Emmanuel Dupoux 26 4 0 14 Dec 2021
Towards Continual Knowledge Learning of Language Models Joel Jang Seonghyeon Ye Sohee Yang Joongbo Shin Janghoon Han Gyeonghun Kim Stanley Jungkyu Choi Minjoon Seo CLL KELM 230 150 0 07 Oct 2021
SEED: Self-supervised Distillation For Visual Representation Zhiyuan Fang Jianfeng Wang Lijuan Wang Lei Zhang Yezhou Yang Zicheng Liu SSL 239 190 0 12 Jan 2021