Continual Pre-Training Mitigates Forgetting in Language and Vision

19 May 2022

Tinne Tuytelaars

Papers citing "Continual Pre-Training Mitigates Forgetting in Language and Vision"

20 / 20 papers shown

Title
Param $Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost Sheng Cao Mingrui Wu Karthik Prasad Yuandong Tian Zechun Liu MoMe 80 0 0 23 Apr 2025
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 93 12 0 31 Dec 2024
Gradient Localization Improves Lifelong Pretraining of Language Models Jared Fernandez Yonatan Bisk Emma Strubell KELM 39 1 0 07 Nov 2024
Assessing Open-world Forgetting in Generative Image Model Customization Héctor Laria Alex Gomez-Villa Imad Eddine Marouf Bogdan Raducanu Bogdan Raducanu VLM DiffM 37 0 0 18 Oct 2024
Exploring Gen-AI applications in building research and industry: A review Hanlong Wan Jian Zhang Yan Chen Weili Xu Fan Feng AI4CE 47 0 0 01 Oct 2024
Towards LifeSpan Cognitive Systems Yu Wang Chi Han Tongtong Wu Xiaoxin He Wangchunshu Zhou ... Zexue He Wei Wang Gholamreza Haffari Heng Ji Julian McAuley KELM CLL 144 1 0 20 Sep 2024
Delayed Bottlenecking: Alleviating Forgetting in Pre-trained Graph Neural Networks Zhe Zhao Pengkun Wang Xu Wang Haibin Wen Xiaolong Xie Zhengyang Zhou Qingfu Zhang Yang Wang AI4CE CLL 20 0 0 23 Apr 2024
From Matching to Generation: A Survey on Generative Information Retrieval Xiaoxi Li Jiajie Jin Yujia Zhou Yuyao Zhang Peitian Zhang Yutao Zhu Zhicheng Dou 3DV 81 46 0 23 Apr 2024
Simple and Scalable Strategies to Continually Pre-train Large Language Models Adam Ibrahim Benjamin Thérien Kshitij Gupta Mats L. Richter Quentin Anthony Timothée Lesort Eugene Belilovsky Irina Rish KELM CLL 44 52 0 13 Mar 2024
Investigating Continual Pretraining in Large Language Models: Insights and Implications cCaugatay Yildiz Nishaanth Kanna Ravichandran Prishruit Punia Matthias Bethge B. Ermiş CLL KELM LRM 58 25 0 27 Feb 2024
Continual Learning: Applications and the Road Forward Eli Verwimp Rahaf Aljundi Shai Ben-David Matthias Bethge Andrea Cossu ... J. Weijer Bing Liu Vincenzo Lomonaco Tinne Tuytelaars Gido M. van de Ven CLL 43 44 0 20 Nov 2023
Class Incremental Learning with Pre-trained Vision-Language Models Xialei Liu Xusheng Cao Haori Lu Jia-Wen Xiao Andrew D. Bagdanov Ming-Ming Cheng VLM 17 12 0 31 Oct 2023
A Comprehensive Empirical Evaluation on Online Continual Learning Albin Soutif--Cormerais Antonio Carta Andrea Cossu J. Hurtado Hamed Hemati Vincenzo Lomonaco Joost van de Weijer CLL 31 20 0 20 Aug 2023
Addressing Distribution Shift at Test Time in Pre-trained Language Models Ayush Singh J. Ortega VLM 24 4 0 05 Dec 2022
Fine-tuned Language Models are Continual Learners Thomas Scialom Tuhin Chakrabarty Smaranda Muresan CLL LRM 145 117 0 24 May 2022
Representational Continuity for Unsupervised Continual Learning Divyam Madaan Jaehong Yoon Yuanchun Li Yunxin Liu Sung Ju Hwang CLL SSL 66 111 0 13 Oct 2021
Towards Continual Knowledge Learning of Language Models Joel Jang Seonghyeon Ye Sohee Yang Joongbo Shin Janghoon Han Gyeonghun Kim Stanley Jungkyu Choi Minjoon Seo CLL KELM 230 151 0 07 Oct 2021
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? Dapeng Hu Shipeng Yan Qizhengqiu Lu Lanqing Hong Hailin Hu Yifan Zhang Zhenguo Li Xinchao Wang Jiashi Feng 53 28 0 25 Apr 2021
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 270 3,375 0 09 Mar 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018