Wings: Learning Multimodal LLMs without Text-only Forgetting

5 June 2024

Yang Li

Papers citing "Wings: Learning Multimodal LLMs without Text-only Forgetting"

33 / 33 papers shown

Title
Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs Zeping Yu Sophia Ananiadou MoMe KELM CLL 65 0 0 22 May 2025
Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing Yi-Kai Zhang De-Chuan Zhan Han-Jia Ye ALM ELM LRM 130 3 0 24 Feb 2025
Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens Zhangqi Jiang Junkai Chen Beier Zhu Tingjin Luo Yankun Shen Xu Yang 126 5 0 23 Nov 2024
CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts Jiachen Li Xinyao Wang Sijie Zhu Chia-Wen Kuo Lu Xu Fan Chen Jitesh Jain Humphrey Shi Longyin Wen MLLM MoE 63 30 0 09 May 2024
Yi: Open Foundation Models by 01.AI 01. AI Alex Young 01.AI Alex Young Bei Chen Chao Li ... Yue Wang Yuxuan Cai Zhenyu Gu Zhiyuan Liu Zonghong Dai OSLM LRM 219 538 0 07 Mar 2024
Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models Lei Li Yuqi Wang Runxin Xu Peiyi Wang Xiachong Feng Lingpeng Kong Qi Liu 69 53 0 01 Mar 2024
Large Language Models: A Survey Shervin Minaee Tomas Mikolov Narjes Nikzad M. Asgari-Chenaghlu R. Socher Xavier Amatriain Jianfeng Gao ALM LM&MA ELM 173 389 0 09 Feb 2024
MobileVLM V2: Faster and Stronger Baseline for Vision Language Model Xiangxiang Chu Limeng Qiao Xinyu Zhang Shuang Xu Fei Wei ... Xiaofei Sun Yiming Hu Xinyang Lin Bo Zhang Chunhua Shen VLM MLLM 51 104 0 06 Feb 2024
Mixtral of Experts Albert Q. Jiang Alexandre Sablayrolles Antoine Roux A. Mensch Blanche Savary ... Théophile Gervet Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LLMAG 108 1,049 0 08 Jan 2024
VCoder: Versatile Vision Encoders for Multimodal Large Language Models Jitesh Jain Jianwei Yang Humphrey Shi MLLM 42 28 0 21 Dec 2023
Honeybee: Locality-enhanced Projector for Multimodal LLM Junbum Cha Wooyoung Kang Jonghwan Mun Byungseok Roh MLLM 60 124 0 11 Dec 2023
mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model Anwen Hu Yaya Shi Haiyang Xu Jiabo Ye Qinghao Ye Mingshi Yan Chenliang Li Qi Qian Ji Zhang Fei Huang MLLM 66 25 0 30 Nov 2023
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI Xiang Yue Yuansheng Ni Kai Zhang Tianyu Zheng Ruoqi Liu ... Yibo Liu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen OSLM ELM VLM 155 833 0 27 Nov 2023
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents Shilong Liu Hao Cheng Haotian Liu Hao Zhang Feng Li ... Hang Su Jun Zhu Lei Zhang Jianfeng Gao Chun-yue Li MLLM VLM 76 118 0 09 Nov 2023
Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP Qi Qian Yuanhong Xu Juhua Hu VLM CLIP 56 19 0 30 Oct 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 174 1,756 0 28 Sep 2023
CMMLU: Measuring massive multitask language understanding in Chinese Haonan Li Yixuan Zhang Fajri Koto Yifei Yang Hai Zhao Yeyun Gong Nan Duan Tim Baldwin ALM ELM 72 253 0 15 Jun 2023
Sigmoid Loss for Language Image Pre-Training Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer CLIP VLM 113 1,076 0 27 Mar 2023
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action Zhengyuan Yang Linjie Li Jianfeng Wang Kevin Qinghong Lin E. Azarnasab Faisal Ahmed Zicheng Liu Ce Liu Michael Zeng Lijuan Wang ReLM KELM LRM 69 379 0 20 Mar 2023
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 302 3,458 0 29 Apr 2022
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language Andy Zeng Maria Attarian Brian Ichter K. Choromanski Adrian S. Wong ... Michael S. Ryoo Vikas Sindhwani Johnny Lee Vincent Vanhoucke Peter R. Florence ReLM LRM 127 578 0 01 Apr 2022
WebQA: Multihop and Multimodal QA Yingshan Chang M. Narang Hisami Suzuki Guihong Cao Jianfeng Gao Yonatan Bisk LRM 40 81 0 01 Sep 2021
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 140 1,893 0 16 Aug 2021
Multimodal Few-Shot Learning with Frozen Language Models Maria Tsimpoukelli Jacob Menick Serkan Cabi S. M. Ali Eslami Oriol Vinyals Felix Hill MLLM 137 766 0 25 Jun 2021
GLM: General Language Model Pretraining with Autoregressive Blank Infilling Zhengxiao Du Yujie Qian Xiao Liu Ming Ding J. Qiu Zhilin Yang Jie Tang BDL AI4CE 100 1,520 0 18 Mar 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 184 4,209 0 01 Jan 2021
OCNLI: Original Chinese Natural Language Inference Hai Hu Kyle Richardson Liang Xu Lu Li Sandra Kübler L. Moss 66 118 0 12 Oct 2020
CLUE: A Chinese Language Understanding Evaluation Benchmark Liang Xu Hai Hu Xuanwei Zhang Lu Li Chenjie Cao ... Cong Yue Xinrui Zhang Zhen-Yi Yang Kyle Richardson Zhenzhong Lan ELM 74 381 0 13 Apr 2020
LEEP: A New Measure to Evaluate Transferability of Learned Representations Cuong V Nguyen Tal Hassner Matthias Seeger Cédric Archambeau 69 213 0 27 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 483 4,662 0 23 Jan 2020
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 434 1,664 0 18 Sep 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 467 24,160 0 26 Jul 2019
ChID: A Large-scale Chinese IDiom Dataset for Cloze Test Chujie Zheng Minlie Huang Aixin Sun 53 86 0 04 Jun 2019