Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

20 February 2023

Yaowei Wang

Yonghong Tian

Papers citing "Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey"

50 / 96 papers shown

Title
Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond Guanyao Wu Haoyu Liu Hongming Fu Yichuan Peng Jinyuan Liu Xin-Yue Fan Risheng Liu 103 0 0 03 Mar 2025
MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts Peijie Wang Zhong-Zhi Li Fei Yin Xin Yang Dekang Ran Cheng-Lin Liu LRM 90 7 0 28 Feb 2025
Mask-informed Deep Contrastive Incomplete Multi-view Clustering Zhenglai Li Yuqi Shi Xiao He Chang-Fu Tang 92 0 0 04 Feb 2025
Towards Visual Grounding: A Survey Linhui Xiao Xiaoshan Yang X. Lan Yaowei Wang Changsheng Xu ObjD 163 4 0 31 Dec 2024
A Survey on Vision-Language-Action Models for Embodied AI Yueen Ma Zixing Song Yuzheng Zhuang Jianye Hao Irwin King LM&Ro 210 49 0 23 May 2024
Unleashing the Power of CNN and Transformer for Balanced RGB-Event Video Recognition Tianlin Li Yao Rong Shiao Wang Yuan Chen Zhe Wu Bowei Jiang Yonghong Tian Jin Tang ViT 99 3 0 18 Dec 2023
VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix Teng Wang Wenhao Jiang Zhichao Lu Feng Zheng Ran Cheng Chengguo Yin Ping Luo VLM 65 43 0 17 Jun 2022
GLIPv2: Unifying Localization and Vision-Language Understanding Haotian Zhang Pengchuan Zhang Xiaowei Hu Yen-Chun Chen Liunian Harold Li Xiyang Dai Lijuan Wang Lu Yuan Lei Li Jianfeng Gao ObjD VLM 83 299 0 12 Jun 2022
A Unified Continuous Learning Framework for Multi-modal Knowledge Discovery and Pre-training Zhihao Fan Zhongyu Wei Jingjing Chen Siyuan Wang Zejun Li Jiarong Xu Xuanjing Huang CLL 35 6 0 11 Jun 2022
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts Basil Mustafa C. Riquelme J. Puigcerver Rodolphe Jenatton N. Houlsby VLM MoE 128 192 0 06 Jun 2022
HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrieval Feilong Chen Xiuyi Chen Jiaxin Shi Duzhen Zhang Jianlong Chang Qi Tian VLM CLIP 41 6 0 24 May 2022
Prompt Tuning for Discriminative Pre-trained Language Models Yuan Yao Bowen Dong Ao Zhang Zhengyan Zhang Ruobing Xie Zhiyuan Liu Leyu Lin Maosong Sun Jianyong Wang VLM 60 34 0 23 May 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 302 3,458 0 29 Apr 2022
PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining Yuting Gao Jinfeng Liu Zihan Xu Jinchao Zhang Ke Li Rongrong Ji Chunhua Shen VLM CLIP 81 102 0 29 Apr 2022
COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval Haoyu Lu Nanyi Fei Yuqi Huo Yizhao Gao Zhiwu Lu Jiaxin Wen CLIP VLM 56 55 0 15 Apr 2022
A Roadmap for Big Model Sha Yuan Hanyu Zhao Shuai Zhao Jiahong Leng Yangxiao Liang ... Yang Liu Xiaodong He Minlie Huang Xiangbo Shu Jie Tang ALM AI4CE 31 16 0 26 Mar 2022
Visual Prompt Tuning Menglin Jia Luming Tang Bor-Chun Chen Claire Cardie Serge Belongie Bharath Hariharan Ser-Nam Lim VLM VPVLM 126 1,576 0 23 Mar 2022
MVP: Multimodality-guided Visual Pre-training Longhui Wei Lingxi Xie Wen-gang Zhou Houqiang Li Qi Tian 47 107 0 10 Mar 2022
Vision-Language Intelligence: Tasks, Representation Learning, and Large Models Feng Li Hao Zhang Yi-Fan Zhang Shixuan Liu Jian Guo L. Ni Pengchuan Zhang Lei Zhang AI4TS VLM 48 37 0 03 Mar 2022
Vision-Language Pre-Training with Triple Contrastive Learning Jinyu Yang Jiali Duan Son N. Tran Yi Xu Sampath Chanda Liqun Chen Belinda Zeng Trishul Chilimbi Junzhou Huang VLM 94 292 0 21 Feb 2022
A Survey of Vision-Language Pre-Trained Models Yifan Du Zikang Liu Junyi Li Wayne Xin Zhao VLM 87 182 0 18 Feb 2022
A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models Da Yin Li Dong Hao Cheng Xiaodong Liu Kai-Wei Chang Furu Wei Jianfeng Gao KELM 46 34 0 17 Feb 2022
Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey Prajjwal Bhargava Vincent Ng ReLM LRM 107 63 0 28 Jan 2022
End-to-end Generative Pretraining for Multimodal Video Captioning Paul Hongsuck Seo Arsha Nagrani Anurag Arnab Cordelia Schmid 59 167 0 20 Jan 2022
A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models Hanqing Zhang Haolin Song Shaoyu Li Ming Zhou Dawei Song 71 219 0 14 Jan 2022
CLIP-Event: Connecting Text and Images with Event Structures Manling Li Ruochen Xu Shuohang Wang Luowei Zhou Xudong Lin Chenguang Zhu Michael Zeng Heng Ji Shih-Fu Chang VLM CLIP 59 126 0 13 Jan 2022
SLIP: Self-supervision meets Language-Image Pre-training Norman Mu Alexander Kirillov David Wagner Saining Xie VLM CLIP 119 485 0 23 Dec 2021
RegionCLIP: Region-based Language-Image Pretraining Yiwu Zhong Jianwei Yang Pengchuan Zhang Chunyuan Li Noel Codella ... Luowei Zhou Xiyang Dai Lu Yuan Yin Li Jianfeng Gao VLM CLIP 116 571 0 16 Dec 2021
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu Baining Guo ViT 86 241 0 24 Nov 2021
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion Chenfei Wu Jian Liang Lei Ji Fan Yang Yuejian Fang Daxin Jiang Nan Duan ViT VGen 59 294 0 24 Nov 2021
RedCaps: web-curated image-text data created by the people, for the people Karan Desai Gaurav Kaul Zubin Aysola Justin Johnson 87 166 0 22 Nov 2021
Attention Mechanisms in Computer Vision: A Survey Meng-Hao Guo Tianhan Xu Jiangjiang Liu Zheng-Ning Liu Peng-Tao Jiang Tai-Jiang Mu Song-Hai Zhang Ralph Robert Martin Ming-Ming Cheng Shimin Hu 98 1,668 0 15 Nov 2021
FILIP: Fine-grained Interactive Language-Image Pre-Training Lewei Yao Runhu Huang Lu Hou Guansong Lu Minzhe Niu Hang Xu Xiaodan Liang Zhenguo Li Xin Jiang Chunjing Xu VLM CLIP 80 627 0 09 Nov 2021
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts Hangbo Bao Wenhui Wang Li Dong Qiang Liu Owais Khan Mohammed Kriti Aggarwal Subhojit Som Furu Wei VLM MLLM MoE 63 543 0 03 Nov 2021
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs Christoph Schuhmann Richard Vencu Romain Beaumont R. Kaczmarczyk Clayton Mullis Aarush Katta Theo Coombes J. Jitsev Aran Komatsuzaki VLM MLLM CLIP 187 1,398 0 03 Nov 2021
Pre-trained Language Models in Biomedical Domain: A Systematic Survey Benyou Wang Qianqian Xie Jiahuan Pei Zhihong Chen Prayag Tiwari Zhao Li Jie Fu LM&MA AI4CE 56 166 0 11 Oct 2021
Hybrid Dynamic Contrast and Probability Distillation for Unsupervised Person Re-Id De Cheng Jingyu Zhou N. Wang Xinbo Gao 51 60 0 29 Sep 2021
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models Yuan Yao Ao Zhang Zhengyan Zhang Zhiyuan Liu Tat-Seng Chua Maosong Sun MLLM VPVLM VLM 273 222 0 24 Sep 2021
Learning to Prompt for Vision-Language Models Kaiyang Zhou Jingkang Yang Chen Change Loy Ziwei Liu VPVLM CLIP VLM 457 2,340 0 02 Sep 2021
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision Zirui Wang Jiahui Yu Adams Wei Yu Zihang Dai Yulia Tsvetkov Yuan Cao VLM MLLM 108 789 0 24 Aug 2021
ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration Yuhao Cui Zhou Yu Chunqi Wang Zhongzhou Zhao Ji Zhang Meng Wang Jun-chen Yu VLM 50 54 0 16 Aug 2021
W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training Yu-An Chung Yu Zhang Wei Han Chung-Cheng Chiu James Qin Ruoming Pang Yonghui Wu SSL VLM 39 421 0 07 Aug 2021
How Much Can CLIP Benefit Vision-and-Language Tasks? Sheng Shen Liunian Harold Li Hao Tan Joey Tianyi Zhou Anna Rohrbach Kai-Wei Chang Z. Yao Kurt Keutzer CLIP VLM MLLM 248 407 0 13 Jul 2021
Focal Self-attention for Local-Global Interactions in Vision Transformers Jianwei Yang Chunyuan Li Pengchuan Zhang Xiyang Dai Bin Xiao Lu Yuan Jianfeng Gao ViT 65 431 0 01 Jul 2021
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training Hongwei Xue Yupan Huang Bei Liu Houwen Peng Jianlong Fu Houqiang Li Jiebo Luo 59 89 0 25 Jun 2021
AudioCLIP: Extending CLIP to Image, Text and Audio A. Guzhov Federico Raue Jörn Hees Andreas Dengel CLIP VLM 78 362 0 24 Jun 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 195 2,790 0 15 Jun 2021
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units Wei-Ning Hsu Benjamin Bolte Yao-Hung Hubert Tsai Kushal Lakhotia Ruslan Salakhutdinov Abdel-rahman Mohamed SSL 140 2,879 0 14 Jun 2021
Pre-Trained Models: Past, Present and Future Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu ... Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao Jun Zhu AIFin MQ AI4MH 123 836 0 14 Jun 2021
MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding Aishwarya Kamath Mannat Singh Yann LeCun Gabriel Synnaeve Ishan Misra Nicolas Carion ObjD VLM 161 876 0 26 Apr 2021