v1v2 (latest)

Self-Supervised Multimodal Learning: A Survey

31 March 2023

Yongshuo Zong

Oisin Mac Aodha

Timothy M. Hospedales

SSL

ArXiv (abs)PDF HTML Github (254★)

Papers citing "Self-Supervised Multimodal Learning: A Survey"

50 / 126 papers shown

Title
Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection SungHeon Jeong Jihong Park Mohsen Imani 172 0 0 05 May 2025
What to align in multimodal contrastive learning? Benoit Dufumier J. Castillo-Navarro D. Tuia Jean-Philippe Thiran 127 4 0 11 Sep 2024
Towards Democratizing Joint-Embedding Self-Supervised Learning Florian Bordes Randall Balestriero Pascal Vincent 75 21 0 03 Mar 2023
Grounding Language Models to Images for Multimodal Inputs and Outputs Jing Yu Koh Ruslan Salakhutdinov Daniel Fried MLLM 100 122 0 31 Jan 2023
MAViL: Masked Audio-Video Learners Po-Yao (Bernie) Huang Vasu Sharma Hu Xu Chaitanya K. Ryali Haoqi Fan Yanghao Li Shang-Wen Li Gargi Ghosh Jitendra Malik Christoph Feichtenhofer 75 54 0 15 Dec 2022
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends Zhe Gan Linjie Li Chunyuan Li Lijuan Wang Zicheng Liu Jianfeng Gao VLM 60 167 0 17 Oct 2022
LAION-5B: An open large-scale dataset for training next generation image-text models Christoph Schuhmann Romain Beaumont Richard Vencu Cade Gordon Ross Wightman ... Srivatsa Kundurthy Katherine Crowson Ludwig Schmidt R. Kaczmarczyk J. Jitsev VLM MLLM CLIP 200 3,500 0 16 Oct 2022
When and why vision-language models behave like bags-of-words, and what to do about it? Mert Yuksekgonul Federico Bianchi Pratyusha Kalluri Dan Jurafsky James Zou VLM CoGe 94 393 0 04 Oct 2022
Contrastive Audio-Visual Masked Autoencoder Yuan Gong Andrew Rouditchenko Alexander H. Liu David Harwath Leonid Karlinsky Hilde Kuehne James R. Glass 89 128 0 02 Oct 2022
Unsupervised Representation Learning in Deep Reinforcement Learning: A Review N. Botteghi M. Poel C. Brune SSL OffRL 88 13 0 27 Aug 2022
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks Wenhui Wang Hangbo Bao Li Dong Johan Bjorck Zhiliang Peng ... Kriti Aggarwal O. Mohammed Saksham Singhal Subhojit Som Furu Wei MLLM VLM ViT 148 644 0 22 Aug 2022
UAVM: Towards Unifying Audio and Visual Models Yuan Gong Alexander H. Liu Andrew Rouditchenko James R. Glass 68 23 0 29 Jul 2022
u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality Wei-Ning Hsu Bowen Shi SSL VLM 84 43 0 14 Jul 2022
Language Modelling with Pixels Phillip Rust Jonas F. Lotz Emanuele Bugliarello Elizabeth Salesky Miryam de Lhoneux Desmond Elliott VLM 97 46 0 14 Jul 2022
Audio-Visual Segmentation Jinxing Zhou Jianyuan Wang Jing Zhang Weixuan Sun Jing Zhang Stan Birchfield Dan Guo Lingpeng Kong Meng Wang Yiran Zhong VOS 87 116 0 11 Jul 2022
Robustness Analysis of Video-Language Models Against Visual and Language Perturbations Madeline Chantry Schiappa Shruti Vyas Hamid Palangi Yogesh S Rawat Vibhav Vineet VLM 149 20 0 05 Jul 2022
Beyond neural scaling laws: beating power law scaling via data pruning Ben Sorscher Robert Geirhos Shashank Shekhar Surya Ganguli Ari S. Morcos 97 444 0 29 Jun 2022
Self-supervised Learning in Remote Sensing: A Review Yi Wang C. Albrecht Nassim Ait Ali Braham Lichao Mou Xiao Xiang Zhu 111 225 0 27 Jun 2022
VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix Teng Wang Wenhao Jiang Zhichao Lu Feng Zheng Ran Cheng Chengguo Yin Ping Luo VLM 72 44 0 17 Jun 2022
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 288 2,518 0 15 Jun 2022
Multimodal Learning with Transformers: A Survey Peng Xu Xiatian Zhu David Clifton ViT 197 568 0 13 Jun 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 418 3,607 0 29 Apr 2022
Sound Localization by Self-Supervised Time Delay Estimation Ziyang Chen David Fouhey Andrew Owens SSL 58 19 0 26 Apr 2022
Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data Corentin Sautier Gilles Puy Spyros Gidaris Alexandre Boulch Andrei Bursuc Renaud Marlet 3DPC 106 118 0 30 Mar 2022
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding Mohamed Afham Isuru Dissanayake Dinithi Dissanayake Amaya Dharmasiri Kanchana Thilakarathna Ranga Rodrigo 3DPC 89 258 0 01 Mar 2022
Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignment Mingyang Zhou Licheng Yu Amanpreet Singh Mengjiao MJ Wang Zhou Yu Ning Zhang VLM 67 31 0 01 Mar 2022
HiP: Hierarchical Perceiver João Carreira Skanda Koppula Daniel Zoran Adrià Recasens Catalin Ionescu ... M. Botvinick Oriol Vinyals Karen Simonyan Andrew Zisserman Andrew Jaegle VLM 98 14 0 22 Feb 2022
Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement L. A. Passos João Paulo Papa Javier Del Ser Amir Hussain Ahsan Adeel 106 35 0 09 Feb 2022
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language Alexei Baevski Wei-Ning Hsu Qiantong Xu Arun Babu Jiatao Gu Michael Auli SSL VLM ViT 99 859 0 07 Feb 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong Guosheng Lin MLLM BDL VLM CLIP 555 4,413 0 28 Jan 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 847 9,683 0 28 Jan 2022
End-to-end Generative Pretraining for Multimodal Video Captioning Paul Hongsuck Seo Arsha Nagrani Anurag Arnab Cordelia Schmid 73 169 0 20 Jan 2022
MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound Rowan Zellers Jiasen Lu Ximing Lu Youngjae Yu Yanpeng Zhao Mohammadreza Salehi Aditya Kusupati Jack Hessel Ali Farhadi Yejin Choi 99 214 0 07 Jan 2022
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction Bowen Shi Wei-Ning Hsu Kushal Lakhotia Abdel-rahman Mohamed SSL 100 321 0 05 Jan 2022
SLIP: Self-supervision meets Language-Image Pre-training Norman Mu Alexander Kirillov David Wagner Saining Xie VLM CLIP 148 490 0 23 Dec 2021
Fine-grained Multi-Modal Self-Supervised Learning Duo Wang S. Karout SSL 59 7 0 22 Dec 2021
Class-aware Sounding Objects Localization via Audiovisual Correspondence Di Hu Yake Wei Rui Qian Weiyao Lin Ruihua Song Ji-Rong Wen 59 41 0 22 Dec 2021
MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning C. Eichenberg Sid Black Samuel Weinbach Letitia Parcalabescu Anette Frank MLLM VLM 58 100 0 09 Dec 2021
FLAVA: A Foundational Language And Vision Alignment Model Amanpreet Singh Ronghang Hu Vedanuj Goswami Guillaume Couairon Wojciech Galuba Marcus Rohrbach Douwe Kiela CLIP VLM 106 719 0 08 Dec 2021
Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval Nina Shvetsova Brian Chen Andrew Rouditchenko Samuel Thomas Brian Kingsbury Rogerio Feris David Harwath James R. Glass Hilde Kuehne ViT 99 132 0 08 Dec 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 477 7,827 0 11 Nov 2021
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts Hangbo Bao Wenhui Wang Li Dong Qiang Liu Owais Khan Mohammed Kriti Aggarwal Subhojit Som Furu Wei VLM MLLM MoE 89 559 0 03 Nov 2021
Self-Supervised Representation Learning: Introduction, Advances and Challenges Linus Ericsson Henry Gouk Chen Change Loy Timothy M. Hospedales SSL OOD AI4TS 79 278 0 18 Oct 2021
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm Yangguang Li Feng Liang Lichen Zhao Yufeng Cui Wanli Ouyang Jing Shao F. Yu Junjie Yan VLM CLIP 154 458 0 11 Oct 2021
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding Hu Xu Gargi Ghosh Po-Yao (Bernie) Huang Dmytro Okhonko Armen Aghajanyan Florian Metze Luke Zettlemoyer Florian Metze Luke Zettlemoyer Christoph Feichtenhofer CLIP VLM 313 582 0 28 Sep 2021
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision Zirui Wang Jiahui Yu Adams Wei Yu Zihang Dai Yulia Tsvetkov Yuan Cao VLM MLLM 136 799 0 24 Aug 2021
Semi-supervised learning for joint SAR and multispectral land cover classification Antonio Montanaro D. Valsesia Giulia Fracastoro E. Magli SSL 51 11 0 20 Aug 2021
Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications Sandhini Agarwal Gretchen Krueger Jack Clark Alec Radford Jong Wook Kim Miles Brundage 63 143 0 05 Aug 2021
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Junnan Li Ramprasaath R. Selvaraju Akhilesh Deepak Gotmare Shafiq Joty Caiming Xiong Guosheng Lin FaML 221 1,975 0 16 Jul 2021
Multimodal Few-Shot Learning with Frozen Language Models Maria Tsimpoukelli Jacob Menick Serkan Cabi S. M. Ali Eslami Oriol Vinyals Felix Hill MLLM 183 789 0 25 Jun 2021