v1v2 (latest)

Self-Supervised Multimodal Learning: A Survey

31 March 2023

Yongshuo Zong

Oisin Mac Aodha

Timothy M. Hospedales

SSL

ArXiv (abs)PDF HTML Github (254★)

Papers citing "Self-Supervised Multimodal Learning: A Survey"

50 / 127 papers shown

Title
AudioCLIP: Extending CLIP to Image, Text and Audio A. Guzhov Federico Raue Jörn Hees Andreas Dengel CLIP VLM 127 370 0 24 Jun 2021
Multi-level Feature Learning for Contrastive Multi-view Clustering Jie Xu Huayi Tang Yazhou Ren Liang Peng Xiao-lan Zhu Lifang He 62 171 0 21 Jun 2021
SelfDoc: Self-Supervised Document Representation Learning Peizhao Li Jiuxiang Gu Jason Kuen Vlad I. Morariu Handong Zhao R. Jain Varun Manjunatha Hongfu Liu ViT SSL 77 162 0 07 Jun 2021
Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model Danfeng Hong Jingliang Hu Jing Yao Jocelyn Chanussot Xiaoxiang Zhu 49 218 0 21 May 2021
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding Hu Xu Gargi Ghosh Po-Yao (Bernie) Huang Prahal Arora Masoumeh Aminzadeh Christoph Feichtenhofer Florian Metze Luke Zettlemoyer 62 133 0 20 May 2021
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos Brian Chen Andrew Rouditchenko Kevin Duarte Hilde Kuehne Samuel Thomas ... Rogerio Feris David Harwath James R. Glass M. Picheny Shih-Fu Chang SSL 60 92 0 26 Apr 2021
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Huayu Chen Boqing Gong ViT 330 589 0 22 Apr 2021
Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models Tejas Srinivasan Yonatan Bisk VLM 69 56 0 18 Apr 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 210 1,024 0 04 Mar 2021
There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge Francisco Rivera Valverde Juana Valeria Hurtado Abhinav Valada 80 73 0 01 Mar 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 420 5,000 0 24 Feb 2021
Self-Supervised Multisensor Change Detection Sudipan Saha Patrick Ebel Xiaoxiang Zhu SSL 57 80 0 12 Feb 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 459 3,901 0 11 Feb 2021
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Wonjae Kim Bokyung Son Ildoo Kim VLM CLIP 134 1,761 0 05 Feb 2021
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers Lisa Anne Hendricks John F. J. Mellor R. Schneider Jean-Baptiste Alayrac Aida Nematzadeh 133 116 0 31 Jan 2021
Adversarial Text-to-Image Synthesis: A Review Stanislav Frolov Tobias Hinz Federico Raue Jörn Hees Andreas Dengel EGVM 64 177 0 25 Jan 2021
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning Wei Li Can Gao Guocheng Niu Xinyan Xiao Hao Liu Jiachen Liu Hua Wu Haifeng Wang 116 380 0 31 Dec 2020
ActBERT: Learning Global-Local Video-Text Representations Linchao Zhu Yi Yang ViT 127 422 0 14 Nov 2020
Learning Representations from Audio-Visual Spatial Alignment Pedro Morgado Yi Li Nuno Vasconcelos SSL 77 123 0 03 Nov 2020
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning Simon Ging Mohammadreza Zolfaghari Hamed Pirsiavash Thomas Brox ViT CLIP 73 172 0 01 Nov 2020
Hard Negative Mixing for Contrastive Learning Yannis Kalantidis Mert Bulent Sariyildiz Noé Pion Philippe Weinzaepfel Diane Larlus SSL 136 646 0 02 Oct 2020
Contrastive Learning of Medical Visual Representations from Paired Images and Text Yuhao Zhang Hang Jiang Yasuhide Miura Christopher D. Manning C. Langlotz MedIm 147 767 0 02 Oct 2020
Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning Ying Cheng Ruize Wang Zhihao Pan Rui Feng Yuejie Zhang SSL 135 108 0 13 Aug 2020
Learning from Noisy Labels with Deep Neural Networks: A Survey Hwanjun Song Minseok Kim Dongmin Park Yooju Shin Jae-Gil Lee NoLa 113 998 0 16 Jul 2020
Debiased Contrastive Learning Ching-Yao Chuang Joshua Robinson Yen-Chen Lin Antonio Torralba Stefanie Jegelka SSL 81 566 0 01 Jul 2020
Self-Supervised MultiModal Versatile Networks Jean-Baptiste Alayrac Adrià Recasens R. Schneider Relja Arandjelović Jason Ramapuram J. Fauw Lucas Smaira Sander Dieleman Andrew Zisserman SSL 146 375 0 29 Jun 2020
Self-supervised Learning: Generative or Contrastive Xiao Liu Fanjin Zhang Zhenyu Hou Zhaoyu Wang Li Mian Jing Zhang Jie Tang SSL 149 1,632 0 15 Jun 2020
Contrastive Multi-View Representation Learning on Graphs Kaveh Hassani Amir Hosein Khas Ahmadi SSL 232 1,305 0 10 Jun 2020
What Makes for Good Views for Contrastive Learning? Yonglong Tian Chen Sun Ben Poole Dilip Krishnan Cordelia Schmid Phillip Isola SSL 116 1,337 0 20 May 2020
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training Linjie Li Yen-Chun Chen Yu Cheng Zhe Gan Licheng Yu Jingjing Liu MLLM VLM OffRL AI4TS 129 504 0 01 May 2020
Audio-Visual Instance Discrimination with Cross-Modal Agreement Pedro Morgado Nuno Vasconcelos Ishan Misra SSL 82 276 0 27 Apr 2020
Music Gesture for Visual Sound Separation Chuang Gan Deng Huang Hang Zhao J. Tenenbaum Antonio Torralba 95 205 0 20 Apr 2020
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks Xiujun Li Xi Yin Chunyuan Li Pengchuan Zhang Xiaowei Hu ... Houdong Hu Li Dong Furu Wei Yejin Choi Jianfeng Gao VLM 140 1,947 0 13 Apr 2020
TextCaps: a Dataset for Image Captioning with Reading Comprehension Oleksii Sidorov Ronghang Hu Marcus Rohrbach Amanpreet Singh 90 418 0 24 Mar 2020
Learning Predictive Representations for Deformable Objects Using Contrastive Estimation Wilson Yan Ashwin Vangipuram Pieter Abbeel Lerrel Pinto 88 190 0 11 Mar 2020
Visual Grounding in Video for Unsupervised Word Translation Gunnar Sigurdsson Jean-Baptiste Alayrac Aida Nematzadeh Lucas Smaira Mateusz Malinowski João Carreira Phil Blunsom Andrew Zisserman VGen 83 50 0 11 Mar 2020
Captioning Images Taken by People Who Are Blind Danna Gurari Yinan Zhao Meng Zhang Nilavra Bhattacharya 77 183 0 20 Feb 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 387 18,897 0 13 Feb 2020
End-to-End Learning of Visual Representations from Uncurated Instructional Videos Antoine Miech Jean-Baptiste Alayrac Lucas Smaira Ivan Laptev Josef Sivic Andrew Zisserman VGen SSL 135 713 0 13 Dec 2019
Self-Supervised Learning by Cross-Modal Audio-Video Clustering Humam Alwassel D. Mahajan Bruno Korbar Lorenzo Torresani Guohao Li Du Tran SSL 113 432 0 28 Nov 2019
Self-labelling via simultaneous clustering and representation learning Yuki M. Asano Christian Rupprecht Andrea Vedaldi SSL 123 777 0 13 Nov 2019
VL-BERT: Pre-training of Generic Visual-Linguistic Representations Weijie Su Xizhou Zhu Yue Cao Bin Li Lewei Lu Furu Wei Jifeng Dai VLM MLLM SSL 177 1,668 0 22 Aug 2019
LXMERT: Learning Cross-Modality Encoder Representations from Transformers Hao Hao Tan Joey Tianyi Zhou VLM MLLM 252 2,493 0 20 Aug 2019
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training Gen Li Nan Duan Yuejian Fang Ming Gong Daxin Jiang Ming Zhou SSL VLM MLLM 211 906 0 16 Aug 2019
VisualBERT: A Simple and Performant Baseline for Vision and Language Liunian Harold Li Mark Yatskar Da Yin Cho-Jui Hsieh Kai-Wei Chang VLM 153 1,965 0 09 Aug 2019
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks Jiasen Lu Dhruv Batra Devi Parikh Stefan Lee SSL VLM 252 3,699 0 06 Aug 2019
Contrastive Multiview Coding Yonglong Tian Dilip Krishnan Phillip Isola SSL 182 2,409 0 13 Jun 2019
ERNIE: Enhanced Language Representation with Informative Entities Zhengyan Zhang Xu Han Zhiyuan Liu Xin Jiang Maosong Sun Qun Liu 113 1,402 0 17 May 2019
Unified Language Model Pre-training for Natural Language Understanding and Generation Li Dong Nan Yang Wenhui Wang Furu Wei Xiaodong Liu Yu Wang Jianfeng Gao M. Zhou H. Hon ELM AI4CE 227 1,560 0 08 May 2019
The Sound of Motions Hang Zhao Chuang Gan Wei-Chiu Ma Antonio Torralba 83 254 0 11 Apr 2019