Preserving Modality Structure Improves Multi-Modal Learning

24 August 2023

ArXiv (abs)PDF HTML Github (5★)

Papers citing "Preserving Modality Structure Improves Multi-Modal Learning"

32 / 32 papers shown

Title
Gramian Multimodal Representation Learning and Alignment Giordano Cicchetti Eleonora Grassucci Luigi Sigillo Danilo Comminiello 175 3 0 16 Dec 2024
Towards Realistic Semi-Supervised Learning Mamshad Nayeem Rizve Navid Kardan M. Shah 84 46 0 05 Jul 2022
Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP) Alex Fang Gabriel Ilharco Mitchell Wortsman Yu Wan Vaishaal Shankar Achal Dave Ludwig Schmidt VLM OOD 102 146 0 03 May 2022
FLAVA: A Foundational Language And Vision Alignment Model Amanpreet Singh Ronghang Hu Vedanuj Goswami Guillaume Couairon Wojciech Galuba Marcus Rohrbach Douwe Kiela CLIP VLM 104 715 0 08 Dec 2021
Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval Nina Shvetsova Brian Chen Andrew Rouditchenko Samuel Thomas Brian Kingsbury Rogerio Feris David Harwath James R. Glass Hilde Kuehne ViT 99 132 0 08 Dec 2021
A Unified Objective for Novel Class Discovery Enrico Fini E. Sangineto Stéphane Lathuilière Zhun Zhong Moin Nabi Elisa Ricci 75 179 0 19 Aug 2021
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos Brian Chen Andrew Rouditchenko Kevin Duarte Hilde Kuehne Samuel Thomas ... Rogerio Feris David Harwath James R. Glass M. Picheny Shih-Fu Chang SSL 60 92 0 26 Apr 2021
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval Max Bain Arsha Nagrani Gül Varol Andrew Zisserman VGen 163 1,186 0 01 Apr 2021
Open World Compositional Zero-Shot Learning Massimiliano Mancini Muhammad Ferjad Naeem Yongqin Xian Zeynep Akata CoGe 138 130 0 29 Jan 2021
ActBERT: Learning Global-Local Video-Text Representations Linchao Zhu Yi Yang ViT 127 422 0 14 Nov 2020
Self-Supervised MultiModal Versatile Networks Jean-Baptiste Alayrac Adrià Recasens R. Schneider Relja Arandjelović Jason Ramapuram J. Fauw Lucas Smaira Sander Dieleman Andrew Zisserman SSL 146 375 0 29 Jun 2020
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin OCL SSL 264 4,098 0 17 Jun 2020
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos Andrew Rouditchenko Angie Boggust David Harwath Brian Chen D. Joshi ... Rogerio Feris Brian Kingsbury M. Picheny Antonio Torralba James R. Glass SSL 68 142 0 16 Jun 2020
VGGSound: A Large-scale Audio-Visual Dataset Honglie Chen Weidi Xie Andrea Vedaldi Andrew Zisserman 89 583 0 29 Apr 2020
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning Elad Amrani Rami Ben-Ari Daniel Rotman A. Bronstein 91 124 0 06 Mar 2020
Audiovisual SlowFast Networks for Video Recognition Fanyi Xiao Yong Jae Lee Kristen Grauman Jitendra Malik Christoph Feichtenhofer 257 209 0 23 Jan 2020
End-to-End Learning of Visual Representations from Uncurated Instructional Videos Antoine Miech Jean-Baptiste Alayrac Lucas Smaira Ivan Laptev Josef Sivic Andrew Zisserman VGen SSL 135 713 0 13 Dec 2019
Self-Supervised Learning by Cross-Modal Audio-Video Clustering Humam Alwassel D. Mahajan Bruno Korbar Lorenzo Torresani Guohao Li Du Tran SSL 111 432 0 28 Nov 2019
SuperGlue: Learning Feature Matching with Graph Neural Networks Paul-Edouard Sarlin Daniel DeTone Tomasz Malisiewicz Andrew Rabinovich 3DPC OffRL 138 1,950 0 26 Nov 2019
Self-labelling via simultaneous clustering and representation learning Yuki M. Asano Christian Rupprecht Andrea Vedaldi SSL 123 777 0 13 Nov 2019
Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck Shuang Ma Daniel J. McDuff Yale Song 60 25 0 19 Aug 2019
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips Antoine Miech Dimitri Zhukov Jean-Baptiste Alayrac Makarand Tapaswi Ivan Laptev Josef Sivic VGen 122 1,207 0 07 Jun 2019
A Joint Sequence Fusion Model for Video Question Answering and Retrieval Youngjae Yu Jongseok Kim Gunhee Kim 96 345 0 07 Aug 2018
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? Kensho Hara Hirokatsu Kataoka Y. Satoh 3DPC 133 1,935 0 27 Nov 2017
Learnable pooling with Context Gating for video classification Antoine Miech Ivan Laptev Josef Sivic 74 327 0 21 Jun 2017
One Model To Learn Them All Lukasz Kaiser Aidan Gomez Noam M. Shazeer Ashish Vaswani Niki Parmar Llion Jones Jakob Uszkoreit VLM ViT 80 334 0 16 Jun 2017
Look, Listen and Learn Relja Arandjelović Andrew Zisserman SSL 127 906 0 23 May 2017
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset João Carreira Andrew Zisserman 240 8,041 0 22 May 2017
Towards Automatic Learning of Procedures from Web Instructional Videos Luowei Zhou Chenliang Xu Jason J. Corso EgoV 75 830 0 28 Mar 2017
Generative Adversarial Text to Image Synthesis Scott E. Reed Zeynep Akata Xinchen Yan Lajanugen Logeswaran Bernt Schiele Honglak Lee GAN 207 3,149 0 17 May 2016
Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances Marco Cuturi OT 220 4,289 0 04 Jun 2013
Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen G. Corrado J. Dean 3DV 687 31,553 0 16 Jan 2013