Understanding the Emergence of Multimodal Representation Alignment

22 February 2025

Papers citing "Understanding the Emergence of Multimodal Representation Alignment"

41 / 41 papers shown

Title
Towards Ambiguity-Free Spatial Foundation Model: Rethinking and Decoupling Depth Ambiguity Xiaohao Xu Feng Xue Xianrui Li Haowei Li Steve Yang Tianze Zhang Matthew Johnson-Roberson Xiaonan Huang 3DV 63 0 0 08 Mar 2025
What to align in multimodal contrastive learning? Benoit Dufumier J. Castillo-Navarro D. Tuia Jean-Philippe Thiran 109 4 0 11 Sep 2024
The Platonic Representation Hypothesis Minyoung Huh Brian Cheung Tongzhou Wang Phillip Isola 110 134 0 13 May 2024
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 296 11,858 0 18 Jul 2023
Factorized Contrastive Learning: Going Beyond Multi-view Redundancy Paul Pu Liang Zihao Deng Martin Q. Ma James Zou Louis-Philippe Morency Ruslan Salakhutdinov SSL 58 53 0 08 Jun 2023
Similarity of Neural Network Models: A Survey of Functional and Representational Measures Max Klabunde Tobias Schumacher M. Strohmaier Florian Lemmerich 133 73 0 10 May 2023
DINOv2: Learning Robust Visual Features without Supervision Maxime Oquab Timothée Darcet Théo Moutakanni Huy Q. Vo Marc Szafraniec ... Hervé Jégou Julien Mairal Patrick Labatut Armand Joulin Piotr Bojanowski VLM CLIP SSL 325 3,410 0 14 Apr 2023
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework Paul Pu Liang Yun Cheng Xiang Fan Chun Kai Ling Suzanne Nie ... Nicholas B. Allen Randy P. Auerbach Faisal Mahmood Ruslan Salakhutdinov Louis-Philippe Morency 64 34 0 23 Feb 2023
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 386 2,385 0 09 Nov 2022
Make-A-Video: Text-to-Video Generation without Text-Video Data Uriel Singer Adam Polyak Thomas Hayes Xiaoyue Yin Jie An ... Oron Ashual Oran Gafni Devi Parikh Sonal Gupta Yaniv Taigman DiffM VGen 81 1,409 0 29 Sep 2022
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions Paul Pu Liang Amir Zadeh Louis-Philippe Morency 89 75 0 07 Sep 2022
How do Variational Autoencoders Learn? Insights from Representational Similarity Lisa Bonheme M. Grzes CoGe SSL DRL 62 10 0 17 May 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 382 3,542 0 29 Apr 2022
Representation Topology Divergence: A Method for Comparing Neural Network Representations S. Barannikov I. Trofimov Nikita Balabin Evgeny Burnaev 3DPC 75 49 0 31 Dec 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 460 7,757 0 11 Nov 2021
Similarity and Matching of Neural Network Representations Adrián Csiszárik Péter Korösi-Szabó Á. Matszangosz Gergely Papp D. Varga 44 70 0 27 Oct 2021
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Junnan Li Ramprasaath R. Selvaraju Akhilesh Deepak Gotmare Shafiq Joty Caiming Xiong Guosheng Lin FaML 194 1,960 0 16 Jul 2021
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning Paul Pu Liang Yiwei Lyu Xiang Fan Zetian Wu Yun Cheng ... Peter Wu Michelle A. Lee Yuke Zhu Ruslan Salakhutdinov Louis-Philippe Morency VLM 85 170 0 15 Jul 2021
Revisiting Model Stitching to Compare Neural Representations Yamini Bansal Preetum Nakkiran Boaz Barak FedML 90 117 0 14 Jun 2021
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning Krishna Srinivasan K. Raman Jiecao Chen Michael Bendersky Marc Najork VLM 258 316 0 02 Mar 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 925 29,436 0 26 Feb 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 395 4,953 0 24 Feb 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 443 3,856 0 11 Feb 2021
Contrastive learning, multi-view redundancy, and linear models Christopher Tosh A. Krishnamurthy Daniel J. Hsu SSL 74 166 0 24 Aug 2020
What Makes for Good Views for Contrastive Learning? Yonglong Tian Chen Sun Ben Poole Dilip Krishnan Cordelia Schmid Phillip Isola SSL 100 1,329 0 20 May 2020
Multimodal End-to-End Autonomous Driving Yi Xiao Felipe Codevilla A. Gurram O. Urfalioglu Antonio M. López 54 244 0 07 Jun 2019
Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper) Santiago Castro Devamanyu Hazarika Verónica Pérez-Rosas Roger Zimmermann Rada Mihalcea Soujanya Poria 62 258 0 05 Jun 2019
Similarity of Neural Network Representations Revisited Simon Kornblith Mohammad Norouzi Honglak Lee Geoffrey E. Hinton 141 1,418 0 01 May 2019
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor M. Hasan Wasifur Rahman Amir Zadeh Jianyuan Zhong Md. Iftekhar Tanveer Louis-Philippe Morency Mohammed Ehsan Hoque 63 185 0 14 Apr 2019
VideoBERT: A Joint Model for Video and Language Representation Learning Chen Sun Austin Myers Carl Vondrick Kevin Patrick Murphy Cordelia Schmid VLM SSL 79 1,246 0 03 Apr 2019
MFAS: Multimodal Fusion Architecture Search Juan-Manuel Perez-Rua Valentin Vielzeuf S. Pateux M. Baccouche F. Jurie 66 178 0 15 Mar 2019
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks Michelle A. Lee Yuke Zhu K. Srinivasan Parth Shah Silvio Savarese Li Fei-Fei Animesh Garg Jeannette Bohg SSL 76 369 0 24 Oct 2018
Insights on representational similarity in neural networks with canonical correlation Ari S. Morcos M. Raghu Samy Bengio DRL 63 446 0 14 Jun 2018
Multimodal Machine Learning: A Survey and Taxonomy T. Baltrušaitis Chaitanya Ahuja Louis-Philippe Morency 99 2,932 0 26 May 2017
Gated Multimodal Units for Information Fusion John Arevalo Thamar Solorio Manuel Montes-y-Gómez Fabio Gonzalez 88 381 0 07 Feb 2017
Fully-Convolutional Siamese Networks for Object Tracking Luca Bertinetto Jack Valmadre João F. Henriques Andrea Vedaldi Philip Torr VOT 77 3,879 0 30 Jun 2016
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos Amir Zadeh Rowan Zellers Eli Pincus Louis-Philippe Morency 78 454 0 20 Jun 2016
On Deep Multi-View Representation Learning: Objectives and Optimization Weiran Wang R. Arora Karen Livescu J. Bilmes SSL DRL 84 914 0 02 Feb 2016
Convergent Learning: Do different neural networks learn the same representations? Yixuan Li J. Yosinski Jeff Clune Hod Lipson John E. Hopcroft SSL 83 370 0 24 Nov 2015
Understanding image representations by measuring their equivariance and equivalence Karel Lenc Andrea Vedaldi SSL FAtt 109 533 0 21 Nov 2014
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,547 0 01 Sep 2014