Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!

13 October 2020

Papers citing "Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!"

17 / 17 papers shown

Title
What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation Michal Golovanevsky William Rudman Vedant Palit Ritambhara Singh Carsten Eickhoff 33 1 0 24 Jun 2024
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition Chen Chen Ruizhe Li Yuchen Hu Sabato Marco Siniscalchi Pin-Yu Chen Ensiong Chng Chao-Han Huck Yang 36 19 0 08 Feb 2024
Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals Yanai Elazar Bhargavi Paranjape Hao Peng Sarah Wiegreffe Khyathi Raghavi Vivek Srikumar Sameer Singh Noah A. Smith AAML OOD 34 0 0 16 Nov 2023
Boosting Multi-modal Model Performance with Adaptive Gradient Modulation Hong Li Xingyu Li Pengbo Hu Yinuo Lei Chunxiao Li Yi Zhou 42 22 0 15 Aug 2023
Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion Isha Rawal Alexander Matyasko Shantanu Jaiswal Basura Fernando Cheston Tan 26 2 0 15 Jun 2023
Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification Chunpu Xu Jing Li VLM 26 5 0 27 Mar 2023
Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answering Paul Lerner O. Ferret C. Guinaudeau 21 9 0 11 Jan 2023
Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation Yaoming Zhu Zewei Sun Shanbo Cheng Yuyang Huang Liwei Wu Mingxuan Wang 28 10 0 20 Dec 2022
Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest Jack Hessel Ana Marasović Jena D. Hwang Lillian Lee Jeff Da Rowan Zellers Robert Mankoff Yejin Choi VLM 24 85 0 13 Sep 2022
SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities Pengbo Hu Xingyu Li Yi Zhou 30 10 0 30 Apr 2022
DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations Yiwei Lyu Paul Pu Liang Zihao Deng Ruslan Salakhutdinov Louis-Philippe Morency 24 31 0 03 Mar 2022
MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound Rowan Zellers Jiasen Lu Ximing Lu Youngjae Yu Yanpeng Zhao Mohammadreza Salehi Aditya Kusupati Jack Hessel Ali Farhadi Yejin Choi 48 207 0 07 Jan 2022
Perceptual Score: What Data Modalities Does Your Model Perceive? Itai Gat Idan Schwartz A. Schwing 33 30 0 27 Oct 2021
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models Peter West Chandrasekhar Bhagavatula Jack Hessel Jena D. Hwang Liwei Jiang Ronan Le Bras Ximing Lu Sean Welleck Yejin Choi SyDa 54 322 0 14 Oct 2021
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining Xiao Dong Xunlin Zhan Yangxin Wu Yunchao Wei Michael C. Kampffmeyer Xiaoyong Wei Minlong Lu Yaowei Wang Xiaodan Liang 33 37 0 09 Sep 2021
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning Paul Pu Liang Yiwei Lyu Xiang Fan Zetian Wu Yun Cheng ... Peter Wu Michelle A. Lee Yuke Zhu Ruslan Salakhutdinov Louis-Philippe Morency VLM 32 159 0 15 Jul 2021
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation Zhiyong Wu Lingpeng Kong W. Bi Xiang Li B. Kao LRM 23 77 0 30 May 2021