Are Multimodal Transformers Robust to Missing Modality?

12 April 2022

Papers citing "Are Multimodal Transformers Robust to Missing Modality?"

46 / 96 papers shown

Title
A Survey on Safe Multi-Modal Learning System Tianyi Zhao Liangliang Zhang Yao Ma Lu Cheng 60 9 0 08 Feb 2024
AM^2-EmoJE: Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learning Naresh Kumar Devulapally Sidharth Anand S. Bhattacharjee Junsong Yuan 30 1 0 26 Jan 2024
Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality Huy Q. Le Chu Myaet Thwal Yu Qiao Ye Lin Tun Minh N. H. Nguyen Choong Seon Hong Choong Seon Hong 70 4 0 25 Jan 2024
Exploring Missing Modality in Multimodal Egocentric Datasets Merey Ramazanova Alejandro Pardo Humam Alwassel Guohao Li EgoV 38 4 0 21 Jan 2024
Toward Robust Multimodal Learning using Multimodal Foundational Models Xianbing Zhao Soujanya Poria Xuejiao Li Yixin Chen Buzhou Tang VLM 37 2 0 20 Jan 2024
Towards Robust Multimodal Prompting With Missing Modalities Jaehyuk Jang Yooseung Wang Changick Kim VLM 30 10 0 26 Dec 2023
Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast Guangyin Bao Qi Zhang Duoqian Miao Zixuan Gong Liang Hu Ke Liu Yang Liu Chongyang Shi 42 8 0 21 Dec 2023
Can Physician Judgment Enhance Model Trustworthiness? A Case Study on Predicting Pathological Lymph Nodes in Rectal Cancer Kazuma Kobayashi Yasuyuki Takamizawa M. Miyake Sono Ito Lin Gu Tatsuya Nakatsuka Yu Akagi Tatsuya Harada Y. Kanemitsu Ryuji Hamamoto 38 2 0 15 Dec 2023
Segment Beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation Renjie Wu Hu Wang Feras Dayoub Hsiang-Ting Chen 30 5 0 14 Dec 2023
Adventures of Trustworthy Vision-Language Models: A Survey Mayank Vatsa Anubhooti Jain Richa Singh 30 4 0 07 Dec 2023
Multimodal Representation Learning by Alternating Unimodal Adaptation Xiaohui Zhang Jaehong Yoon Mohit Bansal Huaxiu Yao 34 22 0 17 Nov 2023
Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition Juan Vazquez-Rodriguez G. Lefebvre Julien Cumin James L. Crowley 34 2 0 16 Nov 2023
MOSEL: Inference Serving Using Dynamic Modality Selection Bodun Hu Le Xu Jeongyoon Moon N. Yadwadkar Aditya Akella 13 4 0 27 Oct 2023
What Makes for Robust Multi-Modal Models in the Face of Missing Modalities? Siting Li Chenzhuang Du Yue Zhao Yu Huang Hang Zhao 24 4 0 10 Oct 2023
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation Md Kaykobad Reza Ashley Prater-Bennette Ulugbek S. Kamilov 28 5 0 06 Oct 2023
MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks Vinitra Swamy Malika Satayeva Jibril Frej Thierry Bossy Thijs Vogels Martin Jaggi Tanja Kaser Mary-Anne Hartley 20 12 0 25 Sep 2023
A Survey on Image-text Multimodal Models Ruifeng Guo Jingxuan Wei Linzhuang Sun Khai Le-Duc Guiyong Chang Dawei Liu Sibo Zhang Zhengbing Yao Mingjun Xu Liping Bu VLM 31 5 0 23 Sep 2023
Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognition Qitong Wang Long Zhao Liangzhe Yuan Ting Liu Xi Peng 36 12 0 22 Aug 2023
Boosting Multi-modal Model Performance with Adaptive Gradient Modulation Hong Li Xingyu Li Pengbo Hu Yinuo Lei Chunxiao Li Yi Zhou 42 22 0 15 Aug 2023
Learning Unseen Modality Interaction Yunhua Zhang Hazel Doughty Cees G. M. Snoek 27 3 0 22 Jun 2023
Modality Influence in Multimodal Machine Learning Abdelhamid Haouhat Slimane Bellaouar A. Nehar H. Cherroun 31 2 0 10 Jun 2023
VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition Ziyi Ni Minglun Han Feilong Chen Linghui Meng Jing Shi Shuang Xu Bo Xu 42 0 0 31 May 2023
FRAMM: Fair Ranking with Missing Modalities for Clinical Trial Site Selection B. Theodorou Lucas Glass Cao Xiao Jimeng Sun 35 7 0 30 May 2023
LANISTR: Multimodal Learning from Structured and Unstructured Data Sayna Ebrahimi Sercan Ö. Arik Yihe Dong Tomas Pfister 20 4 0 26 May 2023
Training Transitive and Commutative Multimodal Transformers with LoReTTa Manuel Tran Yashin Dicente Cid Amal Lahiani Fabian J. Theis Tingying Peng Eldad Klaiman 26 2 0 23 May 2023
Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modality Jialing Yuan Ye Yu Gaurav Mittal Matthew Hall Sandra Sajeev Mei Chen 27 9 0 17 May 2023
MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition Xinyu Gong S. Mohan Naina Dhingra Jean-Charles Bazin Yilei Li Zhangyang Wang Rakesh Ranjan EgoV 56 17 0 12 May 2023
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis Jinsheng Zheng Daqing Liu Chaoyue Wang Minghui Hu Zuopeng Yang Changxing Ding Dacheng Tao 34 1 0 10 May 2023
Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention Kwanhyung Lee Soojeong Lee Sangchul Hahn Heejung Hyun Edward Choi Byungeun Ahn Joohyung Lee 49 16 0 04 May 2023
Incomplete Multimodal Learning for Remote Sensing Data Fusion Yuxing Chen Maofan Zhao Lorenzo Bruzzone 37 3 0 22 Apr 2023
Towards Unified AI Drug Discovery with Multiple Knowledge Modalities Yi Luo Xing Yi Liu Kai Yang Kui Huang Massimo Hong Jiahuan Zhang Yushuai Wu Zaiqing Nie OffRL 40 4 0 17 Apr 2023
Are Data-driven Explanations Robust against Out-of-distribution Data? Tang Li Fengchun Qiao Mengmeng Ma Xiangkai Peng OODD OOD 33 10 0 29 Mar 2023
CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities Konstantinos Kontras Christos Chatzichristos Huy P Phan Johan A. K. Suykens Marina De Vos AI4TS 24 11 0 27 Mar 2023
Multimodal Prompting with Missing Modalities for Visual Recognition Yi-Lun Lee Yi-Hsuan Tsai Wei-Chen Chiu Chen-Yu Lee VPVLM 33 94 0 06 Mar 2023
Rethinking Vision Transformers for MobileNet Size and Speed Yanyu Li Ju Hu Yang Wen Georgios Evangelidis Kamyar Salahi Yanzhi Wang Sergey Tulyakov Jian Ren ViT 35 159 0 15 Dec 2022
Towards Good Practices for Missing Modality Robust Action Recognition Sangmin Woo Sumin Lee Yeonju Park Muhammad Adi Nugroho Changick Kim 22 43 0 25 Nov 2022
Multimodal Transformer for Parallel Concatenated Variational Autoencoders Stephen D. Liang J. Mendel ViT 27 5 0 28 Oct 2022
Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features Gokul Karthik Kumar Karthik Nandakumar VLM CLIP 32 56 0 12 Oct 2022
Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality Haozhe Chi Minghua Yang Junhao Zhu Guanhong Wang Gaoang Wang 41 3 0 07 Oct 2022
Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition Ruiqi Wang Wonse Jo Dezhong Zhao Weizheng Wang B. Yang Guohua Chen Byung-Cheol Min HAI 32 28 0 30 Sep 2022
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions Paul Pu Liang Amir Zadeh Louis-Philippe Morency 18 62 0 07 Sep 2022
Multimodal Learning with Transformers: A Survey P. Xu Xiatian Zhu David A. Clifton ViT 72 528 0 13 Jun 2022
Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review C. Cui Haichun Yang Yaohong Wang Shilin Zhao Zuhayr Asad Lori A. Coburn K. Wilson Bennett A. Landman Yuankai Huo 28 94 0 25 Mar 2022
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Huayu Chen Boqing Gong ViT 251 577 0 22 Apr 2021
Multi-modal Transformer for Video Retrieval Valentin Gabeur Chen Sun Alahari Karteek Cordelia Schmid ViT 427 596 0 21 Jul 2020
Supervised Multimodal Bitransformers for Classifying Images and Text Douwe Kiela Suvrat Bhooshan Hamed Firooz Ethan Perez Davide Testuggine 59 241 0 06 Sep 2019