v1v2v3 (latest)

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

22 April 2021

Papers citing "VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text"

50 / 360 papers shown

Title
RegBN: Batch Normalization of Multimodal Data with Regularization Morteza Ghahremani Christian Wachinger 86 7 0 01 Oct 2023
Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval Hao Li Marie-Jeanne Lesot Lianli Gao Xiaosu Zhu Christophe Marsala EDL 78 15 0 29 Sep 2023
Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback Teresa Yeo Oğuzhan Fatih Kar Zahra Sodagar Amir Zamir TTA OOD 71 4 0 27 Sep 2023
$M$^{3}$3D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understanding$ M $^{3}$ 3D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understanding Muhammad Abdullah Jamal Omid Mohareri 3DPC 66 1 0 26 Sep 2023
SeMAnD: Self-Supervised Anomaly Detection in Multimodal Geospatial Datasets Daria Reshetova Swetava Ganguli C. V. K. Iyer Vipul Pandey 59 3 0 26 Sep 2023
Audio classification with Dilated Convolution with Learnable Spacings Ismail Khalfaoui-Hassani T. Masquelier Thomas Pellegrini 69 1 0 25 Sep 2023
VidChapters-7M: Video Chapters at Scale Antoine Yang Arsha Nagrani Ivan Laptev Josef Sivic Cordelia Schmid VGen 98 28 0 25 Sep 2023
Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers Adam Pardyl Grzegorz Kurzejamski Jan Olszewski Tomasz Trzciñski Bartosz Zieliñski 54 1 0 23 Sep 2023
TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification Meng Liu K. Liang Dayu Hu Hao Yu Yue Liu Lingyuan Meng Wenxuan Tu Sihang Zhou Xinwang Liu 74 26 0 21 Sep 2023
DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation Bo Yin Xuying Zhang Zhongyu Li Li Liu Ming-Ming Cheng Qibin Hou 88 52 0 18 Sep 2023
In-Style: Bridging Text and Uncurated Videos with Style Transfer for Text-Video Retrieval Nina Shvetsova Anna Kukleva Bernt Schiele Hilde Kuehne DiffM 70 4 0 16 Sep 2023
Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals Ran Liu Ellen L. Zippi Hadi Pouransari Chris Sandino Jingping Nie Hanlin Goh Erdrin Azemi Ali Moin 94 12 0 12 Sep 2023
Decoupling Common and Unique Representations for Multimodal Self-supervised Learning Yi Wang C. Albrecht Nassim Ait Ali Braham Chenying Liu Zhitong Xiong Xiaoxiang Zhu SSL 88 19 0 11 Sep 2023
Towards Contrastive Learning in Music Video Domain Karel Veldkamp Mariya Hendriksen Zoltán Szlávik Alexander Keijser SSL 51 2 0 01 Sep 2023
AGS: An Dataset and Taxonomy for Domestic Scene Sound Event Recognition Nan Che Chenrui Liu Fei Yu 57 0 0 30 Aug 2023
CoVR: Learning Composed Video Retrieval from Web Video Captions Lucas Ventura Antoine Yang Cordelia Schmid Gül Varol 73 21 0 28 Aug 2023
MM-AU:Towards Multimodal Understanding of Advertisement Videos Digbalay Bose Rajat Hebbar Tiantian Feng Krishna Somandepalli Anfeng Xu Shrikanth Narayanan 49 7 0 27 Aug 2023
Long-range Multimodal Pretraining for Movie Understanding Dawit Mureja Argaw Joon-Young Lee Markus Woodson In So Kweon Fabian Caba Heilbron VLM 77 9 0 18 Aug 2023
AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes Zhaohui Li Haitao Wang Xinghua Jiang 113 1 0 14 Aug 2023
Which Tokens to Use? Investigating Token Reduction in Vision Transformers Joakim Bruslund Haurum Sergio Escalera Graham W. Taylor T. Moeslund ViT 102 38 0 09 Aug 2023
Cascaded Cross-Modal Transformer for Request and Complaint Detection Nicolae-Cătălin Ristea Radu Tudor Ionescu 90 3 0 27 Jul 2023
Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignment Sarah Ibrahimi Xiaohang Sun Pichao Wang Amanmeet Garg Ashutosh Sanan Mohamed Omar 99 18 0 24 Jul 2023
Robust Visual Question Answering: Datasets, Methods, and Future Challenges Jie Ma Pinghui Wang Dechen Kong Zewei Wang Jun Liu Hongbin Pei Junzhou Zhao OOD 126 23 0 21 Jul 2023
Language-based Action Concept Spaces Improve Video Self-Supervised Learning Kanchana Ranasinghe Michael S. Ryoo SSL VLM 101 12 0 20 Jul 2023
Exploring Transformer Extrapolation Zhen Qin Yiran Zhong Huiyuan Deng 51 9 0 19 Jul 2023
Does Visual Pretraining Help End-to-End Reasoning? Chen Sun Calvin Luo Xingyi Zhou Anurag Arnab Cordelia Schmid OCL LRM ViT 78 3 0 17 Jul 2023
Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution Mostafa Dehghani Basil Mustafa Josip Djolonga Jonathan Heek Matthias Minderer ... Avital Oliver Piotr Padlewski A. Gritsenko Mario Luvcić N. Houlsby ViT 188 119 0 12 Jul 2023
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone Shraman Pramanick Yale Song Sayan Nag Kevin Qinghong Lin Hardik Shah Mike Zheng Shou Ramalingam Chellappa Pengchuan Zhang VLM 118 100 0 11 Jul 2023
One-Versus-Others Attention: Scalable Multimodal Integration for Clinical Data Michal Golovanevsky Eva Schiller Akira Nair Ritambhara Singh Carsten Eickhoff 68 3 0 11 Jul 2023
All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment Chunhui Zhang Xin Sun Li Liu Yiqian Yang Qiong Liu Xiaoping Zhou Yanfeng Wang 218 17 0 07 Jul 2023
VideoGLUE: Video General Understanding Evaluation of Foundation Models Liangzhe Yuan N. B. Gundavarapu Long Zhao Hao Zhou Huayu Chen ... Florian Schroff Hartwig Adam Ming-Hsuan Yang Ting Liu Boqing Gong ELM 85 10 0 06 Jul 2023
ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis Yu Mingjun Zhao Shiguang Qi Feiran Sun Baoxun Wang Weidong Guo Xiaoli Wang Lei Yang Di Niu 56 11 0 27 Jun 2023
Learning Unseen Modality Interaction Yunhua Zhang Hazel Doughty Cees G. M. Snoek 137 6 0 22 Jun 2023
Exploring the Role of Audio in Video Captioning Yuhan Shen Linjie Yang Longyin Wen Haichao Yu Ehsan Elhamifar Heng Wang 65 2 0 21 Jun 2023
Language-Guided Music Recommendation for Video via Prompt Analogies Daniel McKee Justin Salamon Josef Sivic Bryan C. Russell VGen 80 27 0 15 Jun 2023
Global and Local Semantic Completion Learning for Vision-Language Pre-training Rong-Cheng Tu Yatai Ji Jie Jiang Weijie Kong Chengfei Cai Wenzhe Zhao Hongfa Wang Yujiu Yang Wei Liu VLM 96 4 0 12 Jun 2023
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks Saidul Islam Hanae Elmekki Ahmed Elsebai Jamal Bentahar Najat Drawel Gaith Rjoub Witold Pedrycz ViT MedIm 89 207 0 11 Jun 2023
Factorized Contrastive Learning: Going Beyond Multi-view Redundancy Paul Pu Liang Zihao Deng Martin Q. Ma James Zou Louis-Philippe Morency Ruslan Salakhutdinov SSL 94 56 0 08 Jun 2023
Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Work Qiangchang Wang Yilong Yin 98 0 0 02 Jun 2023
There is more to graphs than meets the eye: Learning universal features with self-supervision L. Das Sai Munikoti M. Halappanavar SSL OOD 66 1 0 31 May 2023
A Unified Framework for Slot based Response Generation in a Multimodal Dialogue System Mauajama Firdaus Avinash Madasu Asif Ekbal 120 7 0 27 May 2023
LANISTR: Multimodal Learning from Structured and Unstructured Data Sayna Ebrahimi Sercan O. Arik Yihe Dong Tomas Pfister 57 4 0 26 May 2023
HAAV: Hierarchical Aggregation of Augmented Views for Image Captioning Chia-Wen Kuo Z. Kira 83 23 0 25 May 2023
Training Transitive and Commutative Multimodal Transformers with LoReTTa Manuel Tran Yashin Dicente Cid Amal Lahiani Fabian J. Theis Tingying Peng Eldad Klaiman 54 2 0 23 May 2023
More Perspectives Mean Better: Underwater Target Recognition and Localization with Multimodal Data via Symbiotic Transformer and Multiview Regression Shipei Liu Xiaoya Fan Guowei Wu 61 0 0 22 May 2023
Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios Yuanyuan Jiang Jianqin Yin 62 8 0 21 May 2023
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data Ziyi Yang Mahmoud Khademi Yichong Xu Reid Pryzant Yuwei Fang ... Yu Shi Lu Yuan Takuya Yoshioka Michael Zeng Xuedong Huang 60 2 0 21 May 2023
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities Peng Wang Shijie Wang Junyang Lin Shuai Bai Xiaohuan Zhou Jingren Zhou Xinggang Wang Chang Zhou VLM MLLM ObjD 151 122 0 18 May 2023
UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning Heqing Zou Meng Shen Chen Chen Yuchen Hu D. Rajan Chng Eng Siong SSL 100 17 0 16 May 2023
MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition Xinyu Gong S. Mohan Naina Dhingra Jean-Charles Bazin Yilei Li Zhangyang Wang Rakesh Ranjan EgoV 124 19 0 12 May 2023