VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

22 April 2021

Papers citing "VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text"

50 / 114 papers shown

Title
Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection SungHeon Jeong Jihong Park Mohsen Imani 56 0 0 05 May 2025
Learning Streaming Video Representation via Multitask Training Yibin Yan Jilan Xu Shangzhe Di Yikun Liu Yudi Shi Qirui Chen Zeqian Li Yifei Huang Weidi Xie CLL 84 0 0 28 Apr 2025
Breaking the Barriers: Video Vision Transformers for Word-Level Sign Language Recognition Alexander Brettmann Jakob Grävinghoff Marlene Rüschoff Marie Westhues SLR 51 0 0 10 Apr 2025
Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention Joe Dhanith Shravan Venkatraman Modigari Narendra Vigya Sharma Santhosh Malarvannan 74 0 0 20 Feb 2025
The Sound of Water: Inferring Physical Properties from Pouring Liquids Piyush Bagad Makarand Tapaswi Cees G. M. Snoek Andrew Zisserman 40 0 0 18 Nov 2024
Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding Yunze Man Shuhong Zheng Zhipeng Bao M. Hebert Liang-Yan Gui Yu-xiong Wang 70 15 0 05 Sep 2024
ICSD: An Open-source Dataset for Infant Cry and Snoring Detection Qingyu Liu Longfei Song Dongxing Xu Yanhua Long 37 0 0 20 Aug 2024
VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling Zeyue Tian Zhaoyang Liu Ruibin Yuan Jiahao Pan Xiaoqiang Huang Xu Tan Xu Tan Qifeng Chen Y. Guo VGen 100 16 0 06 Jun 2024
LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate A. Fuller Daniel G. Kyrollos Yousef Yassin James R. Green 43 2 0 22 May 2024
CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering Yuanyuan Jiang Jianqin Yin 38 1 0 13 May 2024
Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition Yash Jain David M. Chan Pranav Dheram Aparna Khare Olabanji Shonibare Venkatesh Ravichandran Shalini Ghosh 37 2 0 28 Mar 2024
Siamese Vision Transformers are Scalable Audio-visual Learners Yan-Bo Lin Gedas Bertasius 37 5 0 28 Mar 2024
VideoPrism: A Foundational Visual Encoder for Video Understanding Long Zhao N. B. Gundavarapu Liangzhe Yuan Hao Zhou Shen Yan ... Huisheng Wang Hartwig Adam Mikhail Sirotenko Ting Liu Boqing Gong VGen 29 29 0 20 Feb 2024
OmniVec: Learning robust representations with cross modal sharing Siddharth Srivastava Gaurav Sharma SSL 21 64 0 07 Nov 2023
CAD -- Contextual Multi-modal Alignment for Dynamic AVQA Asmar Nadeem Adrian Hilton R. Dawes Graham A. Thomas A. Mustafa 21 9 0 25 Oct 2023
AGS: An Dataset and Taxonomy for Domestic Scene Sound Event Recognition Nan Che Chenrui Liu Fei Yu 25 0 0 30 Aug 2023
AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes Zhaohui Li Haitao Wang Xinghua Jiang 40 1 0 14 Aug 2023
Cascaded Cross-Modal Transformer for Request and Complaint Detection Nicolae-Cătălin Ristea Radu Tudor Ionescu 18 3 0 27 Jul 2023
Language-based Action Concept Spaces Improve Video Self-Supervised Learning Kanchana Ranasinghe Michael S. Ryoo SSL VLM 28 12 0 20 Jul 2023
Does Visual Pretraining Help End-to-End Reasoning? Chen Sun Calvin Luo Xingyi Zhou Anurag Arnab Cordelia Schmid OCL LRM ViT 30 3 0 17 Jul 2023
All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment Chunhui Zhang Xin Sun Li Liu Yiqian Yang Qiong Liu Xiaoping Zhou Yanfeng Wang 38 15 0 07 Jul 2023
A Unified Framework for Slot based Response Generation in a Multimodal Dialogue System Mauajama Firdaus Avinash Madasu Asif Ekbal 33 7 0 27 May 2023
HAAV: Hierarchical Aggregation of Augmented Views for Image Captioning Chia-Wen Kuo Z. Kira 27 21 0 25 May 2023
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data Ziyi Yang Mahmoud Khademi Yichong Xu Reid Pryzant Yuwei Fang ... Yu Shi Lu Yuan Takuya Yoshioka Michael Zeng Xuedong Huang 17 2 0 21 May 2023
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities Peng Wang Shijie Wang Junyang Lin Shuai Bai Xiaohuan Zhou Jingren Zhou Xinggang Wang Chang Zhou VLM MLLM ObjD 26 114 0 18 May 2023
A vector quantized masked autoencoder for audiovisual speech emotion recognition Samir Sadok Simon Leglaive Renaud Séguier SSL 79 6 0 05 May 2023
On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence Gengchen Mai Weiming Huang Jin Sun Suhang Song Deepak Mishra ... Yingjie Hu Chris Cundy Ziyuan Li Rui Zhu Ni Lao AI4CE 22 121 0 13 Apr 2023
HypLiLoc: Towards Effective LiDAR Pose Regression with Hyperbolic Fusion Sijie Wang Qiyu Kang Rui She Wei Wang K. Zhao Yang Song Wee Peng Tay 31 18 0 03 Apr 2023
Procedure-Aware Pretraining for Instructional Video Understanding Honglu Zhou Roberto Martín-Martín Mubbasir Kapadia Silvio Savarese Juan Carlos Niebles 23 38 0 31 Mar 2023
What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions Brian Chen Nina Shvetsova Andrew Rouditchenko D. Kondermann Samuel Thomas Shih-Fu Chang Rogerio Feris James R. Glass Hilde Kuehne 27 7 0 29 Mar 2023
Egocentric Auditory Attention Localization in Conversations Fiona Ryan Hao Jiang Abhinav Shukla James M. Rehg V. Ithapu EgoV 24 16 0 28 Mar 2023
3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition Lei Wang Piotr Koniusz ViT 23 45 0 25 Mar 2023
Transformers in Speech Processing: A Survey S. Latif Aun Zaidi Heriberto Cuayáhuitl Fahad Shamshad Moazzam Shoukat Junaid Qadir 42 47 0 21 Mar 2023
Accommodating Audio Modality in CLIP for Multimodal Processing Ludan Ruan Anwen Hu Yuqing Song Liang Zhang S. Zheng Qin Jin VLM 16 10 0 12 Mar 2023
Heterogeneous Graph Learning for Acoustic Event Classification A. Shirian Mona Ahmadian Krishna Somandepalli T. Guha 25 2 0 05 Mar 2023
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning Antoine Yang Arsha Nagrani Paul Hongsuck Seo Antoine Miech Jordi Pont-Tuset Ivan Laptev Josef Sivic Cordelia Schmid AI4TS VLM 23 220 0 27 Feb 2023
Transformadores: Fundamentos teoricos y Aplicaciones J. D. L. Torre 70 0 0 18 Feb 2023
Single Cells Are Spatial Tokens: Transformers for Spatial Transcriptomic Data Imputation Haifang Wen Wenzhuo Tang Wei Jin Jiayuan Ding Renming Liu Xinnan Dai Feng Shi Lulu Shang Jiliang Tang Yuying Xie 27 8 0 06 Feb 2023
What You Say Is What You Show: Visual Narration Detection in Instructional Videos Kumar Ashutosh Rohit Girdhar Lorenzo Torresani Kristen Grauman 16 4 0 05 Jan 2023
UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering Chenlu Zhan Peng Peng Hongsen Wang Tao Chen Hongwei Wang MedIm 23 3 0 21 Dec 2022
Multimodal Vision Transformers with Forced Attention for Behavior Analysis Tanay Agrawal Michal Balazia Philippe Muller Franccois Brémond ViT 15 9 0 07 Dec 2022
See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation Hao Li Yizhi Zhang Junzhe Zhu Shaoxiong Wang Michelle A. Lee Huazhe Xu Edward H. Adelson Li Fei-Fei Ruohan Gao Jiajun Wu 22 58 0 07 Dec 2022
InternVideo: General Video Foundation Models via Generative and Discriminative Learning Yi Wang Kunchang Li Yizhuo Li Yinan He Bingkun Huang ... Junting Pan Jiashuo Yu Yali Wang Limin Wang Yu Qiao VLM VGen 38 309 0 06 Dec 2022
VLG: General Video Recognition with Web Textual Knowledge Jintao Lin Zhaoyang Liu Wenhai Wang Wayne Wu Limin Wang 37 0 0 03 Dec 2022
Token Turing Machines Michael S. Ryoo K. Gopalakrishnan Kumara Kahatapitiya Ted Xiao Kanishka Rao Austin Stone Yao Lu Julian Ibarz Anurag Arnab 27 21 0 16 Nov 2022
Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works Chao Tao Ji Qi Mingning Guo Qing Zhu Haifeng Li SSL 19 56 0 15 Nov 2022
Changes from Classical Statistics to Modern Statistics and Data Science Kai Zhang Shan-Yu Liu M. Xiong 26 0 0 30 Oct 2022
Grafting Vision Transformers Jong Sung Park Kumara Kahatapitiya Donghyun Kim Shivchander Sudalairaj Quanfu Fan Michael S. Ryoo ViT 21 2 0 28 Oct 2022
Masked Vision-Language Transformer in Fashion Ge-Peng Ji Mingchen Zhuge D. Gao Deng-Ping Fan Christos Sakaridis Luc Van Gool 17 25 0 27 Oct 2022
S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces Eric N. D. Nguyen Karan Goel Albert Gu Gordon W. Downs Preey Shah Tri Dao S. Baccus Christopher Ré VLM 22 38 0 12 Oct 2022