VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

22 April 2021

Papers citing "VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text"

50 / 114 papers shown

Title
Contrastive Audio-Visual Masked Autoencoder Yuan Gong Andrew Rouditchenko Alexander H. Liu David F. Harwath Leonid Karlinsky Hilde Kuehne James R. Glass 29 119 0 02 Oct 2022
Data Poisoning Attacks Against Multimodal Encoders Ziqing Yang Xinlei He Zheng Li Michael Backes Mathias Humbert Pascal Berrang Yang Zhang AAML 108 45 0 30 Sep 2022
TVLT: Textless Vision-Language Transformer Zineng Tang Jaemin Cho Yixin Nie Mohit Bansal VLM 49 28 0 28 Sep 2022
Vision Transformers for Action Recognition: A Survey Anwaar Ulhaq Naveed Akhtar Ganna Pogrebna Ajmal Saeed Mian ViT 19 44 0 13 Sep 2022
Fusion of Satellite Images and Weather Data with Transformer Networks for Downy Mildew Disease Detection William Maillet Maryam Ouhami A. Hafiane ViT MedIm 14 6 0 06 Sep 2022
Topic Detection in Continuous Sign Language Videos Álvaro Budria Laia Tarrés Gerard I. Gállego Francesc Moreno-Noguer Jordi Torres Xavier Giró-i-Nieto SLR VLM 33 1 0 01 Sep 2022
UAVM: Towards Unifying Audio and Visual Models Yuan Gong Alexander H. Liu Andrew Rouditchenko James R. Glass 25 20 0 29 Jul 2022
Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training Haoxuan You Luowei Zhou Bin Xiao Noel Codella Yu Cheng Ruochen Xu Shih-Fu Chang Lu Yuan CLIP VLM 19 48 0 26 Jul 2022
LAVA: Language Audio Vision Alignment for Contrastive Video Pre-Training Sumanth Gurram An Fang David M. Chan John F. Canny VLM AI4TS 28 1 0 16 Jul 2022
Robustness Analysis of Video-Language Models Against Visual and Language Perturbations Madeline Chantry Schiappa Shruti Vyas Hamid Palangi Y. S. Rawat Vibhav Vineet VLM 120 17 0 05 Jul 2022
AFT-VO: Asynchronous Fusion Transformers for Multi-View Visual Odometry Estimation Nimet Kaygusuz Oscar Alejandro Mendez Maldonado Richard Bowden 12 5 0 26 Jun 2022
Self-Supervised Learning for Videos: A Survey Madeline Chantry Schiappa Y. S. Rawat M. Shah SSL 34 131 0 18 Jun 2022
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge Linxi Fan Guanzhi Wang Yunfan Jiang Ajay Mandlekar Yuncong Yang Haoyi Zhu Andrew Tang De-An Huang Yuke Zhu Anima Anandkumar LM&Ro 42 347 0 17 Jun 2022
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling Linjie Li Zhe Gan Kevin Qinghong Lin Chung-Ching Lin Zicheng Liu Ce Liu Lijuan Wang MLLM VLM 18 81 0 14 Jun 2022
Multimodal Learning with Transformers: A Survey P. Xu Xiatian Zhu David A. Clifton ViT 50 525 0 13 Jun 2022
Architecture-Agnostic Masked Image Modeling -- From ViT back to CNN Siyuan Li Di Wu Fang Wu Lei Shang Stan.Z.Li 32 48 0 27 May 2022
Non-Parametric Domain Adaptation for End-to-End Speech Translation Yichao Du Weizhi Wang Zhirui Zhang Boxing Chen Tong Bill Xu Jun Xie Enhong Chen 49 18 0 23 May 2022
TransTab: Learning Transferable Tabular Transformers Across Tables Zifeng Wang Jimeng Sun LMTD 25 135 0 19 May 2022
CoCa: Contrastive Captioners are Image-Text Foundation Models Jiahui Yu Zirui Wang Vijay Vasudevan Legg Yeung Mojtaba Seyedhosseini Yonghui Wu VLM CLIP OffRL 59 1,255 0 04 May 2022
i-Code: An Integrative and Composable Multimodal Learning Framework Ziyi Yang Yuwei Fang Chenguang Zhu Reid Pryzant Dongdong Chen ... Bin Xiao Yuanxun Lu Takuya Yoshioka Michael Zeng Xuedong Huang 40 45 0 03 May 2022
Where in the World is this Image? Transformer-based Geo-localization in the Wild Shraman Pramanick E. Nowara Joshua Gleason Carlos D. Castillo Rama Chellappa ViT 13 30 0 29 Apr 2022
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval Yuying Ge Yixiao Ge Xihui Liu Alex Jinpeng Wang Jianping Wu Ying Shan Xiaohu Qie Ping Luo VLM 13 43 0 26 Apr 2022
Are Multimodal Transformers Robust to Missing Modality? Mengmeng Ma Jian Ren Long Zhao Davide Testuggine Xi Peng ViT 26 147 0 12 Apr 2022
Representation Learning by Detecting Incorrect Location Embeddings Sepehr Sameni Simon Jenni Paolo Favaro ViT 29 4 0 10 Apr 2022
ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound Yan-Bo Lin Jie Lei Mohit Bansal Gedas Bertasius 33 39 0 06 Apr 2022
MultiMAE: Multi-modal Multi-task Masked Autoencoders Roman Bachmann David Mizrahi Andrei Atanov Amir Zamir 32 265 0 04 Apr 2022
Deformable Video Transformer Jue Wang Lorenzo Torresani ViT 22 28 0 31 Mar 2022
TubeDETR: Spatio-Temporal Video Grounding with Transformers Antoine Yang Antoine Miech Josef Sivic Ivan Laptev Cordelia Schmid ViT 28 94 0 30 Mar 2022
All in One: Exploring Unified Video-Language Pre-training Alex Jinpeng Wang Yixiao Ge Rui Yan Yuying Ge Xudong Lin Guanyu Cai Jianping Wu Ying Shan Xiaohu Qie Mike Zheng Shou 14 200 0 14 Mar 2022
Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videos Saghir Alfasly Jian Lu C. Xu Yuru Zou 34 18 0 06 Mar 2022
Audio Self-supervised Learning: A Survey Shuo Liu Adria Mallol-Ragolta Emilia Parada-Cabeleiro Kun Qian Xingshuo Jing Alexander Kathan Bin Hu Bjoern W. Schuller SSL 29 106 0 02 Mar 2022
VLP: A Survey on Vision-Language Pre-training Feilong Chen Duzhen Zhang Minglun Han Xiuyi Chen Jing Shi Shuang Xu Bo Xu VLM 82 212 0 18 Feb 2022
Visual Acoustic Matching Changan Chen Ruohan Gao P. Calamia Kristen Grauman 16 55 0 14 Feb 2022
Multimodal data matters: language model pre-training over structured and unstructured electronic health records Sicen Liu Xiaolong Wang Yongshuai Hou Ge Li Hui Wang Huiqin Xu Yang Xiang Buzhou Tang 52 30 0 25 Jan 2022
TriCoLo: Trimodal Contrastive Loss for Text to Shape Retrieval Yue Ruan Han-Hung Lee Yiming Zhang Ke Zhang Angel X. Chang 27 22 0 19 Jan 2022
Video Transformers: A Survey Javier Selva A. S. Johansen Sergio Escalera Kamal Nasrollahi T. Moeslund Albert Clapés ViT 22 103 0 16 Jan 2022
Bridging Video-text Retrieval with Multiple Choice Questions Yuying Ge Yixiao Ge Xihui Liu Dian Li Ying Shan Xiaohu Qie Ping Luo BDL 16 108 0 13 Jan 2022
Multiview Transformers for Video Recognition Shen Yan Xuehan Xiong Anurag Arnab Zhichao Lu Mi Zhang Chen Sun Cordelia Schmid ViT 24 211 0 12 Jan 2022
MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound Rowan Zellers Jiasen Lu Ximing Lu Youngjae Yu Yanpeng Zhao Mohammadreza Salehi Aditya Kusupati Jack Hessel Ali Farhadi Yejin Choi 26 207 0 07 Jan 2022
Co-training Transformer with Videos and Images Improves Action Recognition Bowen Zhang Jiahui Yu Christopher Fifty Wei Han Andrew M. Dai Ruoming Pang Fei Sha ViT 20 54 0 14 Dec 2021
Exploring Temporal Granularity in Self-Supervised Video Representation Learning Rui Qian Yeqing Li Liangzhe Yuan Boqing Gong Ting Liu Matthew A. Brown Serge J. Belongie Ming-Hsuan Yang Hartwig Adam Yin Cui AI4TS 41 6 0 08 Dec 2021
Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval Nina Shvetsova Brian Chen Andrew Rouditchenko Samuel Thomas Brian Kingsbury Rogerio Feris David F. Harwath James R. Glass Hilde Kuehne ViT 23 129 0 08 Dec 2021
BEVT: BERT Pretraining of Video Transformers Rui Wang Dongdong Chen Zuxuan Wu Yinpeng Chen Xiyang Dai Mengchen Liu Yu-Gang Jiang Luowei Zhou Lu Yuan ViT 36 203 0 02 Dec 2021
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks Xizhou Zhu Jinguo Zhu Hao Li Xiaoshi Wu Xiaogang Wang Hongsheng Li Xiaohua Wang Jifeng Dai 45 129 0 02 Dec 2021
SWAT: Spatial Structure Within and Among Tokens Kumara Kahatapitiya Michael S. Ryoo 23 6 0 26 Nov 2021
PolyViT: Co-training Vision Transformers on Images, Videos and Audio Valerii Likhosherstov Anurag Arnab K. Choromanski Mario Lucic Yi Tay Adrian Weller Mostafa Dehghani ViT 33 73 0 25 Nov 2021
V2C: Visual Voice Cloning Qi Chen Yuanqing Li Yuankai Qi Jiaqiu Zhou Mingkui Tan Qi Wu VGen 25 23 0 25 Nov 2021
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling Tsu-jui Fu Linjie Li Zhe Gan Kevin Qinghong Lin W. Wang Lijuan Wang Zicheng Liu VLM 36 216 0 24 Nov 2021
Florence: A New Foundation Model for Computer Vision Lu Yuan Dongdong Chen Yi-Ling Chen Noel Codella Xiyang Dai ... Zhen Xiao Jianwei Yang Michael Zeng Luowei Zhou Pengchuan Zhang VLM 24 878 0 22 Nov 2021
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning Jianfeng Wang Xiaowei Hu Zhe Gan Zhengyuan Yang Xiyang Dai Zicheng Liu Yumao Lu Lijuan Wang ViT 27 57 0 19 Nov 2021