PolyViT: Co-training Vision Transformers on Images, Videos and Audio

25 November 2021

Valerii Likhosherstov

Papers citing "PolyViT: Co-training Vision Transformers on Images, Videos and Audio"

50 / 76 papers shown

Title
Beyond Transfer Learning: Co-finetuning for Action Localisation Anurag Arnab Xuehan Xiong A. Gritsenko Rob Romijnders Josip Djolonga Mostafa Dehghani Chen Sun Mario Lucic Cordelia Schmid 69 8 0 08 Jul 2022
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks Jiasen Lu Christopher Clark Rowan Zellers Roozbeh Mottaghi Aniruddha Kembhavi ObjD VLM MLLM 134 405 0 17 Jun 2022
Simple Open-Vocabulary Object Detection with Vision Transformers Matthias Minderer A. Gritsenko Austin Stone Maxim Neumann Dirk Weissenborn ... Zhuoran Shen Tianlin Li Xiaohua Zhai Thomas Kipf N. Houlsby ObjD CLIP VLM ViT OCL 92 312 0 12 May 2022
The Efficiency Misnomer Daoyuan Chen Liuyi Yao Dawei Gao Ashish Vaswani Yaliang Li 91 103 0 25 Oct 2021
SCENIC: A JAX Library for Computer Vision Research and Beyond Mostafa Dehghani A. Gritsenko Anurag Arnab Matthias Minderer Yi Tay 87 68 0 18 Oct 2021
Exploring the Limits of Large Scale Pre-training Samira Abnar Mostafa Dehghani Behnam Neyshabur Hanie Sedghi AI4CE 91 118 0 05 Oct 2021
Perceiver IO: A General Architecture for Structured Inputs & Outputs Andrew Jaegle Sebastian Borgeaud Jean-Baptiste Alayrac Carl Doersch Catalin Ionescu ... Olivier J. Hénaff M. Botvinick Andrew Zisserman Oriol Vinyals João Carreira MLLM VLM GNN 64 579 0 30 Jul 2021
Attention Bottlenecks for Multimodal Fusion Arsha Nagrani Shan Yang Anurag Arnab A. Jansen Cordelia Schmid Chen Sun 98 565 0 30 Jun 2021
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers Mandela Patrick Dylan Campbell Yuki M. Asano Ishan Misra Ishan Misra Florian Metze Christoph Feichtenhofer Andrea Vedaldi João F. Henriques 80 279 0 09 Jun 2021
Multiscale Vision Transformers Haoqi Fan Bo Xiong K. Mangalam Yanghao Li Zhicheng Yan Jitendra Malik Christoph Feichtenhofer ViT 127 1,259 0 22 Apr 2021
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Huayu Chen Boqing Gong ViT 314 588 0 22 Apr 2021
AST: Audio Spectrogram Transformer Yuan Gong Yu-An Chung James R. Glass ViT 107 863 0 05 Apr 2021
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval Max Bain Arsha Nagrani Gül Varol Andrew Zisserman VGen 136 1,173 0 01 Apr 2021
ViViT: A Video Vision Transformer Anurag Arnab Mostafa Dehghani G. Heigold Chen Sun Mario Lucic Cordelia Schmid ViT 213 2,149 0 29 Mar 2021
Slow-Fast Auditory Streams For Audio Recognition Evangelos Kazakos Arsha Nagrani Andrew Zisserman Dima Damen 74 68 0 05 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 185 1,014 0 04 Mar 2021
UniT: Multimodal Multitask Learning with a Unified Transformer Ronghang Hu Amanpreet Singh ViT 76 300 0 22 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 365 2,048 0 09 Feb 2021
Parameter Efficient Multimodal Transformers for Video Representation Learning Sangho Lee Youngjae Yu Gunhee Kim Thomas Breuel Jan Kautz Yale Song ViT 94 78 0 08 Dec 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 634 41,003 0 22 Oct 2020
AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification Xiaofang Wang Xuehan Xiong Maxim Neumann A. Piergiovanni Michael S. Ryoo A. Angelova Kris Kitani Wei Hua 62 51 0 23 Jul 2020
Self-Supervised MultiModal Versatile Networks Jean-Baptiste Alayrac Adrià Recasens R. Schneider Relja Arandjelović Jason Ramapuram J. Fauw Lucas Smaira Sander Dieleman Andrew Zisserman SSL 121 373 0 29 Jun 2020
Bootstrap your own latent: A new approach to self-supervised Learning Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre Harvey Richemond ... M. G. Azar Bilal Piot Koray Kavukcuoglu Rémi Munos Michal Valko SSL 363 6,797 0 13 Jun 2020
Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data Haytham M. Fayek Anurag Kumar 60 36 0 29 May 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 743 41,932 0 28 May 2020
End-to-End Object Detection with Transformers Nicolas Carion Francisco Massa Gabriel Synnaeve Nicolas Usunier Alexander Kirillov Sergey Zagoruyko ViT 3DV PINN 385 13,035 0 26 May 2020
UnifiedQA: Crossing Format Boundaries With a Single QA System Daniel Khashabi Sewon Min Tushar Khot Ashish Sabharwal Oyvind Tafjord Peter Clark Hannaneh Hajishirzi 124 738 0 02 May 2020
VGGSound: A Large-scale Audio-Visual Dataset Honglie Chen Weidi Xie Andrea Vedaldi Andrew Zisserman 89 576 0 29 Apr 2020
X3D: Expanding Architectures for Efficient Video Recognition Christoph Feichtenhofer 128 1,019 0 09 Apr 2020
TEA: Temporal Excitation and Aggregation for Action Recognition Yan-Ran Li Bin Ji Xintian Shi Jianguo Zhang Bin Kang Limin Wang ViT 84 447 0 03 Apr 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 358 18,752 0 13 Feb 2020
Gradient Surgery for Multi-Task Learning Tianhe Yu Saurabh Kumar Abhishek Gupta Sergey Levine Karol Hausman Chelsea Finn 165 1,213 0 19 Jan 2020
End-to-End Learning of Visual Representations from Uncurated Instructional Videos Antoine Miech Jean-Baptiste Alayrac Lucas Smaira Ivan Laptev Josef Sivic Andrew Zisserman VGen SSL 114 711 0 13 Dec 2019
12-in-1: Multi-Task Vision and Language Representation Learning Jiasen Lu Vedanuj Goswami Marcus Rohrbach Devi Parikh Stefan Lee VLM ObjD 75 481 0 05 Dec 2019
More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation Quanfu Fan Chun-Fu Chen Hilde Kuehne Marco Pistoia David D. Cox 74 126 0 02 Dec 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 419 20,127 0 23 Oct 2019
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark Xiaohua Zhai J. Puigcerver Alexander Kolesnikov P. Ruyssen C. Riquelme ... Michael Tschannen Marcin Michalski Olivier Bousquet Sylvain Gelly N. Houlsby SSL 79 439 0 01 Oct 2019
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training Gen Li Nan Duan Yuejian Fang Ming Gong Daxin Jiang Ming Zhou SSL VLM MLLM 200 902 0 16 Aug 2019
STM: SpatioTemporal and Motion Encoding for Action Recognition Boyuan Jiang Mengmeng Wang Weihao Gan Wei Wu Junjie Yan 79 382 0 07 Aug 2019
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks Jiasen Lu Dhruv Batra Devi Parikh Stefan Lee SSL VLM 224 3,678 0 06 Aug 2019
Learning Spatio-Temporal Representation with Local and Global Diffusion Zhaofan Qiu Ting Yao Chong-Wah Ngo Xinmei Tian Tao Mei 54 171 0 13 Jun 2019
Video Modeling with Correlation Networks Heng Wang Du Tran Lorenzo Torresani Matt Feiszli 59 129 0 07 Jun 2019
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures Michael S. Ryoo A. Piergiovanni Mingxing Tan A. Angelova 52 102 0 30 May 2019
What Makes Training Multi-Modal Classification Networks Hard? Weiyao Wang Du Tran Matt Feiszli 105 451 0 29 May 2019
Attentive Single-Tasking of Multiple Tasks Kevis-Kokitsi Maninis Ilija Radosavovic Iasonas Kokkinos 172 249 0 18 Apr 2019
Video Classification with Channel-Separated Convolutional Networks Du Tran Heng Wang Lorenzo Torresani Matt Feiszli 3DV 61 587 0 04 Apr 2019
Parameter-Efficient Transfer Learning for NLP N. Houlsby A. Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin de Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly 210 4,451 0 02 Feb 2019
SlowFast Networks for Video Recognition Christoph Feichtenhofer Haoqi Fan Jitendra Malik Kaiming He 164 3,273 0 10 Dec 2018
TSM: Temporal Shift Module for Efficient Video Understanding Ji Lin Chuang Gan Song Han 98 1,689 0 20 Nov 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.7K 94,770 0 11 Oct 2018