Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,403 papers shown

Title
SiT: Self-supervised vIsion Transformer Sara Atito Ali Ahmed Muhammad Awais J. Kittler ViT 39 139 0 08 Apr 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 75 1,819 0 05 Apr 2021
AST: Audio Spectrogram Transformer Yuan Gong Yu-An Chung James R. Glass ViT 58 835 0 05 Apr 2021
Deepfake Detection Scheme Based on Vision Transformer and Distillation Young-Jin Heo Y. Choi Young-Woon Lee Byung-Gyu Kim ViT 17 55 0 03 Apr 2021
Keyword Transformer: A Self-Attention Model for Keyword Spotting Axel Berg Mark O'Connor M. T. Cruz 27 133 0 01 Apr 2021
Modular Adaptation for Cross-Domain Few-Shot Learning Xiaoyu Lin Meng Ye Yunye Gong G. Buracas Nikoletta Basiou Ajay Divakaran Yi Yao 26 4 0 01 Apr 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 27 988 0 31 Mar 2021
Rethinking Spatial Dimensions of Vision Transformers Byeongho Heo Sangdoo Yun Dongyoon Han Sanghyuk Chun Junsuk Choe Seong Joon Oh ViT 380 568 0 30 Mar 2021
CvT: Introducing Convolutions to Vision Transformers Haiping Wu Bin Xiao Noel Codella Mengchen Liu Xiyang Dai Lu Yuan Lei Zhang ViT 81 1,878 0 29 Mar 2021
ViViT: A Video Vision Transformer Anurag Arnab Mostafa Dehghani G. Heigold Chen Sun Mario Lucic Cordelia Schmid ViT 30 2,093 0 29 Mar 2021
Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers Hila Chefer Shir Gur Lior Wolf ViT 31 302 0 29 Mar 2021
On the Adversarial Robustness of Vision Transformers Rulin Shao Zhouxing Shi Jinfeng Yi Pin-Yu Chen Cho-Jui Hsieh ViT 33 138 0 29 Mar 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding Pengchuan Zhang Xiyang Dai Jianwei Yang Bin Xiao Lu Yuan Lei Zhang Jianfeng Gao ViT 29 330 0 29 Mar 2021
TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization Wei Gao Fang Wan Xingjia Pan Zhiliang Peng Qi Tian Zhenjun Han Bolei Zhou QiXiang Ye ViT WSOL 32 198 0 27 Mar 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng-Wei Zhang Stephen Lin B. Guo ViT 151 20,774 0 25 Mar 2021
Vision Transformers for Dense Prediction René Ranftl Alexey Bochkovskiy V. Koltun ViT MDE 45 1,667 0 24 Mar 2021
Multi-view 3D Reconstruction with Transformer Dan Wang Xinrui Cui Xun Chen Zhengxia Zou Tianyang Shi Septimiu Salcudean Z. J. Wang Rabab Ward ViT 22 87 0 24 Mar 2021
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search Changlin Li Tao Tang Guangrun Wang Jiefeng Peng Bing Wang Xiaodan Liang Xiaojun Chang ViT 48 105 0 23 Mar 2021
DeepViT: Towards Deeper Vision Transformer Daquan Zhou Bingyi Kang Xiaojie Jin Linjie Yang Xiaochen Lian Zihang Jiang Qibin Hou Jiashi Feng ViT 42 510 0 22 Mar 2021
Incorporating Convolution Designs into Visual Transformers Kun Yuan Shaopeng Guo Ziwei Liu Aojun Zhou F. Yu Wei Wu ViT 56 467 0 22 Mar 2021
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases Stéphane dÁscoli Hugo Touvron Matthew L. Leavitt Ari S. Morcos Giulio Biroli Levent Sagun ViT 58 807 0 19 Mar 2021
Scalable Vision Transformers with Hierarchical Pooling Zizheng Pan Bohan Zhuang Jing Liu Haoyu He Jianfei Cai ViT 27 126 0 19 Mar 2021
UNETR: Transformers for 3D Medical Image Segmentation Ali Hatamizadeh Yucheng Tang Vishwesh Nath Dong Yang Andriy Myronenko Bennett Landman H. Roth Daguang Xu ViT MedIm 95 1,538 0 18 Mar 2021
3D Human Pose Estimation with Spatial and Temporal Transformers Ce Zheng Sijie Zhu Matías Mendieta Taojiannan Yang Cen Chen Zhengming Ding ViT 47 438 0 18 Mar 2021
TransFG: A Transformer Architecture for Fine-grained Recognition Ju He Jieneng Chen Shuai Liu Adam Kortylewski Cheng Yang Yutong Bai Changhu Wang ViT 37 376 0 14 Mar 2021
Unknown Object Segmentation from Stereo Images M. Durner W. Boerdijk M. Sundermeyer W. Friedl Zoltán-Csaba Márton Rudolph Triebel 34 34 0 11 Mar 2021
TransMed: Transformers Advance Multi-modal Medical Image Classification Yin Dai Yifan Gao ViT MedIm 38 281 0 10 Mar 2021
Pretrained Transformers as Universal Computation Engines Kevin Lu Aditya Grover Pieter Abbeel Igor Mordatch 28 217 0 09 Mar 2021
TransBTS: Multimodal Brain Tumor Segmentation Using Transformer Wenxuan Wang Chen Chen Meng Ding Jiangyun Li Hong Yu Sen Zha ViT MedIm 10 701 0 07 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 91 977 0 04 Mar 2021
Transformer in Transformer Kai Han An Xiao Enhua Wu Jianyuan Guo Chunjing Xu Yunhe Wang ViT 319 1,525 0 27 Feb 2021
On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness Eric Mintun A. Kirillov Saining Xie 20 90 0 22 Feb 2021
UniT: Multimodal Multitask Learning with a Unified Transformer Ronghang Hu Amanpreet Singh ViT 25 296 0 22 Feb 2021
LambdaNetworks: Modeling Long-Range Interactions Without Attention Irwan Bello 281 179 0 17 Feb 2021
TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation Yundong Zhang Huiye Liu Qiang Hu ViT MedIm 206 894 0 16 Feb 2021
AlphaNet: Improved Training of Supernets with Alpha-Divergence Dilin Wang Chengyue Gong Meng Li Qiang Liu Vikas Chandra 157 44 0 16 Feb 2021
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up Yi Ding Shiyu Chang Zhangyang Wang ViT 29 383 0 14 Feb 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 226 513 0 11 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 283 1,989 0 09 Feb 2021
Bottleneck Transformers for Visual Recognition A. Srinivas Nayeon Lee Niki Parmar Jonathon Shlens Pieter Abbeel Ashish Vaswani SLR 290 980 0 27 Jan 2021
A Review on Deep Learning in UAV Remote Sensing L. Osco J. M. Junior A. P. Ramos L. Jorge S. Fatholahi J. Silva E. Matsubara H. Pistori W. Gonçalves Jonathan Li AI4Cl 39 342 0 22 Jan 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir Fahad Shahbaz Khan M. Shah ViT 227 2,434 0 04 Jan 2021
TransPose: Keypoint Localization via Transformer Sen Yang Zhibin Quan Mu Nie Wankou Yang ViT 143 263 0 28 Dec 2020
Concept Generalization in Visual Representation Learning Mert Bulent Sariyildiz Yannis Kalantidis Diane Larlus Alahari Karteek SSL 35 50 0 10 Dec 2020
IMAGO: A family photo album dataset for a socio-historical analysis of the twentieth century L. Stacchio Alessia Angeli G. Lisanti Daniela Calanca Gustavo Marfia 31 3 0 03 Dec 2020
Contextualizing Enhances Gradient Based Meta Learning Evan Vogelbaum Rumen Dangovski L. Jing Marin Soljacic 34 3 0 17 Jul 2020
Multi-Head Attention: Collaborate Instead of Concatenate Jean-Baptiste Cordonnier Andreas Loukas Martin Jaggi 6 108 0 29 Jun 2020
Fixing the train-test resolution discrepancy: FixEfficientNet Hugo Touvron Andrea Vedaldi Matthijs Douze Hervé Jégou AAML 196 110 0 18 Mar 2020
A Survey on 3D Skeleton-Based Action Recognition Using Learning Method Bin Ren Mengyuan Liu Runwei Ding Hong Liu 27 121 0 14 Feb 2020
Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey Jérôme Rony Soufiane Belharbi Jose Dolz Ismail Ben Ayed Luke McCaffrey Eric Granger 34 70 0 08 Sep 2019