Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,254 papers shown

Title
ConTNet: Why not use convolution and transformer at the same time? Haotian Yan Zhe Li Weijian Li Changhu Wang Ming Wu Chuang Zhang ViT 20 76 0 27 Apr 2021
Vision Transformers with Patch Diversification Chengyue Gong Dilin Wang Meng Li Vikas Chandra Qiang Liu ViT 42 62 0 26 Apr 2021
Visformer: The Vision-friendly Transformer Zhengsu Chen Lingxi Xie Jianwei Niu Xuefeng Liu Longhui Wei Qi Tian ViT 120 209 0 26 Apr 2021
Diverse Image Inpainting with Bidirectional and Autoregressive Transformers Yingchen Yu Fangneng Zhan Rongliang Wu Jianxiong Pan Kaiwen Cui Shijian Lu Feiying Ma Xuansong Xie C. Miao ViT 29 150 0 26 Apr 2021
Visual Saliency Transformer Nian Liu Ni Zhang Kaiyuan Wan Ling Shao Junwei Han ViT 253 352 0 25 Apr 2021
Multiscale Vision Transformers Haoqi Fan Bo Xiong K. Mangalam Yanghao Li Zhicheng Yan Jitendra Malik Christoph Feichtenhofer ViT 63 1,224 0 22 Apr 2021
All Tokens Matter: Token Labeling for Training Better Vision Transformers Zihang Jiang Qibin Hou Li-xin Yuan Daquan Zhou Yujun Shi Xiaojie Jin Anran Wang Jiashi Feng ViT 19 203 0 22 Apr 2021
Differentiable Model Compression via Pseudo Quantization Noise Alexandre Défossez Yossi Adi Gabriel Synnaeve DiffM MQ 18 47 0 20 Apr 2021
DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning Yuting Gao Jia-Xin Zhuang Xiaowei Guo Hao Cheng Xing Sun Ke Li Feiyue Huang 36 40 0 19 Apr 2021
Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation Rui Cheng Bichen Wu Peizhao Zhang Peter Vajda Joseph E. Gonzalez CLIP VLM 21 31 0 18 Apr 2021
Escaping the Big Data Paradigm with Compact Transformers Ali Hassani Steven Walton Nikhil Shah Abulikemu Abuduweili Jiachen Li Humphrey Shi 56 462 0 12 Apr 2021
SiT: Self-supervised vIsion Transformer Sara Atito Ali Ahmed Muhammad Awais J. Kittler ViT 36 139 0 08 Apr 2021
AST: Audio Spectrogram Transformer Yuan Gong Yu-An Chung James R. Glass ViT 26 830 0 05 Apr 2021
Deepfake Detection Scheme Based on Vision Transformer and Distillation Young-Jin Heo Y. Choi Young-Woon Lee Byung-Gyu Kim ViT 17 55 0 03 Apr 2021
Modular Adaptation for Cross-Domain Few-Shot Learning Xiaoyu Lin Meng Ye Yunye Gong G. Buracas Nikoletta Basiou Ajay Divakaran Yi Yao 23 4 0 01 Apr 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 27 986 0 31 Mar 2021
Rethinking Spatial Dimensions of Vision Transformers Byeongho Heo Sangdoo Yun Dongyoon Han Sanghyuk Chun Junsuk Choe Seong Joon Oh ViT 362 564 0 30 Mar 2021
CvT: Introducing Convolutions to Vision Transformers Haiping Wu Bin Xiao Noel Codella Mengchen Liu Xiyang Dai Lu Yuan Lei Zhang ViT 42 1,876 0 29 Mar 2021
ViViT: A Video Vision Transformer Anurag Arnab Mostafa Dehghani G. Heigold Chen Sun Mario Lucic Cordelia Schmid ViT 30 2,087 0 29 Mar 2021
Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers Hila Chefer Shir Gur Lior Wolf ViT 28 303 0 29 Mar 2021
On the Adversarial Robustness of Vision Transformers Rulin Shao Zhouxing Shi Jinfeng Yi Pin-Yu Chen Cho-Jui Hsieh ViT 33 137 0 29 Mar 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding Pengchuan Zhang Xiyang Dai Jianwei Yang Bin Xiao Lu Yuan Lei Zhang Jianfeng Gao ViT 29 329 0 29 Mar 2021
TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization Wei Gao Fang Wan Xingjia Pan Zhiliang Peng Qi Tian Zhenjun Han Bolei Zhou QiXiang Ye ViT WSOL 30 198 0 27 Mar 2021
Vision Transformers for Dense Prediction René Ranftl Alexey Bochkovskiy V. Koltun ViT MDE 45 1,662 0 24 Mar 2021
Multi-view 3D Reconstruction with Transformer Dan Wang Xinrui Cui Xun Chen Zhengxia Zou Tianyang Shi Septimiu Salcudean Z. J. Wang Rabab Ward ViT 22 87 0 24 Mar 2021
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search Changlin Li Tao Tang Guangrun Wang Jiefeng Peng Bing Wang Xiaodan Liang Xiaojun Chang ViT 46 105 0 23 Mar 2021
DeepViT: Towards Deeper Vision Transformer Daquan Zhou Bingyi Kang Xiaojie Jin Linjie Yang Xiaochen Lian Zihang Jiang Qibin Hou Jiashi Feng ViT 42 510 0 22 Mar 2021
Incorporating Convolution Designs into Visual Transformers Kun Yuan Shaopeng Guo Ziwei Liu Aojun Zhou F. Yu Wei Wu ViT 56 467 0 22 Mar 2021
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases Stéphane dÁscoli Hugo Touvron Matthew L. Leavitt Ari S. Morcos Giulio Biroli Levent Sagun ViT 58 805 0 19 Mar 2021
Scalable Vision Transformers with Hierarchical Pooling Zizheng Pan Bohan Zhuang Jing Liu Haoyu He Jianfei Cai ViT 27 126 0 19 Mar 2021
UNETR: Transformers for 3D Medical Image Segmentation Ali Hatamizadeh Yucheng Tang Vishwesh Nath Dong Yang Andriy Myronenko Bennett Landman H. Roth Daguang Xu ViT MedIm 80 1,535 0 18 Mar 2021
3D Human Pose Estimation with Spatial and Temporal Transformers Ce Zheng Sijie Zhu Matías Mendieta Taojiannan Yang Cheng Chen Zhengming Ding ViT 47 438 0 18 Mar 2021
TransFG: A Transformer Architecture for Fine-grained Recognition Ju He Jieneng Chen Shuai Liu Adam Kortylewski Cheng Yang Yutong Bai Changhu Wang ViT 37 375 0 14 Mar 2021
Unknown Object Segmentation from Stereo Images M. Durner W. Boerdijk M. Sundermeyer W. Friedl Zoltán-Csaba Márton Rudolph Triebel 34 34 0 11 Mar 2021
TransMed: Transformers Advance Multi-modal Medical Image Classification Yin Dai Yifan Gao ViT MedIm 38 280 0 10 Mar 2021
Pretrained Transformers as Universal Computation Engines Kevin Lu Aditya Grover Pieter Abbeel Igor Mordatch 28 217 0 09 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 80 973 0 04 Mar 2021
Transformer in Transformer Kai Han An Xiao Enhua Wu Jianyuan Guo Chunjing Xu Yunhe Wang ViT 289 1,524 0 27 Feb 2021
On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness Eric Mintun A. Kirillov Saining Xie 20 90 0 22 Feb 2021
LambdaNetworks: Modeling Long-Range Interactions Without Attention Irwan Bello 272 179 0 17 Feb 2021
AlphaNet: Improved Training of Supernets with Alpha-Divergence Dilin Wang Chengyue Gong Meng Li Qiang Liu Vikas Chandra 155 44 0 16 Feb 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 223 512 0 11 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 280 1,982 0 09 Feb 2021
Bottleneck Transformers for Visual Recognition A. Srinivas Nayeon Lee Niki Parmar Jonathon Shlens Pieter Abbeel Ashish Vaswani SLR 290 979 0 27 Jan 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,430 0 04 Jan 2021
Concept Generalization in Visual Representation Learning Mert Bulent Sariyildiz Yannis Kalantidis Diane Larlus Alahari Karteek SSL 28 50 0 10 Dec 2020
IMAGO: A family photo album dataset for a socio-historical analysis of the twentieth century L. Stacchio Alessia Angeli G. Lisanti Daniela Calanca Gustavo Marfia 31 2 0 03 Dec 2020
Contextualizing Enhances Gradient Based Meta Learning Evan Vogelbaum Rumen Dangovski L. Jing Marin Soljacic 34 3 0 17 Jul 2020
Fixing the train-test resolution discrepancy: FixEfficientNet Hugo Touvron Andrea Vedaldi Matthijs Douze Hervé Jégou AAML 196 110 0 18 Mar 2020
A Survey on 3D Skeleton-Based Action Recognition Using Learning Method Bin Ren Mengyuan Liu Runwei Ding Hong Liu 27 121 0 14 Feb 2020