Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,171 papers shown

Title
Mesa: A Memory-saving Training Framework for Transformers Zizheng Pan Peng Chen Haoyu He Jing Liu Jianfei Cai Bohan Zhuang 23 20 0 22 Nov 2021
Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism Ihor Vasyltsov Wooseok Chang 25 12 0 21 Nov 2021
Are Vision Transformers Robust to Patch Perturbations? Jindong Gu Volker Tresp Yao Qin AAML ViT 35 60 0 20 Nov 2021
Discrete Representations Strengthen Vision Transformer Robustness Chengzhi Mao Lu Jiang Mostafa Dehghani Carl Vondrick Rahul Sukthankar Irfan Essa ViT 27 43 0 20 Nov 2021
FBNetV5: Neural Architecture Search for Multiple Tasks in One Run Bichen Wu Chaojian Li Hang Zhang Xiaoliang Dai Peizhao Zhang Matthew Yu Jialiang Wang Yingyan Lin Peter Vajda ViT 27 23 0 19 Nov 2021
SimMIM: A Simple Framework for Masked Image Modeling Zhenda Xie Zheng-Wei Zhang Yue Cao Yutong Lin Jianmin Bao Zhuliang Yao Qi Dai Han Hu 42 1,309 0 18 Nov 2021
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng-Wei Zhang Li Dong Furu Wei B. Guo ViT 52 1,747 0 18 Nov 2021
Restormer: Efficient Transformer for High-Resolution Image Restoration Syed Waqas Zamir Aditya Arora Salman Khan Munawar Hayat F. Khan Ming-Hsuan Yang ViT 49 2,123 0 18 Nov 2021
TransMix: Attend to Mix for Vision Transformers Jieneng Chen Shuyang Sun Ju He Philip H. S. Torr Alan Yuille S. Bai ViT 25 103 0 18 Nov 2021
Evaluating Transformers for Lightweight Action Recognition Raivo Koot Markus Hennerbichler Haiping Lu ViT 28 8 0 18 Nov 2021
Dynamically pruning segformer for efficient semantic segmentation Haoli Bai Hongda Mao D. Nair 28 20 0 18 Nov 2021
Achieving Human Parity on Visual Question Answering Ming Yan Haiyang Xu Chenliang Li Junfeng Tian Bin Bi ... Ji Zhang Songfang Huang Fei Huang Luo Si Rong Jin 24 12 0 17 Nov 2021
TRIG: Transformer-Based Text Recognizer with Initial Embedding Guidance Yuefeng Tao Zhiwei Jia Runze Ma Shugong Xu ViT 19 6 0 16 Nov 2021
iBOT: Image BERT Pre-Training with Online Tokenizer Jinghao Zhou Chen Wei Huiyu Wang Wei Shen Cihang Xie Alan Yuille Tao Kong 21 710 0 15 Nov 2021
Full-attention based Neural Architecture Search using Context Auto-regression Yuan Zhou Haiyang Wang Shuwei Huo Boyu Wang 27 3 0 13 Nov 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 305 7,443 0 11 Nov 2021
A Survey of Visual Transformers Yang Liu Yao Zhang Yixin Wang Feng Hou Jin Yuan Jiang Tian Yang Zhang Zhongchao Shi Jianping Fan Zhiqiang He 3DGS ViT 71 330 0 11 Nov 2021
Are Transformers More Robust Than CNNs? Yutong Bai Jieru Mei Alan Yuille Cihang Xie ViT AAML 192 257 0 10 Nov 2021
FILIP: Fine-grained Interactive Language-Image Pre-Training Lewei Yao Runhu Huang Lu Hou Guansong Lu Minzhe Niu Hang Xu Xiaodan Liang Zhenguo Li Xin Jiang Chunjing Xu VLM CLIP 30 614 0 09 Nov 2021
Are we ready for a new paradigm shift? A Survey on Visual Deep MLP Ruiyang Liu Yinghui Li Li Tao Dun Liang Haitao Zheng 85 97 0 07 Nov 2021
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling Renrui Zhang Rongyao Fang Wei Zhang Peng Gao Kunchang Li Jifeng Dai Yu Qiao Hongsheng Li VLM 192 385 0 06 Nov 2021
Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems Victoria Mingote A. Miguel A. O. Giménez EDUARDO LLEIDA SOLANO 31 10 0 06 Nov 2021
Can Vision Transformers Perform Convolution? Shanda Li Xiangning Chen Di He Cho-Jui Hsieh ViT 43 19 0 02 Nov 2021
Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation Jiaqi Gu Hyoukjun Kwon Dilin Wang Wei Ye Meng Li Yu-Hsin Chen Liangzhen Lai Vikas Chandra David Z. Pan ViT 24 182 0 01 Nov 2021
Projected GANs Converge Faster Axel Sauer Kashyap Chitta Jens Muller Andreas Geiger 47 234 0 01 Nov 2021
Blending Anti-Aliasing into Vision Transformer Shengju Qian Hao Shao Yi Zhu Mu Li Jiaya Jia 26 20 0 28 Oct 2021
MVT: Multi-view Vision Transformer for 3D Object Recognition Shuo Chen Tan Yu Ping Li ViT 37 43 0 25 Oct 2021
SOFT: Softmax-free Transformer with Linear Complexity Jiachen Lu Jinghan Yao Junge Zhang Martin Danelljan Hang Xu Weiguo Gao Chunjing Xu Thomas B. Schon Li Zhang 18 161 0 22 Oct 2021
Grafting Transformer on Automatically Designed Convolutional Neural Network for Hyperspectral Image Classification Xizhe Xue Haokui Zhang Bei Fang Zongwen Bai Ying Li ViT 11 22 0 21 Oct 2021
Vis-TOP: Visual Transformer Overlay Processor Wei Hu Dian Xu Zimeng Fan Fang Liu Yanxiang He BDL ViT 20 5 0 21 Oct 2021
No One Representation to Rule Them All: Overlapping Features of Training Methods Raphael Gontijo-Lopes Yann N. Dauphin E. D. Cubuk 20 60 0 20 Oct 2021
SSAST: Self-Supervised Audio Spectrogram Transformer Yuan Gong Cheng-I Jeff Lai Yu-An Chung James R. Glass ViT 38 268 0 19 Oct 2021
TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation Haoyu Ma Liangjian Chen Deying Kong Zhe Wang Xingwei Liu Hao Tang Xiangyi Yan Yusheng Xie Shi-yao Lin Xiaohui Xie ViT 19 61 0 18 Oct 2021
HRFormer: High-Resolution Transformer for Dense Prediction Yuhui Yuan Rao Fu Lang Huang Weihong Lin Chao Zhang Xilin Chen Jingdong Wang ViT 38 227 0 18 Oct 2021
ASFormer: Transformer for Action Segmentation Fangqiu Yi Hongyu Wen Tingting Jiang ViT 76 172 0 16 Oct 2021
Detecting Gender Bias in Transformer-based Models: A Case Study on BERT Bingbing Li Hongwu Peng Rajat Sainju Junhuan Yang Lei Yang Yueying Liang Weiwen Jiang Binghui Wang Hang Liu Caiwen Ding 15 11 0 15 Oct 2021
Self-Supervised Learning by Estimating Twin Class Distributions Feng Wang Tao Kong Rufeng Zhang Huaping Liu Hang Li SSL 55 16 0 14 Oct 2021
Object-Region Video Transformers Roei Herzig Elad Ben-Avraham K. Mangalam Amir Bar Gal Chechik Anna Rohrbach Trevor Darrell Amir Globerson ViT 24 82 0 13 Oct 2021
Dynamic Inference with Neural Interpreters Nasim Rahaman Muhammad Waleed Gondal S. Joshi Peter V. Gehler Yoshua Bengio Francesco Locatello Bernhard Schölkopf 34 31 0 12 Oct 2021
Global Vision Transformer Pruning with Hessian-Aware Saliency Huanrui Yang Hongxu Yin Maying Shen Pavlo Molchanov Hai Helen Li Jan Kautz ViT 30 39 0 10 Oct 2021
CLIP-Adapter: Better Vision-Language Models with Feature Adapters Peng Gao Shijie Geng Renrui Zhang Teli Ma Rongyao Fang Yongfeng Zhang Hongsheng Li Yu Qiao VLM CLIP 71 982 0 09 Oct 2021
Adversarial Token Attacks on Vision Transformers Ameya Joshi Gauri Jagatap C. Hegde ViT 30 19 0 08 Oct 2021
Context-LGM: Leveraging Object-Context Relation for Context-Aware Object Recognition Mingzhou Liu Xinwei Sun Fandong Zhang Yizhou Yu Yizhou Wang 27 0 0 08 Oct 2021
UniNet: Unified Architecture Search with Convolution, Transformer, and MLP Jihao Liu Hongsheng Li Guanglu Song Xin Huang Yu Liu ViT 37 35 0 08 Oct 2021
ViDT: An Efficient and Effective Fully Transformer-based Object Detector Hwanjun Song Deqing Sun Sanghyuk Chun Varun Jampani Dongyoon Han Byeongho Heo Wonjae Kim Ming-Hsuan Yang 87 76 0 08 Oct 2021
Token Pooling in Vision Transformers D. Marin Jen-Hao Rick Chang Anurag Ranjan Anish K. Prabhu Mohammad Rastegari Oncel Tuzel ViT 76 66 0 08 Oct 2021
ATISS: Autoregressive Transformers for Indoor Scene Synthesis Despoina Paschalidou Amlan Kar Maria Shugrina Karsten Kreis Andreas Geiger Sanja Fidler 3DV ViT 33 148 0 07 Oct 2021
Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs Philipp Benz Soomin Ham Chaoning Zhang Adil Karjauv In So Kweon AAML ViT 47 78 0 06 Oct 2021
Ripple Attention for Visual Perception with Sub-quadratic Complexity Lin Zheng Huijie Pan Lingpeng Kong 26 3 0 06 Oct 2021
Spectral Bias in Practice: The Role of Function Frequency in Generalization Sara Fridovich-Keil Raphael Gontijo-Lopes Rebecca Roelofs 38 28 0 06 Oct 2021