Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,402 papers shown

Title
Scaling Vision Transformers Xiaohua Zhai Alexander Kolesnikov N. Houlsby Lucas Beyer ViT 85 1,063 0 08 Jun 2021
MVT: Mask Vision Transformer for Facial Expression Recognition in the wild Hanting Li Ming-Fa Sui Feng Zhao Zhengjun Zha Feng Wu ViT 37 76 0 08 Jun 2021
On the Connection between Local Attention and Dynamic Depth-wise Convolution Qi Han Zejia Fan Qi Dai Lei-huan Sun Ming-Ming Cheng Jiaying Liu Jingdong Wang ViT 29 105 0 08 Jun 2021
Fully Transformer Networks for Semantic Image Segmentation Sitong Wu Tianyi Wu Fangjian Lin Sheng Tian Guodong Guo ViT 34 39 0 08 Jun 2021
Efficient Training of Visual Transformers with Small Datasets Yahui Liu E. Sangineto Wei Bi N. Sebe Bruno Lepri Marco De Nadai ViT 36 167 0 07 Jun 2021
Person Re-Identification with a Locally Aware Transformer Charu Sharma S. R. Kapil David Chapman ViT 48 45 0 07 Jun 2021
Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer Zilong Huang Youcheng Ben Guozhong Luo Pei Cheng Gang Yu Bin-Bin Fu ViT 21 182 0 07 Jun 2021
Self-supervised Depth Estimation Leveraging Global Perception and Geometric Smoothness Using On-board Videos Shaocheng Jia Xin Pei W. Yao S. Wong 3DPC MDE 48 19 0 07 Jun 2021
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias Yufei Xu Qiming Zhang Jing Zhang Dacheng Tao ViT 65 330 0 07 Jun 2021
Rethinking Training from Scratch for Object Detection Yang Li Hong Zhang Yu Zhang VLM OnRL ObjD 25 5 0 06 Jun 2021
Patch Slimming for Efficient Vision Transformers Yehui Tang Kai Han Yunhe Wang Chang Xu Jianyuan Guo Chao Xu Dacheng Tao ViT 26 163 0 05 Jun 2021
CATs: Cost Aggregation Transformers for Visual Correspondence Seokju Cho Sunghwan Hong Sangryul Jeon Yunsung Lee Kwanghoon Sohn Seungryong Kim ViT 26 86 0 04 Jun 2021
Few-Shot Segmentation via Cycle-Consistent Transformer Gengwei Zhang Guoliang Kang Yi Yang Yunchao Wei ViT 19 179 0 04 Jun 2021
Scalable Transformers for Neural Machine Translation Peng Gao Shijie Geng Ping Luo Xiaogang Wang Jifeng Dai Hongsheng Li 31 13 0 04 Jun 2021
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification Yongming Rao Wenliang Zhao Benlin Liu Jiwen Lu Jie Zhou Cho-Jui Hsieh ViT 34 670 0 03 Jun 2021
Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model Jiangning Zhang Chao Xu Jian Li Wenzhou Chen Yabiao Wang Ying Tai Shuo Chen Chengjie Wang Feiyue Huang Yong Liu 40 22 0 31 May 2021
Dual-stream Network for Visual Recognition Mingyuan Mao Renrui Zhang Honghui Zheng Peng Gao Teli Ma Yan Peng Errui Ding Baochang Zhang Shumin Han ViT 25 63 0 31 May 2021
Gaze Estimation using Transformer Yihua Cheng Feng Lu ViT 21 87 0 30 May 2021
FoveaTer: Foveated Transformer for Image Classification Aditya Jonnalagadda Wei Wang B. S. Manjunath Miguel P. Eckstein ViT 38 23 0 29 May 2021
What Is Considered Complete for Visual Recognition? Lingxi Xie Xiaopeng Zhang Longhui Wei Jianlong Chang Qi Tian VLM 23 4 0 28 May 2021
ResT: An Efficient Transformer for Visual Recognition Qing-Long Zhang Yubin Yang ViT 29 229 0 28 May 2021
KVT: k-NN Attention for Boosting Vision Transformers Pichao Wang Xue Wang F. Wang Ming Lin Shuning Chang Hao Li Rong Jin ViT 51 105 0 28 May 2021
Intriguing Properties of Vision Transformers Muzammal Naseer Kanchana Ranasinghe Salman Khan Munawar Hayat Fahad Shahbaz Khan Ming-Hsuan Yang ViT 265 625 0 21 May 2021
Vision Transformer for Fast and Efficient Scene Text Recognition Rowel Atienza ViT 25 144 0 18 May 2021
Pay Attention to MLPs Hanxiao Liu Zihang Dai David R. So Quoc V. Le AI4CE 57 652 0 17 May 2021
Vision Transformers are Robust Learners Sayak Paul Pin-Yu Chen ViT 28 308 0 17 May 2021
Is the aspect ratio of cells important in deep learning? A robust comparison of deep learning methods for multi-scale cytopathology cell image classification: from convolutional neural networks to visual transformers Wanli Liu Chen Li M. Rahaman Tao Jiang Hongzan Sun ... Weiming Hu Hao Chen Changhao Sun Yudong Yao M. Grzegorzek 41 54 0 16 May 2021
Waste detection in Pomerania: non-profit project for detecting waste in environment Sylwia Majchrowska Agnieszka Mikołajczyk M. Ferlin Zuzanna Klawikowska Marta A. Plantykow Arkadiusz Kwasigroch K. Majek 30 125 0 12 May 2021
Self-Supervised Learning with Swin Transformers Zhenda Xie Yutong Lin Zhuliang Yao Zheng-Wei Zhang Qi Dai Yue Cao Han Hu ViT 34 178 0 10 May 2021
Conformer: Local Features Coupling Global Representations for Visual Recognition Zhiliang Peng Wei Huang Shanzhi Gu Lingxi Xie Yaowei Wang Jianbin Jiao QiXiang Ye ViT 21 528 0 09 May 2021
ResMLP: Feedforward networks for image classification with data-efficient training Hugo Touvron Piotr Bojanowski Mathilde Caron Matthieu Cord Alaaeldin El-Nouby ... Gautier Izacard Armand Joulin Gabriel Synnaeve Jakob Verbeek Hervé Jégou VLM 36 656 0 07 May 2021
Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet Luke Melas-Kyriazi ViT 9 102 0 06 May 2021
Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks Meng-Hao Guo Zheng-Ning Liu Tai-Jiang Mu Shimin Hu 28 473 0 05 May 2021
Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries Qingyi Dong Zhuowen Tu Haofu Liao Yuting Zhang Vijay Mahadevan Stefano Soatto ViT 19 38 0 05 May 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 292 2,611 0 04 May 2021
CAT: Cross-Attention Transformer for One-Shot Object Detection Weidong Lin Yuyang Deng Yang Gao Ning Wang Jinghao Zhou Lingqiao Liu Lei Zhang Peng Wang ViT 27 9 0 30 Apr 2021
Perceptual Image Quality Assessment with Transformers Manri Cheon Sung-Jun Yoon Byungyeon Kang Junwoo Lee ViT 28 111 0 30 Apr 2021
Pyramid Medical Transformer for Medical Image Segmentation Zhuangzhuang Zhang Weixiong Zhang ViT MedIm 16 46 0 29 Apr 2021
ConTNet: Why not use convolution and transformer at the same time? Haotian Yan Zhe Li Weijian Li Changhu Wang Ming Wu Chuang Zhang ViT 20 76 0 27 Apr 2021
Vision Transformers with Patch Diversification Chengyue Gong Dilin Wang Meng Li Vikas Chandra Qiang Liu ViT 45 62 0 26 Apr 2021
Visformer: The Vision-friendly Transformer Zhengsu Chen Lingxi Xie Jianwei Niu Xuefeng Liu Longhui Wei Qi Tian ViT 120 209 0 26 Apr 2021
Diverse Image Inpainting with Bidirectional and Autoregressive Transformers Yingchen Yu Fangneng Zhan Rongliang Wu Jianxiong Pan Kaiwen Cui Shijian Lu Feiying Ma Xuansong Xie Chunyan Miao ViT 45 150 0 26 Apr 2021
Visual Saliency Transformer Nian Liu Ni Zhang Kaiyuan Wan Ling Shao Junwei Han ViT 261 352 0 25 Apr 2021
VidTr: Video Transformer Without Convolutions Yanyi Zhang Xinyu Li Chunhui Liu Bing Shuai Yi Zhu Biagio Brattoli Hao Chen I. Marsic Joseph Tighe ViT 148 193 0 23 Apr 2021
Multiscale Vision Transformers Haoqi Fan Bo Xiong K. Mangalam Yanghao Li Zhicheng Yan Jitendra Malik Christoph Feichtenhofer ViT 63 1,224 0 22 Apr 2021
All Tokens Matter: Token Labeling for Training Better Vision Transformers Zihang Jiang Qibin Hou Li-xin Yuan Daquan Zhou Yujun Shi Xiaojie Jin Anran Wang Jiashi Feng ViT 27 203 0 22 Apr 2021
Differentiable Model Compression via Pseudo Quantization Noise Alexandre Défossez Yossi Adi Gabriel Synnaeve DiffM MQ 18 47 0 20 Apr 2021
DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning Yuting Gao Jia-Xin Zhuang Xiaowei Guo Hao Cheng Xing Sun Ke Li Feiyue Huang 54 40 0 19 Apr 2021
Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation Rui Cheng Bichen Wu Peizhao Zhang Peter Vajda Joseph E. Gonzalez CLIP VLM 21 31 0 18 Apr 2021
Escaping the Big Data Paradigm with Compact Transformers Ali Hassani Steven Walton Nikhil Shah Abulikemu Abuduweili Jiachen Li Humphrey Shi 74 462 0 12 Apr 2021