Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,164 papers shown

Title
Improving 360 Monocular Depth Estimation via Non-local Dense Prediction Transformer and Joint Supervised and Self-supervised Learning I. Yun Hyuk-Jae Lee Chae-Eun Rhee ViT MDE 11 27 0 22 Sep 2021
F-CAM: Full Resolution Class Activation Maps via Guided Parametric Upscaling Soufiane Belharbi Aydin Sarraf M. Pedersoli Ismail Ben Ayed Luke McCaffrey Eric Granger WSOL 36 30 0 15 Sep 2021
Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection N. Islam S. Gehlot Zongwei Zhou Michael B. Gotway Jianming Liang OOD 85 11 0 15 Sep 2021
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation Tongkun Xu Weihua Chen Pichao Wang Fan Wang Hao Li R. L. Jin ViT 56 215 0 13 Sep 2021
Compute and Energy Consumption Trends in Deep Learning Inference Radosvet Desislavov Fernando Martínez-Plumed José Hernández Orallo 35 113 0 12 Sep 2021
Sparse MLP for Image Recognition: Is Self-Attention Really Necessary? Chuanxin Tang Yucheng Zhao Guangting Wang Chong Luo Wenxuan Xie Wenjun Zeng MoE ViT 35 98 0 12 Sep 2021
LibFewShot: A Comprehensive Library for Few-shot Learning Wenbin Li Ziyi Ziyi Wang Xuesong Yang C. Dong ... Jing Huo Yinghuan Shi Lei Wang Yang Gao Jiebo Luo VLM 113 66 0 10 Sep 2021
Learning the Physics of Particle Transport via Transformers O. Pastor-Serrano Zoltán Perkó MedIm 21 13 0 08 Sep 2021
Scaled ReLU Matters for Training Vision Transformers Pichao Wang Xue Wang Haowen Luo Jingkai Zhou Zhipeng Zhou Fan Wang Hao Li R. L. Jin 19 41 0 08 Sep 2021
Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimation Ziniu Wan Zhengjia Li Maoqing Tian Jianbo Liu Shuai Yi Hongsheng Li 3DH 35 80 0 06 Sep 2021
Hire-MLP: Vision MLP via Hierarchical Rearrangement Jianyuan Guo Yehui Tang Kai Han Xinghao Chen Han Wu Chao Xu Chang Xu Yunhe Wang 46 105 0 30 Aug 2021
A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP Yucheng Zhao Guangting Wang Chuanxin Tang Chong Luo Wenjun Zeng Zhengjun Zha 33 69 0 30 Aug 2021
Design and Scaffolded Training of an Efficient DNN Operator for Computer Vision on the Edge Vinod Ganesan Pratyush Kumar 34 2 0 25 Aug 2021
TransFER: Learning Relation-aware Facial Expression Representations with Transformers Fanglei Xue Qiangchang Wang G. Guo ViT 39 183 0 25 Aug 2021
SwinIR: Image Restoration Using Swin Transformer Jingyun Liang Jie Cao Guolei Sun K. Zhang Luc Van Gool Radu Timofte ViT 45 2,808 0 23 Aug 2021
Revealing the Distributional Vulnerability of Discriminators by Implicit Generators Zhilin Zhao LongBing Cao Kun-Yu Lin 29 11 0 23 Aug 2021
PatchCleanser: Certifiably Robust Defense against Adversarial Patches for Any Image Classifier Chong Xiang Saeed Mahloujifar Prateek Mittal VLM AAML 24 73 0 20 Aug 2021
Causal Attention for Unbiased Visual Recognition Tan Wang Chan Zhou Qianru Sun Hanwang Zhang OOD CML 32 108 0 19 Aug 2021
End-to-End Dense Video Captioning with Parallel Decoding Teng Wang Ruimao Zhang Zhichao Lu Feng Zheng Ran Cheng Ping Luo 3DV 47 179 0 17 Aug 2021
Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism Shulun Wang Bin Liu Feng Liu 17 16 0 16 Aug 2021
TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation Jinyu Yang Jingjing Liu N. Xu Junzhou Huang 27 125 0 12 Aug 2021
Mobile-Former: Bridging MobileNet and Transformer Yinpeng Chen Xiyang Dai Dongdong Chen Mengchen Liu Xiaoyi Dong Lu Yuan Zicheng Liu ViT 177 476 0 12 Aug 2021
Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations Josh Beal Hao Wu Dong Huk Park Andrew Zhai Dmitry Kislyuk ViT 15 29 0 12 Aug 2021
Video Transformer for Deepfake Detection with Incremental Learning Sohail Ahmed Khan Hang Dai ViT 16 62 0 11 Aug 2021
RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality? Yuki Tatsunami Masato Taki 24 12 0 09 Aug 2021
Understanding the computational demands underlying visual reasoning Mohit Vaishnav Rémi Cadène A. Alamia Drew Linsley Rufin VanRullen Thomas Serre GNN CoGe 37 16 0 08 Aug 2021
Armour: Generalizable Compact Self-Attention for Vision Transformers Lingchuan Meng ViT 21 3 0 03 Aug 2021
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer Yifan Xu Zhijie Zhang Mengdan Zhang Kekai Sheng Ke Li Weiming Dong Liqing Zhang Changsheng Xu Xing Sun ViT 32 201 0 03 Aug 2021
CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention Wenxiao Wang Lulian Yao Long Chen Binbin Lin Deng Cai Xiaofei He Wei Liu 32 257 0 31 Jul 2021
On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals Haizhou Shi Youcai Zhang Siliang Tang Wenjie Zhu Yaqian Li Yandong Guo Yueting Zhuang SyDa 23 14 0 30 Jul 2021
DPT: Deformable Patch-based Transformer for Visual Recognition Zhiyang Chen Yousong Zhu Chaoyang Zhao Guosheng Hu Wei Zeng Jinqiao Wang Ming Tang ViT 16 98 0 30 Jul 2021
Rethinking and Improving Relative Position Encoding for Vision Transformer Kan Wu Houwen Peng Minghao Chen Jianlong Fu Hongyang Chao ViT 42 329 0 29 Jul 2021
A Unified Efficient Pyramid Transformer for Semantic Segmentation Fangrui Zhu Yi Zhu Li Zhang Chongruo Wu Yanwei Fu Mu Li ViT 29 29 0 29 Jul 2021
Spatial-Temporal Transformer for Dynamic Scene Graph Generation Yuren Cong Wentong Liao H. Ackermann Bodo Rosenhahn M. Yang ViT 13 122 0 26 Jul 2021
EAN: Event Adaptive Network for Enhanced Action Recognition Yuan Tian Yichao Yan Guangtao Zhai G. Guo Zhiyong Gao 32 41 0 22 Jul 2021
CycleMLP: A MLP-like Architecture for Dense Prediction Shoufa Chen Enze Xie Chongjian Ge Runjian Chen Ding Liang Ping Luo 19 231 0 21 Jul 2021
Audio Captioning Transformer Xinhao Mei Xubo Liu Qiushi Huang Mark D. Plumbley Wenwu Wang ViT 31 77 0 21 Jul 2021
RAMS-Trans: Recurrent Attention Multi-scale Transformer forFine-grained Image Recognition Yunqing Hu Xuan Jin Yin Zhang Ha Hong Jingfeng Zhang Yuan He Hui Xue ViT 26 97 0 17 Jul 2021
A Comparative Study of Deep Learning Classification Methods on a Small Environmental Microorganism Image Dataset (EMDS-6): from Convolutional Neural Networks to Visual Transformers Penghui Zhao Chen Li M. Rahaman Hao Xu Hechen Yang Hongzan Sun Tao Jiang M. Grzegorzek VLM 27 39 0 16 Jul 2021
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Junnan Li Ramprasaath R. Selvaraju Akhilesh Deepak Gotmare Shafiq R. Joty Caiming Xiong S. Hoi FaML 53 1,886 0 16 Jul 2021
From Show to Tell: A Survey on Deep Learning-based Image Captioning Matteo Stefanini Marcella Cornia Lorenzo Baraldi S. Cascianelli G. Fiameni Rita Cucchiara 3DV VLM MLLM 67 254 0 14 Jul 2021
Transformer with Peak Suppression and Knowledge Guidance for Fine-grained Image Recognition Xinda Liu Lili Wang Xiaoguang Han ViT 39 66 0 14 Jul 2021
Visual Parser: Representing Part-whole Hierarchies with Transformers Shuyang Sun Xiaoyu Yue S. Bai Philip H. S. Torr 50 27 0 13 Jul 2021
TransClaw U-Net: Claw U-Net with Transformers for Medical Image Segmentation Yao Chang Menghan Hu Zhai Guangtao Xiao-Ping Zhang MedIm ViT 74 96 0 12 Jul 2021
Weaving Attention U-net: A Novel Hybrid CNN and Attention-based Method for Organs-at-risk Segmentation in Head and Neck CT Images Zhuangzhuang Zhang T. Zhao Hiram Gay Weixiong Zhang B. Sun 22 19 0 10 Jul 2021
Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World Jiaming Zhang Kailun Yang Angela Constantinescu Kunyu Peng Karin Muller Rainer Stiefelhagen ViT 36 61 0 07 Jul 2021
Learning Vision Transformer with Squeeze and Excitation for Facial Expression Recognition Mouath Aouayeb W. Hamidouche Catherine Soladié K. Kpalma Renaud Séguier ViT 28 57 0 07 Jul 2021
GLiT: Neural Architecture Search for Global and Local Image Transformer Boyu Chen Peixia Li Chuming Li Baopu Li Lei Bai Chen Lin Ming-hui Sun Junjie Yan Wanli Ouyang ViT 24 85 0 07 Jul 2021
Feature Fusion Vision Transformer for Fine-Grained Visual Categorization Jun Wang Xiaohan Yu Yongsheng Gao ViT 35 105 0 06 Jul 2021
Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation Zhiwei Hao Jianyuan Guo Ding Jia Kai Han Yehui Tang Chao Zhang Dacheng Tao Yunhe Wang ViT 33 68 0 03 Jul 2021