Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets

22 October 2022

ArXiv (abs)PDF HTML Github (6★)

Papers citing "Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets"

28 / 28 papers shown

Title
Explicitly Increasing Input Information Density for Vision Transformers on Small Datasets Xiangyu Chen Ying Qin Wenju Xu A. Bur Cuncong Zhong Guanghui Wang ViT 75 3 0 25 Oct 2022
SimA: Simple Softmax-free Attention for Vision Transformers Soroush Abbasi Koohpayegani Hamed Pirsiavash 85 26 0 17 Jun 2022
DaViT: Dual Attention Vision Transformers Mingyu Ding Bin Xiao Noel Codella Ping Luo Jingdong Wang Lu Yuan ViT 130 250 0 07 Apr 2022
Dynamic Label Assignment for Object Detection by Combining Predicted IoUs and Anchor IoUs Tianxiao Zhang Bo Luo A. Sharda Guanghui Wang 86 19 0 23 Jan 2022
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng Zhang Li Dong Furu Wei B. Guo ViT 219 1,825 0 18 Nov 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 477 7,819 0 11 Nov 2021
SOFT: Softmax-free Transformer with Linear Complexity Jiachen Lu Jinghan Yao Junge Zhang Martin Danelljan Hang Xu Weiguo Gao Chunjing Xu Thomas B. Schon Li Zhang 65 166 0 22 Oct 2021
Audio-Visual Transformer Based Crowd Counting Usman Sajid Xiangyu Chen Hasan Sajid Taejoon Kim Guanghui Wang ViT 81 22 0 04 Sep 2021
Shallow Feature Matters for Weakly Supervised Object Localization Junhang Wei Qin Wang Zhen Li Sheng Wang S.Kevin Zhou Shuguang Cui WSOL 60 89 0 02 Aug 2021
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows Xiaoyi Dong Jianmin Bao Dongdong Chen Weiming Zhang Nenghai Yu Lu Yuan Dong Chen B. Guo ViT 154 985 0 01 Jul 2021
Efficient Self-supervised Vision Transformers for Representation Learning Chunyuan Li Jianwei Yang Pengchuan Zhang Mei Gao Bin Xiao Xiyang Dai Lu Yuan Jianfeng Gao ViT 99 213 0 17 Jun 2021
Efficient Training of Visual Transformers with Small Datasets Yahui Liu E. Sangineto Wei Bi N. Sebe Bruno Lepri Marco De Nadai ViT 68 173 0 07 Jun 2021
A Fine-Grained Visual Attention Approach for Fingerspelling Recognition in the Wild Kamala Gajurel Cuncong Zhong Guanghui Wang SLR 57 13 0 17 May 2021
Few-Shot Learning by Integrating Spatial and Frequency Representation Xiangyu Chen Guanghui Wang 61 27 0 11 May 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 713 6,127 0 29 Apr 2021
Twins: Revisiting the Design of Spatial Attention in Vision Transformers Xiangxiang Chu Zhi Tian Yuqing Wang Bo Zhang Haibing Ren Xiaolin K. Wei Huaxia Xia Chunhua Shen ViT 84 1,026 0 28 Apr 2021
Rethinking Spatial Dimensions of Vision Transformers Byeongho Heo Sangdoo Yun Dongyoon Han Sanghyuk Chun Junsuk Choe Seong Joon Oh ViT 514 582 0 30 Mar 2021
CvT: Introducing Convolutions to Vision Transformers Haiping Wu Bin Xiao Noel Codella Mengchen Liu Xiyang Dai Lu Yuan Lei Zhang ViT 154 1,917 0 29 Mar 2021
Scaling Local Self-Attention for Parameter Efficient Visual Backbones Ashish Vaswani Prajit Ramachandran A. Srinivas Niki Parmar Blake A. Hechtman Jonathon Shlens 94 403 0 23 Mar 2021
Incorporating Convolution Designs into Visual Transformers Kun Yuan Shaopeng Guo Ziwei Liu Aojun Zhou F. Yu Wei Wu ViT 111 480 0 22 Mar 2021
Deep Feature Augmentation for Occluded Image Classification Feng Cen Xiaoyu Zhao Wuzhuang Li Guanghui Wang 80 58 0 02 Nov 2020
Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning Yinbo Chen Zhuang Liu Huijuan Xu Trevor Darrell Xiaolong Wang 205 347 0 09 Mar 2020
RandAugment: Practical automated data augmentation with a reduced search space E. D. Cubuk Barret Zoph Jonathon Shlens Quoc V. Le MQ 258 3,502 0 30 Sep 2019
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features Sangdoo Yun Dongyoon Han Seong Joon Oh Sanghyuk Chun Junsuk Choe Y. Yoo OOD 622 4,802 0 13 May 2019
mixup: Beyond Empirical Risk Minimization Hongyi Zhang Moustapha Cissé Yann N. Dauphin David Lopez-Paz NoLa 289 9,803 0 25 Oct 2017
Random Erasing Data Augmentation Zhun Zhong Liang Zheng Guoliang Kang Shaozi Li Yi Yang 98 3,645 0 16 Aug 2017
Deep Networks with Stochastic Depth Gao Huang Yu Sun Zhuang Liu Daniel Sedra Kilian Q. Weinberger 215 2,361 0 30 Mar 2016
Rethinking the Inception Architecture for Computer Vision Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Z. Wojna 3DV BDL 886 27,416 0 02 Dec 2015