Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets

28 July 2024

Papers citing "Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets"

29 / 29 papers shown

Title
Improving Vision Transformers by Overlapping Heads in Multi-Head Self-Attention Tianxiao Zhang Bo Luo G. Wang ViT 54 1 0 18 Oct 2024
SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules Xiangyu Chen Jing Liu Ye Wang Pu Wang Matthew Brand Guanghui Wang T. Koike-Akino 67 7 0 18 Mar 2024
BiFormer: Vision Transformer with Bi-Level Routing Attention Lei Zhu Xinjiang Wang Zhanghan Ke Wayne Zhang Rynson W. H. Lau 172 506 0 15 Mar 2023
Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets Xiangyu Chen Qinghao Hu Kaidong Li Cuncong Zhong Guanghui Wang ViT 57 13 0 22 Oct 2022
Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets Zhiying Lu Hongtao Xie Chuanbin Liu Yongdong Zhang ViT 46 57 0 12 Oct 2022
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond Qiming Zhang Yufei Xu Jing Zhang Dacheng Tao ViT 69 232 0 21 Feb 2022
ResNet strikes back: An improved training procedure in timm Ross Wightman Hugo Touvron Hervé Jégou AI4TS 242 492 0 01 Oct 2021
Audio-Visual Transformer Based Crowd Counting Usman Sajid Xiangyu Chen Hasan Sajid Taejoon Kim Guanghui Wang ViT 76 22 0 04 Sep 2021
Mobile-Former: Bridging MobileNet and Transformer Yinpeng Chen Xiyang Dai Dongdong Chen Mengchen Liu Xiaoyi Dong Lu Yuan Zicheng Liu ViT 236 486 0 12 Aug 2021
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers Andreas Steiner Alexander Kolesnikov Xiaohua Zhai Ross Wightman Jakob Uszkoreit Lucas Beyer ViT 107 629 0 18 Jun 2021
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias Yufei Xu Qiming Zhang Jing Zhang Dacheng Tao ViT 138 336 0 07 Jun 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 131 1,006 0 31 Mar 2021
Rethinking Spatial Dimensions of Vision Transformers Byeongho Heo Sangdoo Yun Dongyoon Han Sanghyuk Chun Junsuk Choe Seong Joon Oh ViT 492 578 0 30 Mar 2021
CvT: Introducing Convolutions to Vision Transformers Haiping Wu Bin Xiao Noel Codella Mengchen Liu Xiyang Dai Lu Yuan Lei Zhang ViT 128 1,901 0 29 Mar 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 398 21,281 0 25 Mar 2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions Wenhai Wang Enze Xie Xiang Li Deng-Ping Fan Kaitao Song Ding Liang Tong Lu Ping Luo Ling Shao ViT 483 3,699 0 24 Feb 2021
Bottleneck Transformers for Visual Recognition A. Srinivas Nayeon Lee Niki Parmar Jonathon Shlens Pieter Abbeel Ashish Vaswani SLR 344 989 0 27 Jan 2021
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 341 6,731 0 23 Dec 2020
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features Sangdoo Yun Dongyoon Han Seong Joon Oh Sanghyuk Chun Junsuk Choe Y. Yoo OOD 604 4,766 0 13 May 2019
AutoAugment: Learning Augmentation Policies from Data E. D. Cubuk Barret Zoph Dandelion Mané Vijay Vasudevan Quoc V. Le 118 1,771 0 24 May 2018
mixup: Beyond Empirical Risk Minimization Hongyi Zhang Moustapha Cissé Yann N. Dauphin David Lopez-Paz NoLa 269 9,743 0 25 Oct 2017
Squeeze-and-Excitation Networks Jie Hu Li Shen Samuel Albanie Gang Sun Enhua Wu 378 26,365 0 05 Sep 2017
Random Erasing Data Augmentation Zhun Zhong Liang Zheng Guoliang Kang Shaozi Li Yi Yang 90 3,630 0 16 Aug 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.1K 20,781 0 17 Apr 2017
Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization Ramprasaath R. Selvaraju Michael Cogswell Abhishek Das Ramakrishna Vedantam Devi Parikh Dhruv Batra FAtt 250 19,929 0 07 Oct 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 334 10,467 0 21 Jul 2016
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 415 43,234 0 11 Feb 2015
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.4K 39,472 0 01 Sep 2014
Microsoft COCO: Common Objects in Context Nayeon Lee Michael Maire Serge J. Belongie Lubomir Bourdev Ross B. Girshick James Hays Pietro Perona Deva Ramanan C. L. Zitnick Piotr Dollár ObjD 367 43,524 0 01 May 2014