DeepViT: Towards Deeper Vision Transformer

22 March 2021

Linjie Yang

Papers citing "DeepViT: Towards Deeper Vision Transformer"

50 / 51 papers shown

Title
AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer Jiquan Shan Junxiao Wang Lifeng Zhao Liang Cai Hongyuan Zhang Ioannis Liritzis ViT 81 0 0 22 May 2025
Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks Kairong Yu Chengting Yu Tianqing Zhang Xiaochen Zhao Shu Yang Hongwei Wang Qiang Zhang Qi Xu 89 4 0 05 Mar 2025
VMamba: Visual State Space Model Yue Liu Yunjie Tian Yuzhong Zhao Hongtian Yu Lingxi Xie Yaowei Wang Qixiang Ye Jianbin Jiao Yunfan Liu Mamba 184 657 0 31 Dec 2024
Improving Vision Transformers by Overlapping Heads in Multi-Head Self-Attention Tianxiao Zhang Bo Luo G. Wang ViT 40 1 0 18 Oct 2024
Interpretability-Aware Vision Transformer Yao Qiang Chengyin Li Prashant Khanduri D. Zhu ViT 113 7 0 14 Sep 2023
Coordinate Attention for Efficient Mobile Network Design Qibin Hou Daquan Zhou Jiashi Feng 50 2,992 0 04 Mar 2021
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet Li-xin Yuan Yunpeng Chen Tao Wang Weihao Yu Yujun Shi Zihang Jiang Francis E. H. Tay Jiashi Feng Shuicheng Yan ViT 80 1,918 0 28 Jan 2021
Bottleneck Transformers for Visual Recognition A. Srinivas Nayeon Lee Niki Parmar Jonathon Shlens Pieter Abbeel Ashish Vaswani SLR 315 986 0 27 Jan 2021
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers Sixiao Zheng Jiachen Lu Hengshuang Zhao Xiatian Zhu Zekun Luo ... Yanwei Fu Jianfeng Feng Tao Xiang Philip Torr Li Zhang ViT 100 2,872 0 31 Dec 2020
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 238 6,657 0 23 Dec 2020
Pre-Trained Image Processing Transformer Hanting Chen Yunhe Wang Tianyu Guo Chang Xu Yiping Deng Zhenhua Liu Siwei Ma Chunjing Xu Chao Xu Wen Gao VLM ViT 120 1,659 0 01 Dec 2020
Point Transformer Nico Engel Vasileios Belagiannis Klaus C. J. Dietmayer 3DPC 135 1,966 0 02 Nov 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 180 40,217 0 22 Oct 2020
ConvBERT: Improving BERT with Span-based Dynamic Convolution Zihang Jiang Weihao Yu Daquan Zhou Yunpeng Chen Jiashi Feng Shuicheng Yan 58 158 0 06 Aug 2020
Rethinking Bottleneck Structure for Efficient Mobile Network Design Zhou Daquan Qibin Hou Yunpeng Chen Jiashi Feng Shuicheng Yan 33 198 0 05 Jul 2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang M. Krikun Noam M. Shazeer Zhiwen Chen MoE 71 1,142 0 30 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 383 41,106 0 28 May 2020
End-to-End Object Detection with Transformers Nicolas Carion Francisco Massa Gabriel Synnaeve Nicolas Usunier Alexander Kirillov Sergey Zagoruyko ViT 3DV PINN 233 12,847 0 26 May 2020
Exploring Self-attention for Image Recognition Hengshuang Zhao Jiaya Jia V. Koltun SSL 63 778 0 28 Apr 2020
ResNeSt: Split-Attention Networks Hang Zhang Chongruo Wu Zhongyue Zhang Yi Zhu Yanghua Peng ... Tong He Jonas W. Mueller R. Manmatha Mu Li Alex Smola 67 1,471 0 19 Apr 2020
Designing Network Design Spaces Ilija Radosavovic Raj Prateek Kosaraju Ross B. Girshick Kaiming He Piotr Dollár GNN 71 1,672 0 30 Mar 2020
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation Huiyu Wang Yukun Zhu Bradley Green Hartwig Adam Alan Yuille Liang-Chieh Chen 3DPC 87 669 0 17 Mar 2020
Axial Attention in Multidimensional Transformers Jonathan Ho Nal Kalchbrenner Dirk Weissenborn Tim Salimans 58 525 0 20 Dec 2019
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 118 42,038 0 03 Dec 2019
RandAugment: Practical automated data augmentation with a reduced search space E. D. Cubuk Barret Zoph Jonathon Shlens Quoc V. Le MQ 169 3,458 0 30 Sep 2019
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks Jiasen Lu Dhruv Batra Devi Parikh Stefan Lee SSL VLM 172 3,659 0 06 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 348 24,160 0 26 Jul 2019
MixConv: Mixed Depthwise Convolutional Kernels Mingxing Tan Quoc V. Le 36 377 0 22 Jul 2019
Stand-Alone Self-Attention in Vision Models Prajit Ramachandran Niki Parmar Ashish Vaswani Irwan Bello Anselm Levskaya Jonathon Shlens VLM SLR ViT 51 1,208 0 13 Jun 2019
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan Quoc V. Le 3DV MedIm 42 17,950 0 28 May 2019
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features Sangdoo Yun Dongyoon Han Seong Joon Oh Sanghyuk Chun Junsuk Choe Y. Yoo OOD 557 4,735 0 13 May 2019
Searching for MobileNetV3 Andrew G. Howard Mark Sandler Grace Chu Liang-Chieh Chen Bo Chen ... Yukun Zhu Ruoming Pang Vijay Vasudevan Quoc V. Le Hartwig Adam 250 6,685 0 06 May 2019
Local Relation Networks for Image Recognition Han Hu Zheng Zhang Zhenda Xie Stephen Lin FAtt 51 499 0 25 Apr 2019
VideoBERT: A Joint Model for Video and Language Representation Learning Chen Sun Austin Myers Carl Vondrick Kevin Patrick Murphy Cordelia Schmid VLM SSL 28 1,238 0 03 Apr 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 808 93,936 0 11 Oct 2018
Network Representation Learning: A Survey Daokun Zhang Jie Yin Xingquan Zhu Chengqi Zhang GNN AI4TS 67 618 0 04 Dec 2017
Non-local Neural Networks Xinyu Wang Ross B. Girshick Abhinav Gupta Kaiming He OffRL 181 8,867 0 21 Nov 2017
mixup: Beyond Empirical Risk Minimization Hongyi Zhang Moustapha Cissé Yann N. Dauphin David Lopez-Paz NoLa 217 9,687 0 25 Oct 2017
Random Erasing Data Augmentation Zhun Zhong Liang Zheng Guoliang Kang Shaozi Li Yi Yang 63 3,614 0 16 Aug 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 317 129,831 0 12 Jun 2017
iCaRL: Incremental Classifier and Representation Learning Sylvestre-Alvise Rebuffi Alexander Kolesnikov G. Sperl Christoph H. Lampert CLL OOD 80 3,713 0 23 Nov 2016
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Zhuowen Tu Kaiming He 407 10,281 0 16 Nov 2016
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 571 36,599 0 25 Aug 2016
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 199 8,030 0 13 Aug 2016
Wide Residual Networks Sergey Zagoruyko N. Komodakis 219 7,951 0 23 May 2016
Identity Mappings in Deep Residual Networks Kaiming He Xinming Zhang Shaoqing Ren Jian Sun 257 10,149 0 16 Mar 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.1K 192,638 0 10 Dec 2015
Rethinking the Inception Architecture for Computer Vision Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Z. Wojna 3DV BDL 390 27,231 0 02 Dec 2015
Going Deeper with Convolutions Christian Szegedy Wei Liu Yangqing Jia P. Sermanet Scott E. Reed Dragomir Anguelov D. Erhan Vincent Vanhoucke Andrew Rabinovich 243 43,511 0 17 Sep 2014
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 711 99,991 0 04 Sep 2014