CvT: Introducing Convolutions to Vision Transformers

29 March 2021

Lu Yuan

Lei Zhang

ViT

ArXiv PDF HTML

Papers citing "CvT: Introducing Convolutions to Vision Transformers"

50 / 818 papers shown

Title
Focal Self-attention for Local-Global Interactions in Vision Transformers Jianwei Yang Chunyuan Li Pengchuan Zhang Xiyang Dai Bin Xiao Lu Yuan Jianfeng Gao ViT 42 428 0 01 Jul 2021
Rethinking Token-Mixing MLP for MLP-based Vision Backbone Tan Yu Xu Li Yunfeng Cai Mingming Sun Ping Li 45 26 0 28 Jun 2021
Early Convolutions Help Transformers See Better Tete Xiao Mannat Singh Eric Mintun Trevor Darrell Piotr Dollár Ross B. Girshick 20 753 0 28 Jun 2021
PVT v2: Improved Baselines with Pyramid Vision Transformer Wenhai Wang Enze Xie Xiang Li Deng-Ping Fan Kaitao Song Ding Liang Tong Lu Ping Luo Ling Shao ViT AI4TS 50 1,615 0 25 Jun 2021
ViTAS: Vision Transformer Architecture Search Xiu Su Shan You Jiyang Xie Mingkai Zheng Fei Wang Chao Qian Changshui Zhang Xiaogang Wang Chang Xu ViT 27 54 0 25 Jun 2021
VOLO: Vision Outlooker for Visual Recognition Li-xin Yuan Qibin Hou Zihang Jiang Jiashi Feng Shuicheng Yan ViT 52 314 0 24 Jun 2021
IA-RED $^2$ : Interpretability-Aware Redundancy Reduction for Vision Transformers Bowen Pan Yikang Shen Yi Ding Zhangyang Wang Rogerio Feris A. Oliva VLM ViT 39 153 0 23 Jun 2021
Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition Qibin Hou Zihang Jiang Li-xin Yuan Mingg-Ming Cheng Shuicheng Yan Jiashi Feng ViT MLLM 24 205 0 23 Jun 2021
P2T: Pyramid Pooling Transformer for Scene Understanding Yu-Huan Wu Yun-Hai Liu Xin Zhan Mingg-Ming Cheng ViT 29 219 0 22 Jun 2021
Encoder-Decoder Architectures for Clinically Relevant Coronary Artery Segmentation Joao Lourencco Silva M. Menezes T. Rodrigues B. Silva F. Pinto Arlindo L. Oliveira MedIm 31 17 0 21 Jun 2021
More than Encoder: Introducing Transformer Decoder to Upsample Yijiang Li Wentian Cai Ying Gao Chengming Li Xiping Hu ViT MedIm 35 51 0 20 Jun 2021
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers Andreas Steiner Alexander Kolesnikov Xiaohua Zhai Ross Wightman Jakob Uszkoreit Lucas Beyer ViT 66 616 0 18 Jun 2021
Efficient Self-supervised Vision Transformers for Representation Learning Chunyuan Li Jianwei Yang Pengchuan Zhang Mei Gao Bin Xiao Xiyang Dai Lu Yuan Jianfeng Gao ViT 37 209 0 17 Jun 2021
S $^2$ -MLP: Spatial-Shift MLP Architecture for Vision Tan Yu Xu Li Yunfeng Cai Mingming Sun Ping Li 41 186 0 14 Jun 2021
Styleformer: Transformer based Generative Adversarial Networks with Style Vector Jeeseung Park Younggeun Kim ViT 29 48 0 13 Jun 2021
MlTr: Multi-label Classification with Transformer Xingyi Cheng Hezheng Lin Xiangyu Wu Fan Yang Dong Shen Zhongyuan Wang Nian Shi Honglin Liu ViT 20 48 0 11 Jun 2021
Transformed CNNs: recasting pre-trained convolutional layers with self-attention Stéphane dÁscoli Levent Sagun Giulio Biroli Ari S. Morcos ViT 18 6 0 10 Jun 2021
CAT: Cross Attention in Vision Transformer Hezheng Lin Xingyi Cheng Xiangyu Wu Fan Yang Dong Shen Zhongyuan Wang Qing Song Wei Yuan ViT 32 149 0 10 Jun 2021
CoAtNet: Marrying Convolution and Attention for All Data Sizes Zihang Dai Hanxiao Liu Quoc V. Le Mingxing Tan ViT 49 1,167 0 09 Jun 2021
TED-net: Convolution-free T2T Vision Transformer-based Encoder-decoder Dilation network for Low-dose CT Denoising Dayang Wang Zhan Wu Hengyong Yu ViT MedIm 19 53 0 08 Jun 2021
On the Connection between Local Attention and Dynamic Depth-wise Convolution Qi Han Zejia Fan Qi Dai Lei-huan Sun Ming-Ming Cheng Jiaying Liu Jingdong Wang ViT 29 105 0 08 Jun 2021
On Improving Adversarial Transferability of Vision Transformers Muzammal Naseer Kanchana Ranasinghe Salman Khan Fahad Shahbaz Khan Fatih Porikli ViT 39 93 0 08 Jun 2021
Fully Transformer Networks for Semantic Image Segmentation Sitong Wu Tianyi Wu Fangjian Lin Sheng Tian Guodong Guo ViT 34 39 0 08 Jun 2021
Efficient Training of Visual Transformers with Small Datasets Yahui Liu E. Sangineto Wei Bi N. Sebe Bruno Lepri Marco De Nadai ViT 36 164 0 07 Jun 2021
Reveal of Vision Transformers Robustness against Adversarial Attacks Ahmed Aldahdooh W. Hamidouche Olivier Déforges ViT 15 56 0 07 Jun 2021
Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer Zilong Huang Youcheng Ben Guozhong Luo Pei Cheng Gang Yu Bin-Bin Fu ViT 19 182 0 07 Jun 2021
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias Yufei Xu Qiming Zhang Jing Zhang Dacheng Tao ViT 65 329 0 07 Jun 2021
Vision Transformers with Hierarchical Attention Yun-Hai Liu Yu-Huan Wu Guolei Sun Le Zhang Ajad Chhatkuli Luc Van Gool ViT 38 32 0 06 Jun 2021
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings Tatiana Likhomanenko Qiantong Xu Gabriel Synnaeve R. Collobert A. Rogozhnikov OOD ViT 33 54 0 06 Jun 2021
Uformer: A General U-Shaped Transformer for Image Restoration Zhendong Wang Xiaodong Cun Jianmin Bao Wengang Zhou Jianzhuang Liu Houqiang Li ViT 51 1,368 0 06 Jun 2021
RegionViT: Regional-to-Local Attention for Vision Transformers Chun-Fu Chen Yikang Shen Quanfu Fan ViT 18 194 0 04 Jun 2021
Glance-and-Gaze Vision Transformer Qihang Yu Yingda Xia Yutong Bai Yongyi Lu Alan Yuille Wei Shen ViT 24 74 0 04 Jun 2021
X-volution: On the unification of convolution and self-attention Xuanhong Chen Hang Wang Bingbing Ni ViT 27 24 0 04 Jun 2021
Attention mechanisms and deep learning for machine vision: A survey of the state of the art A. M. Hafiz S. A. Parah R. A. Bhat 21 45 0 03 Jun 2021
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers Enze Xie Wenhai Wang Zhiding Yu Anima Anandkumar J. Álvarez Ping Luo ViT 50 4,836 0 31 May 2021
MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens Jiemin Fang Lingxi Xie Xinggang Wang Xiaopeng Zhang Wenyu Liu Qi Tian ViT 18 73 0 31 May 2021
Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model Jiangning Zhang Chao Xu Jian Li Wenzhou Chen Yabiao Wang Ying Tai Shuo Chen Chengjie Wang Feiyue Huang Yong Liu 35 22 0 31 May 2021
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition Yulin Wang Rui Huang S. Song Zeyi Huang Gao Huang ViT 30 189 0 31 May 2021
Dual-stream Network for Visual Recognition Mingyuan Mao Renrui Zhang Honghui Zheng Peng Gao Teli Ma Yan Peng Errui Ding Baochang Zhang Shumin Han ViT 25 63 0 31 May 2021
Less is More: Pay Less Attention in Vision Transformers Zizheng Pan Bohan Zhuang Haoyu He Jing Liu Jianfei Cai ViT 24 82 0 29 May 2021
KVT: k-NN Attention for Boosting Vision Transformers Pichao Wang Xue Wang F. Wang Ming Lin Shuning Chang Hao Li R. L. Jin ViT 51 105 0 28 May 2021
Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding Zizhao Zhang Han Zhang Long Zhao Ting Chen Sercan Ö. Arik Tomas Pfister ViT 22 169 0 26 May 2021
Pay Attention to MLPs Hanxiao Liu Zihang Dai David R. So Quoc V. Le AI4CE 57 651 0 17 May 2021
Towards Robust Vision Transformer Xiaofeng Mao Gege Qi YueFeng Chen Xiaodan Li Ranjie Duan Shaokai Ye Yuan He Hui Xue ViT 23 186 0 17 May 2021
Waste detection in Pomerania: non-profit project for detecting waste in environment Sylwia Majchrowska Agnieszka Mikołajczyk M. Ferlin Zuzanna Klawikowska Marta A. Plantykow Arkadiusz Kwasigroch K. Majek 30 125 0 12 May 2021
Homogeneous vector bundles and $G$ -equivariant convolutional neural networks J. Aronsson 29 24 0 12 May 2021
Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet Luke Melas-Kyriazi ViT 9 101 0 06 May 2021
Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks Meng-Hao Guo Zheng-Ning Liu Tai-Jiang Mu Shimin Hu 25 472 0 05 May 2021
Twins: Revisiting the Design of Spatial Attention in Vision Transformers Xiangxiang Chu Zhi Tian Yuqing Wang Bo Zhang Haibing Ren Xiaolin K. Wei Huaxia Xia Chunhua Shen ViT 23 997 0 28 Apr 2021
Vision Transformers with Patch Diversification Chengyue Gong Dilin Wang Meng Li Vikas Chandra Qiang Liu ViT 45 62 0 26 Apr 2021