v1v2v3v4 (latest)

Less is More: Pay Less Attention in Vision Transformers

29 May 2021

Zizheng Pan

Bohan Zhuang

Haoyu He

Jing Liu

Jianfei Cai

ViT

ArXiv (abs)PDF HTML Github (96★)

Papers citing "Less is More: Pay Less Attention in Vision Transformers"

33 / 33 papers shown

Title
Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding Yiming Zhang Zhuokai Zhao Zhaorun Chen Zenghui Ding Xianjun Yang Yining Sun 528 1 0 21 Nov 2024
FViT: A Focal Vision Transformer with Gabor Filter Yulong Shi Mingwei Sun Yongshuai Wang Rui Wang 122 4 0 17 Feb 2024
Morphing Tokens Draw Strong Masked Image Models Taekyung Kim Byeongho Heo Dongyoon Han 135 3 0 30 Dec 2023
Lightweight Vision Transformer with Bidirectional Interaction Qihang Fan Huaibo Huang Xiaoqiang Zhou Ran He ViT 116 29 0 01 Jun 2023
GLiT: Neural Architecture Search for Global and Local Image Transformer Boyu Chen Peixia Li Chuming Li Baopu Li Lei Bai Chen Lin Ming Sun Junjie Yan Wanli Ouyang ViT 76 86 0 07 Jul 2021
Twins: Revisiting the Design of Spatial Attention in Vision Transformers Xiangxiang Chu Zhi Tian Yuqing Wang Bo Zhang Haibing Ren Xiaolin K. Wei Huaxia Xia Chunhua Shen ViT 84 1,026 0 28 Apr 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 160 1,021 0 31 Mar 2021
Rethinking Spatial Dimensions of Vision Transformers Byeongho Heo Sangdoo Yun Dongyoon Han Sanghyuk Chun Junsuk Choe Seong Joon Oh ViT 506 582 0 30 Mar 2021
CvT: Introducing Convolutions to Vision Transformers Haiping Wu Bin Xiao Noel Codella Mengchen Liu Xiyang Dai Lu Yuan Lei Zhang ViT 152 1,915 0 29 Mar 2021
Scaling Local Self-Attention for Parameter Efficient Visual Backbones Ashish Vaswani Prajit Ramachandran A. Srinivas Niki Parmar Blake A. Hechtman Jonathon Shlens 92 403 0 23 Mar 2021
DeepViT: Towards Deeper Vision Transformer Daquan Zhou Bingyi Kang Xiaojie Jin Linjie Yang Xiaochen Lian Zihang Jiang Qibin Hou Jiashi Feng ViT 102 523 0 22 Mar 2021
Incorporating Convolution Designs into Visual Transformers Kun Yuan Shaopeng Guo Ziwei Liu Aojun Zhou F. Yu Wei Wu ViT 111 479 0 22 Mar 2021
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases Stéphane dÁscoli Hugo Touvron Matthew L. Leavitt Ari S. Morcos Giulio Biroli Levent Sagun ViT 129 832 0 19 Mar 2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions Wenhai Wang Enze Xie Xiang Li Deng-Ping Fan Kaitao Song Ding Liang Tong Lu Ping Luo Ling Shao ViT 530 3,734 0 24 Feb 2021
Bottleneck Transformers for Visual Recognition A. Srinivas Nayeon Lee Niki Parmar Jonathon Shlens Pieter Abbeel Ashish Vaswani SLR 360 993 0 27 Jan 2021
Deformable DETR: Deformable Transformers for End-to-End Object Detection Xizhou Zhu Weijie Su Lewei Lu Bin Li Xiaogang Wang Jifeng Dai ViT 234 5,091 0 08 Oct 2020
Are we done with ImageNet? Lucas Beyer Olivier J. Hénaff Alexander Kolesnikov Xiaohua Zhai Aaron van den Oord VLM 128 405 0 12 Jun 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 216 1,713 0 08 Jun 2020
On the Relationship between Self-Attention and Convolutional Layers Jean-Baptiste Cordonnier Andreas Loukas Martin Jaggi 116 535 0 08 Nov 2019
BERT Rediscovers the Classical NLP Pipeline Ian Tenney Dipanjan Das Ellie Pavlick MILM SSeg 138 1,478 0 15 May 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 129 1,908 0 23 Apr 2019
KPConv: Flexible and Deformable Convolution for Point Clouds Hugues Thomas C. Qi Jean-Emmanuel Deschaud B. Marcotegui F. Goulette Leonidas Guibas 3DPC 169 2,541 0 18 Apr 2019
Cascaded Partial Decoder for Fast and Accurate Salient Object Detection Zhe Wu Li Su Qingming Huang ObjD 92 823 0 18 Apr 2019
Do ImageNet Classifiers Generalize to ImageNet? Benjamin Recht Rebecca Roelofs Ludwig Schmidt Vaishaal Shankar OOD SSeg VLM 118 1,726 0 13 Feb 2019
Panoptic Feature Pyramid Networks Alexander Kirillov Ross B. Girshick Kaiming He Piotr Dollár ISeg SSeg 134 1,291 0 08 Jan 2019
Deformable ConvNets v2: More Deformable, Better Results Xizhou Zhu Han Hu Stephen Lin Jifeng Dai ObjD 114 2,018 0 27 Nov 2018
Focal Loss for Dense Object Detection Nayeon Lee Priya Goyal Ross B. Girshick Kaiming He Piotr Dollár ObjD 124 2,998 0 07 Aug 2017
Understanding the Effective Receptive Field in Deep Convolutional Neural Networks Wenjie Luo Yujia Li R. Urtasun R. Zemel HAI 102 1,799 0 15 Jan 2017
Deeply supervised salient object detection with short connections Qibin Hou Ming-Ming Cheng Xiao-Wei Hu Ali Borji Zhuowen Tu Philip Torr 83 1,331 0 15 Nov 2016
Semantic Understanding of Scenes through the ADE20K Dataset Bolei Zhou Hang Zhao Xavier Puig Tete Xiao Sanja Fidler Adela Barriuso Antonio Torralba SSeg 404 1,889 0 18 Aug 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 420 10,526 0 21 Jul 2016
Gaussian Error Linear Units (GELUs) Dan Hendrycks Kevin Gimpel 172 5,037 0 27 Jun 2016
Visualizing and Understanding Convolutional Networks Matthew D. Zeiler Rob Fergus FAtt SSL 595 15,902 0 12 Nov 2013