v1v2v3v4 (latest)

Pruning Self-attentions into Convolutional Layers in Single Path

23 November 2021

Haoyu He

Jianfei Cai

Jing Liu

Bohan Zhuang

ArXiv (abs)PDF HTML Github (113★)

Papers citing "Pruning Self-attentions into Convolutional Layers in Single Path"

46 / 46 papers shown

Title
Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers Siyuan Wei Tianzhu Ye Shen Zhang Yao Tang Jiajun Liang ViT 68 71 0 21 Apr 2023
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization Pavan Kumar Anasosalu Vasu J. Gabriel Jeff J. Zhu Oncel Tuzel Anurag Ranjan ViT 108 166 0 24 Mar 2023
Token Merging: Your ViT But Faster Daniel Bolya Cheng-Yang Fu Xiaoliang Dai Peizhao Zhang Christoph Feichtenhofer Judy Hoffman MoMe 128 470 0 17 Oct 2022
Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets Zhiying Lu Hongtao Xie Chuanbin Liu Yongdong Zhang ViT 81 59 0 12 Oct 2022
ConvMAE: Masked Convolution Meets Masked Autoencoders Peng Gao Teli Ma Hongsheng Li Ziyi Lin Jifeng Dai Yu Qiao ViT 79 126 0 08 May 2022
VSA: Learning Varied-Size Window Attention in Vision Transformers Qiming Zhang Yufei Xu Jing Zhang Dacheng Tao 82 57 0 18 Apr 2022
Unified Visual Transformer Compression Shixing Yu Tianlong Chen Jiayi Shen Huan Yuan Jianchao Tan Sen Yang Ji Liu Zhangyang Wang ViT 59 94 0 15 Mar 2022
Augmenting Convolutional networks with attention-based aggregation Hugo Touvron Matthieu Cord Alaaeldin El-Nouby Piotr Bojanowski Armand Joulin Gabriel Synnaeve Hervé Jégou ViT 102 49 0 27 Dec 2021
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer Sachin Mehta Mohammad Rastegari ViT 290 1,285 0 05 Oct 2021
Fast Convergence of DETR with Spatially Modulated Co-Attention Peng Gao Minghang Zheng Xiaogang Wang Jifeng Dai Hongsheng Li ViT 77 307 0 05 Aug 2021
Per-Pixel Classification is Not All You Need for Semantic Segmentation Bowen Cheng Alex Schwing Alexander Kirillov VLM ViT 214 1,554 0 13 Jul 2021
GLiT: Neural Architecture Search for Global and Local Image Transformer Boyu Chen Peixia Li Chuming Li Baopu Li Lei Bai Chen Lin Ming Sun Junjie Yan Wanli Ouyang ViT 103 86 0 07 Jul 2021
Early Convolutions Help Transformers See Better Tete Xiao Mannat Singh Eric Mintun Trevor Darrell Piotr Dollár Ross B. Girshick 70 774 0 28 Jun 2021
IA-RED $^2$ : Interpretability-Aware Redundancy Reduction for Vision Transformers Bowen Pan Yikang Shen Yi Ding Zhangyang Wang Rogerio Feris A. Oliva VLM ViT 113 164 0 23 Jun 2021
Chasing Sparsity in Vision Transformers: An End-to-End Exploration Tianlong Chen Yu Cheng Zhe Gan Lu Yuan Lei Zhang Zhangyang Wang ViT 70 222 0 08 Jun 2021
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias Yufei Xu Qiming Zhang Jing Zhang Dacheng Tao ViT 177 339 0 07 Jun 2021
X-volution: On the unification of convolution and self-attention Xuanhong Chen Hang Wang Bingbing Ni ViT 49 25 0 04 Jun 2021
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification Yongming Rao Wenliang Zhao Benlin Liu Jiwen Lu Jie Zhou Cho-Jui Hsieh ViT 107 709 0 03 Jun 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 743 6,139 0 29 Apr 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 170 1,022 0 31 Mar 2021
Dynamic Slimmable Network Changlin Li Guangrun Wang Bing Wang Xiaodan Liang Zhihui Li Xiaojun Chang 91 144 0 24 Mar 2021
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search Changlin Li Tao Tang Guangrun Wang Jiefeng Peng Bing Wang Xiaodan Liang Xiaojun Chang ViT 139 107 0 23 Mar 2021
DeepViT: Towards Deeper Vision Transformer Daquan Zhou Bingyi Kang Xiaojie Jin Linjie Yang Xiaochen Lian Zihang Jiang Qibin Hou Jiashi Feng ViT 117 525 0 22 Mar 2021
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases Stéphane dÁscoli Hugo Touvron Matthew L. Leavitt Ari S. Morcos Giulio Biroli Levent Sagun ViT 143 835 0 19 Mar 2021
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth Yihe Dong Jean-Baptiste Cordonnier Andreas Loukas 147 388 0 05 Mar 2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions Wenhai Wang Enze Xie Xiang Li Deng-Ping Fan Kaitao Song Ding Liang Tong Lu Ping Luo Ling Shao ViT 545 3,740 0 24 Feb 2021
Single-path Bit Sharing for Automatic Loss-aware Model Compression Jing Liu Bohan Zhuang Peng Chen Chunhua Shen Jianfei Cai Mingkui Tan MQ 45 8 0 13 Jan 2021
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers Sixiao Zheng Jiachen Lu Hengshuang Zhao Xiatian Zhu Zekun Luo ... Yanwei Fu Jianfeng Feng Tao Xiang Philip Torr Li Zhang ViT 198 2,912 0 31 Dec 2020
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers Huiyu Wang Yukun Zhu Hartwig Adam Alan Yuille Liang-Chieh Chen ViT 133 531 0 01 Dec 2020
Deformable DETR: Deformable Transformers for End-to-End Object Detection Xizhou Zhu Weijie Su Lewei Lu Bin Li Xiaogang Wang Jifeng Dai ViT 262 5,107 0 08 Oct 2020
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning Bingbing Li Zhenglun Kong Tianyun Zhang Ji Li Zechao Li Hang Liu Caiwen Ding VLM 177 65 0 17 Sep 2020
Designing Network Design Spaces Ilija Radosavovic Raj Prateek Kosaraju Ross B. Girshick Kaiming He Piotr Dollár GNN 107 1,697 0 30 Mar 2020
Rigging the Lottery: Making All Tickets Winners Utku Evci Trevor Gale Jacob Menick Pablo Samuel Castro Erich Elsen 199 607 0 25 Nov 2019
On the Relationship between Self-Attention and Convolutional Layers Jean-Baptiste Cordonnier Andreas Loukas Martin Jaggi 119 535 0 08 Nov 2019
Once-for-All: Train One Network and Specialize it for Efficient Deployment Han Cai Chuang Gan Tianzhe Wang Zhekai Zhang Song Han OOD 129 1,283 0 26 Aug 2019
Sparse Networks from Scratch: Faster Training without Losing Performance Tim Dettmers Luke Zettlemoyer 147 340 0 10 Jul 2019
The Generalization-Stability Tradeoff In Neural Network Pruning Brian Bartoldson Ari S. Morcos Adrian Barbu G. Erlebacher 89 76 0 09 Jun 2019
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 114 1,070 0 25 May 2019
Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours Dimitrios Stamoulis Ruizhou Ding Di Wang Dimitrios Lymberopoulos B. Priyantha Jie Liu Diana Marculescu 68 285 0 05 Apr 2019
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet Wieland Brendel Matthias Bethge SSL FAtt 114 561 0 20 Mar 2019
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness Robert Geirhos Patricia Rubisch Claudio Michaelis Matthias Bethge Felix Wichmann Wieland Brendel 143 2,676 0 29 Nov 2018
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Jonathan Frankle Michael Carbin 288 3,489 0 09 Mar 2018
AMC: AutoML for Model Compression and Acceleration on Mobile Devices Yihui He Ji Lin Zhijian Liu Hanrui Wang Li Li Song Han 111 1,349 0 10 Feb 2018
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.2K 20,918 0 17 Apr 2017
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 437 10,548 0 21 Jul 2016
Gaussian Error Linear Units (GELUs) Dan Hendrycks Kevin Gimpel 180 5,056 0 27 Jun 2016