Chasing Sparsity in Vision Transformers: An End-to-End Exploration

8 June 2021

Tianlong Chen

Yu Cheng

Zhe Gan

Lu Yuan

Lei Zhang

Zhangyang Wang

ViT

ArXiv PDF HTML

Papers citing "Chasing Sparsity in Vision Transformers: An End-to-End Exploration"

50 / 58 papers shown

Title
Accelerate 3D Object Detection Models via Zero-Shot Attention Key Pruning Lizhen Xu Xiuxiu Bai Xiaojun Jia Jianwu Fang Shanmin Pang 63 0 0 13 Mar 2025
Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness Boqian Wu Q. Xiao Shunxin Wang N. Strisciuglio Mykola Pechenizkiy M. V. Keulen Decebal Constantin Mocanu Elena Mocanu OOD 3DH 57 0 0 03 Oct 2024
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition Stephen Zhang Vardan Papyan VLM 51 1 0 20 Sep 2024
Isomorphic Pruning for Vision Models Gongfan Fang Xinyin Ma Michael Bi Mi Xinchao Wang VLM ViT 42 6 0 05 Jul 2024
LPViT: Low-Power Semi-structured Pruning for Vision Transformers Kaixin Xu Zhe Wang Chunyun Chen Xue Geng Jie Lin Xulei Yang Min-man Wu Min Wu Xiaoli Li Weisi Lin ViT VLM 51 7 0 02 Jul 2024
LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate A. Fuller Daniel G. Kyrollos Yousef Yassin James R. Green 52 2 0 22 May 2024
QMViT: A Mushroom is worth 16x16 Words Siddhant Dutta Hemant Singh Kalpita Shankhdhar Sridhar Iyer 34 0 0 11 May 2024
OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning Chu Myaet Thwal Minh N. H. Nguyen Ye Lin Tun Seongjin Kim My T. Thai Choong Seon Hong 64 5 0 22 Jan 2024
Prompt-In-Prompt Learning for Universal Image Restoration Zilong Li Yiming Lei Chenglong Ma Junping Zhang Hongming Shan VLM 39 25 0 08 Dec 2023
Improved TokenPose with Sparsity Anning Li ViT 34 0 0 16 Nov 2023
CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs Ao Wang Hui Chen Zijia Lin Sicheng Zhao J. Han Guiguang Ding ViT 34 6 0 27 Sep 2023
Vision Transformers for Mobile Applications: A Short Survey Nahid Alam Steven Kolawole S. Sethi Nishant Bansali Karina Nguyen ViT 31 3 0 30 May 2023
COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models Jinqi Xiao Miao Yin Yu Gong Xiao Zang Jian Ren Bo Yuan VLM ViT 43 9 0 26 May 2023
Boost Vision Transformer with GPU-Friendly Sparsity and Quantization Chong Yu Tao Chen Zhongxue Gan Jiayuan Fan MQ ViT 30 23 0 18 May 2023
Distilling Token-Pruned Pose Transformer for 2D Human Pose Estimation Feixiang Ren ViT 19 2 0 12 Apr 2023
SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer Xuanyao Chen Zhijian Liu Haotian Tang Li Yi Hang Zhao Song Han ViT 29 46 0 30 Mar 2023
Gradient-Free Structured Pruning with Unlabeled Data Azade Nova H. Dai Dale Schuurmans SyDa 40 20 0 07 Mar 2023
Modular Deep Learning Jonas Pfeiffer Sebastian Ruder Ivan Vulić E. Ponti MoMe OOD 32 73 0 22 Feb 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 31 47 0 02 Feb 2023
GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformer Miao Yin Burak Uzkent Yilin Shen Hongxia Jin Bo Yuan ViT 29 13 0 13 Jan 2023
Skip-Attention: Improving Vision Transformers by Paying Less Attention Shashanka Venkataramanan Amir Ghodrati Yuki M. Asano Fatih Porikli A. Habibian ViT 18 25 0 05 Jan 2023
NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers Yijiang Liu Huanrui Yang Zhen Dong Kurt Keutzer Li Du Shanghang Zhang MQ 31 46 0 29 Nov 2022
Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers Sifan Long Z. Zhao Jimin Pi Sheng-sheng Wang Jingdong Wang 22 29 0 21 Nov 2022
Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training Zhenglun Kong Haoyu Ma Geng Yuan Mengshu Sun Yanyue Xie ... Tianlong Chen Xiaolong Ma Xiaohui Xie Zhangyang Wang Yanzhi Wang ViT 34 22 0 19 Nov 2022
HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers Peiyan Dong Mengshu Sun Alec Lu Yanyue Xie Li-Yu Daisy Liu ... Xin Meng Zechao Li Xue Lin Zhenman Fang Yanzhi Wang ViT 34 59 0 15 Nov 2022
BiViT: Extremely Compressed Binary Vision Transformer Yefei He Zhenyu Lou Luoming Zhang Jing Liu Weijia Wu Hong Zhou Bohan Zhuang ViT MQ 20 28 0 14 Nov 2022
ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention Jyotikrishna Dass Shang Wu Huihong Shi Chaojian Li Zhifan Ye Zhongfeng Wang Yingyan Lin 17 50 0 09 Nov 2022
M $^3$ ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design Hanxue Liang Zhiwen Fan Rishov Sarkar Ziyu Jiang Tianlong Chen Kai Zou Yu Cheng Cong Hao Zhangyang Wang MoE 42 81 0 26 Oct 2022
SaiT: Sparse Vision Transformers through Adaptive Token Pruning Ling Li D. Thorsley Joseph Hassoun ViT 27 17 0 11 Oct 2022
Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning Weicong Liang Yuhui Yuan Henghui Ding Xiao Luo Weihong Lin Ding Jia Zheng-Wei Zhang Chao Zhang Hanhua Hu 32 25 0 03 Oct 2022
PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimation Haoyu Ma Zhe Wang Yifei Chen Deying Kong Liangjian Chen Xingwei Liu Xiangyi Yan Hao Tang Xiaohui Xie ViT 35 47 0 16 Sep 2022
Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deployment Jie Zhu Leye Wang Xiao Han 25 9 0 11 Aug 2022
Towards Efficient Adversarial Training on Vision Transformers Boxi Wu Jindong Gu Zhifeng Li Deng Cai Xiaofei He Wei Liu ViT AAML 46 38 0 21 Jul 2022
Spartan: Differentiable Sparsity via Regularized Transportation Kai Sheng Tai Taipeng Tian Ser-Nam Lim 31 11 0 27 May 2022
MiniViT: Compressing Vision Transformers with Weight Multiplexing Jinnian Zhang Houwen Peng Kan Wu Mengchen Liu Bin Xiao Jianlong Fu Lu Yuan ViT 28 123 0 14 Apr 2022
Accelerating Attention through Gradient-Based Learned Runtime Pruning Zheng Li Soroush Ghodrati Amir Yazdanbakhsh H. Esmaeilzadeh Mingu Kang 24 17 0 07 Apr 2022
Dynamic Focus-aware Positional Queries for Semantic Segmentation Haoyu He Jianfei Cai Zizheng Pan Jing Liu Jing Zhang Dacheng Tao Bohan Zhuang 34 17 0 04 Apr 2022
Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning Yang He Weihan Liang Dongyang Zhao Hong-Yu Zhou Weifeng Ge Yizhou Yu Wenqiang Zhang ViT 32 45 0 17 Mar 2022
The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy Tianlong Chen Zhenyu (Allen) Zhang Yu Cheng Ahmed Hassan Awadallah Zhangyang Wang ViT 41 37 0 12 Mar 2022
Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice Peihao Wang Wenqing Zheng Tianlong Chen Zhangyang Wang ViT 30 127 0 09 Mar 2022
CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction Zhuoran Song Yihong Xu Zhezhi He Li Jiang Naifeng Jing Xiaoyao Liang ViT 32 39 0 09 Mar 2022
VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer Mengshu Sun Haoyu Ma Guoliang Kang Yi Ding Tianlong Chen Xiaolong Ma Zhangyang Wang Yanzhi Wang ViT 33 45 0 17 Jan 2022
Multi-Dimensional Model Compression of Vision Transformer Zejiang Hou S. Kung ViT 25 16 0 31 Dec 2021
Self-slimmed Vision Transformer Zhuofan Zong Kunchang Li Guanglu Song Yali Wang Yu Qiao B. Leng Yu Liu ViT 21 30 0 24 Nov 2021
Pruning Self-attentions into Convolutional Layers in Single Path Haoyu He Jianfei Cai Jing Liu Zizheng Pan Jing Zhang Dacheng Tao Bohan Zhuang ViT 34 40 0 23 Nov 2021
Global Vision Transformer Pruning with Hessian-Aware Saliency Huanrui Yang Hongxu Yin Maying Shen Pavlo Molchanov Hai Helen Li Jan Kautz ViT 30 39 0 10 Oct 2021
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer Yifan Xu Zhijie Zhang Mengdan Zhang Kekai Sheng Ke Li Weiming Dong Liqing Zhang Changsheng Xu Xing Sun ViT 32 202 0 03 Aug 2021
CMT: Convolutional Neural Networks Meet Vision Transformers Jianyuan Guo Kai Han Han Wu Yehui Tang Chunjing Xu Yunhe Wang Chang Xu ViT 351 633 0 13 Jul 2021
Rethinking Token-Mixing MLP for MLP-based Vision Backbone Tan Yu Xu Li Yunfeng Cai Mingming Sun Ping Li 45 26 0 28 Jun 2021
Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity Shiwei Liu Tianlong Chen Zahra Atashgahi Xiaohan Chen Ghada Sokar Elena Mocanu Mykola Pechenizkiy Zhangyang Wang Decebal Constantin Mocanu OOD 31 49 0 28 Jun 2021