Skip-Attention: Improving Vision Transformers by Paying Less Attention

5 January 2023

Shashanka Venkataramanan

Fatih Porikli

Papers citing "Skip-Attention: Improving Vision Transformers by Paying Less Attention"

23 / 23 papers shown

Title
Image Recognition with Online Lightweight Vision Transformer: A Survey Zherui Zhang Rongtao Xu Jie Zhou Changwei Wang Xingtian Pei ... Jiguang Zhang Li Guo Longxiang Gao W. Xu Shibiao Xu ViT 139 0 0 06 May 2025
Enhancing Layer Attention Efficiency through Pruning Redundant Retrievals Hanze Li Xiande Huang 41 0 0 09 Mar 2025
Memory Efficient Matting with Adaptive Token Routing Yiheng Lin Yihan Hu Chenyi Zhang Ting Liu Xiaochao Qu Luoqi Liu Yao Zhao Y. X. Wei 66 0 0 14 Dec 2024
SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization Jintao Zhang Haofeng Huang Pengle Zhang Jia wei Jun-Jie Zhu Jianfei Chen VLM MQ 63 2 0 17 Nov 2024
On the Surprising Effectiveness of Attention Transfer for Vision Transformers Alexander C. Li Yuandong Tian B. Chen Deepak Pathak Xinlei Chen 40 0 0 14 Nov 2024
ED-ViT: Splitting Vision Transformer for Distributed Inference on Edge Devices Xiang Liu Yijun Song Xia Li Yifei Sun Huiying Lan Zemin Liu Linshan Jiang Jialin Li 17 1 0 15 Oct 2024
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration Jintao Zhang Jia wei Pengle Zhang Jun-Jie Zhu Jun Zhu Jianfei Chen VLM MQ 82 18 0 03 Oct 2024
Replacement Learning: Training Vision Tasks with Fewer Learnable Parameters Yuming Zhang Peizhe Wang Shouxin Zhang Dongzhi Guan Jiabin Liu Junhao Su 33 0 0 02 Oct 2024
Sharing Key Semantics in Transformer Makes Efficient Image Restoration Bin Ren Yawei Li Jingyun Liang Rakesh Ranjan Mengyuan Liu Rita Cucchiara Luc Van Gool Ming-Hsuan Yang N. Sebe 34 3 0 30 May 2024
Linearly-evolved Transformer for Pan-sharpening Junming Hou Zihan Cao Naishan Zheng Xuan Li Xiaoyu Chen Xinyang Liu Xiaofeng Cong Man Zhou Danfeng Hong ViT 26 7 0 19 Apr 2024
Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit? Bill Psomas Ioannis Kakogeorgiou Konstantinos Karantzalos Yannis Avrithis ViT 25 8 0 13 Sep 2023
ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection Jifeng Shen Yifei Chen Yue Liu Xin Zuo Heng Fan Wankou Yang ViT 21 88 0 15 Aug 2023
Efficiency 360: Efficient Vision Transformers Badri N. Patro Vijay Srinivas Agneeswaran 26 6 0 16 Feb 2023
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer Sachin Mehta Mohammad Rastegari ViT 200 1,212 0 05 Oct 2021
Dynamic Attentive Graph Learning for Image Restoration Chong Mou Jian Andrew Zhang Zhuoyuan Wu DiffM 63 81 0 14 Sep 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 314 5,775 0 29 Apr 2021
Transformer in Transformer Kai Han An Xiao Enhua Wu Jianyuan Guo Chunjing Xu Yunhe Wang ViT 284 1,524 0 27 Feb 2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions Wenhai Wang Enze Xie Xiang Li Deng-Ping Fan Kaitao Song Ding Liang Tong Lu Ping Luo Ling Shao ViT 277 3,622 0 24 Feb 2021
Evolving Attention with Residual Convolutions Yujing Wang Yaming Yang Jiangang Bai Mingliang Zhang Jing Bai J. Yu Ce Zhang Gao Huang Yunhai Tong ViT 26 32 0 20 Feb 2021
Multi-Stage Progressive Image Restoration Syed Waqas Zamir Aditya Arora Salman Khan Munawar Hayat F. Khan Ming-Hsuan Yang Ling Shao 125 1,450 0 04 Feb 2021
NBNet: Noise Basis Learning for Image Denoising with Subspace Projection Shuyang Cheng Yuzhi Wang Haibin Huang Donghao Liu Haoqiang Fan Shuaicheng Liu 107 187 0 30 Dec 2020
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Z. Tu Kaiming He 297 10,216 0 16 Nov 2016
Xception: Deep Learning with Depthwise Separable Convolutions François Chollet MDE BDL PINN 206 14,367 0 07 Oct 2016