Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers

Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers

18 August 2023

Tobias Christian Nauen

Sebastián M. Palacio

Papers citing "Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers"

13 / 13 papers shown

Title
ForAug: Recombining Foregrounds and Backgrounds to Improve Vision Transformer Training with Bias Mitigation Tobias Christian Nauen Brian B. Moser Federico Raue Stanislav Frolov Andreas Dengel ViT 55 0 0 12 Mar 2025
TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax Tobias Christian Nauen Sebastián M. Palacio Andreas Dengel 51 3 0 05 Mar 2024
Making Vision Transformers Efficient from A Token Sparsification View Shuning Chang Pichao Wang Ming Lin Fan Wang David Junhao Zhang Rong Jin Mike Zheng Shou ViT 43 24 0 15 Mar 2023
Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers Nurullah Sevim Ege Ozan Özyedek Furkan Şahinuç Aykut Koç 30 11 0 26 Sep 2022
Hydra Attention: Efficient Attention with Many Heads Daniel Bolya Cheng-Yang Fu Xiaoliang Dai Peizhao Zhang Judy Hoffman 99 76 0 15 Sep 2022
Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning Ting Yao Yingwei Pan Yehao Li Chong-Wah Ngo Tao Mei ViT 146 137 0 11 Jul 2022
Transformers in Medical Imaging: A Survey Fahad Shamshad Salman Khan Syed Waqas Zamir Muhammad Haris Khan Munawar Hayat F. Khan H. Fu ViT LM&MA MedIm 106 663 0 24 Jan 2022
A Survey of Visual Transformers Yang Liu Yao Zhang Yixin Wang Feng Hou Jin Yuan Jiang Tian Yang Zhang Zhongchao Shi Jianping Fan Zhiqiang He 3DGS ViT 71 330 0 11 Nov 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 271 2,603 0 04 May 2021
ImageNet-21K Pretraining for the Masses T. Ridnik Emanuel Ben-Baruch Asaf Noy Lihi Zelnik-Manor SSeg VLM CLIP 176 686 0 22 Apr 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,428 0 04 Jan 2021
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting Haoyi Zhou Shanghang Zhang J. Peng Shuai Zhang Jianxin Li Hui Xiong Wan Zhang AI4TS 169 3,876 0 14 Dec 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 243 579 0 12 Mar 2020