Hydra Attention: Efficient Attention with Many Heads

Hydra Attention: Efficient Attention with Many Heads

15 September 2022

Papers citing "Hydra Attention: Efficient Attention with Many Heads"

15 / 15 papers shown

Title
PolaFormer: Polarity-aware Linear Attention for Vision Transformers Weikang Meng Yadan Luo Xin Li D. Jiang Zheng Zhang 388 2 0 25 Jan 2025
Breaking the Low-Rank Dilemma of Linear Attention Qihang Fan Huaibo Huang Ran He 83 1 0 12 Nov 2024
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers Tobias Christian Nauen Sebastián M. Palacio Federico Raue Andreas Dengel 81 4 0 18 Aug 2023
AdaViT: Adaptive Tokens for Efficient Vision Transformer Hongxu Yin Arash Vahdat J. Álvarez Arun Mallya Jan Kautz Pavlo Molchanov ViT 73 327 0 14 Dec 2021
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection Yanghao Li Chaoxia Wu Haoqi Fan K. Mangalam Bo Xiong Jitendra Malik Christoph Feichtenhofer ViT 142 685 0 02 Dec 2021
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer Sachin Mehta Mohammad Rastegari ViT 271 1,246 0 05 Oct 2021
Mobile-Former: Bridging MobileNet and Transformer Yinpeng Chen Xiyang Dai Dongdong Chen Mengchen Liu Xiaoyi Dong Lu Yuan Zicheng Liu ViT 231 484 0 12 Aug 2021
Multiscale Vision Transformers Haoqi Fan Bo Xiong K. Mangalam Yanghao Li Zhicheng Yan Jitendra Malik Christoph Feichtenhofer ViT 125 1,248 0 22 Apr 2021
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 163 1,548 0 30 Sep 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 157 1,734 0 29 Jun 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 179 1,678 0 08 Jun 2020
Quantifying Attention Flow in Transformers Samira Abnar Willem H. Zuidema 117 789 0 02 May 2020
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 95 1,051 0 25 May 2019
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita David Talbot F. Moiseev Rico Sennrich Ivan Titov 89 1,120 0 23 May 2019
The Cityscapes Dataset for Semantic Urban Scene Understanding Marius Cordts Mohamed Omran Sebastian Ramos Timo Rehfeld Markus Enzweiler Rodrigo Benenson Uwe Franke Stefan Roth Bernt Schiele 860 11,540 0 06 Apr 2016