On The Computational Complexity of Self-Attention

11 September 2022

Papers citing "On The Computational Complexity of Self-Attention"

26 / 26 papers shown

Title
Hadamard product in deep learning: Introduction, Advances and Challenges Grigorios G. Chrysos Yongtao Wu Razvan Pascanu Philip Torr V. Cevher AAML 98 0 0 17 Apr 2025
Geometric Median Matching for Robust k-Subset Selection from Noisy Data Anish Acharya Sujay Sanghavi Alexandros G. Dimakis Inderjit S Dhillon AAML 59 0 0 01 Apr 2025
Masked Sensory-Temporal Attention for Sensor Generalization in Quadruped Locomotion Dikai Liu Tianwei Zhang Jianxiong Yin Simon See 92 1 0 13 Mar 2025
TabNSA: Native Sparse Attention for Efficient Tabular Data Learning Ali Eslamian Qiang Cheng 45 0 0 12 Mar 2025
Attention Condensation via Sparsity Induced Regularized Training Eli Sason Darya Frolova Boris Nazarov Felix Goldberd 181 0 0 03 Mar 2025
Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation Yang Cao Zhao-quan Song Chiwun Yang VGen 46 2 0 01 Feb 2025
PolaFormer: Polarity-aware Linear Attention for Vision Transformers Weikang Meng Yadan Luo Xin Li D. Jiang Zheng Zhang 151 0 0 25 Jan 2025
Fast Gradient Computation for RoPE Attention in Almost Linear Time Yifang Chen Jiayan Huo Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 61 11 0 03 Jan 2025
RAM: Replace Attention with MLP for Efficient Multivariate Time Series Forecasting Suhan Guo Jiahong Deng Yi Wei Hui Dou F. Shen Jian Zhao AI4TS 139 0 0 31 Oct 2024
MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI Arya Tschand Arun Tejusve Raghunath Rajan S. Idgunji Anirban Ghosh J. Holleman ... Rowan Taubitz Sean Zhan Scott Wasson David Kanter Vijay Janapa Reddi 62 3 0 15 Oct 2024
MambaRecon: MRI Reconstruction with Structured State Space Models Yilmaz Korkmaz V. Patel Mamba 35 2 0 19 Sep 2024
T1-contrast Enhanced MRI Generation from Multi-parametric MRI for Glioma Patients with Latent Tumor Conditioning Zach Eidex Mojtaba Safari Richard L. J. Qiu D. Yu Hui-Kuo Shu H. Mao Xiaofeng Yang MedIm 45 1 0 03 Sep 2024
DeMansia: Mamba Never Forgets Any Tokens Ricky Fang Mamba 21 0 0 04 Aug 2024
Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learning Subhojyoti Mukherjee Josiah P. Hanna Qiaomin Xie Robert Nowak 72 2 0 07 Jun 2024
EchoMamba4Rec: Harmonizing Bidirectional State Space Models with Spectral Filtering for Advanced Sequential Recommendation Yuda Wang Xuxin He Shengxin Zhu Mamba 34 8 0 04 Jun 2024
MamMIL: Multiple Instance Learning for Whole Slide Images with State Space Models Zijie Fang Yifeng Wang Zhi Wang Jian Andrew Zhang Xiangyang Ji Yongbing Zhang Mamba 39 6 0 08 Mar 2024
QuadraNet: Improving High-Order Neural Interaction Efficiency with Hardware-Aware Quadratic Neural Networks Chenhui Xu Fuxun Yu Zirui Xu Chenchen Liu Jinjun Xiong Xiang Chen 33 4 0 29 Nov 2023
Language Model Tokenizers Introduce Unfairness Between Languages Aleksandar Petrov Emanuele La Malfa Philip H. S. Torr Adel Bibi 16 96 0 17 May 2023
A Contrastive Learning Scheme with Transformer Innate Patches S. Jyhne Per-Arne Andersen Morten Goodwin Olsen ViT 26 0 0 26 Mar 2023
Hungry Hungry Hippos: Towards Language Modeling with State Space Models Daniel Y. Fu Tri Dao Khaled Kamal Saab A. Thomas Atri Rudra Christopher Ré 70 370 0 28 Dec 2022
MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers Mohammadmahdi Nouriborji Omid Rohanian Samaneh Kouchaki David A. Clifton 24 8 0 12 Oct 2022
SimA: Simple Softmax-free Attention for Vision Transformers Soroush Abbasi Koohpayegani Hamed Pirsiavash 16 25 0 17 Jun 2022
Deciphering the Language of Nature: A transformer-based language model for deleterious mutations in proteins Theodore Jiang Li Fang Kai Wang MedIm 25 17 0 27 Oct 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,430 0 04 Jan 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 280 2,015 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 243 580 0 12 Mar 2020