On The Computational Complexity of Self-Attention

11 September 2022

Papers citing "On The Computational Complexity of Self-Attention"

28 / 28 papers shown

Title
Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform Josh Alman Zhao-quan Song 0 0 0 17 May 2025
Hadamard product in deep learning: Introduction, Advances and Challenges Grigorios G. Chrysos Yongtao Wu Razvan Pascanu Philip Torr V. Cevher AAML 98 0 0 17 Apr 2025
Geometric Median Matching for Robust k-Subset Selection from Noisy Data Anish Acharya Sujay Sanghavi Alexandros G. Dimakis Inderjit S Dhillon AAML 62 0 0 01 Apr 2025
Masked Sensory-Temporal Attention for Sensor Generalization in Quadruped Locomotion Dikai Liu Tianwei Zhang Jianxiong Yin Simon See 92 1 0 13 Mar 2025
TabNSA: Native Sparse Attention for Efficient Tabular Data Learning Ali Eslamian Qiang Cheng 48 0 0 12 Mar 2025
Attention Condensation via Sparsity Induced Regularized Training Eli Sason Darya Frolova Boris Nazarov Felix Goldberd 183 0 0 03 Mar 2025
Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation Yang Cao Zhao-quan Song Chiwun Yang VGen 46 2 0 01 Feb 2025
PolaFormer: Polarity-aware Linear Attention for Vision Transformers Weikang Meng Yadan Luo Xin Li D. Jiang Zheng Zhang 159 0 0 25 Jan 2025
Fast Gradient Computation for RoPE Attention in Almost Linear Time Yifang Chen Jiayan Huo Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 61 11 0 03 Jan 2025
RAM: Replace Attention with MLP for Efficient Multivariate Time Series Forecasting Suhan Guo Jiahong Deng Yi Wei Hui Dou Furao Shen Jian Zhao AI4TS 141 0 0 31 Oct 2024
MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI Arya Tschand Arun Tejusve Raghunath Rajan S. Idgunji Anirban Ghosh J. Holleman ... Rowan Taubitz Sean Zhan Scott Wasson David Kanter Vijay Janapa Reddi 62 3 0 15 Oct 2024
MambaRecon: MRI Reconstruction with Structured State Space Models Yilmaz Korkmaz V. Patel Mamba 35 2 0 19 Sep 2024
T1-contrast Enhanced MRI Generation from Multi-parametric MRI for Glioma Patients with Latent Tumor Conditioning Zach Eidex Mojtaba Safari Richard L. J. Qiu D. Yu Hui-Kuo Shu H. Mao Xiaofeng Yang MedIm 53 1 0 03 Sep 2024
DeMansia: Mamba Never Forgets Any Tokens Ricky Fang Mamba 24 0 0 04 Aug 2024
Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learning Subhojyoti Mukherjee Josiah P. Hanna Qiaomin Xie Robert Nowak 74 2 0 07 Jun 2024
EchoMamba4Rec: Harmonizing Bidirectional State Space Models with Spectral Filtering for Advanced Sequential Recommendation Yuda Wang Xuxin He Shengxin Zhu Mamba 34 8 0 04 Jun 2024
MamMIL: Multiple Instance Learning for Whole Slide Images with State Space Models Zijie Fang Yifeng Wang Zhi Wang Jian Zhang Xiangyang Ji Yongbing Zhang Mamba 39 6 0 08 Mar 2024
QuadraNet: Improving High-Order Neural Interaction Efficiency with Hardware-Aware Quadratic Neural Networks Chenhui Xu Fuxun Yu Zirui Xu Chenchen Liu Jinjun Xiong Xiang Chen 33 4 0 29 Nov 2023
Transformers in Reinforcement Learning: A Survey Pranav Agarwal A. Rahman P. St-Charles Simon J. D. Prince Samira Ebrahimi Kahou OffRL 24 18 0 12 Jul 2023
Language Model Tokenizers Introduce Unfairness Between Languages Aleksandar Petrov Emanuele La Malfa Philip Torr Adel Bibi 23 97 0 17 May 2023
A Contrastive Learning Scheme with Transformer Innate Patches S. Jyhne Per-Arne Andersen Morten Goodwin Olsen ViT 26 0 0 26 Mar 2023
Hungry Hungry Hippos: Towards Language Modeling with State Space Models Daniel Y. Fu Tri Dao Khaled Kamal Saab A. Thomas Atri Rudra Christopher Ré 73 370 0 28 Dec 2022
MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers Mohammadmahdi Nouriborji Omid Rohanian Samaneh Kouchaki David A. Clifton 32 8 0 12 Oct 2022
SimA: Simple Softmax-free Attention for Vision Transformers Soroush Abbasi Koohpayegani Hamed Pirsiavash 21 25 0 17 Jun 2022
Deciphering the Language of Nature: A transformer-based language model for deleterious mutations in proteins Theodore Jiang Li Fang Kai Wang MedIm 33 17 0 27 Oct 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,430 0 04 Jan 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 285 2,015 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 252 580 0 12 Mar 2020