Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 1,049 papers shown

Title
Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform Josh Alman Zhao Song 9 0 0 17 May 2025
Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain Hyowon Wi Jeongwhan Choi Noseong Park 33 0 0 13 May 2025
Hierarchical Sparse Attention Framework for Computationally Efficient Classification of Biological Cells Elad Yoshai Dana Yagoda-Aharoni Eden Dotan N. Shaked 28 0 0 12 May 2025
Accurate and Efficient Multivariate Time Series Forecasting via Offline Clustering Yiming Niu Jinliang Deng L. Zhang Zimu Zhou Yongxin Tong AI4TS 26 0 0 09 May 2025
Graph Laplacian Wavelet Transformer via Learnable Spectral Decomposition Andrew Kiruluta Eric Lundy Priscilla Burity 29 0 0 09 May 2025
T-T: Table Transformer for Tagging-based Aspect Sentiment Triplet Extraction Kun Peng Chaodong Tong Cong Cao Hao Peng Yue Liu Guanlin Wu Lei Jiang Yanbing Liu Philip S. Yu LMTD 48 0 0 08 May 2025
Image Recognition with Online Lightweight Vision Transformer: A Survey Zherui Zhang Rongtao Xu Jie Zhou Changwei Wang Xingtian Pei ... Jiguang Zhang Li Guo Longxiang Gao Wenyuan Xu Shibiao Xu ViT 172 0 0 06 May 2025
From Attention to Atoms: Spectral Dictionary Learning for Fast, Interpretable Language Models Andrew Kiruluta 29 0 0 29 Apr 2025
DYNAMAX: Dynamic computing for Transformers and Mamba based architectures Miguel Nogales Matteo Gambella Manuel Roveri 56 0 0 29 Apr 2025
RSFR: A Coarse-to-Fine Reconstruction Framework for Diffusion Tensor Cardiac MRI with Semantic-Aware Refinement Jiahao Huang Fanwen Wang Pedro F. Ferreira Haiqi Zhang Yinzhe Wu ... R. Rajakulasingam Ranil De Silva D. Pennell Guang Yang S. Nielles-Vallespin DiffM MedIm 48 0 0 25 Apr 2025
An Empirical Study on Prompt Compression for Large Language Models Zhenru Zhang Jinyi Li Yihuai Lan Qing Guo Hao Wang MQ 51 0 0 24 Apr 2025
Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light Ali Hassani Fengzhe Zhou Aditya Kane Jiannan Huang Chieh-Yun Chen ... Bing Xu Haicheng Wu Wen-mei W. Hwu Xuan Li Humphrey Shi 31 0 0 23 Apr 2025
Landmark-Free Preoperative-to-Intraoperative Registration in Laparoscopic Liver Resection Jun Zhou Bingchen Gao Kai Wang Jialun Pei Pheng-Ann Heng Jing Qin MedIm 37 0 0 21 Apr 2025
ECViT: Efficient Convolutional Vision Transformer with Local-Attention and Multi-scale Stages Zhoujie Qian ViT 29 0 0 21 Apr 2025
CacheFormer: High Attention-Based Segment Caching Sushant Singh A. Mahmood 41 0 0 18 Apr 2025
Packing Input Frame Context in Next-Frame Prediction Models for Video Generation Lvmin Zhang Maneesh Agrawala DiffM VGen 75 0 0 17 Apr 2025
MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models Junyang Zhang Tianyi Zhu Cheng Luo A. Anandkumar RALM 47 0 0 16 Apr 2025
A Review of YOLOv12: Attention-Based Enhancements vs. Previous Versions Rahima Khanam Muhammad Hussain 36 0 0 16 Apr 2025
Millions of States: Designing a Scalable MoE Architecture with RWKV-7 Meta-learner Liu Xiao Li Zhiyuan Lin Yueyu 38 0 0 11 Apr 2025
Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention Andrew Kiruluta Priscilla Burity Samantha Williams 27 3 0 08 Apr 2025
Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision Xiaofeng Han Shunpeng Chen Zenghuang Fu Zhe Feng Lue Fan ... Li Guo Weiliang Meng Xiaopeng Zhang Rongtao Xu Shibiao Xu 74 1 0 03 Apr 2025
FT-Transformer: Resilient and Reliable Transformer with End-to-End Fault Tolerant Attention Huangliang Dai Shixun Wu Hairui Zhao Jiajun Huang Zizhe Jian Yue Zhu Haiyang Hu Zizhong Chen 51 0 0 03 Apr 2025
TransMamba: Flexibly Switching between Transformer and Mamba Yixing Li Ruobing Xie Zhen Yang Xingchen Sun Shuaipeng Li ... Zhanhui Kang Yu Cheng C. Xu Di Wang Jie Jiang Mamba 65 1 0 31 Mar 2025
ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Models Guoyizhe Wei Rama Chellappa 46 0 0 30 Mar 2025
Function Fitting Based on Kolmogorov-Arnold Theorem and Kernel Functions Jianpeng Liu Qizhi Pan 40 0 0 29 Mar 2025
Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendation Jiakai Tang Sunhao Dai Teng Shi Jun Xu X. Chen Wen Chen Wu Jian Yuning Jiang LRM 75 5 0 28 Mar 2025
Burst Image Super-Resolution with Mamba Ozan Unal Steven Marty Dengxin Dai Mamba 48 0 0 25 Mar 2025
Selecting and Pruning: A Differentiable Causal Sequentialized State-Space Model for Two-View Correspondence Learning Xiang Fang Shanghang Zhang Hao Zhang Tao Lu Huabing Zhou Jiayi Ma Mamba 77 0 0 23 Mar 2025
Fractal-IR: A Unified Framework for Efficient and Scalable Image Restoration Yawei Li Bin Ren Christos Sakaridis Rakesh Ranjan Mengyuan Liu N. Sebe Ming-Hsuan Yang Luca Benini 61 0 0 22 Mar 2025
From S4 to Mamba: A Comprehensive Survey on Structured State Space Models Shriyank Somvanshi Md Monzurul Islam Mahmuda Sultana Mimi Sazzad Bin Bashar Polock Gaurab Chhetri Subasish Das Mamba AI4TS 45 0 0 22 Mar 2025
InhibiDistilbert: Knowledge Distillation for a ReLU and Addition-based Transformer Tony Zhang Rickard Brännvall 43 0 0 20 Mar 2025
ATTENTION2D: Communication Efficient Distributed Self-Attention Mechanism Venmugil Elango 53 0 0 20 Mar 2025
PSA-MIL: A Probabilistic Spatial Attention-Based Multiple Instance Learning for Whole Slide Image Classification Sharon Peled Y. Maruvka Moti Freiman 46 0 0 20 Mar 2025
Is Discretization Fusion All You Need for Collaborative Perception? Kang Yang Tianci Bu L. Li Chunxu Li Yanjie Wang Deying Li 68 0 0 18 Mar 2025
Learning Shape-Independent Transformation via Spherical Representations for Category-Level Object Pose Estimation Huan Ren Wenfei Yang Xiang Liu Shifeng Zhang Tianzhu Zhang 69 2 0 18 Mar 2025
The Power of Context: How Multimodality Improves Image Super-Resolution Kangfu Mei Hossein Talebi Mojtaba Ardakani Vishal M. Patel P. Milanfar M. Delbracio DiffM 85 1 0 18 Mar 2025
CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences Ziran Qin Yuchen Cao Mingbao Lin Wen Hu Shixuan Fan Ke Cheng Weiyao Lin Jianguo Li 71 3 0 16 Mar 2025
TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models Xudong Tan Peng Ye Chongjun Tu Jianjian Cao Yaoxin Yang Lin Zhang Dongzhan Zhou Tao Chen VLM 56 0 0 13 Mar 2025
STEAD: Spatio-Temporal Efficient Anomaly Detection for Time and Compute Sensitive Applications Andrew Gao Jun Liu AI4TS 58 0 0 11 Mar 2025
MIRAM: Masked Image Reconstruction Across Multiple Scales for Breast Lesion Risk Prediction H. Q. Vo Pengyu Yuan Zheng Yin Kelvin K. Wong Chika F. Ezeana S. Ly Stephen T. C. Wong H. Nguyen 46 0 0 10 Mar 2025
Similarity-Guided Layer-Adaptive Vision Transformer for UAV Tracking Chaocan Xue Bineng Zhong Qihua Liang Yaozong Zheng Ning Li Yuanliang Xue Shuxiang Song 41 0 0 09 Mar 2025
SED2AM: Solving Multi-Trip Time-Dependent Vehicle Routing Problem using Deep Reinforcement Learning Arash Mozhdehi Yansen Wang Sun Sun Xin Wang AI4TS 68 0 0 06 Mar 2025
ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task Vittorio Pippi Matthieu Guillaumin S. Cascianelli Rita Cucchiara M. Jaritz Loris Bazzani 64 0 0 06 Mar 2025
Partial Convolution Meets Visual Attention Haiduo Huang Fuwei Yang D. Li Ji Liu Lu Tian Jinzhang Peng Pengju Ren E. Barsoum 3DH 198 0 0 05 Mar 2025
Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer Yujiao Yang Jing Lian Linhui Li MoE 82 0 0 04 Mar 2025
Walking the Web of Concept-Class Relationships in Incrementally Trained Interpretable Models Susmit Agrawal Deepika Vemuri S. Paul Vineeth N. Balasubramanian CLL 67 0 0 27 Feb 2025
Minds on the Move: Decoding Trajectory Prediction in Autonomous Driving with Cognitive Insights Haicheng Liao Chengyue Wang Kaiqun Zhu Yilong Ren Bolin Gao Shengbo Eben Li Chengzhong Xu Zehan Li 72 2 0 27 Feb 2025
The FFT Strikes Again: An Efficient Alternative to Self-Attention Jacob Fein-Ashley Rajgopal Kannan Viktor Prasanna 68 2 0 25 Feb 2025
Self-Adjust Softmax Chuanyang Zheng Yihang Gao Guoxuan Chen Han Shi Jing Xiong Xiaozhe Ren Chao Huang Xin Jiang Zhiyu Li Yu Li 50 0 0 25 Feb 2025
Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignment Pedram Zaree Md Abdullah Al Mamun Quazi Mishkatul Alam Yue Dong Ihsen Alouani Nael B. Abu-Ghazaleh AAML 41 0 0 24 Feb 2025