Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 1,050 papers shown

Title
A Cookbook of Self-Supervised Learning Randall Balestriero Mark Ibrahim Vlad Sobal Ari S. Morcos Shashank Shekhar ... Pierre Fernandez Amir Bar Hamed Pirsiavash Yann LeCun Micah Goldblum SyDa FedML SSL 50 275 0 24 Apr 2023
An Unbiased Transformer Source Code Learning with Semantic Vulnerability Graph Nafis Tanveer Islam G. Parra Dylan Manuel E. Bou-Harb Peyman Najafirad 28 8 0 17 Apr 2023
Improving Autoregressive NLP Tasks via Modular Linearized Attention Victor Agostinelli Lizhong Chen 27 1 0 17 Apr 2023
Long-term Forecasting with TiDE: Time-series Dense Encoder Abhimanyu Das Weihao Kong Andrew B. Leach Shaan Mathur Rajat Sen Rose Yu AI4TS 53 242 0 17 Apr 2023
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference Tao Lei Junwen Bai Siddhartha Brahma Joshua Ainslie Kenton Lee ... Vincent Zhao Yuexin Wu Yue Liu Yu Zhang Ming-Wei Chang BDL AI4CE 32 55 0 11 Apr 2023
Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimension Yichuan Deng Sridhar Mahadevan Zhao Song 22 35 0 10 Apr 2023
SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers Alberto Marchisio David Durà Maurizio Capra Maurizio Martina Guido Masera Mohamed Bennai 41 20 0 08 Apr 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 35 41 0 07 Apr 2023
Devil's on the Edges: Selective Quad Attention for Scene Graph Generation Deunsol Jung Sanghyun Kim Wonhui Kim Minsu Cho 3DPC GNN 32 32 0 07 Apr 2023
EGA-Depth: Efficient Guided Attention for Self-Supervised Multi-Camera Depth Estimation Y. Shi H. Cai Amin Ansari Fatih Porikli MDE 88 17 0 06 Apr 2023
DIR-AS: Decoupling Individual Identification and Temporal Reasoning for Action Segmentation Peiyao Wang Haibin Ling 15 2 0 04 Apr 2023
Dialogue-Contextualized Re-ranking for Medical History-Taking Jian Zhu Ilya Valmianski Anitha Kannan 30 1 0 04 Apr 2023
Astronomical image time series classification using CONVolutional attENTION (ConvEntion) Anass Bairouk Marc Chaumont D. Fouchez Jerome Paquet Frédéric Comby J. Bautista 16 1 0 03 Apr 2023
Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognition Saumya Yashmohini Sahai Jing Liu Thejaswi Muniyappa Kanthashree Mysore Sathyendra Anastasios Alexandridis ... Ross McGowan Ariya Rastrow Feng-Ju Chang Athanasios Mouchtaris Siegfried Kunzmann 44 5 0 03 Apr 2023
Adaptive Spot-Guided Transformer for Consistent Local Feature Matching Jiahuan Yu Jiahao Chang Jianfeng He Tianzhu Zhang Feng Wu 40 20 0 29 Mar 2023
Solving Regularized Exp, Cosh and Sinh Regression Problems Zhihang Li Zhao Song Dinesh Manocha 36 39 0 28 Mar 2023
Accelerating Trajectory Generation for Quadrotors Using Transformers Srinath Tankasala Mitch Pryor 23 1 0 27 Mar 2023
SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications Abdelrahman M. Shaker Muhammad Maaz H. Rasheed Salman Khan Ming-Hsuan Yang Fahad Shahbaz Khan ViT 53 84 0 27 Mar 2023
TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference Shikhar Tuli N. Jha 40 5 0 27 Mar 2023
You Only Segment Once: Towards Real-Time Panoptic Segmentation Jie Hu Linyan Huang Tianhe Ren Shengchuan Zhang Rongrong Ji Liujuan Cao SSeg 46 55 0 26 Mar 2023
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization Pavan Kumar Anasosalu Vasu J. Gabriel Jeff J. Zhu Oncel Tuzel Anurag Ranjan ViT 42 155 0 24 Mar 2023
Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers Cong Wei Brendan Duke R. Jiang P. Aarabi Graham W. Taylor Florian Shkurti ViT 51 15 0 24 Mar 2023
EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms Shikhar Tuli N. Jha 41 3 0 24 Mar 2023
ReBotNet: Fast Real-time Video Enhancement Jeya Maria Jose Valanarasu Rahul Garg Andeep S. Toor Xin Tong Weijuan Xi Andreas Lugmayr Vishal M. Patel A. Menini 39 0 0 23 Mar 2023
Multiscale Attention via Wavelet Neural Operators for Vision Transformers Anahita Nekoozadeh M. Ahmadzadeh Zahra Mardani ViT 43 2 0 22 Mar 2023
The Multiscale Surface Vision Transformer Simon Dahan Logan Z. J. Williams Daniel Rueckert E. C. Robinson MedIm ViT 17 2 0 21 Mar 2023
Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers Jaehoon Yoo Semin Kim Doyup Lee Chiheon Kim Seunghoon Hong 36 3 0 20 Mar 2023
Robustifying Token Attention for Vision Transformers Yong Guo David Stutz Bernt Schiele ViT 28 24 0 20 Mar 2023
CerviFormer: A Pap-smear based cervical cancer classification method using cross attention and latent transformer Bhaswati Singha Deo M. Pal P. Panigrahi A. Pradhan MedIm 36 22 0 17 Mar 2023
HDformer: A Higher Dimensional Transformer for Diabetes Detection Utilizing Long Range Vascular Signals Ella Lan MedIm 20 1 0 17 Mar 2023
CoLT5: Faster Long-Range Transformers with Conditional Computation Joshua Ainslie Tao Lei Michiel de Jong Santiago Ontañón Siddhartha Brahma ... Mandy Guo James Lee-Thorp Yi Tay Yun-hsuan Sung Sumit Sanghai LLMAG 39 63 0 17 Mar 2023
BiFormer: Vision Transformer with Bi-Level Routing Attention Lei Zhu Xinjiang Wang Zhanghan Ke Wayne Zhang Rynson W. H. Lau 134 490 0 15 Mar 2023
HYBRIDFORMER: improving SqueezeFormer with hybrid attention and NSR mechanism Yuguang Yang Yu Pan Jingjing Yin Jiangyu Han Lei Ma Heng Lu 31 8 0 15 Mar 2023
Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classification Honglin Li Chenglu Zhu Yunlong Zhang Yuxuan Sun Zhongyi Shui Wenwei Kuang S. Zheng Ling Yang 74 57 0 15 Mar 2023
Gradient-Free Structured Pruning with Unlabeled Data Azade Nova H. Dai Dale Schuurmans SyDa 40 20 0 07 Mar 2023
Modular Safety-Critical Control of Legged Robots Berk Tosun Evren Samur 25 0 0 04 Mar 2023
AutoMatch: A Large-scale Audio Beat Matching Benchmark for Boosting Deep Learning Assistant Video Editing Sen Pei Jingya Yu Qi Chen Wozhou He 52 3 0 03 Mar 2023
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration Yawei Li Yuchen Fan Xiaoyu Xiang D. Demandolx Rakesh Ranjan Radu Timofte Luc Van Gool 37 173 0 01 Mar 2023
AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers Shikhar Tuli N. Jha 41 32 0 28 Feb 2023
A Survey on Long Text Modeling with Transformers Zican Dong Tianyi Tang Lunyi Li Wayne Xin Zhao VLM 31 54 0 28 Feb 2023
Elementwise Language Representation Du-Yeong Kim Jeeeun Kim 41 0 0 27 Feb 2023
Fast Attention Requires Bounded Entries Josh Alman Zhao Song 30 81 0 26 Feb 2023
Hyena Hierarchy: Towards Larger Convolutional Language Models Michael Poli Stefano Massaroli Eric Q. Nguyen Daniel Y. Fu Tri Dao S. Baccus Yoshua Bengio Stefano Ermon Christopher Ré VLM 28 286 0 21 Feb 2023
FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification Mingyue Cheng Qi Liu Zhiding Liu Zhi Li Yucong Luo Enhong Chen AI4TS 24 31 0 20 Feb 2023
Efficiency 360: Efficient Vision Transformers Badri N. Patro Vijay Srinivas Agneeswaran 33 6 0 16 Feb 2023
Speculative Decoding with Big Little Decoder Sehoon Kim K. Mangalam Suhong Moon Jitendra Malik Michael W. Mahoney A. Gholami Kurt Keutzer MoE 38 100 0 15 Feb 2023
A Unified View of Long-Sequence Models towards Modeling Million-Scale Dependencies Hongyu Hè Marko Kabić 32 2 0 13 Feb 2023
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment Jared Fernandez Jacob Kahn Clara Na Yonatan Bisk Emma Strubell FedML 38 10 0 13 Feb 2023
In-Context Learning with Many Demonstration Examples Mukai Li Shansan Gong Jiangtao Feng Yiheng Xu Jinchao Zhang Zhiyong Wu Lingpeng Kong 42 32 0 09 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 39 18 0 09 Feb 2023