Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 1,050 papers shown

Title
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 81 221 0 21 Feb 2022
Joint Learning of Frequency and Spatial Domains for Dense Predictions Shaocheng Jia Wei-Ting Yao 33 0 0 18 Feb 2022
cosFormer: Rethinking Softmax in Attention Zhen Qin Weixuan Sun Huicai Deng Dongxu Li Yunshen Wei Baohong Lv Junjie Yan Lingpeng Kong Yiran Zhong 38 212 0 17 Feb 2022
ActionFormer: Localizing Moments of Actions with Transformers Chen-Da Liu-Zhang Jianxin Wu Yin Li ViT 31 333 0 16 Feb 2022
The NLP Task Effectiveness of Long-Range Transformers Guanghui Qin Yukun Feng Benjamin Van Durme 18 28 0 16 Feb 2022
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations Youwei Liang Chongjian Ge Zhan Tong Yibing Song Jue Wang P. Xie ViT 25 237 0 16 Feb 2022
General-purpose, long-context autoregressive modeling with Perceiver AR Curtis Hawthorne Andrew Jaegle Cătălina Cangea Sebastian Borgeaud C. Nash ... Hannah R. Sheahan Neil Zeghidour Jean-Baptiste Alayrac João Carreira Jesse Engel 43 65 0 15 Feb 2022
MuLD: The Multitask Long Document Benchmark G. Hudson Noura Al Moubayed 27 10 0 15 Feb 2022
Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens Felix Ott David Rügamer Lucas Heublein Tim Hamann Jens Barth Bernd Bischl Christopher Mutschler 32 18 0 14 Feb 2022
CATs++: Boosting Cost Aggregation with Convolutions and Transformers Seokju Cho Sunghwan Hong Seung Wook Kim ViT 29 34 0 14 Feb 2022
Flowformer: Linearizing Transformers with Conservation Flows Haixu Wu Jialong Wu Jiehui Xu Jianmin Wang Mingsheng Long 14 90 0 13 Feb 2022
Exploiting Spatial Sparsity for Event Cameras with Visual Transformers Zuowen Wang Yuhuang Hu Shih-Chii Liu ViT 36 33 0 10 Feb 2022
Memory-based gaze prediction in deep imitation learning for robot manipulation Heecheol Kim Yoshiyuki Ohmura Yasuo Kuniyoshi 24 14 0 10 Feb 2022
Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models Beren Millidge Tommaso Salvatori Yuhang Song Thomas Lukasiewicz Rafal Bogacz VLM 24 52 0 09 Feb 2022
Patch-Based Stochastic Attention for Image Editing Nicolas Cherel Andrés Almansa Y. Gousseau A. Newson 25 6 0 07 Feb 2022
Machine Translation from Signed to Spoken Languages: State of the Art and Challenges Mathieu De Coster D. Shterionov Mieke Van Herreweghe J. Dambre SLR 24 40 0 07 Feb 2022
Structure-Aware Transformer for Graph Representation Learning Dexiong Chen Leslie O’Bray Karsten Borgwardt 36 241 0 07 Feb 2022
Exploring Self-Attention Mechanisms for Speech Separation Cem Subakan Mirco Ravanelli Samuele Cornell François Grondin Mirko Bronzi 40 23 0 06 Feb 2022
Local Feature Matching with Transformers for low-end devices Kyrylo Kolodiazhnyi 21 0 0 01 Feb 2022
Improving Sample Efficiency of Value Based Models Using Attention and Vision Transformers Amir Ardalan Kalantari Mohammad Amini Sarath Chandar Doina Precup 54 4 0 01 Feb 2022
Fast Monte-Carlo Approximation of the Attention Mechanism Hyunjun Kim Jeonggil Ko 22 2 0 30 Jan 2022
FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting Tian Zhou Ziqing Ma Qingsong Wen Xue Wang Liang Sun Rong Jin AI4TS 30 1,317 0 30 Jan 2022
Transformers in Medical Imaging: A Survey Fahad Shamshad Salman Khan Syed Waqas Zamir Muhammad Haris Khan Munawar Hayat Fahad Shahbaz Khan Huazhu Fu ViT LM&MA MedIm 111 663 0 24 Jan 2022
Dual-Flattening Transformers through Decomposed Row and Column Queries for Semantic Segmentation Ying Wang C. Ho Wenju Xu Ziwei Xuan Xudong Liu Guo-Jun Qi ViT 28 5 0 22 Jan 2022
Continual Transformers: Redundancy-Free Attention for Online Inference Lukas Hedegaard Arian Bakhtiarnia Alexandros Iosifidis CLL 27 11 0 17 Jan 2022
Transformers in Action: Weakly Supervised Action Segmentation John Ridley Huseyin Coskun D. Tan Nassir Navab F. Tombari ViT 25 5 0 14 Jan 2022
GateFormer: Speeding Up News Feed Recommendation with Input Gated Transformers Peitian Zhang Zheng liu AI4TS 30 1 0 12 Jan 2022
Latency Adjustable Transformer Encoder for Language Understanding Sajjad Kachuee M. Sharifkhani 37 0 0 10 Jan 2022
QuadTree Attention for Vision Transformers Shitao Tang Jiahui Zhang Siyu Zhu Ping Tan ViT 171 156 0 08 Jan 2022
Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networks Lei Cheng Ruslan Khalitov Tong Yu Zhirong Yang 25 32 0 06 Jan 2022
Learning Operators with Coupled Attention Georgios Kissas Jacob H. Seidman Leonardo Ferreira Guilhoto V. Preciado George J. Pappas P. Perdikaris 32 110 0 04 Jan 2022
Multi-Dimensional Model Compression of Vision Transformer Zejiang Hou S. Kung ViT 30 16 0 31 Dec 2021
Self-Supervised Graph Representation Learning for Neuronal Morphologies Marissa A. Weis Laura Hansel Timo Lüddecke Alexander S. Ecker MedIm 25 7 0 23 Dec 2021
Cost Aggregation Is All You Need for Few-Shot Segmentation Sunghwan Hong Seokju Cho Jisu Nam Seungryong Kim ViT 29 23 0 22 Dec 2021
Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization Md Tahmid Rahman Laskar Enamul Hoque J. Huang 45 45 0 22 Dec 2021
LocFormer: Enabling Transformers to Perform Temporal Moment Localization on Long Untrimmed Videos With a Feature Sampling Approach Cristian Rodriguez-Opazo Edison Marrese-Taylor Basura Fernando Hiroya Takamura Qi Wu ViT 19 3 0 19 Dec 2021
Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction Guang-Sheng Chen Meiling Wang Yufeng Yue Qingxiang Zhang Li-xin Yuan ViT 39 17 0 17 Dec 2021
Neural Architectures for Biological Inter-Sentence Relation Extraction Enrique Noriega-Atala Peter Lovett Clayton T. Morrison Mihai Surdeanu NAI 33 3 0 17 Dec 2021
LongT5: Efficient Text-To-Text Transformer for Long Sequences Mandy Guo Joshua Ainslie David C. Uthus Santiago Ontanon Jianmo Ni Yun-hsuan Sung Yinfei Yang VLM 31 306 0 15 Dec 2021
AdaViT: Adaptive Tokens for Efficient Vision Transformer Hongxu Yin Arash Vahdat J. Álvarez Arun Mallya Jan Kautz Pavlo Molchanov ViT 35 320 0 14 Dec 2021
Simple Local Attentions Remain Competitive for Long-Context Tasks Wenhan Xiong Barlas Ouguz Anchit Gupta Xilun Chen Diana Liskovich Omer Levy Wen-tau Yih Yashar Mehdad 49 29 0 14 Dec 2021
Embracing Single Stride 3D Object Detector with Sparse Transformer Lue Fan Ziqi Pang Tianyuan Zhang Yu-xiong Wang Hang Zhao Feng Wang Naiyan Wang Zhaoxiang Zhang ViT 27 255 0 13 Dec 2021
Discourse-Aware Soft Prompting for Text Generation Marjan Ghazvininejad Vladimir Karpukhin Vera Gor Asli Celikyilmaz 31 6 0 10 Dec 2021
Self-attention Does Not Need $O(n^2)$ Memory M. Rabe Charles Staats LRM 26 142 0 10 Dec 2021
Couplformer:Rethinking Vision Transformer with Coupling Attention Map Hai Lan Xihao Wang Xian Wei ViT 34 3 0 10 Dec 2021
Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences Yifan Chen Qi Zeng Dilek Z. Hakkani-Tür Di Jin Heng Ji Yun Yang 27 4 0 10 Dec 2021
FaceFormer: Speech-Driven 3D Facial Animation with Transformers Yingruo Fan Zhaojiang Lin Jun Saito Wenping Wang Taku Komura CVBM 57 197 0 10 Dec 2021
PE-former: Pose Estimation Transformer Paschalis Panteleris Antonis Argyros ViT 21 12 0 09 Dec 2021
Attention-Based Model and Deep Reinforcement Learning for Distribution of Event Processing Tasks A. Mazayev F. Al-Tam N. Correia 36 5 0 07 Dec 2021
A deep language model to predict metabolic network equilibria Franccois Charton Amaury Hayat Sean T. McQuade Nathaniel J. Merrill B. Piccoli GNN 6 5 0 07 Dec 2021