Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

29 June 2020

Angelos Katharopoulos

Papers citing "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention"

50 / 346 papers shown

Title
The Role of Global and Local Context in Named Entity Recognition Arthur Amalvy Vincent Labatut Richard Dufour 38 4 0 04 May 2023
Application of Transformers for Nonlinear Channel Compensation in Optical Systems Behnam Behinaein Hamgini H. Najafi Ali Bakhshali Zhuhong Zhang 29 1 0 25 Apr 2023
Improving Autoregressive NLP Tasks via Modular Linearized Attention Victor Agostinelli Lizhong Chen 24 1 0 17 Apr 2023
Inductive biases in deep learning models for weather prediction Jannik Thümmel Matthias Karlbauer S. Otte C. Zarfl Georg Martius ... Thomas Scholten Ulrich Friedrich V. Wulfmeyer B. Goswami Martin Volker Butz AI4CE 43 5 0 06 Apr 2023
Learning Neural Eigenfunctions for Unsupervised Semantic Segmentation Zhijie Deng Yucen Luo 26 6 0 06 Apr 2023
DIR-AS: Decoupling Individual Identification and Temporal Reasoning for Action Segmentation Peiyao Wang Haibin Ling 15 2 0 04 Apr 2023
PMatch: Paired Masked Image Modeling for Dense Geometric Matching Shengjie Zhu Xiaoming Liu 30 23 0 30 Mar 2023
Adaptive Spot-Guided Transformer for Consistent Local Feature Matching Jiahuan Yu Jiahao Chang Jianfeng He Tianzhu Zhang Feng Wu 30 20 0 29 Mar 2023
You Only Segment Once: Towards Real-Time Panoptic Segmentation Jie Hu Linyan Huang Tianhe Ren Shengchuan Zhang Rongrong Ji Liujuan Cao SSeg 46 54 0 26 Mar 2023
An Evaluation of Memory Optimization Methods for Training Neural Networks Xiaoxuan Liu Siddharth Jha Alvin Cheung 29 0 0 26 Mar 2023
3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition Lei Wang Piotr Koniusz ViT 28 45 0 25 Mar 2023
Towards Accurate Post-Training Quantization for Vision Transformer Yifu Ding Haotong Qin Qing-Yu Yan Z. Chai Junjie Liu Xiaolin K. Wei Xianglong Liu MQ 54 68 0 25 Mar 2023
Bridging Stereo Geometry and BEV Representation with Reliable Mutual Interaction for Semantic Scene Completion Bohan Li Yasheng Sun Zhujin Liang Dalong Du Zhengbiao Zhu Xiaoefeng Wang Yunpeng Zhang Han Xiao Wenjun Zeng 3DV 33 10 0 24 Mar 2023
Depression Detection Using Digital Traces on Social Media: A Knowledge-aware Deep Learning Approach Wenli Zhang Jiaheng Xie Zhuocheng Zhang Xiang Liu 24 9 0 06 Mar 2023
Improving Transformer-based Image Matching by Cascaded Capturing Spatially Informative Keypoints Chenjie Cao Yanwei Fu ViT 28 9 0 06 Mar 2023
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration Yawei Li Yuchen Fan Xiaoyu Xiang D. Demandolx Rakesh Ranjan Radu Timofte Luc Van Gool 26 173 0 01 Mar 2023
Elementwise Language Representation Du-Yeong Kim Jeeeun Kim 33 0 0 27 Feb 2023
Theory of coupled neuronal-synaptic dynamics David G. Clark L. F. Abbott 24 18 0 17 Feb 2023
Efficiency 360: Efficient Vision Transformers Badri N. Patro Vijay Srinivas Agneeswaran 26 6 0 16 Feb 2023
OAMatcher: An Overlapping Areas-based Network for Accurate Local Feature Matching Kun Dai Tao Xie K. Wang Zhiqiang Jiang Ruifeng Li Lijun Zhao 22 14 0 12 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 34 18 0 09 Feb 2023
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers K. Choromanski Shanda Li Valerii Likhosherstov Kumar Avinava Dubey Shengjie Luo Di He Yiming Yang Tamás Sarlós Thomas Weingarten Adrian Weller 37 8 0 03 Feb 2023
FAVOR#: Sharp Attention Kernel Approximations via New Classes of Positive Random Features Valerii Likhosherstov K. Choromanski Kumar Avinava Dubey Frederick Liu Tamás Sarlós Adrian Weller 20 3 0 01 Feb 2023
On the Connection Between MPNN and Graph Transformer Chen Cai Truong-Son Hy Rose Yu Yusu Wang 39 51 0 27 Jan 2023
Open Problems in Applied Deep Learning M. Raissi AI4CE 42 2 0 26 Jan 2023
A Survey of Meta-Reinforcement Learning Jacob Beck Risto Vuorio E. Liu Zheng Xiong L. Zintgraf Chelsea Finn Shimon Whiteson OOD OffRL 37 122 0 19 Jan 2023
Msanii: High Fidelity Music Synthesis on a Shoestring Budget Kinyugo Maina 24 5 0 16 Jan 2023
Self-Attention Amortized Distributional Projection Optimization for Sliced Wasserstein Point-Cloud Reconstruction Khai Nguyen Dang Nguyen N. Ho 32 9 0 12 Jan 2023
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference Haoxin Li Phillip Keung Daniel Cheng Jungo Kasai Noah A. Smith 17 3 0 11 Jan 2023
Dynamic Grained Encoder for Vision Transformers Lin Song Songyang Zhang Songtao Liu Zeming Li Xuming He Hongbin Sun Jian Sun Nanning Zheng ViT 26 34 0 10 Jan 2023
Black-box language model explanation by context length probing Ondřej Cífka Antoine Liutkus MILM LRM 21 6 0 30 Dec 2022
Hungry Hungry Hippos: Towards Language Modeling with State Space Models Daniel Y. Fu Tri Dao Khaled Kamal Saab A. Thomas Atri Rudra Christopher Ré 73 370 0 28 Dec 2022
End-to-End Modeling Hierarchical Time Series Using Autoregressive Transformer and Conditional Normalizing Flow based Reconciliation Shiyu Wang Fan Zhou Yinbo Sun Lintao Ma James Y. Zhang Yang Zheng BDL AI4TS 32 8 0 28 Dec 2022
A Length-Extrapolatable Transformer Yutao Sun Li Dong Barun Patra Shuming Ma Shaohan Huang Alon Benhaim Vishrav Chaudhary Xia Song Furu Wei 30 115 0 20 Dec 2022
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 125 36 0 15 Dec 2022
Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentation Loic Themyr Clément Rambour Nicolas Thome Toby Collins Alexandre Hostettler ViT 27 10 0 15 Dec 2022
LMEC: Learnable Multiplicative Absolute Position Embedding Based Conformer for Speech Recognition Yuguang Yang Y. Pan Jingjing Yin Heng Lu 32 3 0 05 Dec 2022
A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling Z. Guo J. Kang Dorien Herremans 14 18 0 02 Dec 2022
Lightweight Structure-Aware Attention for Visual Understanding Heeseung Kwon F. M. Castro M. Marín-Jiménez N. Guil Alahari Karteek 28 2 0 29 Nov 2022
FsaNet: Frequency Self-attention for Semantic Segmentation Fengyu Zhang Ashkan Panahi Guangjun Gao AI4TS 32 28 0 28 Nov 2022
TransVCL: Attention-enhanced Video Copy Localization Network with Flexible Supervision Sifeng He Yue He Minlong Lu Chen Jiang Xudong Yang Feng Qian Xuanyang Zhang Lei Yang Jiandong Zhang ViT 15 18 0 23 Nov 2022
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention Zineng Tang Jaemin Cho Jie Lei Joey Tianyi Zhou VLM 24 9 0 21 Nov 2022
ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention Jyotikrishna Dass Shang Wu Huihong Shi Chaojian Li Zhifan Ye Zhongfeng Wang Yingyan Lin 17 49 0 09 Nov 2022
Structured State Space Decoder for Speech Recognition and Synthesis Koichi Miyazaki Masato Murata Tomoki Koriyama 34 12 0 31 Oct 2022
Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost Sungjun Cho Seonwoo Min Jinwoo Kim Moontae Lee Honglak Lee Seunghoon Hong 40 3 0 27 Oct 2022
A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives Carlos Hernandez-Olivan Javier Hernandez-Olivan J. R. Beltrán MGen 40 6 0 25 Oct 2022
Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation Botao Yu Peiling Lu Rui Wang Wei Hu Xu Tan Wei Ye Shikun Zhang Tao Qin Tie-Yan Liu MGen 25 55 0 19 Oct 2022
The Devil in Linear Transformer Zhen Qin Xiaodong Han Weixuan Sun Dongxu Li Lingpeng Kong Nick Barnes Yiran Zhong 36 70 0 19 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 43 9 0 14 Oct 2022
An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification Ilias Chalkidis Xiang Dai Manos Fergadiotis Prodromos Malakasiotis Desmond Elliott 34 33 0 11 Oct 2022