Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

29 June 2020

Angelos Katharopoulos

Papers citing "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention"

50 / 346 papers shown

Title
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues Riccardo Grazzi Julien N. Siems Jörg Franke Arber Zela Frank Hutter Massimiliano Pontil 92 11 0 19 Nov 2024
Breaking the Low-Rank Dilemma of Linear Attention Qihang Fan Huaibo Huang Ran He 47 1 0 12 Nov 2024
Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass Tong Chen Hao Fang Patrick Xia Xiaodong Liu Benjamin Van Durme Luke Zettlemoyer Jianfeng Gao Hao Cheng KELM 53 2 0 08 Nov 2024
ETO:Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses Junjie Ni Guofeng Zhang Guanglin Li Yijin Li Xinyang Liu Zhaoyang Huang Hujun Bao ViT 63 2 0 30 Oct 2024
Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation Zhaochong An Guolei Sun Yun Liu Runjia Li Min Wu Ming-Ming Cheng Ender Konukoglu Serge Belongie 64 4 0 29 Oct 2024
Multi-view biomedical foundation models for molecule-target and property prediction Parthasarathy Suryanarayanan Yunguang Qiu Shreyans Sethi Diwakar Mahajan Hongyang Li ... Bum Chul Kwon Pablo Meyer Feixiong Cheng Jianying Hu Joseph A. Morrone AI4CE 36 0 0 25 Oct 2024
Mixture of Parrots: Experts improve memorization more than reasoning Samy Jelassi Clara Mohri David Brandfonbrener Alex Gu Nikhil Vyas Nikhil Anand David Alvarez-Melis Yuanzhi Li Sham Kakade Eran Malach MoE 33 4 0 24 Oct 2024
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination Jerry Huang Prasanna Parthasarathi Mehdi Rezagholizadeh Boxing Chen Sarath Chandar 53 0 0 22 Oct 2024
Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Models Qitan Lv Jie Wang Hanzhu Chen Bin Li Yongdong Zhang Feng Wu HILM 28 3 0 19 Oct 2024
Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusion Chaodong Xiao Minghan Li Zhengqiang Zhang Deyu Meng Lei Zhang Mamba 60 4 0 19 Oct 2024
Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Pingyi Chen Zhongyi Shui Chenglu Zhu Lin Yang MedIm 44 4 0 18 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 149 0 0 17 Oct 2024
SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs Yizhao Gao Zhichen Zeng Dayou Du Shijie Cao Hayden Kwok-Hay So ... Junjie Lai Mao Yang Ting Cao Fan Yang M. Yang 52 19 0 17 Oct 2024
State-space models can learn in-context by gradient descent Neeraj Mohan Sushma Yudou Tian Harshvardhan Mestha Nicolo Colombo David Kappel Anand Subramoney 41 3 0 15 Oct 2024
ControlMM: Controllable Masked Motion Generation Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue Cheng Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 37 4 0 14 Oct 2024
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers Enze Xie Junsong Chen Junyu Chen Han Cai Haotian Tang ... Zhekai Zhang Muyang Li Ligeng Zhu Yaojie Lu Song Han VLM 46 49 0 14 Oct 2024
A Consistency-Aware Spot-Guided Transformer for Versatile and Hierarchical Point Cloud Registration Renlang Huang Yufan Tang Jiming Chen Liang Li 3DPC 44 1 0 14 Oct 2024
Lambda-Skip Connections: the architectural component that prevents Rank Collapse Federico Arangath Joseph Jerome Sieber M. Zeilinger Carmen Amo Alonso 33 0 0 14 Oct 2024
Parameter-Efficient Fine-Tuning of State Space Models Kevin Galim Wonjun Kang Yuchen Zeng H. Koo Kangwook Lee 31 4 0 11 Oct 2024
Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling Yingfa Chen Xinrong Zhang Shengding Hu Xu Han Zhiyuan Liu Maosong Sun Mamba 59 2 0 09 Oct 2024
Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity Mutian He Philip N. Garner 82 0 0 09 Oct 2024
Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions Zhihao He Hang Yu Zi Gong Shizhan Liu J. Li Weiyao Lin VLM 38 1 0 09 Oct 2024
Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective Jinhao Li Jiaming Xu Shan Huang Yonghua Chen Wen Li ... Jiayi Pan Li Ding Hao Zhou Yu Wang Guohao Dai 62 16 0 06 Oct 2024
S7: Selective and Simplified State Space Layers for Sequence Modeling Taylan Soydan Nikola Zubić Nico Messikommer Siddhartha Mishra Davide Scaramuzza 44 4 0 04 Oct 2024
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration Jintao Zhang Jia wei Pengle Zhang Jun-Jie Zhu Jun Zhu Jianfei Chen VLM MQ 82 19 0 03 Oct 2024
Towards Understanding the Universality of Transformers for Next-Token Prediction Michael E. Sander Gabriel Peyré CML 39 0 0 03 Oct 2024
Selective Attention Improves Transformer Yaniv Leviathan Matan Kalman Yossi Matias 51 9 0 03 Oct 2024
Tuning Frequency Bias of State Space Models Annan Yu Dongwei Lyu S. H. Lim Michael W. Mahoney N. Benjamin Erichson 47 3 0 02 Oct 2024
Token Turing Machines are Efficient Vision Models Purvish Jajal Nick Eliopoulos Benjamin Shiue-Hal Chou George K. Thiravathukal James C. Davis Yung-Hsiang Lu 98 0 0 11 Sep 2024
Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer Jinghan Yao Sam Ade Jacobs Masahiro Tanaka Olatunji Ruwase Hari Subramoni D. Panda 33 2 0 30 Aug 2024
Audio xLSTMs: Learning Self-Supervised Audio Representations with xLSTMs Sarthak Yadav Sergios Theodoridis Zheng-Hua Tan 48 2 0 29 Aug 2024
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models Aviv Bick Kevin Y. Li Eric P. Xing J. Zico Kolter Albert Gu Mamba 56 24 0 19 Aug 2024
ELASTIC: Efficient Linear Attention for Sequential Interest Compression Jiaxin Deng Shiyao Wang Song Lu Yinfeng Li Xinchen Luo Yuanjun Liu Peixing Xu Guorui Zhou 44 0 0 18 Aug 2024
End-to-end Semantic-centric Video-based Multimodal Affective Computing Ronghao Lin Ying Zeng Sijie Mai Haifeng Hu VGen 45 0 0 14 Aug 2024
Sampling Foundational Transformer: A Theoretical Perspective Viet Anh Nguyen Minh Lenhat Khoa Nguyen Duong Duc Hieu Dao Huu Hung Truong-Son Hy 44 0 0 11 Aug 2024
MacFormer: Semantic Segmentation with Fine Object Boundaries Guoan Xu Wenfeng Huang Tao Wu Ligeng Chen Wenjing Jia Guangwei Gao Xiatian Zhu Stuart W. Perry 40 0 0 11 Aug 2024
Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness Xiaojing Fan Chunliang Tao AAML 39 28 0 08 Aug 2024
MambaGesture: Enhancing Co-Speech Gesture Generation with Mamba and Disentangled Multi-Modality Fusion Chencan Fu Yabiao Wang Jiangning Zhang Zhengkai Jiang Xiaofeng Mao Jiafu Wu Weijian Cao Chengjie Wang Yanhao Ge Yong Liu Mamba 43 2 0 29 Jul 2024
Continuity Preserving Online CenterLine Graph Learning Yunhui Han Kun Yu Zhiwei Li GNN 3DPC 48 2 0 16 Jul 2024
Low-Rank Interconnected Adaptation Across Layers Yibo Zhong Yao Zhou OffRL MoE 48 1 0 13 Jul 2024
How Effective are State Space Models for Machine Translation? Hugo Pitorro Pavlo Vasylenko Marcos Vinícius Treviso André F. T. Martins Mamba 45 2 0 07 Jul 2024
Linear Attention Based Deep Nonlocal Means Filtering for Multiplicative Noise Removal Xiao Siyao Huang Libing Zhang Shunsheng 44 0 0 06 Jul 2024
Learning to (Learn at Test Time): RNNs with Expressive Hidden States Yu Sun Xinhao Li Karan Dalal Jiarui Xu Arjun Vikram ... Xinlei Chen Xiaolong Wang Sanmi Koyejo Tatsunori Hashimoto Carlos Guestrin 63 92 0 05 Jul 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 36 0 0 03 Jul 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 43 17 0 01 Jul 2024
Vision Mamba-based autonomous crack segmentation on concrete, asphalt, and masonry surfaces Zhaohui Chen Elyas Asadi Shamsabadi Sheng Jiang Luming Shen Daniel Dias-da-Costa Mamba 39 3 0 24 Jun 2024
Separations in the Representational Capabilities of Transformers and Recurrent Architectures S. Bhattamishra Michael Hahn Phil Blunsom Varun Kanade GNN 44 9 0 13 Jun 2024
Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences Zicheng Liu Siyuan Li Li Wang Zedong Wang Yunfan Liu Stan Z. Li 35 7 0 12 Jun 2024
Attention as a Hypernetwork Simon Schug Seijin Kobayashi Yassir Akram João Sacramento Razvan Pascanu GNN 37 3 0 09 Jun 2024
Accelerating Transformers with Spectrum-Preserving Token Merging Hoai-Chau Tran D. M. Nguyen Duy M. Nguyen Trung Thanh Nguyen Ngan Le Pengtao Xie Daniel Sonntag James Y. Zou Binh T. Nguyen Mathias Niepert 42 8 0 25 May 2024