Transformers are Multi-State RNNs

Transformers are Multi-State RNNs

11 January 2024

Yossi Adi

Papers citing "Transformers are Multi-State RNNs"

15 / 15 papers shown

Title
Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity Ruifeng Ren Yong Liu 132 0 0 26 Apr 2025
Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers Markus J. Buehler AI4CE 35 1 0 04 Jan 2025
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 146 0 0 17 Oct 2024
In-context KV-Cache Eviction for LLMs via Attention-Gate Zihao Zeng Bokai Lin Tianqi Hou Hao Zhang Zhijie Deng 38 1 0 15 Oct 2024
Towards LifeSpan Cognitive Systems Yu Wang Chi Han Tongtong Wu Xiaoxin He Wangchunshu Zhou ... Zexue He Wei Wang Gholamreza Haffari Heng Ji Julian McAuley KELM CLL 141 1 0 20 Sep 2024
Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification Jungmin Yun Mihyeon Kim Youngbin Kim 74 9 0 03 Jun 2024
SnapKV: LLM Knows What You are Looking for Before Generation Yuhong Li Yingbing Huang Bowen Yang Bharat Venkitesh Acyr F. Locatelli Hanchen Ye Tianle Cai Patrick Lewis Deming Chen VLM 77 153 0 22 Apr 2024
SubGen: Token Generation in Sublinear Time and Memory A. Zandieh Insu Han Vahab Mirrokni Amin Karbasi 23 15 0 08 Feb 2024
Repeat After Me: Transformers are Better than State Space Models at Copying Samy Jelassi David Brandfonbrener Sham Kakade Eran Malach 100 78 0 01 Feb 2024
Optimizing Retrieval-augmented Reader Models via Token Elimination Moshe Berchansky Peter Izsak Avi Caciularu Ido Dagan Moshe Wasserblat RALM 50 12 0 20 Oct 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 88 266 0 11 Mar 2023
ABC: Attention with Bounded-memory Control Hao Peng Jungo Kasai Nikolaos Pappas Dani Yogatama Zhaofeng Wu Lingpeng Kong Roy Schwartz Noah A. Smith 76 22 0 06 Oct 2021
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 85 152 0 17 Sep 2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 250 695 0 27 Aug 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 280 2,015 0 28 Jul 2020