HGRN2: Gated Linear RNNs with State Expansion

11 April 2024

Zhen Qin

Papers citing "HGRN2: Gated Linear RNNs with State Expansion"

24 / 24 papers shown

Title
Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism Aviv Bick Eric P. Xing Albert Gu RALM 88 0 0 22 Apr 2025
Hadamard product in deep learning: Introduction, Advances and Challenges Grigorios G. Chrysos Yongtao Wu Razvan Pascanu Philip Torr V. Cevher AAML 98 0 0 17 Apr 2025
Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels M. Beck Korbinian Poppel Phillip Lippe Sepp Hochreiter 61 1 0 18 Mar 2025
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts Weigao Sun Disen Lan Tong Zhu Xiaoye Qu Yu-Xi Cheng MoE 100 1 0 07 Mar 2025
Liger: Linearizing Large Language Models to Gated Recurrent Structures Disen Lan Weigao Sun Jiaxi Hu Jusen Du Yu-Xi Cheng 64 0 0 03 Mar 2025
MoM: Linear Sequence Modeling with Mixture-of-Memories Jusen Du Weigao Sun Disen Lan Jiaxi Hu Yu-Xi Cheng KELM 75 3 0 19 Feb 2025
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models Aviv Bick Kevin Y. Li Eric P. Xing J. Zico Kolter Albert Gu Mamba 50 24 0 19 Aug 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 38 17 0 01 Jul 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 68 55 0 11 Jun 2024
Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks Jerome Sieber Carmen Amo Alonso A. Didier M. Zeilinger Antonio Orvieto AAML 44 8 0 24 May 2024
Linear Attention Sequence Parallelism Weigao Sun Zhen Qin Dong Li Xuyang Shen Yu Qiao Yiran Zhong 68 2 0 03 Apr 2024
State Space Models as Foundation Models: A Control Theoretic Overview Carmen Amo Alonso Jerome Sieber M. Zeilinger AI4CE Mamba 36 13 0 25 Mar 2024
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models Soham De Samuel L. Smith Anushan Fernando Aleksandar Botev George-Christian Muraru ... David Budden Yee Whye Teh Razvan Pascanu Nando de Freitas Çağlar Gülçehre Mamba 58 117 0 29 Feb 2024
Theoretical Foundations of Deep Selective State-Space Models Nicola Muca Cirone Antonio Orvieto Benjamin Walker C. Salvi Terry Lyons Mamba 53 25 0 29 Feb 2024
Simple linear attention language models balance the recall-throughput tradeoff Simran Arora Sabri Eyuboglu Michael Zhang Aman Timalsina Silas Alberti Dylan Zinsley James Zou Atri Rudra Christopher Ré 41 18 0 28 Feb 2024
Investigating Recurrent Transformers with Dynamic Halt Jishnu Ray Chowdhury Cornelia Caragea 39 1 0 01 Feb 2024
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models Zhen Qin Weigao Sun Dong Li Xuyang Shen Weixuan Sun Yiran Zhong 64 22 0 09 Jan 2024
Zoology: Measuring and Improving Recall in Efficient Language Models Simran Arora Sabri Eyuboglu Aman Timalsina Isys Johnson Michael Poli James Zou Atri Rudra Christopher Ré 59 66 0 08 Dec 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 88 265 0 11 Mar 2023
Fine-Tuning Pre-trained Transformers into Decaying Fast Weights H. H. Mao 63 20 0 09 Oct 2022
Hydra Attention: Efficient Attention with Many Heads Daniel Bolya Cheng-Yang Fu Xiaoliang Dai Peizhao Zhang Judy Hoffman 99 76 0 15 Sep 2022
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 73 222 0 21 Feb 2022
ABC: Attention with Bounded-memory Control Hao Peng Jungo Kasai Nikolaos Pappas Dani Yogatama Zhaofeng Wu Lingpeng Kong Roy Schwartz Noah A. Smith 61 22 0 06 Oct 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 253 1,986 0 31 Dec 2020