LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement

22 April 2025

Papers citing "LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement"

22 / 22 papers shown

Title
Long-Context State-Space Video World Models Ryan Po Yotam Nitzan Richard Zhang Berlin Chen Tri Dao Eli Shechtman Gordon Wetzstein Xun Huang 59 2 0 26 May 2025
Block-Biased Mamba for Long-Range Sequence Processing Annan Yu N. Benjamin Erichson Mamba 91 0 0 13 May 2025
Overflow Prevention Enhances Long-Context Recurrent LLMs Assaf Ben-Kish Itamar Zimerman M. Jehanzeb Mirza James R. Glass Leonid Karlinsky Raja Giryes LRM 67 0 0 12 May 2025
Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation Junyu Ma Tianqing Fang Zizhuo Zhang Hongming Zhang Haitao Mi Dong Yu ReLM RALM LRM 448 1 0 06 May 2025
Shifting Long-Context LLMs Research from Input to Output Yuhao Wu Yushi Bai Zhiqing Hu Shangqing Tu Ming Shan Hee Juanzi Li Roy Ka-wei Lee 104 5 0 06 Mar 2025
Hymba: A Hybrid-head Architecture for Small Language Models Xin Dong Y. Fu Shizhe Diao Wonmin Byeon Zijia Chen ... Min-Hung Chen Yoshi Suhara Y. Lin Jan Kautz Pavlo Molchanov Mamba 141 27 0 20 Nov 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 181 19 0 20 Jun 2024
An Empirical Study of Mamba-based Language Models R. Waleffe Wonmin Byeon Duncan Riach Brandon Norick V. Korthikanti ... Vartika Singh Jared Casper Jan Kautz Mohammad Shoeybi Bryan Catanzaro 119 78 0 12 Jun 2024
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality Tri Dao Albert Gu Mamba 116 535 0 31 May 2024
Zamba: A Compact 7B SSM Hybrid Model Paolo Glorioso Quentin G. Anthony Yury Tokpanov James Whittington Jonathan Pilault Adam Ibrahim Beren Millidge 78 49 0 26 May 2024
A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods Hanlei Jin Yang Zhang Dan Meng Jun Wang Jinghua Tan 242 96 0 05 Mar 2024
Resonance RoPE: Improving Context Length Generalization of Large Language Models Suyuchen Wang I. Kobyzev Peng Lu Mehdi Rezagholizadeh Bang Liu 68 13 0 29 Feb 2024
YaRN: Efficient Context Window Extension of Large Language Models Bowen Peng Jeffrey Quesnelle Honglu Fan Enrico Shippole OSLM 78 264 0 31 Aug 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 435 4,444 0 09 Jun 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,748 0 15 Mar 2023
Recurrent Memory Transformer Aydar Bulatov Yuri Kuratov Andrey Kravchenko CLL 47 111 0 14 Jul 2022
On the Parameterization and Initialization of Diagonal State Space Models Albert Gu Ankit Gupta Karan Goel Christopher Ré 91 324 0 23 Jun 2022
Competition-Level Code Generation with AlphaCode Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser ... Esme Sutherland Robson Pushmeet Kohli Nando de Koray Kavukcuoglu Oriol Vinyals 148 1,425 0 08 Feb 2022
Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu Karan Goel Christopher Ré 217 1,829 0 31 Oct 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 475 2,121 0 31 Dec 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 203 1,793 0 29 Jun 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 185 4,100 0 10 Apr 2020