Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

11 June 2024

Yadong Lu

Weizhu Chen

Papers citing "Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling"

48 / 48 papers shown

Title
Overflow Prevention Enhances Long-Context Recurrent LLMs Assaf Ben-Kish Itamar Zimerman M. Jehanzeb Mirza James R. Glass Leonid Karlinsky Raja Giryes LRM 29 0 0 12 May 2025
Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation Junyu Ma Tianqing Fang Z. Zhang Hongming Zhang Haitao Mi Dong Yu ReLM RALM LRM 131 0 0 06 May 2025
Random Long-Context Access for Mamba via Hardware-aligned Hierarchical Sparse Attention Xiang Hu Jiaqi Leng Jun Zhao Kewei Tu Wei Wu Mamba 50 0 0 23 Apr 2025
Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism Aviv Bick Eric P. Xing Albert Gu RALM 88 0 0 22 Apr 2025
It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization Ali Behrouz Meisam Razaviyayn Peilin Zhong Vahab Mirrokni 38 0 0 17 Apr 2025
M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models Junxiong Wang Wen-Ding Li Daniele Paliotta Daniel Ritter Alexander M. Rush Tri Dao LRM 26 0 0 14 Apr 2025
From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models C. Xu Wei Ping P. Xu Z. Liu Boxin Wang M. Shoeybi Bo Li Bryan Catanzaro 17 1 0 08 Apr 2025
Stochastic Engrams for Efficient Continual Learning with Binarized Neural Networks Isabelle Aguilar Luis Fernando Herbozo Contreras Omid Kavehei 59 0 0 27 Mar 2025
vGamba: Attentive State Space Bottleneck for efficient Long-range Dependencies in Visual Recognition Yunusa Haruna A. Lawan Mamba 52 0 0 27 Mar 2025
iFlame: Interleaving Full and Linear Attention for Efficient Mesh Generation Hanxiao Wang Biao Zhang Weize Quan Dong-ming Yan Peter Wonka 51 0 0 20 Mar 2025
xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference M. Beck Korbinian Poppel Phillip Lippe Richard Kurle P. Blies G. Klambauer Sebastian Böck Sepp Hochreiter LRM 40 1 0 17 Mar 2025
Small Vision-Language Models: A Survey on Compact Architectures and Techniques Nitesh Patnaik Navdeep Nayak Himani Bansal Agrawal Moinak Chinmoy Khamaru Gourav Bal Saishree Smaranika Panda Rishi Raj Vishal Meena Kartheek Vadlamani VLM 58 0 0 09 Mar 2025
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts Weigao Sun Disen Lan Tong Zhu Xiaoye Qu Yu-Xi Cheng MoE 100 1 0 07 Mar 2025
Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners Daniele Paliotta Junxiong Wang Matteo Pagliardini Kevin Y. Li Aviv Bick J. Zico Kolter Albert Gu F. Fleuret Tri Dao ReLM LRM 43 7 0 27 Feb 2025
PICASO: Permutation-Invariant Context Composition with State Space Models Tian Yu Liu Alessandro Achille Matthew Trager Aditya Golatkar L. Zancato Stefano Soatto LRM 58 0 0 24 Feb 2025
Multilingual State Space Models for Structured Question Answering in Indic Languages A. Vats Rahul Raja Mrinal Mathur Vinija Jain Aman Chadha 68 1 0 01 Feb 2025
SSD4Rec: A Structured State Space Duality Model for Efficient Sequential Recommendation Haohao Qu Yifeng Zhang Liangbo Ning Wenqi Fan Qing Li Mamba 94 7 0 17 Jan 2025
LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity Hongjie Wang Chih-Yao Ma Yen-Cheng Liu Ji Hou Tao Xu ... Peizhao Zhang Tingbo Hou Peter Vajda N. Jha Xiaoliang Dai LMTD DiffM VGen VLM 81 5 0 13 Dec 2024
Marconi: Prefix Caching for the Era of Hybrid LLMs Rui Pan Zhuang Wang Zhen Jia Can Karakus Luca Zancato Tri Dao Ravi Netravali Yida Wang 92 4 0 28 Nov 2024
Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation Seokil Ham H. Kim Sangmin Woo Changick Kim Mamba 180 0 0 21 Nov 2024
Hymba: A Hybrid-head Architecture for Small Language Models Xin Dong Y. Fu Shizhe Diao Wonmin Byeon Zijia Chen ... Min-Hung Chen Yoshi Suhara Y. Lin Jan Kautz Pavlo Molchanov Mamba 100 21 0 20 Nov 2024
How much do contextualized representations encode long-range context? Simeng Sun Cheng-Ping Hsieh 39 0 0 16 Oct 2024
State-space models can learn in-context by gradient descent Neeraj Mohan Sushma Yudou Tian Harshvardhan Mestha Nicolo Colombo David Kappel Anand Subramoney 37 3 0 15 Oct 2024
Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions Zhihao He Hang Yu Zi Gong Shizhan Liu Jianguo Li Weiyao Lin VLM 36 1 0 09 Oct 2024
Falcon Mamba: The First Competitive Attention-free 7B Language Model Jingwei Zuo Maksim Velikanov Dhia Eddine Rhaiem Ilyas Chahed Younes Belkada Guillaume Kunsch Hakim Hacid ALM 52 14 0 07 Oct 2024
A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts Suyu Ge Xihui Lin Yunan Zhang Jiawei Han Hao Peng 31 4 0 02 Oct 2024
Inference-Friendly Models With MixAttention Shashank Rajput Ying Sheng Sean Owen Vitaliy Chiley 76 1 0 23 Sep 2024
Exploring Scaling Laws for Local SGD in Large Language Model Training Qiaozhi He Xiaomin Zhuang Zhihua Wu 20 4 0 20 Sep 2024
Flash STU: Fast Spectral Transform Units Y. Isabel Liu Windsor Nguyen Yagiz Devre Evan Dogariu Anirudha Majumdar Elad Hazan AI4TS 72 1 0 16 Sep 2024
Famba-V: Fast Vision Mamba with Cross-Layer Token Fusion Hui Shen Zhongwei Wan Xin Wang Mi Zhang Mamba 32 6 0 15 Sep 2024
Mamba-YOLO-World: Marrying YOLO-World with Mamba for Open-Vocabulary Detection Haoxuan Wang Q. He Jinlong Peng Hao Yang Mingmin Chi Yabiao Wang Mamba 39 1 0 13 Sep 2024
Gated Slot Attention for Efficient Linear-Time Sequence Modeling Yu Zhang Songlin Yang Ruijie Zhu Yue Zhang Leyang Cui ... Freda Shi Bailin Wang Wei Bi P. Zhou Guohong Fu 60 16 0 11 Sep 2024
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models Aviv Bick Kevin Y. Li Eric P. Xing J. Zico Kolter Albert Gu Mamba 50 24 0 19 Aug 2024
A Survey of Mamba Shuwei Shi Shibing Chu Rui An Wenqi Fan Yuee Xie Hui Liu Yuanping Chen Qing Li AI4CE 40 26 0 02 Aug 2024
Longhorn: State Space Models are Amortized Online Learners Bo Liu Rui Wang Lemeng Wu Yihao Feng Peter Stone Qian Liu 46 10 0 19 Jul 2024
GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression Daniel Goldstein Fares Obeid Eric Alcaide Guangyu Song Eugene Cheah VLM AI4TS 26 7 0 16 Jul 2024
On the Power of Convolution Augmented Transformer Mingchen Li Xuechen Zhang Yixiao Huang Samet Oymak 32 0 0 08 Jul 2024
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention Huiqiang Jiang Yucheng Li Chengruidong Zhang Qianhui Wu Xufang Luo ... Amir H. Abdi Dongsheng Li Chin-Yew Lin Yuqing Yang L. Qiu 67 82 0 02 Jul 2024
Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba Yuchen Zou Yineng Chen Zuchao Li Lefei Zhang Hai Zhao 49 1 0 24 Jun 2024
HGRN2: Gated Linear RNNs with State Expansion Zhen Qin Songlin Yang Weixuan Sun Xuyang Shen Dong Li Weigao Sun Yiran Zhong LRM 44 47 0 11 Apr 2024
Simple linear attention language models balance the recall-throughput tradeoff Simran Arora Sabri Eyuboglu Michael Zhang Aman Timalsina Silas Alberti Dylan Zinsley James Zou Atri Rudra Christopher Ré 41 18 0 28 Feb 2024
Zoology: Measuring and Improving Recall in Efficient Language Models Simran Arora Sabri Eyuboglu Aman Timalsina Isys Johnson Michael Poli James Zou Atri Rudra Christopher Ré 59 66 0 08 Dec 2023
Sparse Modular Activation for Efficient Sequence Modeling Liliang Ren Yang Liu Shuohang Wang Yichong Xu Chenguang Zhu Chengxiang Zhai 43 13 0 19 Jun 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 289 3,003 0 22 Mar 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 88 265 0 11 Mar 2023
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way Alex Jinpeng Wang Richard Yuanzhe Pang Angelica Chen Jason Phang Samuel R. Bowman 74 44 0 23 May 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 355 8,457 0 28 Jan 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 245 695 0 27 Aug 2021