v1v2v3 (latest)

Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

11 June 2024

Yadong Lu

Weizhu Chen

Papers citing "Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling"

50 / 132 papers shown

Title
LoLA: Low-Rank Linear Attention With Sparse Caching Luke McDermott Robert W. Heath Jr. Rahul Parhi RALM 40 0 0 29 May 2025
ATLAS: Learning to Optimally Memorize the Context at Test Time Ali Behrouz Zeman Li Praneeth Kacham Majid Daliri Yuan Deng Peilin Zhong Meisam Razaviyayn Vahab Mirrokni 60 2 0 29 May 2025
RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding Yuichiro Hoshino Hideyuki Tachibana Muneyoshi Inahara Hiroto Takegawa 56 0 0 28 May 2025
Long-Context State-Space Video World Models Ryan Po Yotam Nitzan Richard Zhang Berlin Chen Tri Dao Eli Shechtman Gordon Wetzstein Xun Huang 57 2 0 26 May 2025
Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks Mohammad Mahdi Moradi Walid Ahmed Shuangyue Wen Sudhir Mudur Weiwei Zhang Yang Liu 62 0 0 26 May 2025
Zebra-Llama: Towards Extremely Efficient Hybrid Models Mingyu Yang Mehdi Rezagholizadeh Guihong Li Vikram Appia Emad Barsoum 62 0 0 22 May 2025
Overflow Prevention Enhances Long-Context Recurrent LLMs Assaf Ben-Kish Itamar Zimerman M. Jehanzeb Mirza James R. Glass Leonid Karlinsky Raja Giryes LRM 67 0 0 12 May 2025
Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation Junyu Ma Tianqing Fang Zizhuo Zhang Hongming Zhang Haitao Mi Dong Yu ReLM RALM LRM 445 1 0 06 May 2025
Random Long-Context Access for Mamba via Hardware-aligned Hierarchical Sparse Attention Xiang Hu Jiaqi Leng Jun Zhao Kewei Tu Wei Wu Mamba 103 0 0 23 Apr 2025
Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism Aviv Bick Eric P. Xing Albert Gu RALM 123 1 0 22 Apr 2025
It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization Ali Behrouz Meisam Razaviyayn Peilin Zhong Vahab Mirrokni 94 5 0 17 Apr 2025
M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models Junxiong Wang Wen-Ding Li Daniele Paliotta Daniel Ritter Alexander M. Rush Tri Dao LRM 84 2 0 14 Apr 2025
From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models C. Xu Ming-Yu Liu Peng Xu Ziwei Liu Wei Ping Mohammad Shoeybi Bo Li Bryan Catanzaro 95 4 0 08 Apr 2025
vGamba: Attentive State Space Bottleneck for efficient Long-range Dependencies in Visual Recognition Yunusa Haruna A. Lawan Mamba 116 0 0 27 Mar 2025
Stochastic Engrams for Efficient Continual Learning with Binarized Neural Networks Isabelle Aguilar Luis Fernando Herbozo Contreras Omid Kavehei 102 1 0 27 Mar 2025
iFlame: Interleaving Full and Linear Attention for Efficient Mesh Generation Hanxiao Wang Biao Zhang Weize Quan Dong-ming Yan Peter Wonka 99 2 0 20 Mar 2025
xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference M. Beck Korbinian Poppel Phillip Lippe Richard Kurle P. Blies Günter Klambauer Sebastian Böck Sepp Hochreiter LRM 90 1 0 17 Mar 2025
Small Vision-Language Models: A Survey on Compact Architectures and Techniques Nitesh Patnaik Navdeep Nayak Himani Bansal Agrawal Moinak Chinmoy Khamaru Gourav Bal Saishree Smaranika Panda Rishi Raj Vishal Meena Kartheek Vadlamani VLM 97 0 0 09 Mar 2025
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts Weigao Sun Disen Lan Tong Zhu Xiaoye Qu Yu Cheng MoE 222 4 0 07 Mar 2025
Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners Daniele Paliotta Junxiong Wang Matteo Pagliardini Kevin Y. Li Aviv Bick J. Zico Kolter Albert Gu François Fleuret Tri Dao ReLM LRM 118 10 0 27 Feb 2025
PICASO: Permutation-Invariant Context Composition with State Space Models Tian Yu Liu Alessandro Achille Matthew Trager Aditya Golatkar Luca Zancato Stefano Soatto LRM 134 0 0 24 Feb 2025
Multilingual State Space Models for Structured Question Answering in Indic Languages A. Vats Rahul Raja Mrinal Mathur Vinija Jain Aman Chadha 157 1 0 01 Feb 2025
SSD4Rec: A Structured State Space Duality Model for Efficient Sequential Recommendation Haohao Qu Yifeng Zhang Liangbo Ning Wenqi Fan Qing Li Mamba 174 9 0 17 Jan 2025
LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity Hongjie Wang Chih-Yao Ma Yen-Cheng Liu Ji Hou Tao Xu ... Peizhao Zhang Tingbo Hou Peter Vajda N. Jha Xiaoliang Dai LMTD VGen VLM DiffM 168 11 0 13 Dec 2024
Marconi: Prefix Caching for the Era of Hybrid LLMs Rui Pan Zhuang Wang Zhen Jia Can Karakus Luca Zancato Tri Dao Ravi Netravali Yida Wang 169 4 0 28 Nov 2024
Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation Seokil Ham H. Kim Sangmin Woo Changick Kim Mamba 481 0 0 21 Nov 2024
Hymba: A Hybrid-head Architecture for Small Language Models Xin Dong Y. Fu Shizhe Diao Wonmin Byeon Zijia Chen ... Min-Hung Chen Yoshi Suhara Y. Lin Jan Kautz Pavlo Molchanov Mamba 141 27 0 20 Nov 2024
How much do contextualized representations encode long-range context? Simeng Sun Cheng-Ping Hsieh 103 0 0 16 Oct 2024
State-space models can learn in-context by gradient descent Neeraj Mohan Sushma Yudou Tian Harshvardhan Mestha Nicolo Colombo David Kappel Anand Subramoney 109 3 0 15 Oct 2024
Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions Zhihao He Hang Yu Zi Gong Shizhan Liu Jia-Nan Li Weiyao Lin VLM 88 2 0 09 Oct 2024
Falcon Mamba: The First Competitive Attention-free 7B Language Model Jingwei Zuo Maksim Velikanov Dhia Eddine Rhaiem Ilyas Chahed Younes Belkada Guillaume Kunsch Hakim Hacid ALM 75 17 0 07 Oct 2024
A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts Suyu Ge Xihui Lin Yunan Zhang Jiawei Han Hao Peng 110 4 0 02 Oct 2024
Inference-Friendly Models With MixAttention Shashank Rajput Ying Sheng Sean Owen Vitaliy Chiley 137 1 0 23 Sep 2024
Exploring Scaling Laws for Local SGD in Large Language Model Training Qiaozhi He Xiaomin Zhuang Zhihua Wu 69 4 0 20 Sep 2024
Flash STU: Fast Spectral Transform Units Y. Isabel Liu Windsor Nguyen Yagiz Devre Evan Dogariu Anirudha Majumdar Elad Hazan AI4TS 126 1 0 16 Sep 2024
Famba-V: Fast Vision Mamba with Cross-Layer Token Fusion Hui Shen Zhongwei Wan Xin Wang Mi Zhang Mamba 66 6 0 15 Sep 2024
Mamba-YOLO-World: Marrying YOLO-World with Mamba for Open-Vocabulary Detection Haoxuan Wang Qu He Jinlong Peng Hao Yang Mingmin Chi Yabiao Wang Mamba 76 2 0 13 Sep 2024
Gated Slot Attention for Efficient Linear-Time Sequence Modeling Yu Zhang Aaron Courville Ruijie Zhu Yue Zhang Leyang Cui ... Freda Shi Bailin Wang Wei Bi P. Zhou Guohong Fu 98 24 0 11 Sep 2024
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models Aviv Bick Kevin Y. Li Eric P. Xing J. Zico Kolter Albert Gu Mamba 119 31 0 19 Aug 2024
A Survey of Mamba Shuwei Shi Shibing Chu Rui An Wenqi Fan Yuee Xie Hui Liu Yuanping Chen Qing Li AI4CE 91 30 0 02 Aug 2024
Longhorn: State Space Models are Amortized Online Learners Bo Liu Rui Wang Lemeng Wu Yihao Feng Peter Stone Qian Liu 88 14 0 19 Jul 2024
GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression Daniel Goldstein Fares Obeid Eric Alcaide Guangyu Song Eugene Cheah VLM AI4TS 88 8 0 16 Jul 2024
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision Jay Shah Ganesh Bikshandi Ying Zhang Vijay Thakkar Pradeep Ramani Tri Dao 127 156 0 11 Jul 2024
On the Power of Convolution Augmented Transformer Mingchen Li Xuechen Zhang Yixiao Huang Samet Oymak 58 3 0 08 Jul 2024
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention Huiqiang Jiang Yucheng Li Chengruidong Zhang Qianhui Wu Xufang Luo ... Amir H. Abdi Dongsheng Li Chin-Yew Lin Yuqing Yang L. Qiu 138 118 0 02 Jul 2024
Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba Yuchen Zou Yineng Chen Zuchao Li Lefei Zhang Hai Zhao 107 1 0 24 Jun 2024
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark Yubo Wang Xueguang Ma Ge Zhang Yuansheng Ni Abhranil Chandra ... Kai Wang Alex Zhuang Rongqi Fan Xiang Yue Wenhu Chen LRM ELM 124 461 0 03 Jun 2024
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Marah Abdin Sam Ade Jacobs A. A. Awan J. Aneja Ahmed Hassan Awadallah ... Li Zhang Yi Zhang Yue Zhang Yunan Zhang Xiren Zhou LRM ALM 154 1,257 0 22 Apr 2024
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length Xuezhe Ma Xiaomeng Yang Wenhan Xiong Beidi Chen Lili Yu Hao Zhang Jonathan May Luke Zettlemoyer Omer Levy Chunting Zhou 66 31 0 12 Apr 2024
The Illusion of State in State-Space Models William Merrill Jackson Petty Ashish Sabharwal 96 58 0 12 Apr 2024