Zamba: A Compact 7B SSM Hybrid Model

26 May 2024

Papers citing "Zamba: A Compact 7B SSM Hybrid Model"

21 / 21 papers shown

Title
RWKV-X: A Linear Complexity Hybrid Language Model Haowen Hou Zhiyi Huang Kaifeng Tan Rongchang Lu Fei Richard Yu VLM 108 0 0 30 Apr 2025
vGamba: Attentive State Space Bottleneck for efficient Long-range Dependencies in Visual Recognition Yunusa Haruna A. Lawan Mamba 94 0 0 27 Mar 2025
Liger: Linearizing Large Language Models to Gated Recurrent Structures Disen Lan Weigao Sun Jiaxi Hu Jusen Du Yu Cheng 92 0 0 03 Mar 2025
Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models J. P. Muñoz Jinjie Yuan Nilesh Jain Mamba 101 1 0 28 Jan 2025
Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models Elvis Nunez Luca Zancato Benjamin Bowman Aditya Golatkar Wei Xia Stefano Soatto 139 3 0 17 Dec 2024
Marconi: Prefix Caching for the Era of Hybrid LLMs Rui Pan Zhuang Wang Zhen Jia Can Karakus Luca Zancato Tri Dao Ravi Netravali Yida Wang 130 4 0 28 Nov 2024
Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation Seokil Ham H. Kim Sangmin Woo Changick Kim Mamba 414 0 0 21 Nov 2024
DiMSUM: Diffusion Mamba -- A Scalable and Unified Spatial-Frequency Method for Image Generation Hao Phung Quan Dao T. Dao Hoang Phan Dimitris Metaxas Anh Tran Mamba 99 4 0 06 Nov 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 95 6 0 28 Oct 2024
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture Jiayi Han Liang Du Hongwei Du Xiangguo Zhou Yiwen Wu Weibo Zheng Donghong Han CLL MoMe MoE 54 2 0 10 Oct 2024
Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models Lynn Chua Badih Ghazi Yangsibo Huang Pritish Kamath Ravi Kumar Pasin Manurangsi Amer Sinha Chulin Xie Chiyuan Zhang 86 2 0 23 Jun 2024
Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks Jongho Park Jaeseung Park Zheyang Xiong Nayoung Lee Jaewoong Cho Samet Oymak Kangwook Lee Dimitris Papailiopoulos 71 71 0 06 Feb 2024
LLM360: Towards Fully Transparent Open-Source LLMs Zhengzhong Liu Aurick Qiao Willie Neiswanger Hongyi Wang Bowen Tan ... Zhiting Hu Mark Schulze Preslav Nakov Timothy Baldwin Eric Xing 83 75 0 11 Dec 2023
RWKV: Reinventing RNNs for the Transformer Era Bo Peng Eric Alcaide Quentin G. Anthony Alon Albalak Samuel Arcadinho ... Qihang Zhao P. Zhou Qinghua Zhou Jian Zhu Rui-Jie Zhu 165 585 0 22 May 2023
Downstream Datasets Make Surprisingly Good Pretraining Corpora Kundan Krishna Saurabh Garg Jeffrey P. Bigham Zachary Chase Lipton 65 33 0 28 Sep 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 130 1,915 0 29 Mar 2022
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale Samyam Rajbhandari Conglong Li Z. Yao Minjia Zhang Reza Yazdani Aminabadi A. A. Awan Jeff Rasley Yuxiong He 55 292 0 14 Jan 2022
Relating transformers to models and neural representations of the hippocampal formation James C. R. Whittington Joseph Warren Timothy Edward John Behrens 80 80 0 07 Dec 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 383 2,051 0 31 Dec 2020
Going in circles is the way forward: the role of recurrence in visual inference R. S. V. Bergen N. Kriegeskorte 34 82 0 26 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 458 4,662 0 23 Jan 2020