Zebra-Llama: Towards Extremely Efficient Hybrid Models

22 May 2025

Papers citing "Zebra-Llama: Towards Extremely Efficient Hybrid Models"

20 / 20 papers shown

Title
X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression Guihong Li Mehdi Rezagholizadeh Mingyu Yang Vikram Appia Emad Barsoum VLM 76 1 0 14 Mar 2025
Hymba: A Hybrid-head Architecture for Small Language Models Xin Dong Y. Fu Shizhe Diao Wonmin Byeon Zijia Chen ... Min-Hung Chen Yoshi Suhara Y. Lin Jan Kautz Pavlo Molchanov Mamba 116 22 0 20 Nov 2024
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models Aviv Bick Kevin Y. Li Eric P. Xing J. Zico Kolter Albert Gu Mamba 80 28 0 19 Aug 2024
Compact Language Models via Pruning and Knowledge Distillation Saurav Muralidharan Sharath Turuvekere Sreenivas Raviraj Joshi Marcin Chochowski M. Patwary Mohammad Shoeybi Bryan Catanzaro Jan Kautz Pavlo Molchanov SyDa MQ 64 49 0 19 Jul 2024
GenQA: Generating Millions of Instructions from a Handful of Prompts Jiuhai Chen Rifaa Qadri Yuxin Wen Neel Jain John Kirchenbauer Dinesh Manocha Tom Goldstein ALM 126 19 0 14 Jun 2024
An Empirical Study of Mamba-based Language Models R. Waleffe Wonmin Byeon Duncan Riach Brandon Norick V. Korthikanti ... Vartika Singh Jared Casper Jan Kautz Mohammad Shoeybi Bryan Catanzaro 83 72 0 12 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 97 63 0 11 Jun 2024
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality Tri Dao Albert Gu Mamba 57 480 0 31 May 2024
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI Aixin Liu Bei Feng Bin Wang Bingxuan Wang ... Zhuoshu Li Zihan Wang Zihui Gu Zilin Li Ziwei Xie MoE 76 447 0 07 May 2024
Mamba: Linear-Time Sequence Modeling with Selective State Spaces Albert Gu Tri Dao Mamba 93 2,552 0 01 Dec 2023
TruthfulQA: Measuring How Models Mimic Human Falsehoods Stephanie C. Lin Jacob Hilton Owain Evans HILM 89 1,825 0 08 Sep 2021
Annealing Knowledge Distillation A. Jafari Mehdi Rezagholizadeh Pranav Sharma A. Ghodsi 38 79 0 14 Apr 2021
Measuring Massive Multitask Language Understanding Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika D. Song Jacob Steinhardt ELM RALM 137 4,222 0 07 Sep 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 121 1,734 0 29 Jun 2020
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 100 1,724 0 26 Nov 2019
HellaSwag: Can a Machine Really Finish Your Sentence? Rowan Zellers Ari Holtzman Yonatan Bisk Ali Farhadi Yejin Choi 83 2,373 0 19 May 2019
Improved Knowledge Distillation via Teacher Assistant Seyed Iman Mirzadeh Mehrdad Farajtabar Ang Li Nir Levine Akihiro Matsukawa H. Ghasemzadeh 79 1,073 0 09 Feb 2019
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering Todor Mihaylov Peter Clark Tushar Khot Ashish Sabharwal 74 1,475 0 08 Sep 2018
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick Oyvind Tafjord ELM RALM LRM 77 2,474 0 14 Mar 2018
RACE: Large-scale ReAding Comprehension Dataset From Examinations Guokun Lai Qizhe Xie Hanxiao Liu Yiming Yang Eduard H. Hovy ELM 144 1,329 0 15 Apr 2017