Fast Inference of Mixture-of-Experts Language Models with Offloading

28 December 2023

Papers citing "Fast Inference of Mixture-of-Experts Language Models with Offloading"

19 / 19 papers shown

Title
MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems Yao Fu Yao Fu Yeqi Huang Ping Nie Zhan Lu ... Dayou Du Tairan Xu Dayou Du Edoardo Ponti Luo Mai MoE 94 1 0 16 May 2025
MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators Beichen Huang Yueming Yuan Zelei Shao Minjia Zhang MQ MoE 111 0 0 03 Apr 2025
Mixture of Lookup Experts Shibo Jie Yehui Tang Kai Han Yongqian Li Duyu Tang Zhi-Hong Deng Yunhe Wang MoE 104 1 0 20 Mar 2025
CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory Jiashun Suo Xiaojian Liao Limin Xiao Li Ruan Jinquan Wang Xiao Su Zhisheng Huo 102 0 0 04 Mar 2025
DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference Yujie Zhang Shivam Aggarwal T. Mitra MoE 125 1 0 16 Dec 2024
ProMoE: Fast MoE-based LLM Serving using Proactive Caching Xiaoniu Song Zihang Zhong Rong Chen Haibo Chen MoE 98 6 0 29 Oct 2024
Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models Keisuke Kamahori Tian Tang Yile Gu Kan Zhu Baris Kasikci 122 24 0 10 Feb 2024
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration Ji Lin Jiaming Tang Haotian Tang Shang Yang Wei-Ming Chen Wei-Chen Wang Guangxuan Xiao Xingyu Dang Chuang Gan Song Han EDL MQ 95 574 0 01 Jun 2023
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 394 2,388 0 09 Nov 2022
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers M. Lewis Younes Belkada Luke Zettlemoyer MQ 98 653 0 15 Aug 2022
Language model compression with weighted low-rank factorization Yen-Chang Hsu Ting Hua Sung-En Chang Qiang Lou Yilin Shen Hongxia Jin 60 107 0 30 Jun 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 509 6,279 0 05 Apr 2022
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin ... Kun Zhang Quoc V. Le Yonghui Wu Zhiwen Chen Claire Cui ALM MoE 222 819 0 13 Dec 2021
BASE Layers: Simplifying Training of Large, Sparse Models M. Lewis Shruti Bhosale Tim Dettmers Naman Goyal Luke Zettlemoyer MoE 197 281 0 30 Mar 2021
Pruning and Quantization for Deep Neural Network Acceleration: A Survey Tailin Liang C. Glossner Lei Wang Shaobo Shi Xiaotong Zhang MQ 208 698 0 24 Jan 2021
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang M. Krikun Noam M. Shazeer Zhiwen Chen MoE 103 1,184 0 30 Jun 2020
Up or Down? Adaptive Rounding for Post-Training Quantization Markus Nagel Rana Ali Amjad M. V. Baalen Christos Louizos Tijmen Blankevoort MQ 88 585 0 22 Apr 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 445 20,298 0 23 Oct 2019
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 328 2,895 0 26 Sep 2016