Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

10 February 2024

Papers citing "Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models"

21 / 21 papers shown

Title
MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems Yinsicheng Jiang Yao Fu Yeqi Huang Ping Nie Zhan Lu ... Dayou Du Tairan Xu Kai Zou Edoardo Ponti Luo Mai MoE 12 0 0 16 May 2025
QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration HamidReza Imani Jiaxin Peng Peiman Mohseni Abdolah Amirany Tarek A. El-Ghazawi MoE 31 0 0 10 May 2025
FloE: On-the-Fly MoE Inference on Memory-constrained GPU Yuxin Zhou Zheng Li J. Zhang Jue Wang Yuanda Wang Zhongle Xie Ke Chen Lidan Shou MoE 50 0 0 09 May 2025
$D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving$ D $^{2}$ MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving Haodong Wang Qihua Zhou Zicong Hong Song Guo MoE 58 0 0 17 Apr 2025
MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints Yichao Yuan Lin Ma Nishil Talati MoE 64 0 0 12 Apr 2025
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference Shuzhang Zhong Yizhou Sun Ling Liang Runsheng Wang R. Huang Meng Li MoE 61 0 0 08 Apr 2025
MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching Tairan Xu Leyang Xue Zhan Lu Adrian Jackson Luo Mai MoE 90 1 0 12 Mar 2025
Seesaw: High-throughput LLM Inference via Model Re-sharding Qidong Su Wei Zhao X. Li Muralidhar Andoorveedu Chenhao Jiang Zhanda Zhu Kevin Song Christina Giannoula Gennady Pekhimenko LRM 77 0 0 09 Mar 2025
CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory Jiashun Suo Xiaojian Liao Limin Xiao Li Ruan Jinquan Wang Xiao Su Zhisheng Huo 67 0 0 04 Mar 2025
Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline Zhiyuan Fang Yuegui Huang Zicong Hong Yufeng Lyu Wuhui Chen Yue Yu Fan Yu Zibin Zheng MoE 48 0 0 09 Feb 2025
Deploying Foundation Model Powered Agent Services: A Survey Wenchao Xu Jinyu Chen Peirong Zheng Xiaoquan Yi Tianyi Tian ... Quan Wan Yining Qi Yunfeng Fan Qinliang Su Xuemin Shen AI4CE 119 1 0 18 Dec 2024
DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference Yujie Zhang Shivam Aggarwal T. Mitra MoE 74 0 0 16 Dec 2024
BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching Yilong Zhao Shuo Yang Kan Zhu Lianmin Zheng Baris Kasikci Yang Zhou Jiarong Xing Ion Stoica 118 5 0 25 Nov 2024
HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference Peng Tang Jiacheng Liu X. Hou Yifei Pu Jing Wang Pheng-Ann Heng Chuan Li M. Guo MoE 59 7 0 03 Nov 2024
ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference Xin He Shunkang Zhang Yuxin Wang Haiyan Yin Zihao Zeng Shaohuai Shi Zhenheng Tang Xiaowen Chu Ivor Tsang Ong Yew Soon MoE 63 3 0 23 Oct 2024
Mixture of Experts with Mixture of Precisions for Tuning Quality of Service HamidReza Imani Abdolah Amirany Tarek A. El-Ghazawi MoE 56 6 0 19 Jul 2024
LLM Inference Serving: Survey of Recent Advances and Opportunities Baolin Li Yankai Jiang V. Gadepally Devesh Tiwari 78 18 0 17 Jul 2024
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU Yixin Song Zeyu Mi Haotong Xie Haibo Chen BDL 125 120 0 16 Dec 2023
LLM in a flash: Efficient Large Language Model Inference with Limited Memory Keivan Alizadeh-Vahid Iman Mirzadeh Dmitry Belenko Karen Khatamifard Minsik Cho C. C. D. Mundo Mohammad Rastegari Mehrdad Farajtabar 72 112 0 12 Dec 2023
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models Iman Mirzadeh Keivan Alizadeh-Vahid Sachin Mehta C. C. D. Mundo Oncel Tuzel Golnoosh Samei Mohammad Rastegari Mehrdad Farajtabar 123 60 0 06 Oct 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 149 368 0 13 Mar 2023