Taming the Titans: A Survey of Efficient LLM Inference Serving

28 April 2025

Papers citing "Taming the Titans: A Survey of Efficient LLM Inference Serving"

29 / 29 papers shown

Title
Towards Sustainable Large Language Model Serving Sophia Nguyen Beihao Zhou Yi Ding Sihang Liu 197 8 0 31 Dec 2024
Marconi: Prefix Caching for the Era of Hybrid LLMs Rui Pan Zhuang Wang Zhen Jia Can Karakus Luca Zancato Tri Dao Ravi Netravali Yida Wang 169 4 0 28 Nov 2024
InstCache: A Predictive Cache for LLM Serving Longwei Zou Tingfeng Liu Kai Chen Jiangang Kong Yangdong Deng 84 1 0 21 Nov 2024
Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection Vima Gupta Kartik Sinha Ada Gavrilovska Anand Padmanabha Iyer MoE 56 2 0 13 Nov 2024
Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs Ferdi Kossmann Bruce Fontaine Daya Khudia Michael Cafarella Samuel Madden 266 2 0 23 Oct 2024
A First Look At Efficient And Secure On-Device LLM Inference Against KV Leakage Huan Yang Deyu Zhang Yudong Zhao Yuanchun Li Yunxin Liu 54 2 0 06 Sep 2024
Teola: Towards End-to-End Optimization of LLM-based Applications Xin Tan Yimin Jiang Yitao Yang Hong-Yu Xu 135 7 0 29 Jun 2024
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure Jikai Wang Yi Su Juntao Li Qingrong Xia Zi Ye Xinyu Duan Zhefeng Wang Min Zhang 112 19 0 25 Jun 2024
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving Ruoyu Qin Zheming Li Weiran He Mingxing Zhang Yongwei Wu Weimin Zheng Xinran Xu 106 66 0 24 Jun 2024
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion Jiayi Yao Hanchen Li Yuhan Liu Siddhant Ray Yihua Cheng Qizheng Zhang Kuntai Du Shan Lu Junchen Jiang 104 24 0 26 May 2024
Preble: Efficient Distributed Prompt Scheduling for LLM Serving Vikranth Srivatsa Zijian He Reyna Abhyankar Dongming Li Yiying Zhang 106 20 0 08 May 2024
Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity Tyler Griggs Xiaoxuan Liu Jiaxiang Yu Doyoung Kim Wei-Lin Chiang Alvin Cheung Ion Stoica 82 18 0 22 Apr 2024
Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction Haoran Qiu Weichao Mao Archit Patke Shengkun Cui Saurabh Jha Chen Wang Hubertus Franke Zbigniew T. Kalbarczyk Tamer Basar Ravishankar K. Iyer 62 28 0 12 Apr 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 85 57 0 15 Feb 2024
Accelerating Retrieval-Augmented Language Model Serving with Speculation Zhihao Zhang Alan Zhu Lijie Yang Yihua Xu Lanting Li P. Phothilimthana Zhihao Jia RALM KELM 95 18 0 25 Jan 2024
CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference Suyi Li Hanfeng Lu Tianyuan Wu Minchen Yu Qizhen Weng Xusheng Chen Yizhou Shan Binhang Yuan Wei Wang 89 13 0 20 Jan 2024
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving Yinmin Zhong Shengyu Liu Junda Chen Jianbo Hu Yibo Zhu Xuanzhe Liu Xin Jin Hao Zhang 83 205 0 18 Jan 2024
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding Heming Xia Zhe Yang Qingxiu Dong Peiyi Wang Chak Tou Leong Tao Ge Tianyu Liu Wenjie Li Zhifang Sui LRM 122 129 0 15 Jan 2024
Efficient LLM inference solution on Intel GPU Hui Wu Yi Gan Feng Yuan Jing Ma Wei Zhu ... Hong Zhu Yuhua Zhu Xiaoli Liu Jinghui Gu Peng Zhao 52 3 0 19 Dec 2023
Distributed Inference and Fine-tuning of Large Language Models Over The Internet Alexander Borzunov Max Ryabinin Artem Chumachenko Dmitry Baranchuk Tim Dettmers Younes Belkada Pavel Samygin Colin Raffel MoE ALM 45 42 0 13 Dec 2023
$S$^{3}$: Increasing GPU Utilization during Generative Inference for Higher Throughput$ S $^{3}$ : Increasing GPU Utilization during Generative Inference for Higher Throughput Yunho Jin Chun-Feng Wu David Brooks Gu-Yeon Wei 93 71 0 09 Jun 2023
Reducing Activation Recomputation in Large Transformer Models V. Korthikanti Jared Casper Sangkug Lym Lawrence C. McAfee M. Andersch Mohammad Shoeybi Bryan Catanzaro AI4CE 125 275 0 10 May 2022
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 306 369 0 18 Feb 2022
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale Samyam Rajbhandari Conglong Li Z. Yao Minjia Zhang Reza Yazdani Aminabadi A. A. Awan Jeff Rasley Yuxiong He 112 304 0 14 Jan 2022
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 493 10,526 0 17 Jun 2021
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan Mohammad Shoeybi Jared Casper P. LeGresley M. Patwary ... Prethvi Kashinkunti J. Bernauer Bryan Catanzaro Amar Phanishayee Matei A. Zaharia MoE 124 701 0 09 Apr 2021
FastMoE: A Fast Mixture-of-Expert Training System Jiaao He J. Qiu Aohan Zeng Zhilin Yang Jidong Zhai Jie Tang ALM MoE 88 101 0 24 Mar 2021
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang M. Krikun Noam M. Shazeer Zhiwen Chen MoE 124 1,191 0 30 Jun 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Mohammad Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 336 1,918 0 17 Sep 2019