SpotServe: Serving Generative Large Language Models on Preemptible Instances

27 November 2023

Dahua Lin

Papers citing "SpotServe: Serving Generative Large Language Models on Preemptible Instances"

39 / 39 papers shown

Title
MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems Yinsicheng Jiang Yao Fu Yeqi Huang Ping Nie Zhan Lu ... Dayou Du Tairan Xu Kai Zou Edoardo Ponti Luo Mai MoE 17 0 0 16 May 2025
ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor Seungbeom Choi Jeonghoe Goo Eunjoo Jeon Mingyu Yang Minsung Jang 21 0 0 14 May 2025
Ascendra: Dynamic Request Prioritization for Efficient LLM Serving Azam Ikram Xiang Li Sameh Elnikety S. Bagchi 120 0 0 29 Apr 2025
Tempo: Application-aware LLM Serving with Mixed SLO Requirements Wei Zhang Zhiyu Wu Yi Mu Banruo Liu Myungjin Lee Fan Lai 58 0 0 24 Apr 2025
Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving Shihong Gao Xuzhi Zhang Yanyan Shen Lei Chen 27 1 0 10 Apr 2025
HERA: Hybrid Edge-cloud Resource Allocation for Cost-Efficient AI Agents Shiyi Liu Haiying Shen Shuai Che Mahdi Ghandi Mingqin Li LLMAG 53 0 0 01 Apr 2025
Seesaw: High-throughput LLM Inference via Model Re-sharding Qidong Su Wei Zhao Xuelong Li Muralidhar Andoorveedu Chenhao Jiang Zhanda Zhu Kevin Song Christina Giannoula Gennady Pekhimenko LRM 77 0 0 09 Mar 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 208 0 0 08 Jan 2025
TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms Jovan Stojkovic Chaojie Zhang Íñigo Goiri Esha Choukse Haoran Qiu Rodrigo Fonseca Josep Torrellas Ricardo Bianchini 42 4 0 05 Jan 2025
Deploying Foundation Model Powered Agent Services: A Survey Wenchao Xu Jinyu Chen Peirong Zheng Xiaoquan Yi Tianyi Tian ... Quan Wan Yining Qi Yunfeng Fan Qinliang Su Xuemin Shen AI4CE 119 1 0 18 Dec 2024
Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments Nikoleta Iliakopoulou Jovan Stojkovic Chloe Alverti Tianyin Xu Hubertus Franke Josep Torrellas 77 2 0 24 Nov 2024
DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving Yuhan Liu Esha Choukse Shan Lu Junchen Jiang Madan Musuvathi ... Yihua Cheng Junchen Jiang Shan Lu Madan Musuvathi Esha Choukse 90 2 0 05 Nov 2024
SkyServe: Serving AI Models across Regions and Clouds with Spot Instances Ziming Mao Tian Xia Zhanghao Wu Wei-Lin Chiang Tyler Griggs Romil Bhardwaj Zongheng Yang S. Shenker Ion Stoica 59 2 0 03 Nov 2024
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 31 7 0 08 Oct 2024
TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices Zonghang Li Wenjiao Feng Mohsen Guizani Hongfang Yu 50 3 0 01 Oct 2024
ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments Munkyu Lee Sihoon Seong Minki Kang Jihyuk Lee Gap-Joo Na In-Geol Chun Dimitrios Nikolopoulos Cheol-Ho Hong GNN 29 0 0 22 Sep 2024
LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale Jaehong Cho Minsu Kim Hyunmin Choi Guseul Heo Jongse Park 40 9 0 10 Aug 2024
DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency Jovan Stojkovic Chaojie Zhang Íñigo Goiri Josep Torrellas Esha Choukse 44 31 0 01 Aug 2024
LLM Inference Serving: Survey of Recent Advances and Opportunities Baolin Li Yankai Jiang V. Gadepally Devesh Tiwari 80 18 0 17 Jul 2024
Enabling Elastic Model Serving with MultiWorld Myungjin Lee Akshay Jajoo Ramana Rao Kompella MoE 71 0 0 12 Jul 2024
Teola: Towards End-to-End Optimization of LLM-based Applications Xin Tan Yimin Jiang Yitao Yang Hong-Yu Xu 73 5 0 29 Jun 2024
Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving Ke Cheng Wen Hu Zhi Wang Hongen Peng Jianguo Li Sheng Zhang 57 7 0 19 Jun 2024
Llumnix: Dynamic Scheduling for Large Language Model Serving Biao Sun Ziming Huang Hanyu Zhao Wencong Xiao Xinyi Zhang Yong Li Wei Lin 43 45 0 05 Jun 2024
BlockLLM: Multi-tenant Finer-grained Serving for Large Language Models Jiamin Li Le Xu Hong-Yu Xu Aditya Akella 32 1 0 28 Apr 2024
Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity Tyler Griggs Xiaoxuan Liu Jiaxiang Yu Doyoung Kim Wei-Lin Chiang Alvin Cheung Ion Stoica 54 16 0 22 Apr 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 83 0 22 Apr 2024
MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving Jiangfei Duan Runyu Lu Haojie Duanmu Xiuhong Li Xingcheng Zhang Dahua Lin Ion Stoica Hao Zhang 50 9 0 02 Apr 2024
Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances Jiangfei Duan Ziang Song Xupeng Miao Xiaoli Xi Dahua Lin Harry Xu Minjia Zhang Zhihao Jia 52 10 0 21 Mar 2024
Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inference Baolin Li Yankai Jiang V. Gadepally Devesh Tiwari 29 15 0 19 Mar 2024
DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving F. Strati Sara Mcallister Amar Phanishayee Jakub Tarnawski Ana Klimovic 46 25 0 04 Mar 2024
FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning Xupeng Miao Gabriele Oliaro Xinhao Cheng Vineeth Kada Ruohan Gao ... April Yang Yingcheng Wang Mengdi Wu Colin Unger Zhihao Jia MoE 94 9 0 29 Feb 2024
CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference Suyi Li Hanfeng Lu Tianyuan Wu Minchen Yu Qizhen Weng Xusheng Chen Yizhou Shan Binhang Yuan Wei Wang 56 12 0 20 Jan 2024
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems Xupeng Miao Gabriele Oliaro Zhihao Zhang Xinhao Cheng Hongyi Jin Tianqi Chen Zhihao Jia 67 76 0 23 Dec 2023
SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification Xupeng Miao Gabriele Oliaro Zhihao Zhang Xinhao Cheng Zeyu Wang ... Chunan Shi Zhuoming Chen Daiyaan Arfeen Reyna Abhyankar Zhihao Jia LRM 65 120 0 16 May 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 149 369 0 13 Mar 2023
Varuna: Scalable, Low-cost Training of Massive Deep Learning Models Sanjith Athlur Nitika Saran Muthian Sivathanu Ramachandran Ramjee Nipun Kwatra GNN 33 80 0 07 Nov 2021
What Makes Good In-Context Examples for GPT- $3$ ? Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin Weizhu Chen AAML RALM 275 1,312 0 17 Jan 2021
Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider Mohammad Shahrad Rodrigo Fonseca Íñigo Goiri G. Chaudhry Paul Batum Jason Cooke Eduardo Laureano Colby Tresness M. Russinovich Ricardo Bianchini 89 601 0 06 Mar 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,826 0 17 Sep 2019