DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

18 January 2024

Xin Jin

Papers citing "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving"

50 / 102 papers shown

Title
Chain-of-Model Learning for Language Model Kaitao Song Xiaohua Wang Xu Tan Huiqiang Jiang Chengruidong Zhang ... Xiaoqing Zheng Tao Qin Yuqing Yang Dongsheng Li Lili Qiu LRM AI4CE 2 0 0 17 May 2025
AI Greenferencing: Routing AI Inferencing to Green Modular Data Centers with Heron Tella Rajashekhar Reddy Palak Rohan Gandhi Anjaly Parayil Chaojie Zhang ... Liangcheng Yu Jayashree Mohan Srinivasan Iyengar Shivkumar Kalyanaraman Debopam Bhattacherjee 26 0 0 15 May 2025
Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures Chenggang Zhao Chengqi Deng Chong Ruan Damai Dai Huazuo Gao ... Wenfeng Liang Ying He Yishuo Wang Yuxuan Liu Y. X. Wei MoE 41 0 0 14 May 2025
ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor Seungbeom Choi Jeonghoe Goo Eunjoo Jeon Mingyu Yang Minsung Jang 21 0 0 14 May 2025
SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models Hang Wu Jianian Zhu Yong Li Haojie Wang Biao Hou Jidong Zhai 40 0 0 12 May 2025
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference Yushen Chen J. Zhang Baotong Lu Qianxi Zhang Chengruidong Zhang ... Chen Chen Mingxing Zhang Yuqing Yang Fan Yang Mao Yang 38 0 0 05 May 2025
Taming the Titans: A Survey of Efficient LLM Inference Serving Ranran Zhen J. Li Yixin Ji Zhengyuan Yang Tong Liu Qingrong Xia Xinyu Duan Zehao Wang Baoxing Huai Hao Fei LLMAG 77 0 0 28 Apr 2025
GenTorrent: Scaling Large Language Model Serving with An Overley Network Fei Fang Yifan Hua Shengze Wang Ruilin Zhou Y. Liu Chen Qian Xuzhi Zhang 60 0 0 27 Apr 2025
Tempo: Application-aware LLM Serving with Mixed SLO Requirements Wei Zhang Zhiyu Wu Yi Mu Banruo Liu Myungjin Lee Fan Lai 58 0 0 24 Apr 2025
Circinus: Efficient Query Planner for Compound ML Serving Banruo Liu Wei-Yu Lin Minghao Fang Yihan Jiang Fan Lai LRM 39 0 0 23 Apr 2025
From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs Yaxiong Wu Sheng Liang Chen Zhang Yucheng Wang Wenjie Qu Huifeng Guo Ruiming Tang Yong Liu KELM 47 1 0 22 Apr 2025
Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management Hang Zhang Jiuchen Shi Yixiao Wang Quan Chen Yizhou Shan Minyi Guo 36 0 0 19 Apr 2025
Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs Hyungwoo Lee Kihyun Kim Jinwoo Kim Jungmin So Myung-Hoon Cha H. Kim James J. Kim Youngjae Kim 37 0 0 16 Apr 2025
Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints Ruicheng Ao Gan Luo D. Simchi-Levi Xinshang Wang 31 2 0 15 Apr 2025
AlayaDB: The Data Foundation for Efficient and Effective Long-context LLM Inference Yangshen Deng Zhengxin You Long Xiang Qilong Li Peiqi Yuan ... Man Lung Yiu Huan Li Qiaomu Shen Rui Mao Bo Tang 42 0 0 14 Apr 2025
Understanding and Optimizing Multi-Stage AI Inference Pipelines Abhimanyu Bambhaniya Hanjiang Wu Suvinay Subramanian Sudarshan Srinivasan Souvik Kundu Amir Yazdanbakhsh Suvinay Subramanian Madhu Kumar Tushar Krishna 153 0 0 14 Apr 2025
Efficient LLM Serving on Hybrid Real-time and Best-effort Requests Wan Borui Zhao Juntao Jiang Chenyu Guo Chuanxiong Wu Chuan VLM 82 1 0 13 Apr 2025
MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints Yichao Yuan Lin Ma Nishil Talati MoE 64 0 0 12 Apr 2025
Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents Yueying Li Jim Dai Tianyi Peng 141 1 0 10 Apr 2025
Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving Shihong Gao Xuzhi Zhang Yanyan Shen Lei Chen 27 1 0 10 Apr 2025
SEE: Continual Fine-tuning with Sequential Ensemble of Experts Zhilin Wang Yafu Li Xiaoye Qu Yu Cheng CLL KELM 55 0 0 09 Apr 2025
SLOs-Serve: Optimized Serving of Multi-SLO LLMs Siyuan Chen Zhipeng Jia S. Khan Arvind Krishnamurthy Phillip B. Gibbons 29 3 0 05 Apr 2025
FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling Weiqing Li Guochao Jiang Xiangyong Ding Zhangcheng Tao Chuzhan Hao Chenfeng Xu Yuewei Zhang Hao Wang 31 0 0 03 Apr 2025
A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond Xiaoye Qu Yafu Li Zhaochen Su Weigao Sun Jianhao Yan ... Chaochao Lu Yue Zhang Xian-Sheng Hua Bowen Zhou Yu Cheng ReLM OffRL LRM 88 16 0 27 Mar 2025
Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation Yunkai Liang Zhangyu Chen Pengfei Zuo Zhi Zhou Xu Chen Zhou Yu 88 3 0 26 Mar 2025
Mitigating KV Cache Competition to Enhance User Experience in LLM Inference Haiying Shen Tanmoy Sen Masahiro Tanaka 171 0 0 17 Mar 2025
AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications Haiying Shen Tanmoy Sen 49 0 0 17 Mar 2025
Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference Mohammad Siavashi Faezeh Keshmiri Dindarloo Dejan Kostić Marco Chiesa MoE VLM 47 0 0 13 Mar 2025
Queueing, Predictions, and LLMs: Challenges and Open Problems Michael Mitzenmacher Rana Shahout AI4TS LRM 44 1 0 10 Mar 2025
eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference Suraiya Tairin Shohaib Mahmud Haiying Shen Anand Iyer MoE 176 0 0 10 Mar 2025
Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation Yingfeng Luo Tong Zheng Yongyu Mu Yangqiu Song Qinghong Zhang ... Ziqiang Xu Peinan Feng Xiaoqian Liu Tong Xiao Jingbo Zhu AI4CE 185 0 0 09 Mar 2025
Seesaw: High-throughput LLM Inference via Model Re-sharding Qidong Su Wei Zhao Xuelong Li Muralidhar Andoorveedu Chenhao Jiang Zhanda Zhu Kevin Song Christina Giannoula Gennady Pekhimenko LRM 77 0 0 09 Mar 2025
SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding Kaiyu Huang Yu Wang Zhubo Shi Han Zou Minchen Yu Qingjiang Shi LRM 38 2 0 07 Mar 2025
Alchemist: Towards the Design of Efficient Online Continual Learning System Yuyang Huang Yuhan Liu Haryadi S. Gunawi Beibin Li Changho Hwang CLL OnRL 103 0 0 03 Mar 2025
Efficient Long-Decoding Inference with Reasoning-Aware Attention Sparsity Junhao Hu Wenrui Huang Weidong Wang Zhenwen Li Tiancheng Hu Zhixia Liu Xusheng Chen Tao Xie Yizhou Shan LRM 51 0 0 16 Feb 2025
Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization Bowen Pang Kai Li Ruifeng She Feifan Wang OffRL 48 2 0 14 Feb 2025
fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving Hanfei Yu Xingqi Cui Huan Zhang Han Wang Hao Wang MoE 61 0 0 07 Feb 2025
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang Qirun Dai Hao Peng ALM 117 4 0 06 Feb 2025
KVDirect: Distributed Disaggregated LLM Inference Shiyang Chen Rain Jiang Dezhi Yu Jinlai Xu Mengyuan Chao Fanlong Meng Chenyu Jiang Wei Xu Hang Liu 48 1 0 28 Jan 2025
Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference Weizhi Fei Xueyan Niu Guoqing Xie Yingqing Liu Bo Bai Wei Han 33 1 0 22 Jan 2025
AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding Zikun Li Zhuofu Chen Remi Delacourt Gabriele Oliaro Zeyu Wang ... Zhuoming Chen Sean Lai Xinhao Cheng Xupeng Miao Zhihao Jia 53 6 0 21 Jan 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 194 0 0 08 Jan 2025
TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms Jovan Stojkovic Chaojie Zhang Íñigo Goiri Esha Choukse Haoran Qiu Rodrigo Fonseca Josep Torrellas Ricardo Bianchini 42 4 0 05 Jan 2025
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval Di Liu Meng Chen Baotong Lu Huiqiang Jiang Zhenhua Han ... Kaipeng Zhang Chong Chen Fan Yang Yuqing Yang Lili Qiu 54 30 0 03 Jan 2025
Towards Sustainable Large Language Model Serving Sophia Nguyen Beihao Zhou Yi Ding Sihang Liu 94 6 0 31 Dec 2024
TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication Zongwu Wang Fangxin Liu Mingshuai Li Li Jiang LRM 44 0 0 29 Dec 2024
Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels Mingcong Song Xinru Tang Fengfan Hou Jing Li Wei Wei ... Hongjie Si D. Jiang Shouyi Yin Yang Hu Guoping Long 36 1 0 24 Dec 2024
SYMPHONY: Improving Memory Management for LLM Inference Workloads Saurabh Agarwal Anyong Mao Aditya Akella Shivaram Venkataraman LLMAG 80 0 0 21 Dec 2024
Deploying Foundation Model Powered Agent Services: A Survey Wenchao Xu Jinyu Chen Peirong Zheng Xiaoquan Yi Tianyi Tian ... Quan Wan Yining Qi Yunfeng Fan Qinliang Su Xuemin Shen AI4CE 119 1 0 18 Dec 2024
XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference Weizhuo Li Zhigang Wang Yu Gu Ge Yu MQ 78 0 0 08 Dec 2024