LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism

15 April 2024

Xin Jin

Papers citing "LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism"

25 / 25 papers shown

Title
SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models Hang Wu Jianian Zhu Yicong Li Haojie Wang Biao Hou Jidong Zhai 40 0 0 12 May 2025
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference Y. Chen J. Zhang Baotong Lu Qianxi Zhang Chengruidong Zhang ... Chen Chen Mingxing Zhang Yuqing Yang Fan Yang Mao Yang 38 0 0 05 May 2025
Ascendra: Dynamic Request Prioritization for Efficient LLM Serving Azam Ikram Xiang Li Sameh Elnikety S. Bagchi 108 0 0 29 Apr 2025
Taming the Titans: A Survey of Efficient LLM Inference Serving Ranran Zhen J. Li Yixin Ji Zhengyuan Yang Tong Liu Qingrong Xia Xinyu Duan Zehao Wang Baoxing Huai M. Zhang LLMAG 77 0 0 28 Apr 2025
GenTorrent: Scaling Large Language Model Serving with An Overley Network Fei Fang Yifan Hua Shengze Wang Ruilin Zhou Y. Liu Chen Qian Xuzhi Zhang 60 0 0 27 Apr 2025
StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation Yinmin Zhong Zili Zhang Xiaoniu Song Hanpeng Hu Chao Jin ... Changyi Wan Hongyu Zhou Yimin Jiang Yibo Zhu Daxin Jiang OffRL AI4TS 57 0 0 22 Apr 2025
SLOs-Serve: Optimized Serving of Multi-SLO LLMs Siyuan Chen Zhipeng Jia S. Khan Arvind Krishnamurthy Phillip B. Gibbons 29 3 0 05 Apr 2025
TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms Jovan Stojkovic Chaojie Zhang Íñigo Goiri Esha Choukse Haoran Qiu Rodrigo Fonseca Josep Torrellas Ricardo Bianchini 42 4 0 05 Jan 2025
LoL-PIM: Long-Context LLM Decoding with Scalable DRAM-PIM System Hyucksung Kwon Kyungmo Koo Janghyeon Kim W. Lee Minjae Lee ... Yongkee Kwon Ilkon Kim Euicheol Lim John Kim Jungwook Choi 71 4 0 28 Dec 2024
Marconi: Prefix Caching for the Era of Hybrid LLMs Rui Pan Zhuang Wang Zhen Jia Can Karakus Luca Zancato Tri Dao Ravi Netravali Yida Wang 95 4 0 28 Nov 2024
Context Parallelism for Scalable Million-Token Inference Amy Yang Jingyi Yang Aya Ibrahim Xinfeng Xie Bangsheng Tang Grigory Sizov Jeremy Reizenstein Jongsoo Park Jianyu Huang MoE LRM 64 5 0 04 Nov 2024
POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference Aditya K Kamath Ramya Prabhu Jayashree Mohan Simon Peter Ramachandran Ramjee Ashish Panwar 59 9 0 23 Oct 2024
LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management Yi Xiong Hao Wu Changxu Shao Ziqing Wang Rui Zhang Yuhong Guo Junping Zhao Ke Zhang Zhenxuan Pan 40 4 0 01 Oct 2024
Medha: Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations A. Agrawal Haoran Qiu Junda Chen Íñigo Goiri Chaojie Zhang Rayyan Shahid Ramachandran Ramjee Alexey Tumanov Esha Choukse RALM LRM 32 1 0 25 Sep 2024
P/D-Serve: Serving Disaggregated Large Language Model at Scale Yibo Jin Tao Wang Huimin Lin Mingyang Song Peiyang Li ... Haoliang Cheng Xiaojing Li Jiandong Ding Hefei Guo Zhengyong Zhang MoE 38 9 0 15 Aug 2024
LLM Inference Serving: Survey of Recent Advances and Opportunities Baolin Li Yankai Jiang V. Gadepally Devesh Tiwari 78 18 0 17 Jul 2024
Teola: Towards End-to-End Optimization of LLM-based Applications Xin Tan Yimin Jiang Yitao Yang Hong-Yu Xu 73 5 0 29 Jun 2024
MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool Cunchen Hu Heyang Huang Junhao Hu Jiang Xu Xusheng Chen ... Chenxi Wang Sa Wang Yungang Bao Ninghui Sun Yizhou Shan LLMAG 50 23 0 25 Jun 2024
The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving Pai Zeng Zhenyu Ning Jieru Zhao Weihao Cui Mengwei Xu Liwei Guo Xusheng Chen Yizhou Shan LLMAG 48 4 0 18 May 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 83 0 22 Apr 2024
Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts Weilin Cai Juyong Jiang Le Qin Junwei Cui Sunghun Kim Jiayi Huang 53 7 0 07 Apr 2024
CacheGen: KV Cache Compression and Streaming for Fast Language Model Serving Yuhan Liu Hanchen Li Yihua Cheng Siddhant Ray Yuyang Huang ... Ganesh Ananthanarayanan Michael Maire Henry Hoffmann Ari Holtzman Junchen Jiang 50 41 0 11 Oct 2023
Fast Distributed Inference Serving for Large Language Models Bingyang Wu Yinmin Zhong Zili Zhang Gang Huang Xuanzhe Liu Xin Jin 30 92 0 10 May 2023
Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers Yujing Ma Florin Rusu Kesheng Wu A. Sim 40 3 0 13 Oct 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019