Splitwise: Efficient generative LLM inference using phase splitting

Splitwise: Efficient generative LLM inference using phase splitting

30 November 2023

Íñigo Goiri

Ricardo Bianchini

Papers citing "Splitwise: Efficient generative LLM inference using phase splitting"

12 / 112 papers shown

Title
Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention Bin Gao Zhuomin He Puru Sharma Qingxuan Kang Djordje Jevdjic Junbo Deng Xingkun Yang Zhou Yu Pengfei Zuo 71 45 0 23 Mar 2024
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve Amey Agrawal Nitin Kedia Ashish Panwar Jayashree Mohan Nipun Kwatra Bhargav S. Gulavani Alexey Tumanov Ramachandran Ramjee 55 162 0 04 Mar 2024
DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving F. Strati Sara Mcallister Amar Phanishayee Jakub Tarnawski Ana Klimovic 46 25 0 04 Mar 2024
InferCept: Efficient Intercept Support for Augmented Large Language Model Inference Reyna Abhyankar Zijian He Vikranth Srivatsa Hao Zhang Yiying Zhang RALM 40 13 0 02 Feb 2024
T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives Suchita Pati Shaizeen Aga Mahzabeen Islam Nuwan Jayasena Matthew D. Sinclair 22 13 0 30 Jan 2024
Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads Cunchen Hu Heyang Huang Liangliang Xu Xusheng Chen Jiang Xu ... Chenxi Wang Sa Wang Yungang Bao Ninghui Sun Yizhou Shan DRL 41 63 0 20 Jan 2024
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving Yinmin Zhong Shengyu Liu Junda Chen Jianbo Hu Yibo Zhu Xuanzhe Liu Xin Jin Hao Zhang 44 179 0 18 Jan 2024
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving Yilong Zhao Chien-Yu Lin Kan Zhu Zihao Ye Lequn Chen Wenlei Bao Luis Ceze Arvind Krishnamurthy Tianqi Chen Baris Kasikci MQ 28 133 0 29 Oct 2023
Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models Wenqi Jiang Marco Zeller R. Waleffe Torsten Hoefler Gustavo Alonso 54 14 0 15 Oct 2023
Optimizing Distributed ML Communication with Fused Computation-Collective Operations Kishore Punniyamurthy Khaled Hamidouche Bradford M. Beckmann FedML 34 8 0 11 May 2023
Fast Distributed Inference Serving for Large Language Models Bingyang Wu Yinmin Zhong Zili Zhang Gang Huang Xuanzhe Liu Xin Jin 35 93 0 10 May 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 149 371 0 13 Mar 2023