InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

28 June 2024

Papers citing "InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management"

50 / 50 papers shown

Title
ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor Seungbeom Choi Jeonghoe Goo Eunjoo Jeon Mingyu Yang Minsung Jang 21 0 0 14 May 2025
SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models Hang Wu Jianian Zhu Yongqian Li Haojie Wang Biao Hou Jidong Zhai 40 0 0 12 May 2025
FloE: On-the-Fly MoE Inference on Memory-constrained GPU Yuxin Zhou Zheng Li J. Zhang Jue Wang Yunhong Wang Zhongle Xie Ke Chen Lidan Shou MoE 52 0 0 09 May 2025
Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM Zehao Fan Garrett Gagnon Zhenyu Liu Liu Liu 29 0 0 09 May 2025
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference Y. Chen J. Zhang Baotong Lu Qianxi Zhang Chengruidong Zhang ... Chen Chen Mingxing Zhang Yuqing Yang Fan Yang Mao Yang 38 0 0 05 May 2025
Taming the Titans: A Survey of Efficient LLM Inference Serving Ranran Zhen J. Li Yixin Ji Z. Yang Tong Liu Qingrong Xia Xinyu Duan Z. Wang Baoxing Huai M. Zhang LLMAG 77 0 0 28 Apr 2025
GenTorrent: Scaling Large Language Model Serving with An Overley Network Fei Fang Yifan Hua Shengze Wang Ruilin Zhou Y. Liu Chen Qian Xuzhi Zhang 60 0 0 27 Apr 2025
Tempo: Application-aware LLM Serving with Mixed SLO Requirements Wei Zhang Zhiyu Wu Yi Mu Banruo Liu Myungjin Lee Fan Lai 58 0 0 24 Apr 2025
L3: DIMM-PIM Integrated Architecture and Coordination for Scalable Long-Context LLM Inference Qingyuan Liu Liyan Chen Yanning Yang Haoyu Wang Dong Du Zhigang Mao Naifeng Jing Yubin Xia Haibo Chen 36 0 0 24 Apr 2025
PyGraph: Robust Compiler Support for CUDA Graphs in PyTorch Abhishek Ghosh Ajay Nayak Ashish Panwar Arkaprava Basu GNN 47 0 0 25 Mar 2025
Mitigating KV Cache Competition to Enhance User Experience in LLM Inference Haiying Shen Tanmoy Sen Masahiro Tanaka 159 0 0 17 Mar 2025
Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference Mohammad Siavashi Faezeh Keshmiri Dindarloo Dejan Kostić Marco Chiesa MoE VLM 45 0 0 13 Mar 2025
Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation Wenlong Meng Fan Zhang Wendao Yao Zhenyuan Guo Yongqian Li Chengkun Wei Wenzhi Chen AAML 40 1 0 11 Mar 2025
LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference G. Wang Shubhangi Upasani Chen Henry Wu Darshan Gandhi Jonathan Li Changran Hu Bo Li Urmish Thakker 77 0 0 11 Mar 2025
Alchemist: Towards the Design of Efficient Online Continual Learning System Yuyang Huang Yuhan Liu Haryadi S. Gunawi Beibin Li Changho Hwang CLL OnRL 101 0 0 03 Mar 2025
Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving Qihui Zhou Peiqi Yin Pengfei Zuo James Cheng CLL 40 1 0 01 Mar 2025
HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading Cheng Luo Zefan Cai Hanshi Sun Jinqi Xiao Bo Yuan Wen Xiao Junjie Hu Jiawei Zhao Beidi Chen Anima Anandkumar 69 1 0 18 Feb 2025
APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs Yuxiang Huang Mingye Li Xu Han Chaojun Xiao Weilin Zhao Sun Ao Hao Zhou Jie Zhou Zhiyuan Liu Maosong Sun 44 0 0 17 Feb 2025
Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization Bowen Pang Kai Li Ruifeng She Feifan Wang OffRL 45 2 0 14 Feb 2025
fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving Hanfei Yu Xingqi Cui H. M. Zhang Hairu Wang Hao Wang MoE 61 0 0 07 Feb 2025
GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments Yanyu Chen Ganhong Huang 108 0 0 28 Jan 2025
HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location Ting Sun Penghan Wang Fan Lai 148 1 0 15 Jan 2025
Tensor Product Attention Is All You Need Yifan Zhang Yifeng Liu Huizhuo Yuan Zhen Qin Yang Yuan Q. Gu Andrew Chi-Chih Yao 77 9 0 11 Jan 2025
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval Di Liu Meng Chen Baotong Lu Huiqiang Jiang Zhenhua Han ... Kaipeng Zhang Cheng Chen Fan Yang Yuqing Yang Lili Qiu 52 29 0 03 Jan 2025
Pushing the Envelope of Low-Bit LLM via Dynamic Error Compensation Y. Park Jake Hyun Hojoon Kim Jae W. Lee MQ 46 0 0 31 Dec 2024
SYMPHONY: Improving Memory Management for LLM Inference Workloads Saurabh Agarwal Anyong Mao Aditya Akella Shivaram Venkataraman LLMAG 80 0 0 21 Dec 2024
ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression Guangda Liu Chong Li Jieru Zhao Chenqi Zhang M. Guo 74 8 0 04 Dec 2024
Unifying KV Cache Compression for Large Language Models with LeanKV Yanqi Zhang Yuwei Hu Runyuan Zhao John C. S. Lui Haibo Chen MQ 136 5 0 04 Dec 2024
Marconi: Prefix Caching for the Era of Hybrid LLMs Rui Pan Zhuang Wang Zhen Jia Can Karakus Luca Zancato Tri Dao Ravi Netravali Yida Wang 95 4 0 28 Nov 2024
DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving Yuhan Liu Esha Choukse Shan Lu Junchen Jiang Madan Musuvathi ... Yihua Cheng Junchen Jiang Shan Lu Madan Musuvathi Esha Choukse 88 2 0 05 Nov 2024
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection Wei Yu Wu Zhuoshi Pan Chao Wang L. Chen Y. Bai Kun Fu Zehua Wang Hui Xiong Hui Xiong LLMAG 36 5 0 05 Nov 2024
VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration Dezhan Tu Danylo Vashchilenko Yuzhe Lu Panpan Xu VLM 45 9 0 29 Oct 2024
ProMoE: Fast MoE-based LLM Serving using Proactive Caching Xiaoniu Song Zihang Zhong Rong Chen Haibo Chen MoE 65 4 0 29 Oct 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Yiyuan Ma Wenlei Bao Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 88 16 0 28 Oct 2024
Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching Jie Peng Zhang Cao Huaizhi Qu Zhengyu Zhang Chang Guo Yanyong Zhang Zhichao Cao Tianlong Chen 34 2 0 17 Oct 2024
MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection Bokai Lin Zihao Zeng Zipeng Xiao Siqi Kou Tianqi Hou Xiaofeng Gao Hao Zhang Zhijie Deng 16 2 0 16 Oct 2024
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices Yuxiang Huang Binhang Yuan Xu Han Chaojun Xiao Zhiyuan Liu RALM 81 1 0 02 Oct 2024
TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices Zonghang Li Wenjiao Feng Mohsen Guizani Hongfang Yu 45 2 0 01 Oct 2024
LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management Yi Xiong Hao Wu Changxu Shao Ziqing Wang Rui Zhang Yuhong Guo Junping Zhao Ke Zhang Zhenxuan Pan 40 4 0 01 Oct 2024
Medha: Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations A. Agrawal Haoran Qiu Junda Chen Íñigo Goiri Chaojie Zhang Rayyan Shahid Ramachandran Ramjee Alexey Tumanov Esha Choukse RALM LRM 32 1 0 25 Sep 2024
CSPS: A Communication-Efficient Sequence-Parallelism based Serving System for Transformer based Models with Long Prompts Zeyu Zhang Haiying Shen VLM 29 0 0 23 Sep 2024
InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference Xiurui Pan Endian Li Qiao Li Shengwen Liang Yizhou Shan Ke Zhou Yingwei Luo Xiaolin Wang Jie Zhang 45 10 0 08 Sep 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 33 0 25 Jul 2024
LLM Inference Serving: Survey of Recent Advances and Opportunities Baolin Li Yankai Jiang V. Gadepally Devesh Tiwari 78 18 0 17 Jul 2024
Loki: Low-Rank Keys for Efficient Sparse Attention Prajwal Singhania Siddharth Singh Shwai He S. Feizi A. Bhatele 34 13 0 04 Jun 2024
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling Zefan Cai Yichi Zhang Bofei Gao Yuliang Liu Yongqian Li ... Wayne Xiong Yue Dong Baobao Chang Junjie Hu Wen Xiao 62 84 0 04 Jun 2024
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention Ramya Prabhu Ajay Nayak Jayashree Mohan Ramachandran Ramjee Ashish Panwar VLM 57 25 0 07 May 2024
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU Yixin Song Zeyu Mi Haotong Xie Haibo Chen BDL 125 120 0 16 Dec 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 149 369 0 13 Mar 2023
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018