PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM
Inference

PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference

21 May 2024

Yao Hu

Papers citing "PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference"

16 / 16 papers shown

Title
KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments Junyoung Park Dalton Jones Matthew J Morse Raghavv Goel Mingu Lee Chris Lott 55 0 0 21 Apr 2025
WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference Youhui Zuo Sibo Wei C. Zhang Zhuorui Liu Wenpeng Lu Dawei Song VLM 90 0 0 23 Mar 2025
MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference Zhongwei Wan Jikang Cheng Xin Wang Junfeng Fang Zheda Mai Hao Fei VLM 88 4 0 24 Feb 2025
FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference Bingzhe Zhao Ke Cheng Aomufei Yuan Yuxuan Tian Ruiguang Zhong Chengchen Hu Tong Yang Lian Yu 75 0 0 19 Feb 2025
Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile Hangliang Ding Dacheng Li Runlong Su Peiyuan Zhang Zhijie Deng Ion Stoica Hao Zhang VGen 104 7 0 10 Feb 2025
Can LLMs Maintain Fundamental Abilities under KV Cache Compression? Xiang Liu Zhenheng Tang Hong Chen Peijie Dong Zeyu Li Xiuze Zhou Bo Li Xuming Hu Xiaowen Chu 379 5 0 04 Feb 2025
Twilight: Adaptive Attention Sparsity with Hierarchical Top- $p$ Pruning C. Lin Jiaming Tang Shuo Yang Hanshuo Wang Tian Tang Boyu Tian Ion Stoica Enze Xie Mingyu Gao 119 3 0 04 Feb 2025
Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference Yuan Feng Junlin Lv Yukun Cao Xike Xie S. K. Zhou VLM 85 37 0 28 Jan 2025
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices Yuxiang Huang Binhang Yuan Xu Han Chaojun Xiao Zhiyuan Liu RALM 114 1 0 02 Oct 2024
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding Yao Teng Han Shi Xian Liu Xuefei Ning Guohao Dai Yu Wang Zhenguo Li Xihui Liu 82 12 0 02 Oct 2024
ThinK: Thinner Key Cache by Query-Driven Pruning Yuhui Xu Zhanming Jie Hanze Dong Lei Wang Xudong Lu Aojun Zhou Amrita Saha Caiming Xiong Doyen Sahoo 98 20 0 30 Jul 2024
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling Zefan Cai Yichi Zhang Bofei Gao Yuliang Liu Yongqian Li ... Wayne Xiong Yue Dong Baobao Chang Junjie Hu Wen Xiao 115 97 0 04 Jun 2024
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 312 4,253 0 09 Jun 2023
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 201 5,454 0 07 Jul 2021
Professor Forcing: A New Algorithm for Training Recurrent Networks Alex Lamb Anirudh Goyal Ying Zhang Saizheng Zhang Aaron Courville Yoshua Bengio GAN 119 594 0 27 Oct 2016
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 258 2,842 0 26 Sep 2016

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.