No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

28 February 2024

Papers citing "No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization"

35 / 35 papers shown

Title
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate A. Zandieh Majid Daliri Majid Hadian Vahab Mirrokni MQ 74 0 0 28 Apr 2025
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs Piotr Nawrot Robert Li Renjie Huang Sebastian Ruder Kelly Marchisio E. Ponti 39 0 0 24 Apr 2025
KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments Junyoung Park Dalton Jones Matthew J Morse Raghavv Goel Mingu Lee Chris Lott 27 0 0 21 Apr 2025
KeepKV: Eliminating Output Perturbation in KV Cache Compression for Efficient LLMs Inference Yuxuan Tian Zihan Wang Yebo Peng Aomufei Yuan Zhaoxiang Wang Bairen Yi Xin Liu Yong Cui Tong Yang 39 0 0 14 Apr 2025
Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference Wei Tao Bin Zhang Xiaoyang Qu Jiguang Wan Jianzong Wang 47 1 0 30 Mar 2025
GPU-Accelerated Motion Planning of an Underactuated Forestry Crane in Cluttered Environments M. Vu Gerald Ebmer Alexander Watcher Marc-Philip Ecker Giang Nguyen Tobias Glueck 74 2 0 18 Mar 2025
CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences Ziran Qin Yuchen Cao Mingbao Lin Wen Hu Shixuan Fan Ke Cheng Weiyao Lin Jianguo Li 71 3 0 16 Mar 2025
Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs Ravi Ghadia Avinash Kumar Gaurav Jain Prashant J. Nair Poulami Das 46 1 0 02 Mar 2025
Binary Neural Networks for Large Language Model: A Survey Liangdong Liu Zhitong Zheng Cong Wang TianHuang Su ZhenYu Yang MQ 67 0 0 26 Feb 2025
SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention Hong Yankun Li Xing Zhen Hui-Ling Yu Xianzhi Liu Wulong Yuan Mingxuan MQ 80 0 0 24 Feb 2025
KVCrush: Key value cache size-reduction using similarity in head-behaviour Gopi Krishna Jha Sameh Gobriel Liubov Talamanova Alexander Kozlov Nilesh Jain MQ 39 0 0 24 Feb 2025
CoKV: Optimizing KV Cache Allocation via Cooperative Game Qiheng Sun Hongwei Zhang Haocheng Xia Jiayao Zhang Jinfei Liu Kui Ren VLM 42 0 0 21 Feb 2025
BalanceKV: KV Cache Compression through Discrepancy Theory Insu Han Michael Kapralov Ekaterina Kochetkova Kshiteej Sheth A. Zandieh 86 2 0 11 Feb 2025
MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache Akshat Sharma Hangliang Ding Jianping Li Neel Dani Minjia Zhang 80 1 0 27 Nov 2024
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection Wei Wu Zhuoshi Pan Chao Wang L. Chen Y. Bai Kun Fu Zehua Wang Hui Xiong Hui Xiong LLMAG 39 5 0 05 Nov 2024
VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration Dezhan Tu Danylo Vashchilenko Yuzhe Lu Panpan Xu VLM 48 9 0 29 Oct 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Yiyuan Ma Wenlei Bao Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 88 16 0 28 Oct 2024
Lossless KV Cache Compression to 2% Zhen Yang Jizong Han Kan Wu Ruobing Xie An Wang Xingchen Sun Zhanhui Kang VLM MQ 36 2 0 20 Oct 2024
AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations Qian Tao Wenyuan Yu Jingren Zhou MQ 31 4 0 17 Oct 2024
ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification and KV Cache Compression Yefei He Feng Chen Jing Liu Wenqi Shao Hong Zhou Kaipeng Zhang Bohan Zhuang VLM 52 11 0 11 Oct 2024
Fast State Restoration in LLM Serving with HCache Shiwei Gao Youmin Chen Jiwu Shu 27 4 0 07 Oct 2024
AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization Yifan Tan Haoze Wang Chao Yan Yangdong Deng MQ 31 2 0 25 Sep 2024
Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner Yuzhang Shang Bingxin Xu Weitai Kang Mu Cai Yuheng Li Zehao Wen Zhen Dong Kurt Keutzer Yong Jae Lee Yan Yan 38 7 0 19 Sep 2024
Eigen Attention: Attention in Low-Rank Space for KV Cache Compression Utkarsh Saxena Gobinda Saha Sakshi Choudhary Kaushik Roy 36 8 0 10 Aug 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 33 0 25 Jul 2024
Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks Zheng Wang Boxiao Jin Zhongzhi Yu Minjia Zhang MoMe 45 23 0 11 Jul 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 46 17 0 01 Jul 2024
LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference Zhongwei Wan Ziang Wu Che Liu Jinfa Huang Zhihong Zhu Peng Jin Longyue Wang Li Yuan VLM 38 28 0 26 Jun 2024
D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models Zhongwei Wan Xinjian Wu Yu Zhang Yi Xin Chaofan Tao ... Xin Wang Siqi Luo Jing Xiong Mi Zhang Mi Zhang 29 0 0 18 Jun 2024
QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead A. Zandieh Majid Daliri Insu Han MQ 45 12 0 05 Jun 2024
BiSup: Bidirectional Quantization Error Suppression for Large Language Models Minghui Zou Ronghui Guo Sai Zhang Xiaowang Zhang Zhiyong Feng MQ 36 1 0 24 May 2024
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification Yefei He Luoming Zhang Weijia Wu Jing Liu Hong Zhou Bohan Zhuang MQ 51 25 0 23 May 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 46 47 0 18 Apr 2024
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 149 369 0 13 Mar 2023
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,489 0 23 Jan 2020