WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More

19 February 2024

Papers citing "WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More"

28 / 28 papers shown

Title
MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design Haojie Duanmu Xiuhong Li Zhihang Yuan Size Zheng Jiangfei Duan Xingcheng Zhang Dahua Lin MQ MoE 200 0 0 09 May 2025
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate A. Zandieh Majid Daliri Majid Hadian Vahab Mirrokni MQ 74 0 0 28 Apr 2025
Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling Ziran Qin Youru Lv Mingbao Lin Zeren Zhang Danping Zou Weiyao Lin VLM 43 0 0 12 Apr 2025
SQuat: Subspace-orthogonal KV Cache Quantization Hao Wang Ligong Han Kai Xu Akash Srivastava MQ 51 0 0 31 Mar 2025
Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models Keda Tao Haoxuan You Yang Sui Can Qin Haoyu Wang VLM MQ 91 0 0 20 Mar 2025
CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences Ziran Qin Yuchen Cao Mingbao Lin Wen Hu Shixuan Fan Ke Cheng Weiyao Lin Jianguo Li 74 3 0 16 Mar 2025
Binary Neural Networks for Large Language Model: A Survey Liangdong Liu Zhitong Zheng Cong Wang TianHuang Su ZhenYu Yang MQ 67 0 0 26 Feb 2025
More for Keys, Less for Values: Adaptive KV Cache Quantization Mohsen Hariri Lam Nguyen Sixu Chen Shaochen Zhong Qifan Wang Xia Hu Xiaotian Han V. Chaudhary MQ 48 0 0 24 Feb 2025
GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning Sifan Zhou Shuo Wang Zhihang Yuan Mingjia Shi Yuzhang Shang Dawei Yang ALM MQ 90 0 0 18 Feb 2025
BalanceKV: KV Cache Compression through Discrepancy Theory Insu Han Michael Kapralov Ekaterina Kochetkova Kshiteej Sheth A. Zandieh 86 2 0 11 Feb 2025
CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation Hongxuan Zhang Yao Zhao Jiaqi Zheng Chenyi Zhuang Jinjie Gu Guihai Chen MQ 75 1 0 16 Dec 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Yiyuan Ma Wenlei Bao Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 88 16 0 28 Oct 2024
AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations Qian Tao Wenyuan Yu Jingren Zhou MQ 31 4 0 17 Oct 2024
AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization Yifan Tan Haoze Wang Chao Yan Yangdong Deng MQ 31 2 0 25 Sep 2024
Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner Yuzhang Shang Bingxin Xu Weitai Kang Mu Cai Yuheng Li Zehao Wen Zhen Dong Kurt Keutzer Yong Jae Lee Yan Yan 41 7 0 19 Sep 2024
Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview Yanshu Wang Tong Yang Xiyan Liang Guoan Wang Hanning Lu Xu Zhe Yaoming Li Li Weitao MQ 44 3 0 18 Sep 2024
Palu: Compressing KV-Cache with Low-Rank Projection Chi-Chih Chang Wei-Cheng Lin Chien-Yu Lin Chong-Yan Chen Yu-Fang Hu Pei-Shuo Wang N. Huang Luis Ceze Kai-Chiang Wu 59 0 0 30 Jul 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 33 0 25 Jul 2024
D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models Zhongwei Wan Xinjian Wu Yu Zhang Yi Xin Chaofan Tao ... Xin Wang Siqi Luo Jing Xiong Mi Zhang Mi Zhang 29 0 0 18 Jun 2024
QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead A. Zandieh Majid Daliri Insu Han MQ 48 12 0 05 Jun 2024
I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models Xing Hu Yuan Cheng Dawei Yang Zhihang Yuan Jiangyong Yu Chen Xu Sifan Zhou MQ 36 7 0 28 May 2024
Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis Yao Fu 35 19 0 14 May 2024
SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models Haojie Duanmu Zhihang Yuan Xiuhong Li Jiangfei Duan Xingcheng Zhang Dahua Lin MQ 42 19 0 10 May 2024
Efficient LLM Inference with Kcache Qiaozhi He Zhihua Wu RALM 32 1 0 28 Apr 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 46 47 0 18 Apr 2024
LLM Inference Unveiled: Survey and Roofline Model Insights Zhihang Yuan Yuzhang Shang Yang Zhou Zhen Dong Zhe Zhou ... Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer 54 80 0 26 Feb 2024
A Survey on Model Compression for Large Language Models Xunyu Zhu Jian Li Yong Liu Can Ma Weiping Wang 36 193 0 15 Aug 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 354 12,003 0 04 Mar 2022