SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention

24 February 2025

Papers citing "SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention"

28 / 28 papers shown

Title
Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA Qingyun Jin Xiaohui Song Feng Zhou Zengchang Qin 57 4 0 31 Dec 2024
BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments Xinghao Wang Pengyu Wang Bo Wang Dong Zhang Yunhua Zhou Xipeng Qiu MQ 58 1 0 31 Oct 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Yiyuan Ma Wenlei Bao Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 133 21 0 28 Oct 2024
LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy Rongzhi Zhang Kuang Wang Liyuan Liu Shuohang Wang Hao Cheng Chao Zhang Yelong Shen MQ 79 12 0 04 Oct 2024
AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization Yifan Tan Haoze Wang Chao Yan Yangdong Deng MQ 66 2 0 25 Sep 2024
Eigen Attention: Attention in Low-Rank Space for KV Cache Compression Utkarsh Saxena Gobinda Saha Sakshi Choudhary Kaushik Roy 91 18 0 10 Aug 2024
Palu: Compressing KV-Cache with Low-Rank Projection Chi-Chih Chang Wei-Cheng Lin Chien-Yu Lin Chong-Yan Chen Yu-Fang Hu Pei-Shuo Wang N. Huang Luis Ceze Kai-Chiang Wu 91 2 0 30 Jul 2024
ThinK: Thinner Key Cache by Query-Driven Pruning Yuhui Xu Zhanming Jie Hanze Dong Lei Wang Xudong Lu Aojun Zhou Amrita Saha Caiming Xiong Doyen Sahoo 162 23 0 30 Jul 2024
Optimised Grouped-Query Attention Mechanism for Transformers Yuang Chen Cheng Zhang Xitong Gao Robert D. Mullins George A. Constantinides Yiren Zhao 73 9 0 21 Jun 2024
Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference Jiaming Tang Yilong Zhao Kan Zhu Guangxuan Xiao Baris Kasikci Song Han 94 106 0 16 Jun 2024
Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models Bowen Ping Shuo Wang Hanqing Wang Xu Han Yuzhuang Xu Yukun Yan Yun Chen Baobao Chang Zhiyuan Liu Maosong Sun MQ 102 10 0 13 Jun 2024
Loki: Low-Rank Keys for Efficient Sparse Attention Prajwal Singhania Siddharth Singh Shwai He Soheil Feizi A. Bhatele 97 22 0 04 Jun 2024
Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression Peiyu Liu Zeming Gao Wayne Xin Zhao Yipeng Ma Tao Wang Ji-Rong Wen MQ 138 5 0 21 May 2024
QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving Chengyue Wu Haotian Tang Shang Yang Zhekai Zhang Guangxuan Xiao Chuang Gan Song Han 140 96 0 07 May 2024
SnapKV: LLM Knows What You are Looking for Before Generation Yuhong Li Yingbing Huang Bowen Yang Bharat Venkitesh Acyr Locatelli Hanchen Ye Tianle Cai Patrick Lewis Deming Chen VLM 132 210 0 22 Apr 2024
SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression Xin Wang Yu Zheng Zhongwei Wan Mi Zhang MQ 139 63 0 12 Mar 2024
No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization J. Yang Byeongwook Kim Jeongin Bae Beomseok Kwon Gunho Park Eunho Yang S. Kwon Dongsoo Lee MQ 139 53 0 28 Feb 2024
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache Zirui Liu Jiayi Yuan Hongye Jin Shaochen Zhong Zhaozhuo Xu Vladimir Braverman Beidi Chen Helen Zhou MQ 98 204 0 05 Feb 2024
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization Coleman Hooper Sehoon Kim Hiva Mohammadzadeh Michael W. Mahoney Y. Shao Kurt Keutzer A. Gholami MQ 82 224 0 31 Jan 2024
SparQ Attention: Bandwidth-Efficient LLM Inference Luka Ribar Ivan Chelombiev Luke Hudlass-Galley Charlie Blake Carlo Luschi Douglas Orr 119 54 0 08 Dec 2023
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs Suyu Ge Yunan Zhang Liyuan Liu Minjia Zhang Jiawei Han Jianfeng Gao 58 259 0 03 Oct 2023
Efficient Streaming Language Models with Attention Sinks Michel Lang Yuandong Tian Beidi Chen Song Han Mike Lewis AI4TS RALM 138 790 0 29 Sep 2023
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding Yushi Bai Xin Lv Jiajie Zhang Hong Lyu Jiankai Tang ... Aohan Zeng Lei Hou Yuxiao Dong Jie Tang Juanzi Li LLMAG RALM 97 603 0 28 Aug 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu Zhang Ying Sheng Dinesh Manocha Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 151 313 0 24 Jun 2023
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai 101 695 0 22 May 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,761 0 15 Mar 2023
Efficiently Scaling Transformer Inference Reiner Pope Sholto Douglas Aakanksha Chowdhery Jacob Devlin James Bradbury Anselm Levskaya Jonathan Heek Kefan Xiao Shivani Agrawal J. Dean 106 325 0 09 Nov 2022
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 805 132,725 0 12 Jun 2017