Not All Heads Matter: A Head-Level KV Cache Compression Method with
Integrated Retrieval and Reasoning

v1v2v3 (latest)

Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning

25 October 2024

Abedelkadir Asi

ArXiv (abs)PDF HTML Github (23★)

Papers citing "Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning"

16 / 16 papers shown

Title
ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models Hao Chen Haoze Li Zhiqing Xiao Lirong Gao Qi Zhang Xiaomeng Hu Ningtao Wang Xing Fu Junbo Zhao 187 0 0 24 May 2025
Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions Yiming Du Wenyu Huang Danna Zheng Zhaowei Wang Sébastien Montella Mirella Lapata Kam-Fai Wong Jeff Z. Pan KELM MU 223 5 0 01 May 2025
GPU-Accelerated Motion Planning of an Underactuated Forestry Crane in Cluttered Environments M. Vu Gerald Ebmer Alexander Watcher Marc-Philip Ecker Giang Nguyen Tobias Glueck 125 3 0 18 Mar 2025
FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference Bingzhe Zhao Ke Cheng Aomufei Yuan Yuxuan Tian Ruiguang Zhong Chengchen Hu Tong Yang Lian Yu 113 0 0 19 Feb 2025
Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference Yuan Feng Junlin Lv Yukun Cao Xike Xie S. K. Zhou VLM 135 44 0 28 Jan 2025
SnapKV: LLM Knows What You are Looking for Before Generation Yuhong Li Yingbing Huang Bowen Yang Bharat Venkitesh Acyr Locatelli Hanchen Ye Tianle Cai Patrick Lewis Deming Chen VLM 132 210 0 22 Apr 2024
StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses Jia-Nan Li Quan Tu Cunli Mao Zhengtao Yu Ji-Rong Wen Rui Yan OffRL 76 4 0 13 Mar 2024
A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems Zihao Yi Jiarui Ouyang Yuwen Liu Tianhao Liao Zhe Xu Ying Shen LLMAG LRM 108 72 0 28 Feb 2024
ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition Lu Ye Ze Tao Yong Huang Yang Li 77 33 0 23 Feb 2024
LooGLE: Can Long-Context Language Models Understand Long Contexts? Jiaqi Li Mengmeng Wang Zilong Zheng Muhan Zhang ELM RALM 82 134 0 08 Nov 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu Zhang Ying Sheng Dinesh Manocha Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 151 313 0 24 Jun 2023
A Critical Evaluation of Evaluations for Long-form Question Answering Fangyuan Xu Yixiao Song Mohit Iyyer Eunsol Choi ELM 97 104 0 29 May 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,761 0 15 Mar 2023
Fast Transformer Decoding: One Write-Head is All You Need Noam M. Shazeer 161 476 0 06 Nov 2019
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita David Talbot F. Moiseev Rico Sennrich Ivan Titov 117 1,148 0 23 May 2019
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks Jason Weston Antoine Bordes S. Chopra Alexander M. Rush Bart van Merriënboer Armand Joulin Tomas Mikolov LRM ELM 155 1,182 0 19 Feb 2015