ThinK: Thinner Key Cache by Query-Driven Pruning

ThinK: Thinner Key Cache by Query-Driven Pruning

30 July 2024

Papers citing "ThinK: Thinner Key Cache by Query-Driven Pruning"

15 / 15 papers shown

Title
Scalable Chain of Thoughts via Elastic Reasoning Yuhui Xu Hanze Dong Lei Wang Doyen Sahoo Junnan Li Caiming Xiong OffRL LRM 51 2 0 08 May 2025
KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments Junyoung Park Dalton Jones Matt Morse Raghavv Goel Mingu Lee Chris Lott 24 0 0 21 Apr 2025
Cognitive Memory in Large Language Models Lianlei Shan Shixian Luo Zezhou Zhu Yu Yuan Yong Wu LLMAG KELM 160 1 0 03 Apr 2025
Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding Sakhinana Sagar Srinivas Venkataramana Runkana OffRL 47 1 0 02 Apr 2025
SQuat: Subspace-orthogonal KV Cache Quantization Hao Wang Ligong Han Kai Xu Akash Srivastava MQ 51 0 0 31 Mar 2025
PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference Weisheng Jin Maojia Song Tej Deep Pala Yew Ken Chia Amir Zadeh Chuan Li Soujanya Poria VLM 57 0 0 30 Mar 2025
CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences Ziran Qin Yuchen Cao Mingbao Lin Wen Hu Shixuan Fan Ke Cheng Weiyao Lin Jianguo Li 71 3 0 16 Mar 2025
Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoning Giulio Corallo Orion Weller Fabio Petroni Paolo Papotti MQ VLM 54 0 0 06 Mar 2025
SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention Hong Yankun Li Xing Zhen Hui-Ling Yu Xianzhi Liu Wulong Yuan Mingxuan MQ 80 0 0 24 Feb 2025
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Yiyuan Ma Wenlei Bao Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 88 16 0 28 Oct 2024
KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing Yifei Yang Zouying Cao Qiguang Chen L. Qin Dongjie Yang Hai Zhao Zhi Chen 28 5 0 24 Oct 2024
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction Zhenmei Shi Yifei Ming Xuan-Phi Nguyen Yingyu Liang Shafiq Joty 78 27 0 25 Sep 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 33 0 25 Jul 2024
What is the State of Neural Network Pruning? Davis W. Blalock Jose Javier Gonzalez Ortiz Jonathan Frankle John Guttag 191 1,027 0 06 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 246 4,489 0 23 Jan 2020