MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

23 May 2024

Akide Liu

Jing Liu

Bohan Zhuang

Papers citing "MiniCache: KV Cache Compression in Depth Dimension for Large Language Models"

22 / 22 papers shown

Title
Can LLMs Maintain Fundamental Abilities under KV Cache Compression? Xiang Liu Zhenheng Tang Hong Chen Peijie Dong Zeyu Li Xiuze Zhou Bo Li Xuming Hu Xiaowen Chu 356 5 0 04 Feb 2025
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding Yao Teng Han Shi Xian Liu Xuefei Ning Guohao Dai Yu Wang Zhenguo Li Xihui Liu 77 11 0 02 Oct 2024
ThinK: Thinner Key Cache by Query-Driven Pruning Yuhui Xu Zhanming Jie Hanze Dong Lei Wang Xudong Lu Aojun Zhou Amrita Saha Caiming Xiong Doyen Sahoo 90 19 0 30 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 72 49 0 09 Jul 2024
Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees Sijia Chen Yibo Wang Yi-Feng Wu Qing-Guo Chen Zhao Xu Weihua Luo Kaifu Zhang Lijun Zhang LLMAG LRM 72 14 0 11 Jun 2024
Layer-Condensed KV Cache for Efficient Inference of Large Language Models Haoyi Wu Kewei Tu MQ 58 19 0 17 May 2024
An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models Liang Chen Haozhe Zhao Tianyu Liu Shuai Bai Junyang Lin Chang Zhou Baobao Chang MLLM VLM 82 131 0 11 Mar 2024
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM Hao Kang Qingru Zhang Souvik Kundu Geonhwa Jeong Zaoxing Liu Tushar Krishna Tuo Zhao MQ 122 91 0 08 Mar 2024
Mixtral of Experts Albert Q. Jiang Alexandre Sablayrolles Antoine Roux A. Mensch Blanche Savary ... Théophile Gervet Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LLMAG 88 1,049 0 08 Jan 2024
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models Jing Liu Ruihao Gong Xiuying Wei Zhiwei Dong Jianfei Cai Bohan Zhuang MQ 44 52 0 12 Oct 2023
SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference Luciano Del Corro Allison Del Giorno Sahaj Agarwal Ting Yu Ahmed Hassan Awadallah Subhabrata Mukherjee 63 57 0 05 Jul 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu Zhang Ying Sheng Dinesh Manocha Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 108 275 0 24 Jun 2023
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 268 326 0 11 Sep 2022
Tutel: Adaptive Mixture-of-Experts at Scale Changho Hwang Wei Cui Yifan Xiong Ziyue Yang Ze Liu ... Joe Chau Peng Cheng Fan Yang Mao Yang Y. Xiong MoE 139 115 0 07 Jun 2022
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks R. Entezari Hanie Sedghi O. Saukh Behnam Neyshabur MoMe 63 226 0 12 Oct 2021
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 144 1,548 0 30 Sep 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 170 1,678 0 08 Jun 2020
Up or Down? Adaptive Rounding for Post-Training Quantization Markus Nagel Rana Ali Amjad M. V. Baalen Christos Louizos Tijmen Blankevoort MQ 37 571 0 22 Apr 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 451 4,662 0 23 Jan 2020
Fast Transformer Decoding: One Write-Head is All You Need Noam M. Shazeer 77 452 0 06 Nov 2019
Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization Shashi Narayan Shay B. Cohen Mirella Lapata AILaw 104 1,652 0 27 Aug 2018
CoQA: A Conversational Question Answering Challenge Siva Reddy Danqi Chen Christopher D. Manning RALM HAI 84 1,197 0 21 Aug 2018