v1v2 (latest)

A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference

18 October 2024

Papers citing "A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference"

24 / 24 papers shown

Title
Prefill-Based Jailbreak: A Novel Approach of Bypassing LLM Safety Boundary Yakai Li Jiekang Hu Weiduan Sang Luping Ma Jing Xie Weijuan Zhang Aimin Yu Shijie Zhao Qingjia Huang Qihang Zhou AAML 125 1 0 28 Apr 2025
Tensor Product Attention Is All You Need Yifan Zhang Yifeng Liu Huizhuo Yuan Zhen Qin Yang Yuan Q. Gu Andrew Chi-Chih Yao 214 15 0 11 Jan 2025
GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression Daniel Goldstein Fares Obeid Eric Alcaide Guangyu Song Eugene Cheah VLM AI4TS 95 8 0 16 Jul 2024
Beyond KV Caching: Shared Attention for Efficient LLMs Bingli Liao Danilo Vasconcellos Vargas 60 5 0 13 Jul 2024
Skip-Layer Attention: Bridging Abstract and Detailed Dependencies in Transformers Qian Chen Wen Wang Qinglin Zhang Siqi Zheng Shiliang Zhang Chong Deng Hai Yu Jiaqing Liu Yukun Ma Chong Zhang 69 1 0 17 Jun 2024
MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding Zayd Muhammad Kawakibi Zuhri Muhammad Farid Adilazuarda Ayu Purwarianti Alham Fikri Aji 92 10 0 13 Jun 2024
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention William Brandon Mayank Mishra Aniruddha Nrusimha Yikang Shen Jonathan Ragan-Kelley MQ 115 52 0 21 May 2024
Layer-Condensed KV Cache for Efficient Inference of Large Language Models Haoyi Wu Kewei Tu MQ 126 19 0 17 May 2024
You Only Cache Once: Decoder-Decoder Architectures for Language Models Yutao Sun Li Dong Yi Zhu Shaohan Huang Wenhui Wang Shuming Ma Quanlu Zhang Jianyong Wang Furu Wei VLM 97 64 0 08 May 2024
TinyLlama: An Open-Source Small Language Model Peiyuan Zhang Guangtao Zeng Tianduo Wang Wei Lu ALM LRM 174 409 0 04 Jan 2024
Efficient Streaming Language Models with Attention Sinks Michel Lang Yuandong Tian Beidi Chen Song Han Mike Lewis AI4TS RALM 154 791 0 29 Sep 2023
Efficient Memory Management for Large Language Model Serving with PagedAttention Woosuk Kwon Zhuohan Li Siyuan Zhuang Ying Sheng Lianmin Zheng Cody Hao Yu Joseph E. Gonzalez Haotong Zhang Ion Stoica VLM 202 2,338 0 12 Sep 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao LRM 122 1,336 0 17 Jul 2023
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai 118 703 0 22 May 2023
The MiniPile Challenge for Data-Efficient Language Models Jean Kaddour MoE ALM 133 46 0 17 Apr 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 221 404 0 13 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.6K 13,520 0 27 Feb 2023
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 221 1,849 0 26 Nov 2019
Fast Transformer Decoding: One Write-Head is All You Need Noam M. Shazeer 163 478 0 06 Nov 2019
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 266 1,562 0 24 May 2019
HellaSwag: Can a Machine Really Finish Your Sentence? Rowan Zellers Ari Holtzman Yonatan Bisk Ali Farhadi Yejin Choi 194 2,534 0 19 May 2019
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering Todor Mihaylov Peter Clark Tushar Khot Ashish Sabharwal 125 1,571 0 08 Sep 2018
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick Oyvind Tafjord ELM RALM LRM 237 2,675 0 14 Mar 2018
Crowdsourcing Multiple Choice Science Questions Johannes Welbl Nelson F. Liu Matt Gardner AI4Ed 126 522 0 19 Jul 2017