Hardware-Efficient Attention for Fast Decoding

27 May 2025

Papers citing "Hardware-Efficient Attention for Fast Decoding"

43 / 43 papers shown

Title
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference Pol G. Recasens Ferran Agullo Yue Zhu Chen Wang Eun Kyung Lee Olivier Tardieu Jordi Torres Josep Ll. Berral 83 1 0 11 Mar 2025
Seesaw: High-throughput LLM Inference via Model Re-sharding Qidong Su Wei Zhao Xuelong Li Muralidhar Andoorveedu Chenhao Jiang Zhanda Zhu Kevin Song Christina Giannoula Gennady Pekhimenko LRM 125 2 0 09 Mar 2025
Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHA Nils Graef Matthew Clapp 99 2 0 07 Mar 2025
Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs Tao Ji B. Guo Y. Wu Qipeng Guo Lixing Shen Zhan Chen Xipeng Qiu Qi Zhang Tao Gui 81 7 0 21 Feb 2025
Rope to Nope and Back Again: A New Hybrid Attention Strategy Bowen Yang Bharat Venkitesh Dwarak Talupuru Hangyu Lin David Cairuz Phil Blunsom Acyr Locatelli 167 6 0 30 Jan 2025
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI Daya Guo Dejian Yang Haowei Zhang Junxiao Song ... Shiyu Wang S. Yu Shunfeng Zhou Shuting Pan S.S. Li ReLM VLM OffRL AI4TS LRM 380 1,970 0 22 Jan 2025
Tensor Product Attention Is All You Need Yifan Zhang Yifeng Liu Huizhuo Yuan Zhen Qin Yang Yuan Q. Gu Andrew Chi-Chih Yao 157 14 0 11 Jan 2025
Weight decay induces low-rank attention layers Seijin Kobayashi Yassir Akram J. Oswald 90 12 0 31 Oct 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Yiyuan Ma Wenlei Bao Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 133 20 0 28 Oct 2024
Round and Round We Go! What makes Rotary Positional Encodings useful? Federico Barbero Alex Vitvitskyi Christos Perivolaropoulos Razvan Pascanu Petar Velickovic 117 29 0 08 Oct 2024
Eigen Attention: Attention in Low-Rank Space for KV Cache Compression Utkarsh Saxena Gobinda Saha Sakshi Choudhary Kaushik Roy 91 18 0 10 Aug 2024
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell Jaehoon Lee Kelvin Xu Aviral Kumar LRM 192 692 0 06 Aug 2024
Palu: Compressing KV-Cache with Low-Rank Projection Chi-Chih Chang Wei-Cheng Lin Chien-Yu Lin Chong-Yan Chen Yu-Fang Hu Pei-Shuo Wang N. Huang Luis Ceze Kai-Chiang Wu 86 2 0 30 Jul 2024
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding Haoning Wu Dongxu Li Bei Chen Junnan Li 96 163 0 22 Jul 2024
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision Jay Shah Ganesh Bikshandi Ying Zhang Vijay Thakkar Pradeep Ramani Tri Dao 127 156 0 11 Jul 2024
Effectively Compress KV Heads for LLM Hao Yu Zelan Yang Shen Li Yong Li Jianxin Wu MQ VLM 57 16 0 11 Jun 2024
Loki: Low-Rank Keys for Efficient Sparse Attention Prajwal Singhania Siddharth Singh Shwai He Soheil Feizi A. Bhatele 81 22 0 04 Jun 2024
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality Tri Dao Albert Gu Mamba 114 532 0 31 May 2024
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention William Brandon Mayank Mishra Aniruddha Nrusimha Yikang Shen Jonathan Ragan-Kelley MQ 99 52 0 21 May 2024
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI Aixin Liu Bei Feng Bin Wang Bingxuan Wang ... Zhuoshu Li Zihan Wang Zihui Gu Zilin Li Ziwei Xie MoE 117 490 0 07 May 2024
FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines Jiaao He Jidong Zhai 73 34 0 18 Mar 2024
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization Coleman Hooper Sehoon Kim Hiva Mohammadzadeh Michael W. Mahoney Y. Shao Kurt Keutzer A. Gholami MQ 71 223 0 31 Jan 2024
SGLang: Efficient Execution of Structured Language Model Programs Lianmin Zheng Liangsheng Yin Zhiqiang Xie Chuyue Sun Jeff Huang ... Christos Kozyrakis Ion Stoica Joseph E. Gonzalez Clark W. Barrett Ying Sheng LRM 105 168 0 12 Dec 2023
Gated Linear Attention Transformers with Hardware-Efficient Training Aaron Courville Bailin Wang Songlin Yang Yikang Shen Yoon Kim 106 180 0 11 Dec 2023
Efficient Streaming Language Models with Attention Sinks Michel Lang Yuandong Tian Beidi Chen Song Han Mike Lewis AI4TS RALM 130 784 0 29 Sep 2023
Efficient Memory Management for Large Language Model Serving with PagedAttention Woosuk Kwon Zhuohan Li Siyuan Zhuang Ying Sheng Lianmin Zheng Cody Hao Yu Joseph E. Gonzalez Haotong Zhang Ion Stoica VLM 192 2,311 0 12 Sep 2023
Reducing shared memory footprint to leverage high throughput on Tensor Cores and its flexible API extension library Hiroyuki Ootomo Rio Yokota 44 8 0 29 Aug 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao LRM 115 1,326 0 17 Jul 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 190 400 0 13 Mar 2023
Efficiently Scaling Transformer Inference Reiner Pope Sholto Douglas Aakanksha Chowdhery Jacob Devlin James Bradbury Anselm Levskaya Jonathan Heek Kefan Xiao Shivani Agrawal J. Dean 97 325 0 09 Nov 2022
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale Reza Yazdani Aminabadi Samyam Rajbhandari Minjia Zhang A. A. Awan Cheng-rong Li ... Elton Zheng Jeff Rasley Shaden Smith Olatunji Ruwase Yuxiong He 76 369 0 30 Jun 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 252 2,279 0 27 May 2022
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation Heming Xia Tao Ge Peiyi Wang Si-Qing Chen Furu Wei Zhifang Sui 80 90 0 30 Mar 2022
RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su Yu Lu Shengfeng Pan Ahmed Murtadha Bo Wen Yunfeng Liu 284 2,521 0 20 Apr 2021
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan Mohammad Shoeybi Jared Casper P. LeGresley M. Patwary ... Prethvi Kashinkunti J. Bernauer Bryan Catanzaro Amar Phanishayee Matei A. Zaharia MoE 117 699 0 09 Apr 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 458 2,120 0 31 Dec 2020
Measuring Massive Multitask Language Understanding Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Basel Alomair Jacob Steinhardt ELM RALM 184 4,553 0 07 Sep 2020
Data Movement Is All You Need: A Case Study on Optimizing Transformers A. Ivanov Nikoli Dryden Tal Ben-Nun Shigang Li Torsten Hoefler 104 135 0 30 Jun 2020
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 175 1,839 0 26 Nov 2019
Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering Vikas Yadav Steven Bethard Mihai Surdeanu 114 77 0 17 Nov 2019
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering Todor Mihaylov Peter Clark Tushar Khot Ashish Sabharwal 119 1,562 0 08 Sep 2018
Crowdsourcing Multiple Choice Science Questions Johannes Welbl Nelson F. Liu Matt Gardner AI4Ed 101 517 0 19 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 730 132,363 0 12 Jun 2017