Fast Transformer Decoding: One Write-Head is All You Need

6 November 2019

Papers citing "Fast Transformer Decoding: One Write-Head is All You Need"

50 / 109 papers shown

Title
A3 : an Analytical Low-Rank Approximation Framework for Attention Jeffrey T. H. Wong Cheng Zhang Xinye Cao Pedro Gimenes George A. Constantinides Wayne Luk Yiren Zhao OffRL MQ 7 0 0 19 May 2025
Accurate KV Cache Quantization with Outlier Tokens Tracing Yi Su Yuechi Zhou Quantong Qiu Juntao Li Qingrong Xia Ping Li Xinyu Duan Zhefeng Wang Min Zhang MQ 22 0 0 16 May 2025
Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures Chenggang Zhao Chengqi Deng Chong Ruan Damai Dai Huazuo Gao ... Wenfeng Liang Ying He Yishuo Wang Yuxuan Liu Y. X. Wei MoE 41 0 0 14 May 2025
EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices Arnab Sanyal Prithwish Mukherjee Gourav Datta Sandeep P. Chinchali MQ 207 0 0 05 May 2025
Small Clips, Big Gains: Learning Long-Range Refocused Temporal Information for Video Super-Resolution Xingyu Zhou Wei Long Jingbo Lu Shiyin Jiang Weiyi You Haifeng Wu Shuhang Gu 48 0 0 04 May 2025
Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing Piotr Piekos Róbert Csordás Jürgen Schmidhuber MoE VLM 106 1 0 01 May 2025
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate A. Zandieh Majid Daliri Majid Hadian Vahab Mirrokni MQ 74 0 0 28 Apr 2025
How Effective are Generative Large Language Models in Performing Requirements Classification? Waad Alhoshan Alessio Ferrari Liping Zhao 30 0 0 23 Apr 2025
Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management Hang Zhang Jiuchen Shi Yixiao Wang Quan Chen Yizhou Shan Minyi Guo 38 0 0 19 Apr 2025
Using Attention Sinks to Identify and Evaluate Dormant Heads in Pretrained LLMs Pedro Sandoval-Segura Xijun Wang Ashwinee Panda Micah Goldblum Ronen Basri Tom Goldstein David Jacobs 27 0 0 04 Apr 2025
Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization Minsu Kim Seongmin Hong RyeoWook Ko S. Choi Hunjong Lee Junsoo Kim Joo-Young Kim Jongse Park 59 0 0 24 Mar 2025
WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference Youhui Zuo Sibo Wei C. Zhang Zhuorui Liu Wenpeng Lu Dawei Song VLM 63 0 0 23 Mar 2025
GPU-Accelerated Motion Planning of an Underactuated Forestry Crane in Cluttered Environments M. Vu Gerald Ebmer Alexander Watcher Marc-Philip Ecker Giang Nguyen Tobias Glueck 77 0 0 18 Mar 2025
X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression Guihong Li Mehdi Rezagholizadeh Mingyu Yang Vikram Appia Emad Barsoum VLM 62 0 0 14 Mar 2025
Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques Neusha Javidnia B. Rouhani F. Koushanfar 223 0 0 14 Mar 2025
Cost-Optimal Grouped-Query Attention for Long-Context Modeling Yuxiao Chen Yutong Wu Chenyang Song Zhiyuan Liu Maosong Sun Xu Han Zhiyuan Liu Maosong Sun 73 0 0 12 Mar 2025
FANformer: Improving Large Language Models Through Effective Periodicity Modeling Yihong Dong Ge Li Xue Jiang Yongding Tao Kechi Zhang ... Huanyu Liu Jiazheng Ding Jia Li Jinliang Deng Hong Mei AI4TS 46 0 0 28 Feb 2025
Learning Humanoid Locomotion with World Model Reconstruction Wandong Sun L. Chen Yongbo Su Baoshi Cao Yang Liu Zongwu Xie 63 1 0 22 Feb 2025
C2T: A Classifier-Based Tree Construction Method in Speculative Decoding Feiye Huo Jianchao Tan Kaipeng Zhang Xunliang Cai Shengli Sun 44 0 0 20 Feb 2025
Efficient Long-Decoding Inference with Reasoning-Aware Attention Sparsity Junhao Hu Wenrui Huang Weidong Wang Zhenwen Li Tiancheng Hu Zhixia Liu Xusheng Chen Tao Xie Yizhou Shan LRM 53 0 0 16 Feb 2025
Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge Daniel Tamayo Aitor Gonzalez-Agirre Javier Hernando Marta Villegas KELM 93 4 0 04 Feb 2025
Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment Gregor Bachmann Sotiris Anagnostidis Albert Pumarola Markos Georgopoulos A. Sanakoyeu Yuming Du Edgar Schönfeld Ali K. Thabet Jonas Kohler ALM BDL 114 8 0 31 Jan 2025
Tensor Product Attention Is All You Need Yifan Zhang Yifeng Liu Huizhuo Yuan Zhen Qin Yang Yuan Q. Gu Andrew Chi-Chih Yao 96 9 0 11 Jan 2025
Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions Doaa Mahmud Hadeel Hajmohamed Shamma Almentheri Shamma Alqaydi Lameya Aldhaheri R. A. Khalil Nasir Saeed AI4TS 51 5 0 08 Jan 2025
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving Zihao Ye Lequn Chen Ruihang Lai Wuwei Lin Yineng Zhang ... Tianqi Chen Baris Kasikci Vinod Grover Arvind Krishnamurthy Luis Ceze 67 21 0 02 Jan 2025
Unifying KV Cache Compression for Large Language Models with LeanKV Yanqi Zhang Yuwei Hu Runyuan Zhao John C. S. Lui Haibo Chen MQ 151 5 0 04 Dec 2024
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Akhiad Bercovich Tomer Ronen Talor Abramovich Nir Ailon Nave Assaf ... Ido Shahaf Oren Tropp Omer Ullman Argov Ran Zilberstein Ran El-Yaniv 91 1 0 28 Nov 2024
Context Parallelism for Scalable Million-Token Inference Amy Yang Jingyi Yang Aya Ibrahim Xinfeng Xie Bangsheng Tang Grigory Sizov Jeremy Reizenstein Jongsoo Park Jianyu Huang MoE LRM 72 5 0 04 Nov 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 84 5 0 28 Oct 2024
A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference You Wu Haoyi Wu Kewei Tu 34 3 0 18 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 224 0 0 17 Oct 2024
MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection Bokai Lin Zihao Zeng Zipeng Xiao Siqi Kou Tianqi Hou Xiaofeng Gao Hao Zhang Zhijie Deng 23 2 0 16 Oct 2024
In-context KV-Cache Eviction for LLMs via Attention-Gate Zihao Zeng Bokai Lin Tianqi Hou Hao Zhang Zhijie Deng 38 1 0 15 Oct 2024
Self-Data Distillation for Recovering Quality in Pruned Large Language Models Vithursan Thangarasa Ganesh Venkatesh Mike Lasby Nish Sinnadurai Sean Lie SyDa 38 1 0 13 Oct 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 59 6 0 09 Oct 2024
Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions Zhihao He Hang Yu Zi Gong Shizhan Liu Jia-Nan Li Weiyao Lin VLM 38 1 0 09 Oct 2024
Selective Attention Improves Transformer Yaniv Leviathan Matan Kalman Yossi Matias 51 9 0 03 Oct 2024
Denoising with a Joint-Embedding Predictive Architecture Dengsheng Chen Jie Hu Xiaoming Wei Enhua Wu DiffM 52 2 0 02 Oct 2024
End-to-end Semantic-centric Video-based Multimodal Affective Computing Ronghao Lin Ying Zeng Sijie Mai Haifeng Hu VGen 48 0 0 14 Aug 2024
RazorAttention: Efficient KV Cache Compression Through Retrieval Heads Hanlin Tang Yang Lin Jing Lin Qingsen Han Shikuan Hong Yiwu Yao Gongyi Wang MQ 44 27 0 22 Jul 2024
PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation Branden Butler Sixing Yu Arya Mazaheri Ali Jannesari LRM 48 6 0 16 Jul 2024
Weighted Grouped Query Attention in Transformers Sai Sena Chinnakonduru Astarag Mohapatra 44 5 0 15 Jul 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 48 18 0 01 Jul 2024
A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems Florin Cuconasu Giovanni Trappolini Nicola Tonellotto Fabrizio Silvestri 55 2 0 21 Jun 2024
Autoregressive Image Generation without Vector Quantization Tianhong Li Yonglong Tian He Li Mingyang Deng Kaiming He DiffM 62 184 0 17 Jun 2024
Investigating the translation capabilities of Large Language Models trained on parallel data only Javier García Gilabert Carlos Escolano Aleix Sant Savall Francesca de Luca Fornaciari Audrey Mash Xixian Liao Maite Melero LRM 42 2 0 13 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 77 57 0 11 Jun 2024
DEPTH: Discourse Education through Pre-Training Hierarchically Zachary Bamberger Ofek Glick Chaim Baskin Yonatan Belinkov 67 0 0 13 May 2024
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention Ramya Prabhu Ajay Nayak Jayashree Mohan Ramachandran Ramjee Ashish Panwar VLM 68 26 0 07 May 2024
Granite Code Models: A Family of Open Foundation Models for Code Intelligence Mayank Mishra Matt Stallone Gaoyuan Zhang Songlin Yang Aditya Prasad ... Amith Singhee Nirmit Desai David D. Cox Ruchir Puri Yikang Shen AI4TS 63 58 0 07 May 2024