Fast Transformer Decoding: One Write-Head is All You Need

6 November 2019

Papers citing "Fast Transformer Decoding: One Write-Head is All You Need"

50 / 109 papers shown

Title
Efficient LLM Inference with Kcache Qiaozhi He Zhihua Wu RALM 38 1 0 28 Apr 2024
Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges Badri N. Patro Vijay Srinivas Agneeswaran Mamba 48 38 0 24 Apr 2024
CORM: Cache Optimization with Recent Message for Large Language Model Inference Jincheng Dai Zhuowei Huang Haiyun Jiang Chen Chen Deng Cai Wei Bi Shuming Shi 38 3 0 24 Apr 2024
SPARO: Selective Attention for Robust and Compositional Transformer Encodings for Vision Ankit Vani Bac Nguyen Samuel Lavoie Ranjay Krishna Aaron Courville 39 1 0 24 Apr 2024
Towards smaller, faster decoder-only transformers: Architectural variants and their implications Sathya Krishnan Suresh P. Shunmugapriya 24 0 0 22 Apr 2024
Transformer tricks: Removing weights for skipless transformers Nils Graef 43 2 0 18 Apr 2024
Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies Benjue Weng LM&MA 48 8 0 13 Apr 2024
On Speculative Decoding for Multimodal Large Language Models Mukul Gagrani Raghavv Goel Wonseok Jeon Junyoung Park Mingu Lee Christopher Lott LRM 40 8 0 13 Apr 2024
IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe Biomedical Natural Language Inference for Clinical Trials Shreyasi Mandal Ashutosh Modi LRM ELM 36 3 0 06 Apr 2024
Towards Pareto Optimal Throughput in Small Language Model Serving Pol G. Recasens Yue Zhu Chen Wang Eun Kyung Lee Olivier Tardieu Alaa Youssef Jordi Torres Josep Ll. Berral 40 4 0 04 Apr 2024
DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference Jinwei Yao Kaiqi Chen Kexun Zhang Jiaxuan You Binhang Yuan Zeke Wang Tao Lin 48 2 0 30 Mar 2024
Yi: Open Foundation Models by 01.AI 01. AI Alex Young 01.AI Alex Young Bei Chen Chao Li ... Yue Wang Yuxuan Cai Zhenyu Gu Zhiyuan Liu Zonghong Dai OSLM LRM 150 511 0 07 Mar 2024
CLLMs: Consistency Large Language Models Siqi Kou Lanxiang Hu Zhe He Zhijie Deng Hao Zhang 52 28 0 28 Feb 2024
EAN-MapNet: Efficient Vectorized HD Map Construction with Anchor Neighborhoods Huiyuan Xiong Jun Shen Taohong Zhu Yuelong Pan 30 3 0 28 Feb 2024
A Comprehensive Evaluation of Quantization Strategies for Large Language Models Renren Jin Jiangcun Du Wuwei Huang Wei Liu Jian Luan Bin Wang Deyi Xiong MQ 32 31 0 26 Feb 2024
Data-free Weight Compress and Denoise for Large Language Models Runyu Peng Yunhua Zhou Qipeng Guo Yang Gao Hang Yan Xipeng Qiu Dahua Lin 47 1 0 26 Feb 2024
On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference Siyu Ren Kenny Q. Zhu 26 27 0 09 Feb 2024
Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding Zack Ankner Rishab Parthasarathy Aniruddha Nrusimha Christopher Rinard Jonathan Ragan-Kelley William Brandon 34 26 0 07 Feb 2024
Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awareness Samaneh Shafee A. Bessani Pedro M. Ferreira 31 19 0 26 Jan 2024
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 52 126 0 26 Jan 2024
BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models Feng-Huei Lin Hanling Yi Hongbin Li Yifan Yang Xiaotian Yu Guangming Lu Rong Xiao 41 3 0 23 Jan 2024
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Tianle Cai Yuhong Li Zhengyang Geng Hongwu Peng Jason D. Lee De-huai Chen Tri Dao 60 257 0 19 Jan 2024
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding Heming Xia Zhe Yang Qingxiu Dong Peiyi Wang Yongqi Li Tao Ge Tianyu Liu Wenjie Li Zhifang Sui LRM 38 105 0 15 Jan 2024
Zebra: Extending Context Window with Layerwise Grouped Local-Global Attention Kaiqiang Song Xiaoyang Wang Sangwoo Cho Xiaoman Pan Dong Yu 36 7 0 14 Dec 2023
PaSS: Parallel Speculative Sampling Giovanni Monea Armand Joulin Edouard Grave MoE 24 32 0 22 Nov 2023
GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values Farnoosh Javadi Walid Ahmed Habib Hajimolahoseini Foozhan Ataiefard Mohammad Hassanpour Saina Asani Austin Wen Omar Mohamed Awad Kangling Liu Yang Liu VLM 42 7 0 06 Nov 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 36 15 0 28 Sep 2023
KERMIT: Knowledge Graph Completion of Enhanced Relation Modeling with Inverse Transformation Haotian Li Lingzhi Wang Yuliang Wei Richard Y. D. Xu Bailing Wang Bailing Wang 56 2 0 26 Sep 2023
RecycleGPT: An Autoregressive Language Model with Recyclable Module Yu Jiang Qiaozhi He Xiaomin Zhuang Zhihua Wu Kunpeng Wang Wenlai Zhao Guangwen Yang KELM 28 3 0 07 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 132 11,144 0 18 Jul 2023
Retentive Network: A Successor to Transformer for Large Language Models Yutao Sun Li Dong Shaohan Huang Shuming Ma Yuqing Xia Jilong Xue Jianyong Wang Furu Wei LRM 78 304 0 17 Jul 2023
Leveraging Cross-Utterance Context For ASR Decoding Robert Flynn Anton Ragni 33 1 0 29 Jun 2023
$S$^{3}$: Increasing GPU Utilization during Generative Inference for Higher Throughput$ S $^{3}$ : Increasing GPU Utilization during Generative Inference for Higher Throughput Yunho Jin Chun-Feng Wu David Brooks Gu-Yeon Wei 39 62 0 09 Jun 2023
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time Zichang Liu Aditya Desai Fangshuo Liao Weitao Wang Victor Xie Zhaozhuo Xu Anastasios Kyrillidis Anshumali Shrivastava 33 204 0 26 May 2023
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers Sotiris Anagnostidis Dario Pavllo Luca Biggio Lorenzo Noci Aurelien Lucchi Thomas Hofmann 42 53 0 25 May 2023
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference Tao Lei Junwen Bai Siddhartha Brahma Joshua Ainslie Kenton Lee ... Vincent Zhao Yuexin Wu Bo-wen Li Yu Zhang Ming-Wei Chang BDL AI4CE 30 55 0 11 Apr 2023
Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASR Rami Botros Anmol Gulati Tara N. Sainath K. Choromanski Ruoming Pang Trevor Strohman Weiran Wang Jiahui Yu MQ 28 3 0 31 Mar 2023
Unlocking the Potential of ChatGPT: A Comprehensive Exploration of its Applications, Advantages, Limitations, and Future Directions in Natural Language Processing Walid Hariri AI4MH LM&MA 33 85 0 27 Mar 2023
CoLT5: Faster Long-Range Transformers with Conditional Computation Joshua Ainslie Tao Lei Michiel de Jong Santiago Ontañón Siddhartha Brahma ... Mandy Guo James Lee-Thorp Yi Tay Yun-hsuan Sung Sumit Sanghai LLMAG 39 63 0 17 Mar 2023
Accelerating Large Language Model Decoding with Speculative Sampling Charlie Chen Sebastian Borgeaud G. Irving Jean-Baptiste Lespiau Laurent Sifre J. Jumper BDL LRM 13 384 0 02 Feb 2023
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference Michiel de Jong Yury Zemlyanskiy Joshua Ainslie Nicholas FitzGerald Sumit Sanghai Fei Sha William W. Cohen VLM 23 32 0 15 Dec 2022
Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan Matan Kalman Yossi Matias LRM 49 636 0 30 Nov 2022
Efficiently Scaling Transformer Inference Reiner Pope Sholto Douglas Aakanksha Chowdhery Jacob Devlin James Bradbury Anselm Levskaya Jonathan Heek Kefan Xiao Shivani Agrawal J. Dean 48 297 0 09 Nov 2022
Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models Hao Liu Xinyang Geng Lisa Lee Igor Mordatch Sergey Levine Sharan Narang Pieter Abbeel KELM CLL 35 2 0 24 Oct 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 136 6,035 0 05 Apr 2022
Transframer: Arbitrary Frame Prediction with Generative Models C. Nash João Carreira Jacob Walker Iain Barr Andrew Jaegle Mateusz Malinowski Peter W. Battaglia ViT 27 37 0 17 Mar 2022
Enabling arbitrary translation objectives with Adaptive Tree Search Wang Ling Wojciech Stokowiec Domenic Donato Laurent Sartran Lei Yu Austin Matthews Chris Dyer 16 0 0 23 Feb 2022
ST-MoE: Designing Stable and Transferable Sparse Expert Models Barret Zoph Irwan Bello Sameer Kumar Nan Du Yanping Huang J. Dean Noam M. Shazeer W. Fedus MoE 24 183 0 17 Feb 2022
General-purpose, long-context autoregressive modeling with Perceiver AR Curtis Hawthorne Andrew Jaegle Cătălina Cangea Sebastian Borgeaud C. Nash ... Hannah R. Sheahan Neil Zeghidour Jean-Baptiste Alayrac João Carreira Jesse Engel 43 65 0 15 Feb 2022
Competition-Level Code Generation with AlphaCode Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser ... Esme Sutherland Robson Pushmeet Kohli Nando de Koray Kavukcuoglu Oriol Vinyals 26 1,313 0 08 Feb 2022