Accelerating Large Language Model Decoding with Speculative Sampling

2 February 2023

Charlie Chen

Sebastian Borgeaud

G. Irving

Jean-Baptiste Lespiau

Papers citing "Accelerating Large Language Model Decoding with Speculative Sampling"

50 / 316 papers shown

Title
Tutorial Proposal: Speculative Decoding for Efficient LLM Inference Heming Xia Cunxiao Du Yong Li Qian Liu Wenjie Li 43 0 0 01 Mar 2025
Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff Maximilian Holsman Yukun Huang Bhuwan Dhingra 46 0 0 28 Feb 2025
Long-Context Inference with Retrieval-Augmented Speculative Decoding Guanzheng Chen Qilong Feng Jinjie Ni Xin Li Michael Shieh RALM 57 2 0 27 Feb 2025
Speculative Decoding and Beyond: An In-Depth Survey of Techniques Y. Hu Zining Liu Zhenyuan Dong Tianfan Peng Bradley McDanel Shanghang Zhang 93 0 0 27 Feb 2025
Towards Optimal Multi-draft Speculative Decoding Zhibo Hu Tong Zheng Vignesh Viswanathan Ziyi Chen Ryan Rossi Yihan Wu Dinesh Manocha Heng Huang 47 4 0 26 Feb 2025
From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence Generation up to 100K Tokens Tong Wu Junzhe Shen Zixia Jia Yunhong Wang Zilong Zheng 85 0 0 26 Feb 2025
CodeSwift: Accelerating LLM Inference for Efficient Code Generation Qianhui Zhao L. Zhang Fang Liu Xiaoli Lian Qiaoyuanhe Meng Ziqian Jiao Zetong Zhou Borui Zhang Runlin Guo Jia Li 43 0 0 24 Feb 2025
Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding Tian Jin Ellie Y. Cheng Zack Ankner Nikunj Saunshi Blake M. Elias Amir Yazdanbakhsh Jonathan Ragan-Kelley Suvinay Subramanian Michael Carbin 62 3 0 24 Feb 2025
CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Drafter Yepeng Weng Dianwen Mei Huishi Qiu Xujie Chen Li Liu Jiang Tian Zhongchao Shi 56 0 0 24 Feb 2025
Sequence-level Large Language Model Training with Contrastive Preference Optimization Zhili Feng Dhananjay Ram Cole Hawkins Aditya Rawal Jinman Zhao Sheng Zha 64 0 0 23 Feb 2025
PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System Yintao He Haiyu Mao Christina Giannoula Mohammad Sadrosadati Juan Gómez Luna Huawei Li Xiaowei Li Ying Wang O. Mutlu 45 6 0 21 Feb 2025
C2T: A Classifier-Based Tree Construction Method in Speculative Decoding Feiye Huo Jianchao Tan Kaipeng Zhang Xunliang Cai Shengli Sun 44 0 0 20 Feb 2025
SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models Seanie Lee Dong Bok Lee Dominik Wagner Minki Kang Haebin Seong Tobias Bocklet Juho Lee Sung Ju Hwang 19 1 0 18 Feb 2025
GRIFFIN: Effective Token Alignment for Faster Speculative Decoding Shijing Hu Jingyang Li Xingyu Xie Zhihui Lu Kim-Chuan Toh Pan Zhou 48 0 0 16 Feb 2025
Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization Bowen Pang Kai Li Ruifeng She Feifan Wang OffRL 51 2 0 14 Feb 2025
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding Zhilin Wang Muneeza Azmart Ang Li R. Horesh Mikhail Yurochkin 118 1 0 11 Feb 2025
Auditing Prompt Caching in Language Model APIs Chenchen Gu Xiang Lisa Li Rohith Kuditipudi Percy Liang Tatsunori Hashimoto 78 0 0 11 Feb 2025
Acceleration Multiple Heads Decoding for LLM via Dynamic Tree Attention Zhendong Zhang 66 0 0 09 Feb 2025
Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding Sukmin Cho S. Choi T. Hwang Jeongyeon Seo Soyeong Jeong Huije Lee Hoyun Song Jong C. Park Youngjin Kwon 51 0 0 08 Feb 2025
Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models S. Poddar Paramita Koley Janardan Misra Niloy Ganguly Saptarshi Ghosh Saptarshi Ghosh 64 0 0 08 Feb 2025
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache Rishabh Tiwari Haocheng Xi Aditya Tomar Coleman Hooper Sehoon Kim Maxwell Horton Mahyar Najibi Michael W. Mahoney Kemal Kurniawan Amir Gholami MQ 66 1 0 05 Feb 2025
M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference Nikhil Bhendawade Mahyar Najibi Devang Naik Irina Belousova MoE 85 0 0 04 Feb 2025
Position: AI Scaling: From Up to Down and Out Yunke Wang Yanxi Li Chang Xu HAI 96 2 0 02 Feb 2025
Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment Gregor Bachmann Sotiris Anagnostidis Albert Pumarola Markos Georgopoulos A. Sanakoyeu Yuming Du Edgar Schönfeld Ali K. Thabet Jonas Kohler ALM BDL 114 8 0 31 Jan 2025
Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies Nadav Timor Jonathan Mamou Daniel Korat Moshe Berchansky Oren Pereg Gaurav Jain Roy Schwartz Moshe Wasserblat David Harel 98 2 0 31 Jan 2025
AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding Zikun Li Zhuofu Chen Remi Delacourt Gabriele Oliaro Zeyu Wang ... Zhuoming Chen Sean Lai Xinhao Cheng Xupeng Miao Zhihao Jia 53 6 0 21 Jan 2025
Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models Hao Li C. Bezemer Ahmed E. Hassan 45 2 0 08 Jan 2025
Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels Mingcong Song Xinru Tang Fengfan Hou Jing Li Wei Wei ... Hongjie Si Dengyang Jiang Shouyi Yin Yang Hu Guoping Long 38 1 0 24 Dec 2024
Parallelized Autoregressive Visual Generation Yunhong Wang Shuhuai Ren Zhijie Lin Yujin Han Haoyuan Guo Zhenheng Yang Difan Zou Jiashi Feng Xihui Liu VGen 95 12 0 19 Dec 2024
Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree Xiangxiang Gao Weisheng Xie Yiwei Xiang Feng Ji 91 6 0 17 Dec 2024
Constrained Decoding with Speculative Lookaheads Nishanth Nakshatri Shamik Roy Rajarshi Das Suthee Chaidaroon Leonid Boytsov Rashmi Gangadharaiah 84 0 0 09 Dec 2024
PLD+: Accelerating LLM inference by leveraging Language Model Artifacts Shwetha Somasundaram Anirudh Phukan Apoorv Saxena 85 1 0 02 Dec 2024
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration Zhuofan Wen Shangtong Gui Yang Feng 100 3 0 25 Nov 2024
Debiasing Watermarks for Large Language Models via Maximal Coupling Yangxinyu Xie Xiang Li Tanwi Mallick Weijie J. Su Ruixun Zhang WaLM 55 2 0 17 Nov 2024
SSSD: Simply-Scalable Speculative Decoding Michele Marzollo Jiawei Zhuang Niklas Roemer Lorenz K. Müller Lukas Cavigelli LRM 47 2 0 08 Nov 2024
SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding Ryan Sun Tianyi Zhou Xun Chen Lichao Sun 34 5 0 08 Nov 2024
SuffixDecoding: A Model-Free Approach to Speeding Up Large Language Model Inference Gabriele Oliaro Zhihao Jia Daniel F Campos Aurick Qiao LRM 44 4 0 07 Nov 2024
The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation Lawrence Stewart Matthew Trager Sujan Kumar Gonugondla Stefano Soatto 58 5 0 06 Nov 2024
Privacy Risks of Speculative Decoding in Large Language Models Jiankun Wei Abdulrahman Abdulrazzag Tianchen Zhang Adel Muursepp Gururaj Saileshwar 37 2 0 01 Nov 2024
Interpretable Language Modeling via Induction-head Ngram Models Eunji Kim Sriya Mantena Weiwei Yang Chandan Singh Sungroh Yoon Jianfeng Gao 67 0 0 31 Oct 2024
Accelerated AI Inference via Dynamic Execution Methods Haim Barad Jascha Achterberg Tien Pei Chou Jean Yu 31 0 0 30 Oct 2024
A Theoretical Perspective for Speculative Decoding Algorithm Ming Yin Minshuo Chen Kaixuan Huang Mengdi Wang 32 5 0 30 Oct 2024
Transferable Post-training via Inverse Value Learning Xinyu Lu Xueru Wen Yaojie Lu Bowen Yu Hongyu Lin Haiyang Yu Le Sun Xianpei Han Yongbin Li 25 1 0 28 Oct 2024
FIRP: Faster LLM inference via future intermediate representation prediction Pengfei Wu Jiahao Liu Zhuocheng Gong Qifan Wang Jinpeng Li Jingang Wang Xunliang Cai Dongyan Zhao AI4CE 29 0 0 27 Oct 2024
Fast Best-of-N Decoding via Speculative Rejection Hanshi Sun Momin Haider Ruiqi Zhang Huitao Yang Jiahao Qiu Ming Yin Mengdi Wang Peter L. Bartlett Andrea Zanette BDL 45 28 0 26 Oct 2024
Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning Yu Fu Zefan Cai Abedelkadir Asi Wayne Xiong Yue Dong Wen Xiao 49 17 0 25 Oct 2024
Watermarking Large Language Models and the Generated Content: Opportunities and Challenges Ruisi Zhang F. Koushanfar WaLM 47 0 0 24 Oct 2024
Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits Ashish Khisti MohammadReza Ebrahimi Hassan Dbouk Arash Behboodi Roland Memisevic Christos Louizos 38 0 0 23 Oct 2024
Scaling Diffusion Language Models via Adaptation from Autoregressive Models Shansan Gong Shivam Agarwal Yizhe Zhang Jiacheng Ye Lin Zheng ... Peilin Zhao W. Bi Jiawei Han Hao Peng Lingpeng Kong AI4CE 78 17 0 23 Oct 2024
Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition Artem Basharin Andrei Chertkov Ivan Oseledets 53 1 0 23 Oct 2024