Optimizing Speculative Decoding for Serving Large Language Models Using
Goodput

Optimizing Speculative Decoding for Serving Large Language Models Using Goodput

20 June 2024

Alvin Cheung

Ion Stoica

Hao Zhang

Papers citing "Optimizing Speculative Decoding for Serving Large Language Models Using Goodput"

18 / 18 papers shown

Title
BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms Yunlong Hou Fengzhuo Zhang Cunxiao Du Xuan Zhang Jiachun Pan Tianyu Pang Chao Du Vincent Y. F. Tan Zhuoran Yang OffRL 12 0 0 21 May 2025
Energy Considerations of Large Language Model Inference and Efficiency Optimizations Jared Fernandez Clara Na Vashisth Tiwari Yonatan Bisk Sasha Luccioni Emma Strubell 52 1 0 24 Apr 2025
ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts E. Georganas Dhiraj D. Kalamkar Alexander Kozlov A. Heinecke MQ 216 0 0 17 Mar 2025
Collaborative Speculative Inference for Efficient LLM Inference Serving Luyao Gao Jianchun Liu Hongli Xu Xichong Zhang Yunming Liao Liusheng Huang 46 0 0 13 Mar 2025
SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding Kaiyu Huang Yu Wang Zhubo Shi Han Zou Minchen Yu Qingjiang Shi LRM 49 2 0 07 Mar 2025
From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence Generation up to 100K Tokens Tong Wu Junzhe Shen Zixia Jia Yunhong Wang Zilong Zheng 85 0 0 26 Feb 2025
TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding Zhaoxuan Wu Zijian Zhou Arun Verma Alok Prakash Daniela Rus Bryan Kian Hsiang Low 60 0 0 24 Feb 2025
AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding Zikun Li Zhuofu Chen Remi Delacourt Gabriele Oliaro Zeyu Wang ... Zhuoming Chen Sean Lai Xinhao Cheng Xupeng Miao Zhihao Jia 53 6 0 21 Jan 2025
Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels Mingcong Song Xinru Tang Fengfan Hou Jing Li Wei Wei ... Hongjie Si Dengyang Jiang Shouyi Yin Yang Hu Guoping Long 38 1 0 24 Dec 2024
Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding Hyun Ryu Eric Kim 80 3 0 20 Nov 2024
SSSD: Simply-Scalable Speculative Decoding Michele Marzollo Jiawei Zhuang Niklas Roemer Lorenz K. Müller Lukas Cavigelli LRM 47 2 0 08 Nov 2024
Dynamic Depth Decoding: Faster Speculative Decoding for LLMs Oscar Brown Zhengjie Wang Andrea Do Nikhil Mathew Cheng Yu 34 4 0 30 Aug 2024
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding Jian Chen Vashisth Tiwari Ranajoy Sadhukhan Zhuoming Chen Jinyuan Shi Ian En-Hsu Yen Ian En-Hsu Yen Avner May Tianqi Chen Beidi Chen LRM 41 22 0 20 Aug 2024
LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism Bingya Wu Shengyu Liu Yinmin Zhong Peng Sun Xuanzhe Liu Xin Jin RALM 46 53 0 15 Apr 2024
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding Weilin Zhao Yuxiang Huang Xu Han Wang Xu Chaojun Xiao Xinrong Zhang Yewei Fang Kaihuo Zhang Zhiyuan Liu Maosong Sun 43 11 0 21 Feb 2024
Hydragen: High-Throughput LLM Inference with Shared Prefixes Jordan Juravsky Bradley Brown Ryan Ehrlich Daniel Y. Fu Christopher Ré Azalia Mirhoseini 66 36 0 07 Feb 2024
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding Yichao Fu Peter Bailis Ion Stoica Hao Zhang 133 145 0 03 Feb 2024
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 211 3,517 0 10 Jun 2015