DistillSpec: Improving Speculative Decoding via Knowledge Distillation

DistillSpec: Improving Speculative Decoding via Knowledge Distillation

12 October 2023

Afshin Rostamizadeh

Sanjiv Kumar

Jean-François Kagy

Rishabh Agarwal

Papers citing "DistillSpec: Improving Speculative Decoding via Knowledge Distillation"

18 / 18 papers shown

Title
Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission Seungeun Oh Jinhyuk Kim Jihong Park Seung-Woo Ko Jinho Choi Tony Q. S. Quek Seong-Lyun Kim 9 0 0 17 May 2025
Scaling Laws for Speculative Decoding Siyuan Yan Mo Zhu Guo-qing Jiang Jianfei Wang Jiaxing Chen ... Xiang Liao Xiao Cui Chen Zhang Zhuoran Song Ran Zhu LRM 48 0 0 08 May 2025
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding Zihan Wang Muneeza Azmart Ang Li R. Horesh Mikhail Yurochkin 118 1 0 11 Feb 2025
Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment Gregor Bachmann Sotiris Anagnostidis Albert Pumarola Markos Georgopoulos A. Sanakoyeu Yuming Du Edgar Schönfeld Ali K. Thabet Jonas Kohler ALM BDL 106 7 0 31 Jan 2025
Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models Seungeun Oh Jinhyuk Kim Jihong Park Seung-Woo Ko Tony Q. S. Quek Seong-Lyun Kim 77 5 0 17 Dec 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 56 6 0 09 Oct 2024
Efficient Inference for Large Language Model-based Generative Recommendation Xinyu Lin Chaoqun Yang Wenjie Wang Yongqi Li Cunxiao Du Fuli Feng See-Kiong Ng Tat-Seng Chua 70 4 0 07 Oct 2024
Mixture of Attentions For Speculative Decoding Matthieu Zimmer Milan Gritta Gerasimos Lampouras Haitham Bou Ammar Jun Wang 76 4 0 04 Oct 2024
LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding Doohyuk Jang Sihwan Park J. Yang Yeonsung Jung Jihun Yun Souvik Kundu Sung-Yub Kim Eunho Yang 49 7 0 04 Oct 2024
PEARL: Parallel Speculative Decoding with Adaptive Draft Length Tianyu Liu Yun Li Qitan Lv Kai Liu Jianchen Zhu Winston Hu Xingchen Sun 59 14 0 13 Aug 2024
Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding Zack Ankner Rishab Parthasarathy Aniruddha Nrusimha Christopher Rinard Jonathan Ragan-Kelley William Brandon 32 25 0 07 Feb 2024
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 52 123 0 26 Jan 2024
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Tianle Cai Yuhong Li Zhengyang Geng Hongwu Peng Jason D. Lee De-huai Chen Tri Dao 52 252 0 19 Jan 2024
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding Heming Xia Zhe Yang Qingxiu Dong Peiyi Wang Yongqi Li Tao Ge Tianyu Liu Wenjie Li Zhifang Sui LRM 38 101 0 15 Jan 2024
Online Speculative Decoding Xiaoxuan Liu Lanxiang Hu Peter Bailis Alvin Cheung Zhijie Deng Ion Stoica Hao Zhang 29 51 0 11 Oct 2023
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding Tao Ge Heming Xia Xin Sun Si-Qing Chen Furu Wei 85 18 0 20 May 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 413 8,559 0 28 Jan 2022
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 211 3,513 0 10 Jun 2015