SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths

30 May 2024

Kaixuan Huang

Mengdi Wang

Papers citing "SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths"

21 / 21 papers shown

Title
Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency Ruixiao Li Fahao Chen Peng Li 125 0 0 20 May 2025
Mixture of Attentions For Speculative Decoding Matthieu Zimmer Milan Gritta Gerasimos Lampouras Haitham Bou Ammar Jun Wang 104 4 0 04 Oct 2024
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting Fangcheng Liu Yehui Tang Zhenhua Liu Yunsheng Ni Kai Han Yunhe Wang 87 27 0 29 Apr 2024
SnapKV: LLM Knows What You are Looking for Before Generation Yuhong Li Yingbing Huang Bowen Yang Bharat Venkitesh Acyr Locatelli Hanchen Ye Tianle Cai Patrick Lewis Deming Chen VLM 107 192 0 22 Apr 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 81 61 0 18 Apr 2024
Recurrent Drafter for Fast Speculative Decoding in Large Language Models Aonan Zhang Chong-Jun Wang Yi Wang Xuanyu Zhang Yunfei Cheng 49 17 0 14 Mar 2024
CLLMs: Consistency Large Language Models Siqi Kou Lanxiang Hu Zhe He Zhijie Deng Hao Zhang 74 31 0 28 Feb 2024
Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement Wonseok Jeon Mukul Gagrani Raghavv Goel Junyoung Park Mingu Lee Christopher Lott LRM 37 18 0 21 Feb 2024
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 91 154 0 26 Jan 2024
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Tianle Cai Yuhong Li Zhengyang Geng Hongwu Peng Jason D. Lee De-huai Chen Tri Dao 110 295 0 19 Jan 2024
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding Heming Xia Zhe Yang Qingxiu Dong Peiyi Wang Chak Tou Leong Tao Ge Tianyu Liu Wenjie Li Zhifang Sui LRM 116 121 0 15 Jan 2024
PaSS: Parallel Speculative Sampling Giovanni Monea Armand Joulin Edouard Grave MoE 53 34 0 22 Nov 2023
Accelerating LLM Inference with Staged Speculative Decoding Benjamin Spector Christal Re 63 107 0 08 Aug 2023
Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding Seongjun Yang Gibbeum Lee Jaewoong Cho Dimitris Papailiopoulos Kangwook Lee 66 36 0 12 Jul 2023
Inference with Reference: Lossless Acceleration of Large Language Models Nan Yang Tao Ge Liang Wang Binxing Jiao Daxin Jiang Linjun Yang Rangan Majumder Furu Wei 40 60 0 10 Apr 2023
Learning in POMDPs is Sample-Efficient with Hindsight Observability Jonathan Lee Alekh Agarwal Christoph Dann Tong Zhang 49 20 0 31 Jan 2023
Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan Matan Kalman Yossi Matias LRM 105 701 0 30 Nov 2022
Delayed Feedback in Generalised Linear Bandits Revisited Benjamin Howson Ciara Pike-Burke Sarah Filippi 39 15 0 21 Jul 2022
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 227 4,392 0 27 Oct 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 205 5,513 0 07 Jul 2021
Structured Denoising Diffusion Models in Discrete State-Spaces Jacob Austin Daniel D. Johnson Jonathan Ho Daniel Tarlow Rianne van den Berg DiffM 127 935 0 07 Jul 2021