SDSAT: Accelerating LLM Inference through Speculative Decoding with
Semantic Adaptive Tokens

v1v2 (latest)

SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens

27 March 2024

ArXiv (abs)PDF HTML

Papers citing "SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens"

6 / 6 papers shown

Title
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 123 165 0 26 Jan 2024
Accelerating LLM Inference with Staged Speculative Decoding Benjamin Spector Christal Re 69 112 0 08 Aug 2023
SantaCoder: don't reach for the stars! Loubna Ben Allal Raymond Li Denis Kocetkov Chenghao Mou Christopher Akiki ... Sean M. Hughes Daniel Fried Arjun Guha H. D. Vries Leandro von Werra 166 197 0 09 Jan 2023
Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan Matan Kalman Yossi Matias LRM 147 733 0 30 Nov 2022
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 209 2,004 0 16 Aug 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 236 5,647 0 07 Jul 2021