Speculative Streaming: Fast LLM Inference without Auxiliary Models

Speculative Streaming: Fast LLM Inference without Auxiliary Models

16 February 2024

Nikhil Bhendawade

Irina Belousova

Mohammad Rastegari

ArXiv (abs)PDF HTML

Papers citing "Speculative Streaming: Fast LLM Inference without Auxiliary Models"

14 / 14 papers shown

Title
Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency Ruixiao Li Fahao Chen Peng Li 153 0 0 20 May 2025
Collaborative Speculative Inference for Efficient LLM Inference Serving Luyao Gao Jianchun Liu Hongli Xu Xichong Zhang Yunming Liao Liusheng Huang 88 1 0 13 Mar 2025
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache Rishabh Tiwari Haocheng Xi Aditya Tomar Coleman Hooper Sehoon Kim Maxwell Horton Mahyar Najibi Michael W. Mahoney Kemal Kurniawan Amir Gholami MQ 95 4 0 05 Feb 2025
Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition Artem Basharin Andrei Chertkov Ivan Oseledets 124 1 0 23 Oct 2024
Accelerating LLM Inference with Staged Speculative Decoding Benjamin Spector Christal Re 69 112 0 08 Aug 2023
Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan Matan Kalman Yossi Matias LRM 147 733 0 30 Nov 2022
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers Z. Yao Reza Yazdani Aminabadi Minjia Zhang Xiaoxia Wu Conglong Li Yuxiong He VLM MQ 127 479 0 04 Jun 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 522 6,293 0 05 Apr 2022
LaMDA: Language Models for Dialog Applications R. Thoppilan Daniel De Freitas Jamie Hall Noam M. Shazeer Apoorv Kulshreshtha ... Blaise Aguera-Arcas Claire Cui M. Croak Ed H. Chi Quoc Le ALM 140 1,601 0 20 Jan 2022
DialogSum: A Real-Life Scenario Dialogue Summarization Dataset Yulong Chen Yang Liu Liang Chen Yue Zhang 121 232 0 14 May 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 611 4,905 0 23 Jan 2020
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training Weizhen Qi Yu Yan Yeyun Gong Dayiheng Liu Nan Duan Jiusheng Chen Ruofei Zhang Ming Zhou AI4TS 93 450 0 13 Jan 2020
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 236 8,447 0 19 Jun 2019
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task Tao Yu Rui Zhang Kai-Chou Yang Michihiro Yasunaga Dongxu Wang ... Irene Li Qingning Yao Shanelle Roman Zilin Zhang Dragomir R. Radev RALM 114 1,243 0 24 Sep 2018

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.