Fast Inference from Transformers via Speculative Decoding

30 November 2022

Yossi Matias

Papers citing "Fast Inference from Transformers via Speculative Decoding"

50 / 99 papers shown

Title
Do Large Language Models (Really) Need Statistical Foundations? Weijie Su 251 0 0 25 May 2025
Inference Compute-Optimal Video Vision Language Models Peiqi Wang ShengYun Peng Xuewen Zhang Hanchao Yu Yibo Yang Lifu Huang Fujun Liu Qifan Wang VLM 84 0 0 24 May 2025
L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models Xiaohao Liu Xiaobo Xia Weixiang Zhao Manyi Zhang Xianzhi Yu Xiu Su Shuo Yang See-Kiong Ng Tat-Seng Chua KELM LRM 73 0 0 23 May 2025
Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency Ruixiao Li Fahao Chen Peng Li 142 0 0 20 May 2025
Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps Jie Ou Jinyu Guo Shuaihong Jiang Zhaokun Wang Libo Qin Shunyu Yao Wenhong Tian 3DV 136 0 0 19 May 2025
Policy Contrastive Decoding for Robotic Foundation Models Shihan Wu Ji Zhang Xu Luo Junlin Xie Jingkuan Song Heng Tao Shen Lianli Gao OffRL 222 0 0 19 May 2025
MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models Mugilan Ganesan Siyang Song Ankur Aggarwal Nish Sinnadurai Sean Lie Vithursan Thangarasa VLM 89 0 0 15 May 2025
Efficient Reasoning for LLMs through Speculative Chain-of-Thought Jikai Wang Junlin Li Jianye Hou Hao Fei Lijun Wu Min Zhang LLMAG LRM 103 3 0 27 Apr 2025
GenTorrent: Scaling Large Language Model Serving with An Overley Network Fei Fang Yifan Hua Shengze Wang Ruilin Zhou Y. Liu Chen Qian Wei Wei 95 0 0 27 Apr 2025
PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation Zihao An Huajun Bai Ziqiang Liu Dong Li E. Barsoum 127 0 0 23 Apr 2025
Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding Sakhinana Sagar Srinivas Akash Das Shivam Gupta Venkataramana Runkana OffRL 88 1 0 02 Apr 2025
Adaptive Layer-skipping in Pre-trained LLMs Xuan Luo Weizhi Wang Xifeng Yan 395 1 0 31 Mar 2025
Speculative End-Turn Detector for Efficient Speech Chatbot Assistant Hyunjong Ok Suho Yoo Jaeho Lee 157 0 0 30 Mar 2025
Accelerate Parallelizable Reasoning via Parallel Decoding within One Sequence Yijiong Yu LRM AIMat 132 1 0 26 Mar 2025
Collaborative Speculative Inference for Efficient LLM Inference Serving Luyao Gao Jianchun Liu Hongli Xu Xichong Zhang Yunming Liao Liusheng Huang 67 1 0 13 Mar 2025
Training Domain Draft Models for Speculative Decoding: Best Practices and Insights Fenglu Hong Ravi Raju Jonathan Li Bo Li Urmish Thakker Avinash Ravichandran Swayambhoo Jain Changran Hu 73 0 0 10 Mar 2025
Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation Yingfeng Luo Tong Zheng Yongyu Mu Yangqiu Song Qinghong Zhang ... Ziqiang Xu Peinan Feng Xiaoqian Liu Tong Xiao Jingbo Zhu AI4CE 439 2 0 09 Mar 2025
Exploiting Edited Large Language Models as General Scientific Optimizers Qitan Lv T. Liu Haoyu Wang 141 1 0 08 Mar 2025
Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models Benyamin Jamialahmadi Parsa Kavehzadeh Mehdi Rezagholizadeh Parsa Farinneya Hossein Rajabzadeh A. Jafari Boxing Chen Marzieh S. Tahaei 77 0 0 06 Mar 2025
EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 176 14 0 03 Mar 2025
CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Drafter Yepeng Weng Dianwen Mei Huishi Qiu Xujie Chen Li Liu Jiang Tian Zhongchao Shi 97 0 0 24 Feb 2025
Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding Tian Jin Ellie Y. Cheng Zack Ankner Nikunj Saunshi Blake M. Elias Amir Yazdanbakhsh Jonathan Ragan-Kelley Suvinay Subramanian Michael Carbin 113 4 0 24 Feb 2025
DReSD: Dense Retrieval for Speculative Decoding Milan Gritta Huiyin Xue Gerasimos Lampouras RALM 149 0 0 21 Feb 2025
SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models Seanie Lee Dong Bok Lee Dominik Wagner Minki Kang Haebin Seong Tobias Bocklet Juho Lee Sung Ju Hwang 75 2 0 18 Feb 2025
Energy-Conscious LLM Decoding: Impact of Text Generation Strategies on GPU Energy Consumption Alireza Nik Michael A. Riegler Pål Halvorsen 82 1 0 17 Feb 2025
SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs Yige Xu Xu Guo Zhiwei Zeng Chunyan Miao LLMAG CLL LRM 107 18 0 17 Feb 2025
Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization Bowen Pang Kai Li Ruifeng She Feifan Wang OffRL 82 2 0 14 Feb 2025
Theoretical Benefit and Limitation of Diffusion Language Model Guhao Feng Yihan Geng Jian Guan Wei Wu Liwei Wang Di He DiffM 132 1 0 13 Feb 2025
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding Ziyi Wang Muneeza Azmart Ang Li R. Horesh Mikhail Yurochkin 154 1 0 11 Feb 2025
LANTERN++: Enhancing Relaxed Speculative Decoding with Static Tree Drafting for Visual Auto-regressive Models Sihwan Park Doohyuk Jang Sungyub Kim Souvik Kundu Eunho Yang 117 0 0 10 Feb 2025
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache Rishabh Tiwari Haocheng Xi Aditya Tomar Coleman Hooper Sehoon Kim Maxwell Horton Mahyar Najibi Michael W. Mahoney Kemal Kurniawan Amir Gholami MQ 83 2 0 05 Feb 2025
Intelligent Sensing-to-Action for Robust Autonomy at the Edge: Opportunities and Challenges A. R. Trivedi Sina Tayebati Hemant Kumawat Nastaran Darabi Divake Kumar ... Dinithi Jayasuriya Nethmi Jayasinghe Priyadarshini Panda Saibal Mukhopadhyay Kaushik Roy 127 0 0 04 Feb 2025
Privacy-Preserving Edge Speech Understanding with Tiny Foundation Models A. Benazir Felix Xiaozhu Lin 107 1 0 29 Jan 2025
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Nicolas Boizard Kevin El Haddad C´eline Hudelot Pierre Colombo 116 17 0 28 Jan 2025
Towards Sustainable Large Language Model Serving Sophia Nguyen Beihao Zhou Yi Ding Sihang Liu 182 8 0 31 Dec 2024
Parallelized Autoregressive Visual Generation Yanjie Wang Shuhuai Ren Zhijie Lin Yujin Han Haoyuan Guo Zhenheng Yang Difan Zou Jiashi Feng Xihui Liu VGen 154 12 0 19 Dec 2024
Constrained Decoding with Speculative Lookaheads Nishanth Nakshatri Shamik Roy Rajarshi Das Suthee Chaidaroon Leonid Boytsov Rashmi Gangadharaiah 145 0 0 09 Dec 2024
Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding Bohan Li Hankun Wang Situo Zhang Yiwei Guo Kai Yu 90 8 0 29 Oct 2024
ProMoE: Fast MoE-based LLM Serving using Proactive Caching Xiaoniu Song Zihang Zhong Rong Chen Haibo Chen MoE 89 5 0 29 Oct 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 115 6 0 28 Oct 2024
Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition Artem Basharin Andrei Chertkov Ivan Oseledets 111 1 0 23 Oct 2024
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling Wenyuan Xu Rujun Han Zhenting Wang L. Le Dhruv Madeka Lei Li Wenjie Wang Rishabh Agarwal Chen-Yu Lee Tomas Pfister 135 11 0 15 Oct 2024
Self-Data Distillation for Recovering Quality in Pruned Large Language Models Vithursan Thangarasa Ganesh Venkatesh Mike Lasby Nish Sinnadurai Sean Lie SyDa 93 2 0 13 Oct 2024
Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level Xinyi Zeng Yuying Shang Yutao Zhu Jingyuan Zhang Yu Tian AAML 401 3 0 09 Oct 2024
Efficient Inference for Large Language Model-based Generative Recommendation Xinyu Lin Chaoqun Yang Wenjie Wang Yongqi Li Cunxiao Du Fuli Feng See-Kiong Ng Tat-Seng Chua 123 4 0 07 Oct 2024
Geometric Collaborative Filtering with Convergence Hisham Husain Julien Monteil FedML 109 8 0 04 Oct 2024
Mixture of Attentions For Speculative Decoding Matthieu Zimmer Milan Gritta Gerasimos Lampouras Haitham Bou Ammar Jun Wang 116 4 0 04 Oct 2024
Better Instruction-Following Through Minimum Bayes Risk Ian Wu Patrick Fernandes Amanda Bertsch Seungone Kim Sina Pakazad Graham Neubig 117 11 0 03 Oct 2024
Selective Attention Improves Transformer Yaniv Leviathan Matan Kalman Yossi Matias 91 12 0 03 Oct 2024
Integrative Decoding: Improve Factuality via Implicit Self-consistency Yi Cheng Xiao Liang Yeyun Gong Wen Xiao Song Wang ... Wenjie Li Jian Jiao Qi Chen Peng Cheng Wayne Xiong HILM 99 2 0 02 Oct 2024