REST: Retrieval-Based Speculative Decoding

14 November 2023

Tianle Cai

Papers citing "REST: Retrieval-Based Speculative Decoding"

50 / 59 papers shown

Title
PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding Bradley McDanel S. Zhang Y. Hu Zining Liu MoE 142 0 0 02 May 2025
Efficient Reasoning for LLMs through Speculative Chain-of-Thought Jikai Wang J. Li Lijun Wu Hao Fei LLMAG LRM 69 2 0 27 Apr 2025
PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation Zihao An Huajun Bai Ziqiang Liu Dong Li E. Barsoum 61 0 0 23 Apr 2025
Speculative Decoding for Multi-Sample Inference Yiwei Li Jiayi Shi Shaoxiong Feng Peiwen Yuan Xinyu Wang ... Ji Zhang Chuyi Tan Boyuan Pan Yao Hu Kan Li LRM 44 0 0 07 Mar 2025
SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding Kaiyu Huang Yu Wang Zhubo Shi Han Zou Minchen Yu Qingjiang Shi LRM 41 2 0 07 Mar 2025
RASD: Retrieval-Augmented Speculative Decoding Guofeng Quan Wenfeng Feng Chuzhan Hao Guochao Jiang Yuewei Zhang Hao Wang RALM 82 1 0 05 Mar 2025
DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting Kai Lv Honglin Guo Qipeng Guo Xipeng Qiu 41 0 0 02 Mar 2025
Tutorial Proposal: Speculative Decoding for Efficient LLM Inference Heming Xia Cunxiao Du Yong Li Qian Liu Wenjie Li 39 0 0 01 Mar 2025
Speculative Decoding and Beyond: An In-Depth Survey of Techniques Y. Hu Zining Liu Zhenyuan Dong Tianfan Peng Bradley McDanel S. Zhang 93 0 0 27 Feb 2025
From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence Generation up to 100K Tokens Tong Wu Junzhe Shen Zixia Jia Yanjie Wang Zilong Zheng 85 0 0 26 Feb 2025
Towards Optimal Multi-draft Speculative Decoding Zhibo Hu Tong Zheng Vignesh Viswanathan Ziyi Chen Ryan Rossi Yihan Wu Dinesh Manocha Heng Huang 47 3 0 26 Feb 2025
LongSpec: Long-Context Speculative Decoding with Efficient Drafting and Verification Penghui Yang Cunxiao Du Fengzhuo Zhang Haonan Wang Tianyu Pang Chao Du Bo An RALM 45 0 0 24 Feb 2025
CodeSwift: Accelerating LLM Inference for Efficient Code Generation Qianhui Zhao L. Zhang Fang Liu Xiaoli Lian Qiaoyuanhe Meng Ziqian Jiao Zetong Zhou Borui Zhang Runlin Guo Jia Li 43 0 0 24 Feb 2025
Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding Tian Jin Ellie Y. Cheng Zack Ankner Nikunj Saunshi Blake M. Elias Amir Yazdanbakhsh Jonathan Ragan-Kelley Suvinay Subramanian Michael Carbin 60 3 0 24 Feb 2025
GRIFFIN: Effective Token Alignment for Faster Speculative Decoding Shijing Hu Jingyang Li Xingyu Xie Zhihui Lu Kim-Chuan Toh Pan Zhou 43 0 0 16 Feb 2025
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding Zihan Wang Muneeza Azmart Ang Li R. Horesh Mikhail Yurochkin 118 1 0 11 Feb 2025
Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding Sukmin Cho S. Choi T. Hwang Jeongyeon Seo Soyeong Jeong Huije Lee Hoyun Song Jong C. Park Youngjin Kwon 51 0 0 08 Feb 2025
Constrained Decoding with Speculative Lookaheads Nishanth Nakshatri Shamik Roy Rajarshi Das Suthee Chaidaroon Leonid Boytsov Rashmi Gangadharaiah 79 0 0 09 Dec 2024
SAM Decoding: Speculative Decoding via Suffix Automaton Yuxuan Hu Ke Wang Jing Zhang Fanjin Zhang C. Li Hongyu Chen Jing Zhang 52 1 0 16 Nov 2024
The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation Lawrence Stewart Matthew Trager Sujan Kumar Gonugondla Stefano Soatto 53 5 0 06 Nov 2024
Privacy Risks of Speculative Decoding in Large Language Models Jiankun Wei Abdulrahman Abdulrazzag Tianchen Zhang Adel Muursepp Gururaj Saileshwar 35 2 0 01 Nov 2024
Interpretable Language Modeling via Induction-head Ngram Models Eunji Kim Sriya Mantena Weiwei Yang Chandan Singh Sungroh Yoon Jianfeng Gao 56 0 0 31 Oct 2024
A Theoretical Perspective for Speculative Decoding Algorithm Ming Yin Minshuo Chen Kaixuan Huang Mengdi Wang 32 4 0 30 Oct 2024
AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration Bradley McDanel LRM 20 2 0 22 Oct 2024
DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure Yunfan Xiong Ruoyu Zhang Yanzeng Li Tianhao Wu Lei Zou 32 5 0 15 Oct 2024
CursorCore: Assist Programming through Aligning Anything Hao Jiang Qi Liu Rui Li Shengyu Ye Shijin Wang 53 1 0 09 Oct 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 53 6 0 09 Oct 2024
ParallelSpec: Parallel Drafter for Efficient Speculative Decoding Zilin Xiao Hongming Zhang Tao Ge Siru Ouyang Vicente Ordonez Dong Yu 39 5 0 08 Oct 2024
Mixture of Attentions For Speculative Decoding Matthieu Zimmer Milan Gritta Gerasimos Lampouras Haitham Bou Ammar Jun Wang 76 4 0 04 Oct 2024
CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decoding Sophia Ho Jinsol Park Patrick Wang 34 0 0 08 Aug 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 36 0 0 03 Jul 2024
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure Jikai Wang Yi Su Juntao Li Qingrong Xia Zi Ye Xinyu Duan Zhefeng Wang Min Zhang 46 12 0 25 Jun 2024
SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices Ruslan Svirschevski Avner May Zhuoming Chen Beidi Chen Zhihao Jia Max Ryabinin 39 12 0 04 Jun 2024
SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths Kaixuan Huang Xudong Guo Mengdi Wang 43 19 0 30 May 2024
Nearest Neighbor Speculative Decoding for LLM Generation and Attribution Minghan Li Xilun Chen Ari Holtzman Beidi Chen Jimmy Lin Wen-tau Yih Xi Lin RALM BDL 108 10 0 29 May 2024
Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference Hao Mark Chen Wayne Luk Ka-Fai Cedric Yiu Rui Li Konstantin Mishchenko Stylianos I. Venieris Hongxiang Fan 39 7 0 28 May 2024
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting Fangcheng Liu Yehui Tang Zhenhua Liu Yunsheng Ni Kai Han Yunhe Wang 46 24 0 29 Apr 2024
Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models Chen Zhang Zhuorui Liu Dawei Song LRM 38 3 0 23 Apr 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 83 0 22 Apr 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 46 47 0 18 Apr 2024
SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens Chengbo Liu Yong Zhu 23 0 0 27 Mar 2024
Recurrent Drafter for Fast Speculative Decoding in Large Language Models Aonan Zhang Chong-Jun Wang Yi Wang Xuanyu Zhang Yunfei Cheng 37 16 0 14 Mar 2024
Reliable, Adaptable, and Attributable Language Models with Retrieval Akari Asai Zexuan Zhong Danqi Chen Pang Wei Koh Luke Zettlemoyer Hanna Hajishirzi Wen-tau Yih KELM RALM 49 54 0 05 Mar 2024
Accelerating Greedy Coordinate Gradient via Probe Sampling Yiran Zhao Wenyue Zheng Tianle Cai Xuan Long Do Kenji Kawaguchi Anirudh Goyal Michael Shieh 48 11 0 02 Mar 2024
Retrieval-Augmented Generation for AI-Generated Content: A Survey Penghao Zhao Hailin Zhang Qinhan Yu Zhengren Wang Yunteng Geng Fangcheng Fu Ling Yang Wentao Zhang Jie Jiang Bin Cui 3DV 126 232 0 29 Feb 2024
Grounding Language Models for Visual Entity Recognition Zilin Xiao Ming Gong Paola Cascante-Bonilla Xingyao Zhang Jie Wu Vicente Ordonez VLM 46 8 0 28 Feb 2024
LLM Inference Unveiled: Survey and Roofline Model Insights Zhihang Yuan Yuzhang Shang Yang Zhou Zhen Dong Zhe Zhou ... Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer 50 80 0 26 Feb 2024
HiRE: High Recall Approximate Top- $k$ Estimation for Efficient LLM Inference Yashas Samaga Varun Yerram Chong You Srinadh Bhojanapalli Sanjiv Kumar Prateek Jain Praneeth Netrapalli 59 4 0 14 Feb 2024
Tandem Transformers for Inference Efficient LLMs S. AishwaryaP Pranav Ajit Nair Yashas Samaga Toby Boyd Sanjiv Kumar Prateek Jain Praneeth Netrapalli 17 6 0 13 Feb 2024
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding Yichao Fu Peter Bailis Ion Stoica Hao Zhang 133 143 0 03 Feb 2024