SpecTr: Fast Speculative Decoding via Optimal Transport

23 October 2023

Papers citing "SpecTr: Fast Speculative Decoding via Optimal Transport"

50 / 53 papers shown

Title
BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms Yunlong Hou Fengzhuo Zhang Cunxiao Du Xuan Zhang Jiachun Pan Tianyu Pang Chao Du Vincent Y. F. Tan Zhuoran Yang OffRL 17 0 0 21 May 2025
Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks Yang Liu Bingjie Yan Tianyuan Zou Jianqing Zhang Zixuan Gu ... Jiajian Li Xiaozhou Ye Ye Ouyang Qiang Yang Yanzhe Zhang ALM 260 1 0 24 Apr 2025
PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation Zihao An Huajun Bai Ziqiang Liu Dong Li E. Barsoum 61 0 0 23 Apr 2025
PCM : Picard Consistency Model for Fast Parallel Sampling of Diffusion Models Junhyuk So Jiwoong Shin Chaeyeon Jang Eunhyeok Park DiffM 55 0 0 25 Mar 2025
Speculative Decoding and Beyond: An In-Depth Survey of Techniques Y. Hu Zining Liu Zhenyuan Dong Tianfan Peng Bradley McDanel Shanghang Zhang 93 0 0 27 Feb 2025
Towards Optimal Multi-draft Speculative Decoding Zhibo Hu Tong Zheng Vignesh Viswanathan Ziyi Chen Ryan Rossi Yihan Wu Dinesh Manocha Heng Huang 47 4 0 26 Feb 2025
LongSpec: Long-Context Speculative Decoding with Efficient Drafting and Verification Penghui Yang Cunxiao Du Fengzhuo Zhang Haonan Wang Tianyu Pang Chao Du Bo An RALM 50 0 0 24 Feb 2025
GRIFFIN: Effective Token Alignment for Faster Speculative Decoding Shijing Hu Jingyang Li Xingyu Xie Zhihui Lu Kim-Chuan Toh Pan Zhou 53 0 0 16 Feb 2025
AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding Zikun Li Zhuofu Chen Remi Delacourt Gabriele Oliaro Zeyu Wang ... Zhuoming Chen Sean Lai Xinhao Cheng Xupeng Miao Zhihao Jia 53 6 0 21 Jan 2025
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration Zhuofan Wen Shangtong Gui Yang Feng 108 3 0 25 Nov 2024
Privacy Risks of Speculative Decoding in Large Language Models Jiankun Wei Abdulrahman Abdulrazzag Tianchen Zhang Adel Muursepp Gururaj Saileshwar 40 2 0 01 Nov 2024
A Theoretical Perspective for Speculative Decoding Algorithm Ming Yin Minshuo Chen Kaixuan Huang Mengdi Wang 37 5 0 30 Oct 2024
Fast Best-of-N Decoding via Speculative Rejection Hanshi Sun Momin Haider Ruiqi Zhang Huitao Yang Jiahao Qiu Ming Yin Mengdi Wang Peter L. Bartlett Andrea Zanette BDL 50 30 0 26 Oct 2024
Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits Ashish Khisti MohammadReza Ebrahimi Hassan Dbouk Arash Behboodi Roland Memisevic Christos Louizos 38 0 0 23 Oct 2024
TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling Jiahao Qiu Yifu Lu Yifan Zeng Jiacheng Guo Jiayi Geng Huazheng Wang Kaixuan Huang Yue Wu Mengdi Wang 61 23 0 18 Oct 2024
DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure Yunfan Xiong Ruoyu Zhang Yanzeng Li Tianhao Wu Lei Zou 42 5 0 15 Oct 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 62 6 0 09 Oct 2024
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 33 7 0 08 Oct 2024
Efficient Inference for Large Language Model-based Generative Recommendation Xinyu Lin Chaoqun Yang Wenjie Wang Yongqi Li Cunxiao Du Fuli Feng See-Kiong Ng Tat-Seng Chua 70 4 0 07 Oct 2024
Integrative Decoding: Improve Factuality via Implicit Self-consistency Yi Cheng Xiao Liang Yeyun Gong Wen Xiao Song Wang ... Wenjie Li Jian Jiao Qi Chen Peng Cheng Wayne Xiong HILM 65 1 0 02 Oct 2024
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding Yao Teng Han Shi Xian Liu Xuefei Ning Guohao Dai Yu Wang Zhenguo Li Xihui Liu 63 10 0 02 Oct 2024
Learning Harmonized Representations for Speculative Sampling Lefan Zhang Xiaodan Wang Yanhua Huang Ruiwen Xu 26 10 0 28 Aug 2024
Coupling without Communication and Drafter-Invariant Speculative Decoding Majid Daliri Christopher Musco A. Suresh 54 1 0 15 Aug 2024
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 96 57 0 24 Jun 2024
Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving Ke Cheng Wen Hu Zhi Wang Hongen Peng Jianguo Li Sheng Zhang 57 7 0 19 Jun 2024
Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding Kaiyan Zhang Jianyu Wang Ning Ding Biqing Qi Ermo Hua Xingtai Lv Bowen Zhou 48 9 0 18 Jun 2024
Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction Ke Cheng Wen Hu Zhi Wang Peng Du Jianguo Li Sheng Zhang 47 10 0 07 Jun 2024
SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices Ruslan Svirschevski Avner May Zhuoming Chen Beidi Chen Zhihao Jia Max Ryabinin 39 12 0 04 Jun 2024
SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths Kaixuan Huang Xudong Guo Mengdi Wang 52 20 0 30 May 2024
Faster Cascades via Speculative Decoding Harikrishna Narasimhan Wittawat Jitkrittum A. S. Rawat Seungyeon Kim Neha Gupta A. Menon Sanjiv Kumar LRM 46 6 0 29 May 2024
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting Fangcheng Liu Yehui Tang Zhenhua Liu Yunsheng Ni Kai Han Yunhe Wang 51 24 0 29 Apr 2024
Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models Chen Zhang Zhuorui Liu Dawei Song LRM 43 3 0 23 Apr 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu Wang 51 86 0 22 Apr 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 51 49 0 18 Apr 2024
Language Model Cascades: Token-level uncertainty and beyond Neha Gupta Harikrishna Narasimhan Wittawat Jitkrittum A. S. Rawat A. Menon Sanjiv Kumar UQLM 58 42 0 15 Apr 2024
Exploring and Improving Drafts in Blockwise Parallel Decoding Taehyeon Kim A. Suresh Kishore Papineni Michael Riley Sanjiv Kumar Adrian Benton AI4TS 52 2 0 14 Apr 2024
On Speculative Decoding for Multimodal Large Language Models Mukul Gagrani Raghavv Goel Wonseok Jeon Junyoung Park Mingu Lee Christopher Lott LRM 40 8 0 13 Apr 2024
Recurrent Drafter for Fast Speculative Decoding in Large Language Models Aonan Zhang Chong-Jun Wang Yi Wang Xuanyu Zhang Yunfei Cheng 42 17 0 14 Mar 2024
LLM Inference Unveiled: Survey and Roofline Model Insights Zhihang Yuan Yuzhang Shang Yang Zhou Zhen Dong Zhe Zhou ... Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer 61 82 0 26 Feb 2024
Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens Ziqian Zeng Jiahong Yu Qianshi Pang Zihao Wang Huiping Zhuang Cen Chen Xiaofeng Zou 40 4 0 24 Feb 2024
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding Zhuoming Chen Avner May Ruslan Svirschevski Yuhsun Huang Max Ryabinin Zhihao Jia Beidi Chen 53 41 0 19 Feb 2024
Speculative Streaming: Fast LLM Inference without Auxiliary Models Nikhil Bhendawade Irina Belousova Qichen Fu Henry Mason Mohammad Rastegari Mahyar Najibi LRM 36 29 0 16 Feb 2024
Accelerating Parallel Sampling of Diffusion Models Zhiwei Tang Jiasheng Tang Hao Luo Fan Wang Tsung-Hui Chang 37 13 0 15 Feb 2024
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding Cunxiao Du Jing Jiang Yuanchen Xu Jiawei Wu Sicheng Yu ... Shenggui Li Kai Xu Liqiang Nie Zhaopeng Tu Yang You 42 30 0 03 Feb 2024
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding Yichao Fu Peter Bailis Ion Stoica Hao Zhang 133 145 0 03 Feb 2024
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 52 128 0 26 Jan 2024
BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models Feng-Huei Lin Hanling Yi Hongbin Li Yifan Yang Xiaotian Yu Guangming Lu Rong Xiao 43 3 0 23 Jan 2024
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding Heming Xia Zhe Yang Qingxiu Dong Peiyi Wang Yongqi Li Tao Ge Tianyu Liu Wenjie Li Zhifang Sui LRM 40 106 0 15 Jan 2024
Multi-Candidate Speculative Decoding Sen Yang Shujian Huang Xinyu Dai Jiajun Chen BDL 30 16 0 12 Jan 2024
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems Xupeng Miao Gabriele Oliaro Zhihao Zhang Xinhao Cheng Hongyi Jin Tianqi Chen Zhihao Jia 73 77 0 23 Dec 2023