Accelerating Large Language Model Decoding with Speculative Sampling

2 February 2023

Charlie Chen

Sebastian Borgeaud

G. Irving

Jean-Baptiste Lespiau

Papers citing "Accelerating Large Language Model Decoding with Speculative Sampling"

50 / 316 papers shown

Title
σ-GPTs: A New Approach to Autoregressive Models Arnaud Pannatier Evann Courdier Franccois Fleuret AI4TS 28 7 0 15 Apr 2024
Mitigating Hallucination in Abstractive Summarization with Domain-Conditional Mutual Information Kyubyung Chae Jaepill Choi Yohan Jo Taesup Kim HILM 30 1 0 15 Apr 2024
Exploring and Improving Drafts in Blockwise Parallel Decoding Taehyeon Kim A. Suresh Kishore Papineni Michael Riley Sanjiv Kumar Adrian Benton AI4TS 52 2 0 14 Apr 2024
Leveraging Large Language Model as Simulated Patients for Clinical Education Yaneng Li Cheng Zeng Jialun Zhong Ruoyu Zhang Minhao Zhang Lei Zou AI4Ed LM&MA 51 15 0 13 Apr 2024
On Speculative Decoding for Multimodal Large Language Models Mukul Gagrani Raghavv Goel Wonseok Jeon Junyoung Park Mingu Lee Christopher Lott LRM 40 8 0 13 Apr 2024
Reducing hallucination in structured outputs via Retrieval-Augmented Generation Patrice Béchard Orlando Marquez Ayala LLMAG 42 51 0 12 Apr 2024
Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding Jie Ou Yueming Chen Wenhong Tian 53 12 0 10 Apr 2024
CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers Longwei Zou Qingyang Wang Han Zhao Jiangang Kong Yi Yang Yangdong Deng 50 0 0 10 Apr 2024
Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models Bowen Pan Songlin Yang Haokun Liu Mayank Mishra Gaoyuan Zhang Aude Oliva Colin Raffel Yikang Shen MoE 46 19 0 08 Apr 2024
The Larger the Better? Improved LLM Code-Generation via Budget Reallocation Michael Hassid Tal Remez Jonas Gehring Roy Schwartz Yossi Adi 36 20 0 31 Mar 2024
SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens Chengbo Liu Yong Zhu 28 0 0 27 Mar 2024
Efficient Encoder-Decoder Transformer Decoding for Decomposable Tasks Bo-Ru Lu Nikita Haduong Chien-Yu Lin Hao Cheng Noah A. Smith Mari Ostendorf AI4CE 40 0 0 19 Mar 2024
Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inference Baolin Li Yankai Jiang V. Gadepally Devesh Tiwari 31 15 0 19 Mar 2024
MELTing point: Mobile Evaluation of Language Transformers Stefanos Laskaridis Kleomenis Katevas Lorenzo Minto Hamed Haddadi 29 21 0 19 Mar 2024
Repoformer: Selective Retrieval for Repository-Level Code Completion Di Wu W. Ahmad Dejiao Zhang M. K. Ramanathan Xiaofei Ma 25 26 0 15 Mar 2024
Recurrent Drafter for Fast Speculative Decoding in Large Language Models Aonan Zhang Chong-Jun Wang Yi Wang Xuanyu Zhang Yunfei Cheng 37 17 0 14 Mar 2024
GiT: Towards Generalist Vision Transformer through Universal Language Interface Haiyang Wang Hao Tang Li Jiang Shaoshuai Shi Muhammad Ferjad Naeem Hongsheng Li Bernt Schiele Liwei Wang VLM 46 10 0 14 Mar 2024
Token Alignment via Character Matching for Subword Completion Ben Athiwaratkun Shiqi Wang Mingyue Shang Yuchen Tian Zijian Wang Sujan Kumar Gonugondla Sanjay Krishna Gouda Rob Kwiatowski Ramesh Nallapati Bing Xiang 50 4 0 13 Mar 2024
Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs Ben Athiwaratkun Sujan Kumar Gonugondla Sanjay Krishna Gouda Haifeng Qian Hantian Ding ... Liangfu Chen Parminder Bhatia Ramesh Nallapati Sudipta Sengupta Bing Xiang 59 4 0 13 Mar 2024
Rethinking Generative Large Language Model Evaluation for Semantic Comprehension Fangyun Wei Xi Chen Linzi Luo ELM ALM LRM 38 7 0 12 Mar 2024
SynCode: LLM Generation with Grammar Augmentation Shubham Ugare Tarun Suresh Hangoo Kang Sasa Misailovic Gagandeep Singh 40 12 0 03 Mar 2024
Accelerating Greedy Coordinate Gradient via Probe Sampling Yiran Zhao Wenyue Zheng Tianle Cai Xuan Long Do Kenji Kawaguchi Anirudh Goyal Michael Shieh 51 11 0 02 Mar 2024
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact Ruikang Liu Haoli Bai Haokun Lin Yuening Li Han Gao Zheng-Jun Xu Lu Hou Jun Yao Chun Yuan MQ 23 29 0 02 Mar 2024
Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs Raghavv Goel Mukul Gagrani Wonseok Jeon Junyoung Park Mingu Lee Christopher Lott ALM 34 6 0 29 Feb 2024
CLLMs: Consistency Large Language Models Siqi Kou Lanxiang Hu Zhe He Zhijie Deng Hao Zhang 52 28 0 28 Feb 2024
Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding Benjamin Bergner Andrii Skliar Amelie Royer Tijmen Blankevoort Yuki Markus Asano B. Bejnordi 58 5 0 26 Feb 2024
LLM Inference Unveiled: Survey and Roofline Model Insights Zhihang Yuan Yuzhang Shang Yang Zhou Zhen Dong Zhe Zhou ... Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer 58 82 0 26 Feb 2024
Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens Ziqian Zeng Jiahong Yu Qianshi Pang Zihao Wang Huiping Zhuang Cen Chen Xiaofeng Zou 38 4 0 24 Feb 2024
RelayAttention for Efficient Large Language Model Serving with Long System Prompts Lei Zhu Xinjiang Wang Wayne Zhang Rynson W. H. Lau 33 6 0 22 Feb 2024
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding Weilin Zhao Yuxiang Huang Xu Han Wang Xu Chaojun Xiao Xinrong Zhang Yewei Fang Kaihuo Zhang Zhiyuan Liu Maosong Sun 43 11 0 21 Feb 2024
ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models Chenyang Song Xu Han Zhengyan Zhang Shengding Hu Xiyu Shi ... Chen Chen Zhiyuan Liu Guanglin Li Tao Yang Maosong Sun 53 25 0 21 Feb 2024
ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding Shuzhang Zhong Zebin Yang Meng Li Ruihao Gong Runsheng Wang Ru Huang 39 6 0 21 Feb 2024
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding Zhuoming Chen Avner May Ruslan Svirschevski Yuhsun Huang Max Ryabinin Zhihao Jia Beidi Chen 48 41 0 19 Feb 2024
Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing? Marco Gaido Sara Papi Matteo Negri L. Bentivogli 49 13 0 19 Feb 2024
Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding Hanling Yi Feng-Huei Lin Hongbin Li Peiyang Ning Xiaotian Yu Rong Xiao LRM 29 10 0 19 Feb 2024
Speculative Streaming: Fast LLM Inference without Auxiliary Models Nikhil Bhendawade Irina Belousova Qichen Fu Henry Mason Mohammad Rastegari Mahyar Najibi LRM 34 28 0 16 Feb 2024
Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs Yeonhong Park Jake Hyun SangLyul Cho Bonggeun Sim Jae W. Lee MQ 51 16 0 16 Feb 2024
Chain-of-Thought Reasoning Without Prompting Xuezhi Wang Denny Zhou ReLM LRM 152 104 0 15 Feb 2024
BitDelta: Your Fine-Tune May Only Be Worth One Bit James Liu Guangxuan Xiao Kai Li Jason D. Lee Song Han Tri Dao Tianle Cai 45 21 0 15 Feb 2024
Permute-and-Flip: An optimally stable and watermarkable decoder for LLMs Xuandong Zhao Lei Li Yu-Xiang Wang 55 10 0 08 Feb 2024
Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding Zack Ankner Rishab Parthasarathy Aniruddha Nrusimha Christopher Rinard Jonathan Ragan-Kelley William Brandon 34 26 0 07 Feb 2024
PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition Jinghui Lu Ziwei Yang Yanjie Wang Xuejing Liu Brian Mac Namee Can Huang MoE 53 5 0 07 Feb 2024
Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation Luca Beurer-Kellner Marc Fischer Martin Vechev 44 38 0 07 Feb 2024
Linear-time Minimum Bayes Risk Decoding with Reference Aggregation Jannis Vamvas Rico Sennrich 50 15 0 06 Feb 2024
ReLU $^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs Zhengyan Zhang Yixin Song Guanghui Yu Xu Han Yankai Lin Chaojun Xiao Chenyang Song Zhiyuan Liu Zeyu Mi Maosong Sun 24 31 0 06 Feb 2024
Decoding-time Realignment of Language Models Tianlin Liu Shangmin Guo Leonardo Bianco Daniele Calandriello Quentin Berthet Felipe Llinares-López Jessica Hoffmann Lucas Dixon Michal Valko Mathieu Blondel AI4CE 54 37 0 05 Feb 2024
A Survey on Transformer Compression Yehui Tang Yunhe Wang Jianyuan Guo Zhijun Tu Kai Han Hailin Hu Dacheng Tao 41 30 0 05 Feb 2024
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding Cunxiao Du Jing Jiang Yuanchen Xu Jiawei Wu Sicheng Yu ... Shenggui Li Kai Xu Liqiang Nie Zhaopeng Tu Yang You 42 30 0 03 Feb 2024
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding Yichao Fu Peter Bailis Ion Stoica Hao Zhang 133 144 0 03 Feb 2024
Decoding Speculative Decoding Minghao Yan Saurabh Agarwal Shivaram Venkataraman LRM 42 6 0 02 Feb 2024