Fast Inference from Transformers via Speculative Decoding

30 November 2022

Yossi Matias

Papers citing "Fast Inference from Transformers via Speculative Decoding"

49 / 99 papers shown

Title
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding Yao Teng Han Shi Xian Liu Xuefei Ning Guohao Dai Yu Wang Zhenguo Li Xihui Liu 91 12 0 02 Oct 2024
The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems Linke Song Zixuan Pang Wenhao Wang Zihao Wang XiaoFeng Wang Hongbo Chen Wei Song Yier Jin Dan Meng Rui Hou 95 8 0 30 Sep 2024
Characterizing and Efficiently Accelerating Multimodal Generation Model Inference Yejin Lee Anna Y. Sun Basil Hosmer Bilge Acun Can Balioglu ... Ram Pasunuru Scott Yih Sravya Popuri Xing Liu Carole-Jean Wu 95 2 0 30 Sep 2024
Dynamic-Width Speculative Beam Decoding for Efficient LLM Inference Zongyue Qin Zifan He Neha Prakriya Jason Cong Yizhou Sun 69 5 0 25 Sep 2024
What is the Role of Small Models in the LLM Era: A Survey Lihu Chen Gaël Varoquaux ALM 169 29 0 10 Sep 2024
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding Jian Chen Vashisth Tiwari Ranajoy Sadhukhan Zhuoming Chen Jinyuan Shi Ian En-Hsu Yen Ian En-Hsu Yen Avner May Tianqi Chen Beidi Chen LRM 101 27 0 20 Aug 2024
Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling Xianzhen Luo Yixuan Wang Qingfu Zhu Zhiming Zhang Xuanyu Zhang Qing Yang Dongliang Xu 56 8 0 16 Aug 2024
Coupling without Communication and Drafter-Invariant Speculative Decoding Majid Daliri Christopher Musco A. Suresh 61 2 0 15 Aug 2024
PEARL: Parallel Speculative Decoding with Adaptive Draft Length Tianyu Liu Yun Li Qitan Lv Kai Liu Jianchen Zhu Winston Hu Xingwu Sun 104 17 0 13 Aug 2024
Efficiency Unleashed: Inference Acceleration for LLM-based Recommender Systems with Speculative Decoding Yunjia Xi Hangyu Wang Bo Chen Jianghao Lin Menghui Zhu Wen Liu Ruiming Tang Zhewei Wei Weinan Zhang Yong Yu OffRL 127 4 0 11 Aug 2024
Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion Jacob K Christopher Brian Bartoldson Tal Ben-Nun Michael Cardei B. Kailkhura Ferdinando Fioretto DiffM 88 4 0 10 Aug 2024
ThinK: Thinner Key Cache by Query-Driven Pruning Yuhui Xu Zhanming Jie Hanze Dong Lei Wang Xudong Lu Aojun Zhou Amrita Saha Caiming Xiong Doyen Sahoo 109 20 0 30 Jul 2024
Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting Zilong Wang Zifeng Wang Long Le Huaixiu Steven Zheng Swaroop Mishra ... Anush Mattapalli Ankur Taly Jingbo Shang Chen-Yu Lee Tomas Pfister RALM 110 43 0 11 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 102 53 0 09 Jul 2024
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure Jikai Wang Yi Su Juntao Li Qingrong Xia Zi Ye Xinyu Duan Zhefeng Wang Min Zhang 96 17 0 25 Jun 2024
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph Roman Vashurin Ekaterina Fadeeva Artem Vazhentsev Akim Tsvigun Daniil Vasilev ... Timothy Baldwin Timothy Baldwin Maxim Panov Artem Shelmanov Artem Shelmanov HILM 95 28 0 21 Jun 2024
CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Models Yuetai Li Zhangchen Xu Fengqing Jiang Luyao Niu D. Sahabandu Bhaskar Ramasubramanian Radha Poovendran SILM AAML 91 8 0 18 Jun 2024
Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL Zijin Hong Zheng Yuan Qinggang Zhang Hao Chen Junnan Dong Feiran Huang Xiao Huang 135 72 0 12 Jun 2024
Nearest Neighbor Speculative Decoding for LLM Generation and Attribution Minghan Li Xilun Chen Ari Holtzman Beidi Chen Jimmy Lin Wen-tau Yih Xi Lin RALM BDL 161 14 0 29 May 2024
The Unreasonable Ineffectiveness of the Deeper Layers Andrey Gromov Kushal Tirumala Hassan Shapourian Paolo Glorioso Daniel A. Roberts 113 99 0 26 Mar 2024
On the Challenges and Opportunities in Generative AI Laura Manduchi Kushagra Pandey Robert Bamler Ryan Cotterell Sina Daubener ... F. Wenzel Frank Wood Stephan Mandt Vincent Fortuin Vincent Fortuin 214 21 0 28 Feb 2024
Decoding Speculative Decoding Minghao Yan Saurabh Agarwal Shivaram Venkataraman LRM 73 10 0 02 Feb 2024
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 103 155 0 26 Jan 2024
Hot PATE: Private Aggregation of Distributions for Diverse Task Edith Cohen Benjamin Cohen-Wang Xin Lyu Jelani Nelson Tamas Sarlos Uri Stemmer 89 4 0 04 Dec 2023
GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values Farnoosh Javadi Walid Ahmed Habib Hajimolahoseini Foozhan Ataiefard Mohammad Hassanpour Saina Asani Austin Wen Omar Mohamed Awad Kangling Liu Yang Liu VLM 78 8 0 06 Nov 2023
Accelerating Large Language Model Decoding with Speculative Sampling Charlie Chen Sebastian Borgeaud G. Irving Jean-Baptiste Lespiau Laurent Sifre J. Jumper BDL LRM 82 421 0 02 Feb 2023
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation Jiahui Yu Yuanzhong Xu Jing Yu Koh Thang Luong Gunjan Baid ... Zarana Parekh Xin Li Han Zhang Jason Baldridge Yonghui Wu EGVM 181 1,117 0 22 Jun 2022
$Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$$ Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$ Adam Roberts Hyung Won Chung Anselm Levskaya Gaurav Mishra James Bradbury ... Brennan Saeta Ryan Sepassi A. Spiridonov Joshua Newlan Andrea Gesmundo ALM 95 196 0 31 Mar 2022
Sparse is Enough in Scaling Transformers Sebastian Jaszczur Aakanksha Chowdhery Afroz Mohiuddin Lukasz Kaiser Wojciech Gajewski Henryk Michalewski Jonni Kanerva MoE 56 102 0 24 Nov 2021
The Efficiency Misnomer Daoyuan Chen Liuyi Yao Dawei Gao Ashish Vaswani Yaliang Li 91 103 0 25 Oct 2021
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 251 156 0 17 Sep 2021
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding Xin Sun Tao Ge Furu Wei Houfeng Wang 56 63 0 09 Jun 2021
Consistent Accelerated Inference via Confident Adaptive Transformers Tal Schuster Adam Fisch Tommi Jaakkola Regina Barzilay AI4TS 225 72 0 18 Apr 2021
Dynamic Neural Networks: A Survey Yizeng Han Gao Huang Shiji Song Le Yang Honghui Wang Yulin Wang 3DH AI4TS AI4CE 109 649 0 09 Feb 2021
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 743 41,932 0 28 May 2020
Why should we add early exits to neural networks? Simone Scardapane M. Scarpiniti E. Baccarelli A. Uncini 62 123 0 27 Apr 2020
The Right Tool for the Job: Matching Model and Instance Complexities Roy Schwartz Gabriel Stanovsky Swabha Swayamdipta Jesse Dodge Noah A. Smith 95 169 0 16 Apr 2020
Controlling Computation versus Quality for Neural Sequence Models Ankur Bapna N. Arivazhagan Orhan Firat 65 30 0 17 Feb 2020
Fast Transformer Decoding: One Write-Head is All You Need Noam M. Shazeer 151 461 0 06 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 419 20,127 0 23 Oct 2019
Depth-Adaptive Transformer Maha Elbayad Jiatao Gu Edouard Grave Michael Auli 86 192 0 22 Oct 2019
Adaptive Attention Span in Transformers Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski Armand Joulin 76 285 0 19 May 2019
Blockwise Parallel Decoding for Deep Autoregressive Models Mitchell Stern Noam M. Shazeer Ashley J. Llorens 51 233 0 07 Nov 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.7K 94,770 0 11 Oct 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 690 131,526 0 12 Jun 2017
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations Itay Hubara Matthieu Courbariaux Daniel Soudry Ran El-Yaniv Yoshua Bengio MQ 149 1,864 0 22 Sep 2016
Gaussian Error Linear Units (GELUs) Dan Hendrycks Kevin Gimpel 169 5,000 0 27 Jun 2016
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 347 19,643 0 09 Mar 2015
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling Ciprian Chelba Tomas Mikolov M. Schuster Qi Ge T. Brants P. Koehn T. Robinson 175 1,103 0 11 Dec 2013