Optimized Speculative Sampling for GPU Hardware Accelerators

16 June 2024

Seanie Lee

Papers citing "Optimized Speculative Sampling for GPU Hardware Accelerators"

22 / 22 papers shown

Title
SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models Seanie Lee Dong Bok Lee Dominik Wagner Minki Kang Haebin Seong Tobias Bocklet Juho Lee Sung Ju Hwang 86 2 0 18 Feb 2025
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Tianle Cai Yuhong Li Zhengyang Geng Hongwu Peng Jason D. Lee De-huai Chen Tri Dao 172 314 0 19 Jan 2024
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding Heming Xia Zhe Yang Qingxiu Dong Peiyi Wang Chak Tou Leong Tao Ge Tianyu Liu Wenjie Li Zhifang Sui LRM 141 129 0 15 Jan 2024
Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling Sanchit Gandhi Patrick von Platen Alexander M. Rush VLM 68 62 0 01 Nov 2023
Replacing softmax with ReLU in Vision Transformers Mitchell Wortsman Jaehoon Lee Justin Gilmer Simon Kornblith ViT 77 33 0 15 Sep 2023
Accelerating LLM Inference with Staged Speculative Decoding Benjamin Spector Christal Re 75 112 0 08 Aug 2023
A Study on ReLU and Softmax in Transformer Kai Shen Junliang Guo Xuejiao Tan Siliang Tang Rui Wang Jiang Bian 101 57 0 13 Feb 2023
Robust Speech Recognition via Large-Scale Weak Supervision Alec Radford Jong Wook Kim Tao Xu Greg Brockman C. McLeavey Ilya Sutskever OffRL 215 3,750 0 06 Dec 2022
Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan Matan Kalman Yossi Matias LRM 151 736 0 30 Nov 2022
A White Paper on Neural Network Quantization Markus Nagel Marios Fournarakis Rana Ali Amjad Yelysei Bondarenko M. V. Baalen Tijmen Blankevoort MQ 95 546 0 15 Jun 2021
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units Wei-Ning Hsu Benjamin Bolte Yao-Hung Hubert Tsai Kushal Lakhotia Ruslan Salakhutdinov Abdel-rahman Mohamed SSL 184 3,003 0 14 Jun 2021
RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su Yu Lu Shengfeng Pan Ahmed Murtadha Bo Wen Yunfeng Liu 329 2,533 0 20 Apr 2021
Training with Quantization Noise for Extreme Model Compression Angela Fan Pierre Stock Benjamin Graham Edouard Grave Remi Gribonval Hervé Jégou Armand Joulin MQ 102 246 0 15 Apr 2020
Common Voice: A Massively-Multilingual Speech Corpus Rosana Ardila Megan Branson Kelly Davis Michael Henretty M. Kohler Josh Meyer Reuben Morais Lindsay Saunders Francis M. Tyers Gregor Weber VLM 93 1,620 0 13 Dec 2019
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 565 42,639 0 03 Dec 2019
Fast Transformer Decoding: One Write-Head is All You Need Noam M. Shazeer 161 476 0 06 Nov 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 264 7,554 0 02 Oct 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 113 1,872 0 23 Sep 2019
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita David Talbot F. Moiseev Rico Sennrich Ivan Titov 119 1,149 0 23 May 2019
Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization Shashi Narayan Shay B. Cohen Mirella Lapata AILaw 158 1,684 0 27 Aug 2018
Online normalizer calculation for softmax Maxim Milakov N. Gimelshein 94 95 0 08 May 2018
Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking Zhe Jia Marco Maggioni Benjamin Staiger D. Scarpazza 64 311 0 18 Apr 2018