MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large
Language Models

MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models

21 August 2024

Roberto L. Castro

Torsten Hoefler

ArXiv (abs)PDF HTML

Papers citing "MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models"

9 / 9 papers shown

Title
MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems Yao Fu Yao Fu Yeqi Huang Ping Nie Zhan Lu ... Dayou Du Tairan Xu Dayou Du Edoardo Ponti Luo Mai MoE 94 1 0 16 May 2025
Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models Hung-Yueh Chiang Chi-chih Chang N. Frumkin Kai-Chiang Wu Mohamed S. Abdelfattah Diana Marculescu MQ 457 0 0 28 Mar 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 505 0 0 08 Jan 2025
CursorCore: Assist Programming through Aligning Anything Hao Jiang Qi Liu Rui Li Shengyu Ye Shijin Wang 104 1 0 09 Oct 2024
Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU Muhammad Osama D. Merrill C. Cecka M. Garland John Douglas Owens 57 28 0 09 Jan 2023
Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production Young Jin Kim Rawn Henry Raffy Fahim Hany Awadalla MoE 61 23 0 18 Nov 2022
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers Elias Frantar Saleh Ashkboos Torsten Hoefler Dan Alistarh MQ 143 1,005 0 31 Oct 2022
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers M. Lewis Younes Belkada Luke Zettlemoyer MQ 103 662 0 15 Aug 2022
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 547 42,591 0 03 Dec 2019