SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

18 November 2022

Song Han

Papers citing "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models"

50 / 533 papers shown

Title
TinyChirp: Bird Song Recognition Using TinyML Models on Low-power Wireless Acoustic Sensors Zhaolan Huang Adrien Tousnakhoff Polina Kozyr Roman Rehausen Felix Biessmann Robert Lachlan C. Adjih Emmanuel Baccelli 42 1 0 31 Jul 2024
ThinK: Thinner Key Cache by Query-Driven Pruning Yuhui Xu Zhanming Jie Hanze Dong Lei Wang Xudong Lu Aojun Zhou Amrita Saha Caiming Xiong Doyen Sahoo 75 15 0 30 Jul 2024
Efficient Inference of Vision Instruction-Following Models with Elastic Cache Zuyan Liu Benlin Liu Jiahui Wang Yuhao Dong Guangyi Chen Yongming Rao Ranjay Krishna Jiwen Lu VLM 48 9 0 25 Jul 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 61 9 0 24 Jul 2024
Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance Ao Shen Qiang Wang Zhiquan Lai Xionglve Li Dongsheng Li ALM MQ 35 1 0 24 Jul 2024
RazorAttention: Efficient KV Cache Compression Through Retrieval Heads Hanlin Tang Yang Lin Jing Lin Qingsen Han Shikuan Hong Yiwu Yao Gongyi Wang MQ 42 27 0 22 Jul 2024
Mixture of Experts with Mixture of Precisions for Tuning Quality of Service HamidReza Imani Abdolah Amirany Tarek A. El-Ghazawi MoE 64 6 0 19 Jul 2024
SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization Rui Xie Asad Ul Haq Linsen Ma Krystal Sun Sanchari Sen Swagath Venkataramani Liu Liu Tong Zhang MQ 25 1 0 17 Jul 2024
Mamba-PTQ: Outlier Channels in Recurrent Large Language Models Alessandro Pierro Steven Abreu MQ Mamba 45 6 0 17 Jul 2024
Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Models Chen Ju Haicheng Wang Haozhe Cheng Xu Chen Zhonghua Zhai Weilin Huang Jinsong Lan Shuai Xiao Bo Zheng VLM 49 5 0 16 Jul 2024
MINI-LLM: Memory-Efficient Structured Pruning for Large Language Models Hongrong Cheng Miao Zhang J. Q. Shi 57 2 0 16 Jul 2024
Real-Time Anomaly Detection and Reactive Planning with Large Language Models Rohan Sinha Amine Elhafsi Christopher Agia Matthew Foutter Edward Schmerling Marco Pavone OffRL LRM 45 27 0 11 Jul 2024
EfficientQAT: Efficient Quantization-Aware Training for Large Language Models Yonghong Tian Wenqi Shao Peng Xu Jiahao Wang Peng Gao Kaipeng Zhang Ping Luo MQ 52 26 0 10 Jul 2024
Scaling Retrieval-Based Language Models with a Trillion-Token Datastore Rulin Shao Jacqueline He Akari Asai Weijia Shi Tim Dettmers Sewon Min Luke Zettlemoyer Pang Wei Koh RALM 29 22 0 09 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 42 43 0 09 Jul 2024
Composable Interventions for Language Models Arinbjorn Kolbeinsson Kyle O'Brien Tianjin Huang Shanghua Gao Shiwei Liu ... Anurag J. Vaidya Faisal Mahmood Marinka Zitnik Tianlong Chen Thomas Hartvigsen KELM MU 89 5 0 09 Jul 2024
On the Limitations of Compute Thresholds as a Governance Strategy Sara Hooker 58 14 0 08 Jul 2024
Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations Bowen Shen Zheng Lin Daren Zha Wei Liu Jian Luan Bin Wang Weiping Wang 60 1 0 08 Jul 2024
The Impact of Quantization and Pruning on Deep Reinforcement Learning Models Heng Lu Mehdi Alemi Reza Rawassizadeh 42 1 0 05 Jul 2024
Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directions Shumaila Javaid R. A. Khalil Nasir Saeed Bin He Mohamed-Slim Alouini 44 9 0 05 Jul 2024
SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking Xingrun Xing Boyan Gao Zheng Zhang David A. Clifton Shitao Xiao Li Du Guoqi Li Jiajun Zhang 58 5 0 05 Jul 2024
GPTQT: Quantize Large Language Models Twice to Push the Efficiency Yipin Guo Yilin Lang Qinyuan Ren MQ 24 1 0 03 Jul 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 39 0 0 03 Jul 2024
VcLLM: Video Codecs are Secretly Tensor Codecs Ceyu Xu Yongji Wu Xinyu Yang Beidi Chen Matthew Lentz Danyang Zhuo Lisa Wu Wills 50 0 0 29 Jun 2024
Teola: Towards End-to-End Optimization of LLM-based Applications Xin Tan Yimin Jiang Yitao Yang Hong-Yu Xu 73 5 0 29 Jun 2024
InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management Wonbeom Lee Jungi Lee Junghwan Seo Jaewoong Sim RALM 34 75 0 28 Jun 2024
LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference Dong Liu Meng Jiang MQ 38 12 0 28 Jun 2024
OutlierTune: Efficient Channel-Wise Quantization for Large Language Models Jinguang Wang Yuexi Yin Haifeng Sun Qi Qi Jingyu Wang Zirui Zhuang Tingting Yang Jianxin Liao 46 2 0 27 Jun 2024
MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool Cunchen Hu Heyang Huang Junhao Hu Jiang Xu Xusheng Chen ... Chenxi Wang Sa Wang Yungang Bao Ninghui Sun Yizhou Shan LLMAG 53 23 0 25 Jun 2024
Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels Razvan-Gabriel Dumitru Vikas Yadav Rishabh Maheshwary Paul-Ioan Clotan Sathwik Tejaswi Madhusudhan Mihai Surdeanu MQ 46 2 0 25 Jun 2024
Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers Lei Chen Yuan Meng Chen Tang Xinzhu Ma Jingyan Jiang Xin Wang Zhi Wang Wenwu Zhu MQ 31 23 0 25 Jun 2024
T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge Jianyu Wei Shijie Cao Ting Cao Lingxiao Ma Lei Wang Yanyong Zhang Mao Yang MQ 53 11 0 25 Jun 2024
From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models Sean Welleck Amanda Bertsch Matthew Finlayson Hailey Schoelkopf Alex Xie Graham Neubig Ilia Kulikov Zaid Harchaoui 33 51 0 24 Jun 2024
Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs Ashwinee Panda Berivan Isik Xiangyu Qi Sanmi Koyejo Tsachy Weissman Prateek Mittal MoMe 45 14 0 24 Jun 2024
Evaluation of Language Models in the Medical Context Under Resource-Constrained Settings Andrea Posada Daniel Rueckert Felix Meissen Philip Muller LM&MA ELM 37 0 0 24 Jun 2024
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging Deyuan Liu Zhan Qin Han Wang Zhao Yang Zecheng Wang ... Zhao Lv Zhiying Tu Dianhui Chu Bo Li Dianbo Sui 29 2 0 24 Jun 2024
MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression Tianyu Fu Haofeng Huang Xuefei Ning Genghan Zhang Boju Chen ... Shiyao Li Shengen Yan Guohao Dai Huazhong Yang Yu Wang MQ 52 17 0 21 Jun 2024
SDQ: Sparse Decomposed Quantization for LLM Inference Geonhwa Jeong Po-An Tsai S. Keckler Tushar Krishna MQ 45 3 0 19 Jun 2024
Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving Ke Cheng Wen Hu Zhi Wang Hongen Peng Jianguo Li Sheng Zhang 57 7 0 19 Jun 2024
VELO: A Vector Database-Assisted Cloud-Edge Collaborative LLM QoS Optimization Framework Zhi Yao Zhiqing Tang Jiong Lou Ping Shen Weijia Jia 48 7 0 19 Jun 2024
BoA: Attention-aware Post-training Quantization without Backpropagation Junhan Kim Ho-Young Kim Eulrang Cho Chungman Lee Joonyoung Kim Yongkweon Jeon MQ 38 0 0 19 Jun 2024
Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates Cristian Meo Ksenia Sycheva Anirudh Goyal Justin Dauwels MQ 29 4 0 18 Jun 2024
Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference Donghyeon Joo Ramyad Hadidi S. Feizi Bahar Asgari MQ 33 0 0 17 Jun 2024
SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention Qianchao Zhu Jiangfei Duan Chang Chen Siran Liu Xiuhong Li ... Huanqi Cao Xiao Chuanfu Xingcheng Zhang Dahua Lin Chao Yang 30 16 0 17 Jun 2024
ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking Wenshuo Li Xinghao Chen Han Shu Yehui Tang Yunhe Wang MQ 47 2 0 17 Jun 2024
An Analysis on Quantizing Diffusion Transformers Yuewei Yang Jialiang Wang Xiaoliang Dai Peizhao Zhang Hongbo Zhang MQ 37 1 0 16 Jun 2024
Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models Dominik Wagner Ilja Baumann Korbinian Riedhammer Tobias Bocklet MQ 37 1 0 16 Jun 2024
Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization Jungi Lee Wonbeom Lee Jaewoong Sim MQ 46 14 0 16 Jun 2024
Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox Yijun Liu Yuan Meng Fang Wu Shenhao Peng Hang Yao Chaoyu Guan Chen Tang Xinzhu Ma Zhi Wang Wenwu Zhu MQ 62 7 0 15 Jun 2024
ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models Jing Liu Ruihao Gong Mingyang Zhang Yefei He Jianfei Cai Bohan Zhuang MoE 45 0 0 13 Jun 2024