SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

18 November 2022

Song Han

Papers citing "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models"

50 / 533 papers shown

Title
Error Diffusion: Post Training Quantization with Block-Scaled Number Formats for Neural Networks Alireza Khodamoradi K. Denolf Eric Dellinger MQ 39 0 0 15 Oct 2024
QSpec: Speculative Decoding with Complementary Quantization Schemes Juntao Zhao Wenhao Lu Sheng Wang Lingpeng Kong Chuan Wu MQ 74 5 0 15 Oct 2024
TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models Mu Cai Reuben Tan Jianrui Zhang Bocheng Zou Kai Zhang ... Yao Dou J. Park Jianfeng Gao Yong Jae Lee Jianwei Yang 44 13 0 14 Oct 2024
ControlMM: Controllable Masked Motion Generation Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue Chong Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 37 4 0 14 Oct 2024
SLaNC: Static LayerNorm Calibration Mahsa Salmani Nikita Trukhanov I. Soloveychik MQ 33 0 0 14 Oct 2024
big.LITTLE Vision Transformer for Efficient Visual Recognition He Guo Yulong Wang Zixuan Ye Jifeng Dai Yuwen Xiong ViT 54 0 0 14 Oct 2024
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models Haiquan Lu Yefan Zhou Shiwei Liu Zhangyang Wang Michael W. Mahoney Yaoqing Yang 37 1 0 14 Oct 2024
FlatQuant: Flatness Matters for LLM Quantization Yuxuan Sun Ruikang Liu Haoli Bai Han Bao Kang Zhao ... Lu Hou Chun Yuan Xin Jiang Wei Liu Jun Yao MQ 79 4 0 12 Oct 2024
CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression Wenyuan Liu Xindian Ma Peng Zhang Yan Wang MQ 29 1 0 10 Oct 2024
Q-VLM: Post-training Quantization for Large Vision-Language Models Changyuan Wang Ziwei Wang Xiuwei Xu Yansong Tang Jie Zhou Jiwen Lu MQ 32 2 0 10 Oct 2024
Scaling Laws for Mixed quantization in Large Language Models Zeyu Cao Cheng Zhang Pedro Gimenes Jianqiao Lu Jianyi Cheng Yiren Zhao MQ 33 1 0 09 Oct 2024
Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs Ruijia Niu D. Wu Rose Yu Yi Ma 33 1 0 09 Oct 2024
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 31 7 0 08 Oct 2024
Mixture Compressor for Mixture-of-Experts LLMs Gains More Wei Huang Yue Liao Jianhui Liu Ruifei He Haoru Tan Shiming Zhang Hongsheng Li Si Liu Xiaojuan Qi MoE 39 3 0 08 Oct 2024
ESPACE: Dimensionality Reduction of Activations for Model Compression Charbel Sakr Brucek Khailany 36 3 0 07 Oct 2024
Efficient Inference for Large Language Model-based Generative Recommendation Xinyu Lin Chaoqun Yang Wenjie Wang Yongqi Li Cunxiao Du Fuli Feng See-Kiong Ng Tat-Seng Chua 70 4 0 07 Oct 2024
RevMUX: Data Multiplexing with Reversible Adapters for Efficient LLM Batch Inference Yige Xu Xu Guo Zhiwei Zeng Chunyan Miao 39 0 0 06 Oct 2024
Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective Jinhao Li Jiaming Xu Shan Huang Yonghua Chen Wen Li ... Jiayi Pan Li Ding Hao Zhou Yu Wang Guohao Dai 62 17 0 06 Oct 2024
Continuous Approximations for Improving Quantization Aware Training of LLMs He Li Jianhang Hong Yuanzhuo Wu Snehal Adbol Zonglin Li MQ 29 1 0 06 Oct 2024
Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher Yong Guo Shulian Zhang Haolin Pan Jing Liu Yulun Zhang Jian Chen 38 0 0 05 Oct 2024
ARB-LLM: Alternating Refined Binarizations for Large Language Models Zhiteng Li Xinyu Yan Tianao Zhang Haotong Qin Dong Xie Jiang Tian Zhongchao Shi Linghe Kong Yulun Zhang Xiaokang Yang MQ 37 2 0 04 Oct 2024
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration Jintao Zhang Jia wei Pengle Zhang Jun-Jie Zhu Jun Zhu Jianfei Chen VLM MQ 84 19 0 03 Oct 2024
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices Yuxiang Huang Binhang Yuan Xu Han Chaojun Xiao Zhiyuan Liu RALM 87 1 0 02 Oct 2024
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression Jingcun Wang Yu-Guang Chen Ing-Chao Lin Bing Li Grace Li Zhang 35 4 0 02 Oct 2024
Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference Ke Yi Zengke Liu Jianwei Zhang Chengyuan Li Tong Zhang Junyang Lin Jingren Zhou MQ 48 1 0 30 Sep 2024
The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems Linke Song Zixuan Pang Wenhao Wang Zihao Wang XiaoFeng Wang Hongbo Chen Wei Song Yier Jin Dan Meng Rui Hou 56 7 0 30 Sep 2024
Accumulator-Aware Post-Training Quantization Ian Colbert Fabian Grob Giuseppe Franco Jinjie Zhang Rayan Saab MQ 35 4 0 25 Sep 2024
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models Yifei Liu Jicheng Wen Yang Wang Shengyu Ye Li Lyna Zhang Ting Cao Cheng Li Mao Yang MQ 108 10 0 25 Sep 2024
PTQ4RIS: Post-Training Quantization for Referring Image Segmentation Xiaoyan Jiang Hang Yang Kaiying Zhu Xihe Qiu Shibo Zhao Sifan Zhou MQ 26 0 0 25 Sep 2024
Ascend HiFloat8 Format for Deep Learning Yuanyong Luo Zhongxing Zhang Richard Wu Hu Liu Ying Jin ... Korviakov Vladimir Bobrin Maxim Yuhao Hu Guanfu Chen Zeyi Huang MQ 30 1 0 25 Sep 2024
Multi-Modal Generative AI: Multi-modal LLM, Diffusion and Beyond Hong Chen Xin Wang Yuwei Zhou Bin Huang Yipeng Zhang Wei Feng Houlun Chen Zeyang Zhang Siao Tang Wenwu Zhu DiffM 55 7 0 23 Sep 2024
DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation Xuewen Liu Zhikai Li Qingyi Gu MQ 34 4 0 22 Sep 2024
CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information Yuxin Wang Minghua Ma Zekun Wang Jingchang Chen Huiming Fan Liping Shan Qing Yang Dongliang Xu Ming Liu Bing Qin 38 3 0 20 Sep 2024
Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview Yanshu Wang Tong Yang Xiyan Liang Guoan Wang Hanning Lu Xu Zhe Yaoming Li Li Weitao MQ 44 3 0 18 Sep 2024
DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing Zhenyuan Dong Sai Qian Zhang MQ 33 3 0 12 Sep 2024
Understanding Foundation Models: Are We Back in 1924? Alan F. Smeaton AI4CE 43 2 0 11 Sep 2024
Shaping the Future of Endangered and Low-Resource Languages -- Our Role in the Age of LLMs: A Keynote at ECIR 2024 Josiane Mothe 39 2 0 05 Sep 2024
Recent Advances in Attack and Defense Approaches of Large Language Models Jing Cui Yishi Xu Zhewei Huang Shuchang Zhou Jianbin Jiao Junge Zhang PILM AAML 57 1 0 05 Sep 2024
Foundations of Large Language Model Compression -- Part 1: Weight Quantization Sean I. Young MQ 50 1 0 03 Sep 2024
CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification Junhui He Shangyu Wu Weidong Wen Chun Jason Xue Qingan Li 21 5 0 02 Sep 2024
Hyper-Compression: Model Compression via Hyperfunction Fenglei Fan Juntong Fan Dayang Wang Jingbo Zhang Zelin Dong Shijun Zhang Ge Wang Tieyong Zeng 32 0 0 01 Sep 2024
MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models Elias Frantar Roberto L. Castro Jiale Chen Torsten Hoefler Dan Alistarh MQ 24 12 0 21 Aug 2024
Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism Guanchen Li Xiandong Zhao Lian Liu Zeping Li Dong Li Lu Tian Jie He Ashish Sirasao E. Barsoum VLM 34 0 0 20 Aug 2024
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding Jian Chen Vashisth Tiwari Ranajoy Sadhukhan Zhuoming Chen Jinyuan Shi Ian En-Hsu Yen Ian En-Hsu Yen Avner May Tianqi Chen Beidi Chen LRM 39 22 0 20 Aug 2024
P/D-Serve: Serving Disaggregated Large Language Model at Scale Yibo Jin Tao Wang Huimin Lin Mingyang Song Peiyang Li ... Haoliang Cheng Xiaojing Li Jiandong Ding Hefei Guo Zhengyong Zhang MoE 41 10 0 15 Aug 2024
LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference Zhiwen Mo Lei Wang Jianyu Wei Zhichen Zeng Shijie Cao ... Naifeng Jing Ting Cao Jilong Xue Fan Yang Mao Yang 54 0 0 12 Aug 2024
Post-Training Sparse Attention with Double Sparsity Shuo Yang Ying Sheng Joseph E. Gonzalez Ion Stoica Lianmin Zheng 41 7 0 11 Aug 2024
LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale Jaehong Cho Minsu Kim Hyunmin Choi Guseul Heo Jongse Park 49 9 0 10 Aug 2024
Deeploy: Enabling Energy-Efficient Deployment of Small Language Models On Heterogeneous Microcontrollers Moritz Scherer Luka Macan Victor J. B. Jung Philip Wiese Luca Bompani Alessio Burrello Francesco Conti Luca Benini MoE 52 10 0 08 Aug 2024
Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations Leo Donisch Sigurd Schacht Carsten Lanquillon 30 2 0 06 Aug 2024