SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

18 November 2022

Song Han

Papers citing "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models"

50 / 533 papers shown

Title
BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models Qingqing Cao Sewon Min Yizhong Wang Hannaneh Hajishirzi MQ RALM 40 4 0 02 Oct 2023
Do Compressed LLMs Forget Knowledge? An Experimental Study with Practical Implications Duc N. M. Hoang Minsik Cho Thomas Merth Mohammad Rastegari Zhangyang Wang KELM CLL 33 4 0 02 Oct 2023
Efficient Streaming Language Models with Attention Sinks Michel Lang Yuandong Tian Beidi Chen Song Han Mike Lewis AI4TS RALM 44 659 0 29 Sep 2023
PB-LLM: Partially Binarized Large Language Models Yuzhang Shang Zhihang Yuan Qiang Wu Zhen Dong MQ 31 43 0 29 Sep 2023
Training and inference of large language models using 8-bit floating point Sergio P. Perez Yan Zhang James Briggs Charlie Blake Prashanth Krishnamurthy Paul Balanca Carlo Luschi Stephen Barlow Andrew William Fitzgibbon MQ 31 18 0 29 Sep 2023
ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers Junjie Yin Jiahao Dong Yingheng Wang Christopher De Sa Volodymyr Kuleshov MQ 33 4 0 28 Sep 2023
Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models Jung Hwan Heo Jeonghoon Kim Beomseok Kwon Byeongwook Kim Se Jung Kwon Dongsoo Lee MQ 43 9 0 27 Sep 2023
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models Yuhui Xu Lingxi Xie Xiaotao Gu Xin Chen Heng Chang Hengheng Zhang Zhensu Chen Xiaopeng Zhang Qi Tian MQ 21 92 0 26 Sep 2023
Efficient Post-training Quantization with FP8 Formats Haihao Shen Naveen Mellempudi Xin He Q. Gao Chang‐Bao Wang Mengni Wang MQ 23 19 0 26 Sep 2023
LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression Ayush Kaushal Tejas Vaidhya Irina Rish 64 15 0 25 Sep 2023
Scaling Laws for Sparsely-Connected Foundation Models Elias Frantar C. Riquelme N. Houlsby Dan Alistarh Utku Evci 35 36 0 15 Sep 2023
Large Language Models for Compiler Optimization Chris Cummins Volker Seeker Dejan Grubisic Mostafa Elhoushi Youwei Liang ... Jonas Gehring Fabian Gloeckle Kim M. Hazelwood Gabriel Synnaeve Hugh Leather 26 48 0 11 Sep 2023
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs Wenhua Cheng Weiwei Zhang Haihao Shen Yiyang Cai Xin He Kaokao Lv Yi. Liu MQ 36 22 0 11 Sep 2023
Understanding the Impact of Post-Training Quantization on Large Language Models Somnath Roy MQ 38 3 0 11 Sep 2023
Norm Tweaking: High-performance Low-bit Quantization of Large Language Models Liang Li Qingyuan Li Bo-Wen Zhang Xiangxiang Chu MQ 47 29 0 06 Sep 2023
QuantEase: Optimization-based Quantization for Language Models Kayhan Behdin Ayan Acharya Aman Gupta Qingquan Song Siyu Zhu S. Keerthi Rahul Mazumder MQ 30 20 0 05 Sep 2023
Concepts is All You Need: A More Direct Path to AGI Peter Voss M. Jovanovic 25 4 0 04 Sep 2023
Memory Efficient Optimizers with 4-bit States Bingrui Li Jianfei Chen Jun Zhu MQ 30 34 0 04 Sep 2023
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills Amey Agrawal Ashish Panwar Jayashree Mohan Nipun Kwatra Bhargav S. Gulavani Ramachandran Ramjee AI4TS LRM 39 94 0 31 Aug 2023
FPTQ: Fine-grained Post-Training Quantization for Large Language Models Qingyuan Li Yifan Zhang Liang Li Peng Yao Bo-Wen Zhang Xiangxiang Chu Yerui Sun Li-Qiang Du Yuchen Xie MQ 49 12 0 30 Aug 2023
Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models Kaiyuan Gao Su He Zhenyu He Jiacheng Lin Qizhi Pei Jie Shao Wei Zhang LM&MA SyDa 38 4 0 27 Aug 2023
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models Wenqi Shao Yonghong Tian Zhaoyang Zhang Peng Xu Lirui Zhao Zhiqiang Li Kaipeng Zhang Peng Gao Yu Qiao Ping Luo MQ 30 178 0 25 Aug 2023
FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs Young Jin Kim Rawn Henry Raffy Fahim Hany Awadalla MQ 37 19 0 16 Aug 2023
A Survey on Model Compression for Large Language Models Xunyu Zhu Jian Li Yong Liu Can Ma Weiping Wang 36 193 0 15 Aug 2023
Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling Marija vSakota Maxime Peyrard Robert West 29 46 0 11 Aug 2023
Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs Or Sharir Anima Anandkumar 32 0 0 27 Jul 2023
QuIP: 2-Bit Quantization of Large Language Models With Guarantees Jerry Chee Yaohui Cai Volodymyr Kuleshov Chris De Sa MQ 51 189 0 25 Jul 2023
ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats Xiaoxia Wu Z. Yao Yuxiong He MQ 35 43 0 19 Jul 2023
A Survey of Techniques for Optimizing Transformer Inference Krishna Teja Chitty-Venkata Sparsh Mittal M. Emani V. Vishwanath Arun Somani 45 62 0 16 Jul 2023
EFL Students' Attitudes and Contradictions in a Machine-in-the-loop Activity System David James Woo Hengky Susanto Kai Guo AI4CE 19 1 0 13 Jul 2023
A Comprehensive Overview of Large Language Models Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Naveed Akhtar Nick Barnes Ajmal Mian OffRL 70 538 0 12 Jul 2023
QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models Tommaso Pegolotti Elias Frantar Dan Alistarh Markus Püschel MQ 24 3 0 07 Jul 2023
INT-FP-QSim: Mixed Precision and Formats For Large Language Models and Vision Transformers Lakshmi Nair Mikhail Bernadskiy Arulselvan Madhavan Craig Chan Ayon Basumallik D. Bunandar MQ 46 2 0 07 Jul 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu Zhang Ying Sheng Dinesh Manocha Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 61 255 0 24 Jun 2023
Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing Yelysei Bondarenko Markus Nagel Tijmen Blankevoort MQ 23 87 0 22 Jun 2023
Training Transformers with 4-bit Integers Haocheng Xi Changhao Li Jianfei Chen Jun Zhu MQ 25 48 0 21 Jun 2023
A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun Zhuang Liu Anna Bair J. Zico Kolter 82 359 0 20 Jun 2023
SqueezeLLM: Dense-and-Sparse Quantization Sehoon Kim Coleman Hooper A. Gholami Zhen Dong Xiuyu Li Sheng Shen Michael W. Mahoney Kurt Keutzer MQ 36 168 0 13 Jun 2023
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression Tim Dettmers Ruslan Svirschevski Vage Egiazarian Denis Kuznedelev Elias Frantar Saleh Ashkboos Alexander Borzunov Torsten Hoefler Dan Alistarh MQ 37 231 0 05 Jun 2023
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration Ji Lin Jiaming Tang Haotian Tang Shang Yang Wei-Ming Chen Wei-Chen Wang Guangxuan Xiao Xingyu Dang Chuang Gan Song Han EDL MQ 41 474 0 01 Jun 2023
FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization J. H. Lee Jeonghoon Kim S. Kwon Dongsoo Lee MQ 28 33 0 01 Jun 2023
Intriguing Properties of Quantization at Scale Arash Ahmadian Saurabh Dash Hongyu Chen Bharat Venkitesh Stephen Gou Phil Blunsom Ahmet Üstün Sara Hooker MQ 54 38 0 30 May 2023
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models Zechun Liu Barlas Oğuz Changsheng Zhao Ernie Chang Pierre Stock Yashar Mehdad Yangyang Shi Raghuraman Krishnamoorthi Vikas Chandra MQ 60 190 0 29 May 2023
CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers Dachuan Shi Chaofan Tao Anyi Rao Zhendong Yang Chun Yuan Jiaqi Wang VLM 34 21 0 27 May 2023
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time Zichang Liu Aditya Desai Fangshuo Liao Weitao Wang Victor Xie Zhaozhuo Xu Anastasios Kyrillidis Anshumali Shrivastava 28 204 0 26 May 2023
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers Sotiris Anagnostidis Dario Pavllo Luca Biggio Lorenzo Noci Aurelien Lucchi Thomas Hofmann 42 53 0 25 May 2023
Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model Zirui Liu Guanchu Wang Shaochen Zhong Zhaozhuo Xu Daochen Zha ... Zhimeng Jiang Kaixiong Zhou V. Chaudhary Shuai Xu Xia Hu 47 12 0 24 May 2023
QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers Artidoro Pagnoni Ari Holtzman Luke Zettlemoyer ALM 73 2,361 0 23 May 2023
Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization Jeonghoon Kim J. H. Lee Sungdong Kim Joonsuk Park Kang Min Yoo S. Kwon Dongsoo Lee MQ 44 99 0 23 May 2023
Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models Yijia Zhang Lingran Zhao Shijie Cao Wenqiang Wang Ting Cao Fan Yang Mao Yang Shanghang Zhang Ningyi Xu MQ 29 17 0 21 May 2023