SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

18 November 2022

Song Han

Papers citing "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models"

50 / 534 papers shown

Title
QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving Chengyue Wu Haotian Tang Shang Yang Zhekai Zhang Guangxuan Xiao Chuang Gan Song Han 90 77 0 07 May 2024
Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer Huihong Shi Haikuo Shao Wendong Mao Zhongfeng Wang ViT MQ 44 3 0 06 May 2024
Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment Abhinav Agarwalla Abhay Gupta Alexandre Marques Shubhra Pandit Michael Goin ... Tuan Nguyen Mahmoud Salem Dan Alistarh Sean Lie Mark Kurtz MoE SyDa 45 11 0 06 May 2024
PTQ4SAM: Post-Training Quantization for Segment Anything Chengtao Lv Hong Chen Jinyang Guo Yifu Ding Xianglong Liu VLM MQ 31 14 0 06 May 2024
Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs Jordan Dotzel Yuzong Chen Bahaa Kotb Sushma Prasad Gang Wu Sheng Li Mohamed S. Abdelfattah Zhiru Zhang 31 9 0 06 May 2024
Torch2Chip: An End-to-end Customizable Deep Neural Network Compression and Deployment Toolkit for Prototype Hardware Accelerator Design Jian Meng Yuan Liao Anupreetham Anupreetham Ahmed Hassan Shixing Yu Han-Sok Suh Xiaofeng Hu Jae-sun Seo MQ 49 2 0 02 May 2024
Investigating Automatic Scoring and Feedback using Large Language Models G. Katuka Alexander Gain Yen-Yun Yu AI4Ed ALM 33 3 0 01 May 2024
Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey Dayou Du Gu Gong Xiaowen Chu MQ 38 7 0 01 May 2024
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting Fangcheng Liu Yehui Tang Zhenhua Liu Yunsheng Ni Kai Han Yunhe Wang 46 24 0 29 Apr 2024
PatentGPT: A Large Language Model for Intellectual Property Zilong Bai Ruiji Zhang Linqing Chen Qijun Cai Yuan Zhong ... Fu Bian Xiaolong Gu Lisha Zhang Weilei Wang Changyang Tu 49 5 0 28 Apr 2024
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding Mostafa Elhoushi Akshat Shrivastava Diana Liskovich Basil Hosmer Bram Wasti ... Saurabh Agarwal Ahmed Roman Ahmed Aly Beidi Chen Carole-Jean Wu LRM 41 86 0 25 Apr 2024
How to Parameterize Asymmetric Quantization Ranges for Quantization-Aware Training Jaeseong You Minseop Park Kyunggeun Lee Seokjun An Chirag I. Patel Markus Nagel MQ 46 1 0 25 Apr 2024
zkLLM: Zero Knowledge Proofs for Large Language Models Haochen Sun Jason Li Hongyang Zhang ALM 36 22 0 24 Apr 2024
Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity Tyler Griggs Xiaoxuan Liu Jiaxiang Yu Doyoung Kim Wei-Lin Chiang Alvin Cheung Ion Stoica 54 16 0 22 Apr 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 86 0 22 Apr 2024
An empirical study of LLaMA3 quantization: from LLMs to MLLMs Wei Huang Xingyu Zheng Xudong Ma Haotong Qin Chengtao Lv Hong Chen Jie Luo Xiaojuan Qi Xianglong Liu Michele Magno MQ 63 38 0 22 Apr 2024
MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts Dengchun Li Yingzi Ma Naizheng Wang Zhengmao Ye Zhiyuan Cheng ... Yan Zhang Lei Duan Jie Zuo Cal Yang Mingjie Tang MoE 40 44 0 22 Apr 2024
decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points Yi Guo Fanliu Kong Xiaoyang Li Hui Li Wei Chen Xiaogang Tian Jinping Cai Yang Zhang Shouda Liu MQ 34 6 0 19 Apr 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 46 49 0 18 Apr 2024
Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models Siyan Zhao Daniel Israel Mathias Niepert Aditya Grover KELM VLM 36 5 0 15 Apr 2024
Exploring and Improving Drafts in Blockwise Parallel Decoding Taehyeon Kim A. Suresh Kishore Papineni Michael Riley Sanjiv Kumar Adrian Benton AI4TS 52 2 0 14 Apr 2024
Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation Thomas Merth Qichen Fu Mohammad Rastegari Mahyar Najibi LRM RALM 39 9 0 10 Apr 2024
CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers Longwei Zou Qingyang Wang Han Zhao Jiangang Kong Yi Yang Yangdong Deng 50 0 0 10 Apr 2024
Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models Bowen Pan Songlin Yang Haokun Liu Mayank Mishra Gaoyuan Zhang Aude Oliva Colin Raffel Yikang Shen MoE 46 19 0 08 Apr 2024
What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models Busayo Awobade Mardiyyah Oduwole Steven Kolawole 52 0 0 06 Apr 2024
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization Aniruddha Nrusimha Mayank Mishra Naigang Wang Dan Alistarh Yikang Shen Yoon Kim MQ 68 9 0 04 Apr 2024
Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models Wanyun Cui Qianle Wang MQ 42 2 0 03 Apr 2024
Minimize Quantization Output Error with Bias Compensation Cheng Gong Haoshuai Zheng Mengting Hu Zheng Lin Deng-Ping Fan Yuzhi Zhang Tao Li MQ 38 2 0 02 Apr 2024
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs Saleh Ashkboos Amirkeivan Mohtashami Maximilian L. Croci Bo Li Martin Jaggi Dan Alistarh Torsten Hoefler James Hensman MQ 36 142 0 30 Mar 2024
Accurate Block Quantization in LLMs with Outliers Nikita Trukhanov I. Soloveychik MQ 28 4 0 29 Mar 2024
Tiny Machine Learning: Progress and Futures Ji Lin Ligeng Zhu Wei-Ming Chen Wei-Chen Wang Song Han 52 51 0 28 Mar 2024
Recommendation of data-free class-incremental learning algorithms by simulating future data Eva Feillet Adrian Daniel Popescu C´eline Hudelot 54 0 0 26 Mar 2024
The Need for Speed: Pruning Transformers with One Recipe Samir Khaki Konstantinos N. Plataniotis 42 10 0 26 Mar 2024
The Unreasonable Ineffectiveness of the Deeper Layers Andrey Gromov Kushal Tirumala Hassan Shapourian Paolo Glorioso Daniel A. Roberts 54 83 0 26 Mar 2024
Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models Qiong Wu Weihao Ye Yiyi Zhou Xiaoshuai Sun Rongrong Ji MoE 52 1 0 22 Mar 2024
AffineQuant: Affine Transformation Quantization for Large Language Models Yuexiao Ma Huixia Li Xiawu Zheng Feng Ling Xuefeng Xiao Rui Wang Shilei Wen Rongrong Ji Rongrong Ji MQ 43 20 0 19 Mar 2024
Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization Haocheng Xi Yuxiang Chen Kang Zhao Kaijun Zheng Jianfei Chen Jun Zhu MQ 45 21 0 19 Mar 2024
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression Junyuan Hong Jinhao Duan Chenhui Zhang Zhangheng Li Chulin Xie ... B. Kailkhura Dan Hendrycks Dawn Song Zhangyang Wang Bo-wen Li 39 25 0 18 Mar 2024
ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference Hyungjun Oh Kihong Kim Jaemin Kim Sungkyun Kim Junyeol Lee Du-Seong Chang Jiwon Seo 41 28 0 15 Mar 2024
Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs Ben Athiwaratkun Sujan Kumar Gonugondla Sanjay Krishna Gouda Haifeng Qian Hantian Ding ... Liangfu Chen Parminder Bhatia Ramesh Nallapati Sudipta Sengupta Bing Xiang 59 4 0 13 Mar 2024
CHAI: Clustered Head Attention for Efficient LLM Inference Saurabh Agarwal Bilge Acun Basil Homer Mostafa Elhoushi Yejin Lee Shivaram Venkataraman Dimitris Papailiopoulos Carole-Jean Wu 58 8 0 12 Mar 2024
IM-Unpack: Training and Inference with Arbitrarily Low Precision Integers Zhanpeng Zeng Karthikeyan Sankaralingam Vikas Singh 58 1 0 12 Mar 2024
SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression Xin Wang Yu Zheng Zhongwei Wan Mi Zhang MQ 57 44 0 12 Mar 2024
COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization Aozhong Zhang Zi Yang Naigang Wang Yingyong Qin Jack Xin Xin Li Penghang Yin VLM MQ 46 3 0 11 Mar 2024
QuantTune: Optimizing Model Quantization with Adaptive Outlier-Driven Fine Tuning Jiun-Man Chen Yu-Hsuan Chao Yu-Jie Wang Ming-Der Shieh Chih-Chung Hsu Wei-Fen Lin MQ 40 1 0 11 Mar 2024
FrameQuant: Flexible Low-Bit Quantization for Transformers Harshavardhan Adepu Zhanpeng Zeng Li Zhang Vikas Singh MQ 45 6 0 10 Mar 2024
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM Hao Kang Qingru Zhang Souvik Kundu Geonhwa Jeong Zaoxing Liu Tushar Krishna Tuo Zhao MQ 43 79 0 08 Mar 2024
Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities Kaiwen Cai Zhekai Duan Gaowen Liu Charles Fleming Chris Xiaoxuan Lu VLM 41 4 0 07 Mar 2024
Do You Trust Your Model? Emerging Malware Threats in the Deep Learning Ecosystem Dorjan Hitaj Giulio Pagnotta Fabio De Gaspari Sediola Ruko Briland Hitaj Luigi V. Mancini Fernando Perez-Cruz 42 4 0 06 Mar 2024
Large Language Model-Based Evolutionary Optimizer: Reasoning with elitism Shuvayan Brahmachary Subodh M. Joshi Aniruddha Panda K. Koneripalli A. Sagotra Harshil Patel Ankush Sharma Ameya Dilip Jagtap Kaushic Kalyanaraman LRM 52 19 0 04 Mar 2024