SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

18 November 2022

Song Han

Papers citing "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models"

50 / 533 papers shown

Title
Scaling Laws for Floating Point Quantization Training Xingchen Sun Shuaipeng Li Ruobing Xie Weidong Han Kan Wu ... Yangyu Tao Zhanhui Kang C. Xu Di Wang Jie Jiang MQ AIFin 62 1 0 05 Jan 2025
FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance Haicheng Wang Zhemeng Yu Gabriele Spadaro Chen Ju Victor Quétu Enzo Tartaglione Enzo Tartaglione VLM 161 3 0 05 Jan 2025
Pushing the Envelope of Low-Bit LLM via Dynamic Error Compensation Y. Park Jake Hyun Hojoon Kim Jae W. Lee MQ 46 0 0 31 Dec 2024
PTQ4VM: Post-Training Quantization for Visual Mamba Younghyun Cho Changhun Lee Seonggon Kim Eunhyeok Park MQ Mamba 46 2 0 29 Dec 2024
LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment Binrui Zeng Bin Ji Xiaodong Liu Jie Yu Shasha Li Jun Ma Xiaopeng Li Shangwen Wang Xinran Hong Yongtao Tang MQ 42 1 0 24 Dec 2024
TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers Savitha Viswanadh Kandala Pramuka Medaranga Ambuj Varshney 76 2 0 19 Dec 2024
Deploying Foundation Model Powered Agent Services: A Survey Wenchao Xu Jinyu Chen Peirong Zheng Xiaoquan Yi Tianyi Tian ... Quan Wan Yining Qi Yunfeng Fan Qinliang Su Xuemin Shen AI4CE 119 1 0 18 Dec 2024
Accelerating Retrieval-Augmented Generation Derrick Quinn Mohammad Nouri Neel Patel John Salihu Alireza Salemi Sukhan Lee Hamed Zamani Mohammad Alian RALM 3DV 90 4 0 14 Dec 2024
SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization Runsheng Bai Qiang Liu B. Liu MQ 72 1 0 05 Dec 2024
PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation Ao Wang Hui Chen Jianchao Tan Kaipeng Zhang Xunliang Cai Zijia Lin J. Han Guiguang Ding VLM 77 3 0 04 Dec 2024
Multi-Bin Batching for Increasing LLM Inference Throughput Ozgur Guldogan Jackson Kunde Kangwook Lee Ramtin Pedarsani LRM 70 2 0 03 Dec 2024
MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache Akshat Sharma Hangliang Ding Jianping Li Neel Dani Minjia Zhang 80 1 0 27 Nov 2024
SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors M. Rakka Jiyang Li Guohao Dai A. Eltawil M. Fouda Fadi J. Kurdahi 72 1 0 26 Nov 2024
PIM-AI: A Novel Architecture for High-Efficiency LLM Inference Cristobal Ortega Yann Falevoz Renaud Ayrignac 86 1 0 26 Nov 2024
Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format Chao Fang Man Shi Robin Geens Arne Symons Zhongfeng Wang Marian Verhelst 76 0 0 24 Nov 2024
Nimbus: Secure and Efficient Two-Party Inference for Transformers Zhengyi Li Kang Yang Jin Tan Wen-jie Lu Haoqi Wu ... Yu Yu Derun Zhao Yancheng Zheng M. Guo Jingwen Leng 74 2 0 24 Nov 2024
UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages Bethel Melesse Tessema Akhil Kedia Tae-Sun Chung 77 0 0 21 Nov 2024
FuseGPT: Learnable Layers Fusion of Generative Pre-trained Transformers Zehua Pei Hui-Ling Zhen Xianzhi Yu Sinno Jialin Pan M. Yuan Bei Yu AI4CE 89 0 0 21 Nov 2024
BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration Yuzong Chen Ahmed F. AbouElhamayed Xilai Dai Yang Wang Marta Andronic George A. Constantinides Mohamed S. Abdelfattah MQ 108 1 0 18 Nov 2024
Towards Accurate and Efficient Sub-8-Bit Integer Training Wenjin Guo Donglai Liu Weiying Xie Yunsong Li Xuefei Ning Zihan Meng Shulin Zeng Jie Lei Zhenman Fang Yu Wang MQ 39 1 0 17 Nov 2024
Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms Minghe Gao Wendong Bu Bingchen Miao Yang Wu Yunfei Li Juncheng Billy Li Siliang Tang Qi Wu Yueting Zhuang Meng Wang LM&Ro 53 3 0 17 Nov 2024
SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization Jintao Zhang Haofeng Huang Pengle Zhang Jia wei Jun-Jie Zhu Jianfei Chen VLM MQ 65 15 0 17 Nov 2024
SAM Decoding: Speculative Decoding via Suffix Automaton Yuxuan Hu Ke Wang Jing Zhang Fanjin Zhang C. Li Hongyu Chen Jing Zhang 52 2 0 16 Nov 2024
AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment Y. Fu Zhongzhi Yu Junwei Li Jiayi Qian Yongan Zhang Xiangchi Yuan Dachuan Shi Roman Yakunin Y. Lin 43 2 0 15 Nov 2024
AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference Janghwan Lee Jiwoong Park Jinseok Kim Yongjik Kim Jungju Oh Jinwook Oh Jungwook Choi 44 2 0 15 Nov 2024
The Super Weight in Large Language Models Mengxia Yu De Wang Qi Shan Colorado Reed Alvin Wan MQ MILM 42 10 0 11 Nov 2024
Scaling Laws for Precision Tanishq Kumar Zachary Ankner Benjamin Spector Blake Bordelon Niklas Muennighoff Mansheej Paul Cengiz Pehlevan Christopher Ré Aditi Raghunathan AIFin MoMe 54 14 0 07 Nov 2024
Interactions Across Blocks in Post-Training Quantization of Large Language Models Khasmamad Shabanovi Lukas Wiest Vladimir Golkov Daniel Cremers Thomas Pfeil MQ 33 1 0 06 Nov 2024
The Unreasonable Effectiveness of LLMs for Query Optimization Peter Akioyamen Zixuan Yi Ryan Marcus 34 2 0 05 Nov 2024
Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment Jason Vega Junsheng Huang Gaokai Zhang Hangoo Kang Minjia Zhang Gagandeep Singh 39 0 0 05 Nov 2024
DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution Yang Yue Yulin Wang Bingyi Kang Yizeng Han Shenzhi Wang Shiji Song Jiashi Feng Gao Huang VLM 45 16 0 04 Nov 2024
Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference Shashank Nag Alan T. L. Bacellar Zachary Susskind Anshul Jha Logan Liberty ... Krishnan Kailas P. Lima Neeraja J. Yadwadkar F. M. G. França L. John 43 0 0 04 Nov 2024
Two-Timescale Model Caching and Resource Allocation for Edge-Enabled AI-Generated Content Services Zhang Liu Hongyang Du Xiangwang Hou Lianfen Huang Seyyedali Hosseinalipour Dusit Niyato K. B. Letaief DiffM 49 1 0 03 Nov 2024
HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference Peng Tang Jiacheng Liu X. Hou Yifei Pu Jing Wang Pheng-Ann Heng Chong Li M. Guo MoE 72 7 0 03 Nov 2024
NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference Xuanlin Jiang Yang Zhou Shiyi Cao Ion Stoica Minlan Yu 50 8 0 02 Nov 2024
A Comprehensive Study on Quantization Techniques for Large Language Models Jiedong Lang Zhehao Guo Shuyu Huang MQ 44 10 0 30 Oct 2024
LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment Ge Yang Changyi He J. Guo Jianyu Wu Yifu Ding Aishan Liu Haotong Qin Pengliang Ji Xianglong Liu MQ 33 4 0 28 Oct 2024
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks Yongchang Hao Yanshuai Cao Lili Mou MQ 36 2 0 28 Oct 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Yiyuan Ma Wenlei Bao Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 88 16 0 28 Oct 2024
Ripple: Accelerating LLM Inference on Smartphones with Correlation-Aware Neuron Management Tuowei Wang Ruwen Fan Minxing Huang Zixu Hao Kun Li Ting Cao Youyou Lu Yaoxue Zhang Ju Ren 53 2 0 25 Oct 2024
Beware of Calibration Data for Pruning Large Language Models Yixin Ji Yang Xiang Juntao Li Qingrong Xia Ping Li Xinyu Duan Zhefeng Wang Min Zhang 42 2 0 23 Oct 2024
From Attention to Activation: Unravelling the Enigmas of Large Language Models Prannay Kaul Chengcheng Ma Ismail Elezi Jiankang Deng 34 2 0 22 Oct 2024
SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training Jinda Jia Cong Xie Hanlin Lu Daoce Wang Hao Feng ... Baixi Sun Yanghua Peng Zhi-Li Zhang Xin Liu Dingwen Tao MQ 30 4 0 20 Oct 2024
Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs Zifei Xu Sayeh Sharify W. Yazar T. Webb Xin Wang MQ 43 0 0 18 Oct 2024
Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs Tianyu Guo Druv Pai Yu Bai Jiantao Jiao Michael I. Jordan Song Mei 34 10 0 17 Oct 2024
Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching Jie Peng Zhang Cao Huaizhi Qu Zhengyu Zhang Chang Guo Yanyong Zhang Zhichao Cao Tianlong Chen 39 2 0 17 Oct 2024
AERO: Softmax-Only LLMs for Efficient Private Inference N. Jha Brandon Reagen 32 1 0 16 Oct 2024
Channel-Wise Mixed-Precision Quantization for Large Language Models Zihan Chen Bike Xie Jundong Li Cong Shen MQ 39 2 0 16 Oct 2024
Scaling laws for post-training quantized large language models Zifei Xu Alexander Lan W. Yazar T. Webb Sayeh Sharify Xin Wang MQ 35 0 0 15 Oct 2024
Sorted Weight Sectioning for Energy-Efficient Unstructured Sparse DNNs on Compute-in-Memory Crossbars Matheus Farias H. T. Kung 23 1 0 15 Oct 2024