v1v2 (latest)

Training Transformers with 4-bit Integers

21 June 2023

Haocheng Xi

Changhao Li

Jianfei Chen

Jun Zhu

ArXiv (abs)PDF HTML

Papers citing "Training Transformers with 4-bit Integers"

35 / 35 papers shown

Title
Quartet: Native FP4 Training Can Be Optimal for Large Language Models Roberto L. Castro Andrei Panferov Soroush Tabesh Oliver Sieberling Jiale Chen Mahdi Nikdan Saleh Ashkboos Dan Alistarh MQ 82 0 0 20 May 2025
Pychop: Emulating Low-Precision Arithmetic in Numerical Methods and Neural Networks Erin Carson Xinye Chen 129 0 0 10 Apr 2025
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam Tianjin Huang Haotian Hu Zhenyu Zhang Gaojie Jin Xianrui Li ... Tianlong Chen Lu Liu Qingsong Wen Zhangyang Wang Shiwei Liu MQ 96 1 0 24 Feb 2025
GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning Sifan Zhou Shuo Wang Zhihang Yuan Mingjia Shi Yuzhang Shang Dawei Yang MQ ALM 166 0 0 18 Feb 2025
Optimizing Large Language Model Training Using FP4 Quantization Ruizhe Wang Yeyun Gong Xiao Liu Guoshuai Zhao Ziyue Yang Baining Guo Zhengjun Zha Peng Cheng MQ 151 12 0 28 Jan 2025
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training Haocheng Xi Han Cai Ligeng Zhu Yaojie Lu Kurt Keutzer Jianfei Chen Song Han MQ 127 10 0 25 Oct 2024
FlatQuant: Flatness Matters for LLM Quantization Yuxuan Sun Ruikang Liu Haoli Bai Han Bao Kang Zhao ... Lu Hou Chun Yuan Xin Jiang Wen Liu Jun Yao MQ 134 10 0 12 Oct 2024
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models Xiuying Wei Yunchen Zhang Xiangguo Zhang Ruihao Gong Shanghang Zhang Qi Zhang F. Yu Xianglong Liu MQ 96 152 0 27 Sep 2022
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 427 2,685 0 04 May 2021
I-BERT: Integer-only BERT Quantization Sehoon Kim A. Gholami Z. Yao Michael W. Mahoney Kurt Keutzer MQ 160 352 0 05 Jan 2021
BinaryBERT: Pushing the Limit of BERT Quantization Haoli Bai Wei Zhang Lu Hou Lifeng Shang Jing Jin Xin Jiang Qun Liu Michael Lyu Irwin King MQ 211 227 0 31 Dec 2020
Sharpness-Aware Minimization for Efficiently Improving Generalization Pierre Foret Ariel Kleiner H. Mobahi Behnam Neyshabur AAML 199 1,358 0 03 Oct 2020
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 186 1,600 0 30 Sep 2020
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions Zechun Liu Zhiqiang Shen Marios Savvides Kwang-Ting Cheng MQ 146 354 0 07 Mar 2020
Towards Unified INT8 Training for Convolutional Neural Network Feng Zhu Ruihao Gong F. Yu Xianglong Liu Yanfei Wang Zhelong Li Xiuqi Yang Junjie Yan MQ 88 152 0 29 Dec 2019
HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks Zhen Dong Z. Yao Yaohui Cai Daiyaan Arfeen A. Gholami Michael W. Mahoney Kurt Keutzer MQ 91 282 0 10 Nov 2019
Reducing Transformer Depth on Demand with Structured Dropout Angela Fan Edouard Grave Armand Joulin 120 596 0 25 Sep 2019
Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers Yukuan Yang Shuang Wu Lei Deng Tianyi Yan Yuan Xie Guoqi Li MQ 145 112 0 05 Sep 2019
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks Ruihao Gong Xianglong Liu Shenghu Jiang Tian-Hao Li Peng Hu Jiazhen Lin F. Yu Junjie Yan MQ 70 459 0 14 Aug 2019
Deep Learning Training on the Edge with Low-Precision Posits H. F. Langroudi Zachariah Carmichael Dhireesha Kudithipudi MQ 53 14 0 30 Jul 2019
Learned Step Size Quantization S. K. Esser J. McKinstry Deepika Bablani R. Appuswamy D. Modha MQ 75 809 0 21 Feb 2019
Improving Neural Network Quantization without Retraining using Outlier Channel Splitting Ritchie Zhao Yuwei Hu Jordan Dotzel Christopher De Sa Zhiru Zhang OODD MQ 99 311 0 28 Jan 2019
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks Dongqing Zhang Jiaolong Yang Dongqiangzi Ye G. Hua MQ 63 703 0 26 Jul 2018
Know What You Don't Know: Unanswerable Questions for SQuAD Pranav Rajpurkar Robin Jia Percy Liang RALM ELM 292 2,853 0 11 Jun 2018
Scalable Methods for 8-bit Training of Neural Networks Ron Banner Itay Hubara Elad Hoffer Daniel Soudry MQ 84 339 0 25 May 2018
Faster Neural Network Training with Approximate Tensor Operations Menachem Adelman Kfir Y. Levy Ido Hakimi M. Silberstein 66 27 0 21 May 2018
PACT: Parameterized Clipping Activation for Quantized Neural Networks Jungwook Choi Zhuo Wang Swagath Venkataramani P. Chuang Vijayalakshmi Srinivasan K. Gopalakrishnan MQ 68 955 0 16 May 2018
A Call for Clarity in Reporting BLEU Scores Matt Post 177 2,996 0 23 Apr 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,196 0 20 Apr 2018
Training DNNs with Hybrid Block Floating Point M. Drumond Tao R. Lin Martin Jaggi Babak Falsafi 60 98 0 04 Apr 2018
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference Benoit Jacob S. Kligys Bo Chen Menglong Zhu Matthew Tang Andrew G. Howard Hartwig Adam Dmitry Kalenichenko MQ 162 3,141 0 15 Dec 2017
Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights Aojun Zhou Anbang Yao Yiwen Guo Lin Xu Yurong Chen MQ 395 1,055 0 10 Feb 2017
Semi-Supervised Classification with Graph Convolutional Networks Thomas Kipf Max Welling GNN SSL 657 29,154 0 09 Sep 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 314 8,169 0 16 Jun 2016
Deep Networks with Stochastic Depth Gao Huang Yu Sun Zhuang Liu Daniel Sedra Kilian Q. Weinberger 215 2,361 0 30 Mar 2016