Mixed Precision Training With 8-bit Floating Point

29 May 2019

Papers citing "Mixed Precision Training With 8-bit Floating Point"

22 / 22 papers shown

Title
Optimizing Large Language Model Training Using FP4 Quantization Ruizhe Wang Yeyun Gong Xiao Liu Guoshuai Zhao Ziyue Yang Baining Guo Zhengjun Zha Peng Cheng MQ 151 12 0 28 Jan 2025
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 105 10 0 24 Jul 2024
Training Deep Neural Networks with 8-bit Floating Point Numbers Naigang Wang Jungwook Choi D. Brand Chia-Yu Chen K. Gopalakrishnan MQ 65 503 0 19 Dec 2018
Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq Oleksii Kuchaiev Boris Ginsburg Igor Gitman Vitaly Lavrukhin Jason Chun Lok Li Huyen Nguyen Carl Case Paulius Micikevicius VLM 59 49 0 25 May 2018
A Call for Clarity in Reporting BLEU Scores Matt Post 179 2,996 0 23 Apr 2018
NVIDIA Tensor Core Programmability, Performance & Precision Stefano Markidis Steven W. D. Chien Erwin Laure Ivy Bo Peng Jeffrey S. Vetter 42 374 0 11 Mar 2018
High-Accuracy Low-Precision Training Christopher De Sa Megan Leszczynski Jian Zhang Alana Marzoev Christopher R. Aberger K. Olukotun Christopher Ré 67 109 0 09 Mar 2018
Training and Inference with Integers in Deep Neural Networks Shuang Wu Guoqi Li F. Chen Luping Shi MQ 65 391 0 13 Feb 2018
Mixed Precision Training of Convolutional Neural Networks using Integer Operations Dipankar Das Naveen Mellempudi Dheevatsa Mudigere Dhiraj D. Kalamkar Sasikanth Avancha ... J. Corbal N. Shustrov R. Dubtsov Evarist Fomenko V. Pirogov MQ 63 154 0 03 Feb 2018
Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks Urs Koster T. Webb Xin Eric Wang Marcel Nassar Arjun K. Bansal ... Luke Hornof A. Khosrowshahi Carey Kloss Ruby J. Pai N. Rao MQ 47 262 0 06 Nov 2017
Mixed Precision Training Paulius Micikevicius Sharan Narang Jonah Alben G. Diamos Erich Elsen ... Boris Ginsburg Michael Houston Oleksii Kuchaiev Ganesh Venkatesh Hao Wu 174 1,805 0 10 Oct 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 786 132,363 0 12 Jun 2017
Deep Learning with Low Precision by Half-wave Gaussian Quantization Zhaowei Cai Xiaodong He Jian Sun Nuno Vasconcelos MQ 138 505 0 03 Feb 2017
Neural Architecture Search with Reinforcement Learning Barret Zoph Quoc V. Le 478 5,381 0 05 Nov 2016
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Zhiwen Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 911 6,796 0 26 Sep 2016
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations Itay Hubara Matthieu Courbariaux Daniel Soudry Ran El-Yaniv Yoshua Bengio MQ 155 1,868 0 22 Sep 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 429 2,945 0 15 Sep 2016
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients Shuchang Zhou Yuxin Wu Zekun Ni Xinyu Zhou He Wen Yuheng Zou MQ 129 2,090 0 20 Jun 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,426 0 10 Dec 2015
Deep Learning with Limited Numerical Precision Suyog Gupta A. Agrawal K. Gopalakrishnan P. Narayanan HAI 207 2,049 0 09 Feb 2015
Deep Speech: Scaling up end-to-end speech recognition Awni Y. Hannun Carl Case Jared Casper Bryan Catanzaro G. Diamos ... R. Prenger S. Satheesh Shubho Sengupta Adam Coates A. Ng 188 2,128 0 17 Dec 2014
Speech Recognition with Deep Recurrent Neural Networks Alex Graves Abdel-rahman Mohamed Geoffrey E. Hinton 230 8,526 0 22 Mar 2013