Unit Scaling: Out-of-the-Box Low-Precision Training

20 March 2023

Carlo Luschi

Papers citing "Unit Scaling: Out-of-the-Box Low-Precision Training"

21 / 21 papers shown

Title
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 68 10 0 24 Jul 2024
Constitutional AI: Harmlessness from AI Feedback Yuntao Bai Saurav Kadavath Sandipan Kundu Amanda Askell John Kernion ... Dario Amodei Nicholas Joseph Sam McCandlish Tom B. Brown Jared Kaplan SyDa MoMe 152 1,583 0 15 Dec 2022
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model A. Luccioni S. Viguier Anne-Laure Ligozat 84 272 0 03 Nov 2022
Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training Charbel Sakr Steve Dai Rangharajan Venkatesan B. Zimmer W. Dally Brucek Khailany MQ 42 41 0 13 Jun 2022
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer Greg Yang J. E. Hu Igor Babuschkin Szymon Sidor Xiaodong Liu David Farhi Nick Ryder J. Pachocki Weizhu Chen Jianfeng Gao 62 155 0 07 Mar 2022
Ethical and social risks of harm from Language Models Laura Weidinger John F. J. Mellor Maribeth Rauh Conor Griffin J. Uesato ... Lisa Anne Hendricks William S. Isaac Sean Legassick G. Irving Iason Gabriel PILM 64 1,009 0 08 Dec 2021
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts Alisa Liu Maarten Sap Ximing Lu Swabha Swayamdipta Chandra Bhagavatula Noah A. Smith Yejin Choi MU 85 369 0 07 May 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 252 514 0 11 Feb 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 406 1,868 0 14 Dec 2020
Recipes for Safety in Open-domain Chatbots Jing Xu Da Ju Margaret Li Y-Lan Boureau Jason Weston Emily Dinan 51 232 0 14 Oct 2020
Multi-node Bert-pretraining: Cost-efficient Approach Jiahuang Lin Xuelong Li Gennady Pekhimenko 23 13 0 01 Aug 2020
Dissecting the Graphcore IPU Architecture via Microbenchmarking Zhe Jia Blake Tillman Marco Maggioni D. Scarpazza 39 134 0 07 Dec 2019
A Study of BFLOAT16 for Deep Learning Training Dhiraj D. Kalamkar Dheevatsa Mudigere Naveen Mellempudi Dipankar Das K. Banerjee ... Sudarshan Srinivasan Abhisek Kundu M. Smelyanskiy Bharat Kaul Pradeep Dubey MQ 67 340 0 29 May 2019
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes Yang You Jing Li Sashank J. Reddi Jonathan Hseu Sanjiv Kumar Srinadh Bhojanapalli Xiaodan Song J. Demmel Kurt Keutzer Cho-Jui Hsieh ODL 163 991 0 01 Apr 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 140 3,714 0 09 Jan 2019
Know What You Don't Know: Unanswerable Questions for SQuAD Pranav Rajpurkar Robin Jia Percy Liang RALM ELM 192 2,830 0 11 Jun 2018
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference Benoit Jacob S. Kligys Bo Chen Menglong Zhu Matthew Tang Andrew G. Howard Hartwig Adam Dmitry Kalenichenko MQ 124 3,090 0 15 Dec 2017
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 168 2,814 0 26 Sep 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 251 10,412 0 21 Jul 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 153 8,067 0 16 Jun 2016
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 200 18,534 0 06 Feb 2015