Stable and low-precision training for large-scale vision-language models

25 April 2023

Mitchell Wortsman

Tim Dettmers

Luke Zettlemoyer

Papers citing "Stable and low-precision training for large-scale vision-language models"

36 / 36 papers shown

Title
Quartet: Native FP4 Training Can Be Optimal for Large Language Models Roberto L. Castro Andrei Panferov Soroush Tabesh Oliver Sieberling Jiale Chen Mahdi Nikdan Saleh Ashkboos Dan Alistarh MQ 56 0 0 20 May 2025
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam Tianjin Huang Haotian Hu Zhenyu Zhang Gaojie Jin Xianrui Li ... Tianlong Chen Lu Liu Qingsong Wen Zhangyang Wang Shiwei Liu MQ 81 1 0 24 Feb 2025
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training Haocheng Xi Han Cai Ligeng Zhu Yaojie Lu Kurt Keutzer Jianfei Chen Song Han MQ 93 10 0 25 Oct 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 82 10 0 24 Jul 2024
Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detection Jia Guo Shuai Lu Weihang Zhang Huiqi Li Huiqi Li Hongen Liao ViT 106 12 0 23 May 2024
A Theory on Adam Instability in Large-Scale Machine Learning Igor Molybog Peter Albert Moya Chen Zach DeVito David Esiobu ... Puxin Xu Yuchen Zhang Melanie Kambadur Stephen Roller Susan Zhang AI4CE 59 33 0 19 Apr 2023
Effective Theory of Transformers at Initialization Emily Dinan Sho Yaida Susan Zhang 55 16 0 04 Apr 2023
Sigmoid Loss for Language Image Pre-Training Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer CLIP VLM 128 1,119 0 27 Mar 2023
Symbolic Discovery of Optimization Algorithms Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang ... Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le 136 373 0 13 Feb 2023
Dual PatchNorm Manoj Kumar Mostafa Dehghani N. Houlsby UQCV ViT 37 11 0 02 Feb 2023
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 347 2,377 0 09 Nov 2022
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng Zhang Yuxiao Dong Jie Tang BDL LRM 342 1,090 0 05 Oct 2022
FP8 Formats for Deep Learning Paulius Micikevicius Dusan Stosic N. Burgess Marius Cornea Pradeep Dubey ... Naveen Mellempudi S. Oberman Mohammad Shoeybi Michael Siu Hao Wu BDL VLM MQ 94 134 0 12 Sep 2022
Adaptive Gradient Methods at the Edge of Stability Jeremy M. Cohen Behrooz Ghorbani Shankar Krishnan Naman Agarwal Sourabh Medapati ... Daniel Suo David E. Cardoze Zachary Nado George E. Dahl Justin Gilmer ODL 80 53 0 29 Jul 2022
Hierarchical Text-Conditional Image Generation with CLIP Latents Aditya A. Ramesh Prafulla Dhariwal Alex Nichol Casey Chu Mark Chen VLM DiffM 339 6,830 0 13 Apr 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 414 6,202 0 05 Apr 2022
Automatic Mixed-Precision Quantization Search of BERT Changsheng Zhao Ting Hua Yilin Shen Qian Lou Hongxia Jin MQ 35 21 0 30 Dec 2021
Combined Scaling for Zero-shot Transfer Learning Hieu H. Pham Zihang Dai Golnaz Ghiasi Kenji Kawaguchi Hanxiao Liu ... Yi-Ting Chen Minh-Thang Luong Yonghui Wu Mingxing Tan Quoc V. Le VLM 58 198 0 19 Nov 2021
8-bit Optimizers via Block-wise Quantization Tim Dettmers M. Lewis Sam Shleifer Luke Zettlemoyer MQ 105 295 0 06 Oct 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 144 1,857 0 05 Apr 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 131 1,006 0 31 Mar 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 382 4,919 0 24 Feb 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 259 517 0 11 Feb 2021
FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference D. Khudia Jianyu Huang Protonu Basu Summer Deng Haixin Liu Jongsoo Park M. Smelyanskiy FedML MQ 81 46 0 13 Jan 2021
I-BERT: Integer-only BERT Quantization Sehoon Kim A. Gholami Z. Yao Michael W. Mahoney Kurt Keutzer MQ 137 351 0 05 Jan 2021
BinaryBERT: Pushing the Limit of BERT Quantization Haoli Bai Wei Zhang Lu Hou Lifeng Shang Jing Jin Xin Jiang Qun Liu Michael Lyu Irwin King MQ 187 225 0 31 Dec 2020
Training with Quantization Noise for Extreme Model Compression Angela Fan Pierre Stock Benjamin Graham Edouard Grave Remi Gribonval Hervé Jégou Armand Joulin MQ 68 245 0 15 Apr 2020
Binary Neural Networks: A Survey Haotong Qin Ruihao Gong Xianglong Liu Xiao Bai Jingkuan Song N. Sebe MQ 100 466 0 31 Mar 2020
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks Léopold Cambier Anahita Bhiwandiwalla Ting Gong M. Nekuii Oguz H. Elibol Hanlin Tang MQ 85 48 0 16 Jan 2020
Towards Unified INT8 Training for Convolutional Neural Network Feng Zhu Ruihao Gong F. Yu Xianglong Liu Yanfei Wang Zhelong Li Xiuqi Yang Junjie Yan MQ 70 151 0 29 Dec 2019
Mixed Precision Training With 8-bit Floating Point Naveen Mellempudi Sudarshan Srinivasan Dipankar Das Bharat Kaul MQ 37 69 0 29 May 2019
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost Noam M. Shazeer Mitchell Stern ODL 69 1,043 0 11 Apr 2018
Training DNNs with Hybrid Block Floating Point M. Drumond Tao R. Lin Martin Jaggi Babak Falsafi 45 96 0 04 Apr 2018
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference Benoit Jacob S. Kligys Bo Chen Menglong Zhu Matthew Tang Andrew G. Howard Hartwig Adam Dmitry Kalenichenko MQ 136 3,111 0 15 Dec 2017
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 288 8,091 0 13 Aug 2016
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 274 18,587 0 06 Feb 2015