Stable and low-precision training for large-scale vision-language models

25 April 2023

Mitchell Wortsman

Tim Dettmers

Luke Zettlemoyer

Papers citing "Stable and low-precision training for large-scale vision-language models"

29 / 29 papers shown

Title
Quartet: Native FP4 Training Can Be Optimal for Large Language Models Roberto L. Castro Andrei Panferov Soroush Tabesh Oliver Sieberling Jiale Chen Mahdi Nikdan Saleh Ashkboos Dan Alistarh MQ 40 0 0 20 May 2025
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam Tianjin Huang Haotian Hu Zhenyu Zhang Gaojie Jin Xianrui Li ... Tianlong Chen Lu Liu Qingsong Wen Zhangyang Wang Shiwei Liu MQ 79 1 0 24 Feb 2025
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training Haocheng Xi Han Cai Ligeng Zhu Yaojie Lu Kurt Keutzer Jianfei Chen Song Han MQ 93 9 0 25 Oct 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 74 10 0 24 Jul 2024
Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detection Jia Guo Shuai Lu Weihang Zhang Huiqi Li Huiqi Li Hongen Liao ViT 103 10 0 23 May 2024
A Theory on Adam Instability in Large-Scale Machine Learning Igor Molybog Peter Albert Moya Chen Zach DeVito David Esiobu ... Puxin Xu Yuchen Zhang Melanie Kambadur Stephen Roller Susan Zhang AI4CE 59 32 0 19 Apr 2023
Effective Theory of Transformers at Initialization Emily Dinan Sho Yaida Susan Zhang 52 15 0 04 Apr 2023
Sigmoid Loss for Language Image Pre-Training Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer CLIP VLM 116 1,076 0 27 Mar 2023
Symbolic Discovery of Optimization Algorithms Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang ... Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le 125 367 0 13 Feb 2023
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 319 2,364 0 09 Nov 2022
Adaptive Gradient Methods at the Edge of Stability Jeremy M. Cohen Behrooz Ghorbani Shankar Krishnan Naman Agarwal Sourabh Medapati ... Daniel Suo David E. Cardoze Zachary Nado George E. Dahl Justin Gilmer ODL 77 52 0 29 Jul 2022
Hierarchical Text-Conditional Image Generation with CLIP Latents Aditya A. Ramesh Prafulla Dhariwal Alex Nichol Casey Chu Mark Chen VLM DiffM 312 6,768 0 13 Apr 2022
Automatic Mixed-Precision Quantization Search of BERT Changsheng Zhao Ting Hua Yilin Shen Qian Lou Hongxia Jin MQ 35 20 0 30 Dec 2021
Combined Scaling for Zero-shot Transfer Learning Hieu H. Pham Zihang Dai Golnaz Ghiasi Kenji Kawaguchi Hanxiao Liu ... Yi-Ting Chen Minh-Thang Luong Yonghui Wu Mingxing Tan Quoc V. Le VLM 53 198 0 19 Nov 2021
8-bit Optimizers via Block-wise Quantization Tim Dettmers M. Lewis Sam Shleifer Luke Zettlemoyer MQ 101 286 0 06 Oct 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 125 998 0 31 Mar 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 339 4,873 0 24 Feb 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 256 514 0 11 Feb 2021
FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference D. Khudia Jianyu Huang Protonu Basu Summer Deng Haixin Liu Jongsoo Park M. Smelyanskiy FedML MQ 70 46 0 13 Jan 2021
I-BERT: Integer-only BERT Quantization Sehoon Kim A. Gholami Z. Yao Michael W. Mahoney Kurt Keutzer MQ 137 348 0 05 Jan 2021
BinaryBERT: Pushing the Limit of BERT Quantization Haoli Bai Wei Zhang Lu Hou Lifeng Shang Jing Jin Xin Jiang Qun Liu Michael Lyu Irwin King MQ 178 223 0 31 Dec 2020
Training with Quantization Noise for Extreme Model Compression Angela Fan Pierre Stock Benjamin Graham Edouard Grave Remi Gribonval Hervé Jégou Armand Joulin MQ 65 244 0 15 Apr 2020
Binary Neural Networks: A Survey Haotong Qin Ruihao Gong Xianglong Liu Xiao Bai Jingkuan Song N. Sebe MQ 97 463 0 31 Mar 2020
Towards Unified INT8 Training for Convolutional Neural Network Feng Zhu Ruihao Gong F. Yu Xianglong Liu Yanfei Wang Zhelong Li Xiuqi Yang Junjie Yan MQ 62 151 0 29 Dec 2019
Mixed Precision Training With 8-bit Floating Point Naveen Mellempudi Sudarshan Srinivasan Dipankar Das Bharat Kaul MQ 37 69 0 29 May 2019
Training DNNs with Hybrid Block Floating Point M. Drumond Tao R. Lin Martin Jaggi Babak Falsafi 45 96 0 04 Apr 2018
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference Benoit Jacob S. Kligys Bo Chen Menglong Zhu Matthew Tang Andrew G. Howard Hartwig Adam Dmitry Kalenichenko MQ 136 3,090 0 15 Dec 2017
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 252 8,030 0 13 Aug 2016
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 238 18,534 0 06 Feb 2015