Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients

11 July 2024

Ajay Jaiswal

Jiawei Zhao

Zhangyang Wang

Papers citing "Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients"

27 / 27 papers shown

Title
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam Tianjin Huang Haotian Hu Zhenyu Zhang Gaojie Jin Xianrui Li ... Tianlong Chen Lu Liu Qingsong Wen Zhangyang Wang Shiwei Liu MQ 101 2 0 24 Feb 2025
SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training Tianjin Huang Ziquan Zhu Gaojie Jin Lu Liu Zhangyang Wang Shiwei Liu 107 6 0 12 Jan 2025
AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning Yehonathan Refael Jonathan Svirsky Boris Shustin Wasim Huleihel Ofir Lindenbaum 91 4 0 31 Dec 2024
CompAct: Compressed Activations for Memory-Efficient LLM Training Yara Shamshoum Nitzan Hodos Yuval Sieradzki Assaf Schuster MQ VLM 103 4 0 20 Oct 2024
SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning Yuze Zhao Jintao Huang Jinghan Hu Xingjun Wang Yunlin Mao ... Zhikai Wu Baole Ai Ang Wang Wenmeng Zhou Yingda Chen 97 46 0 10 Aug 2024
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning Boyao Wang Xiang Liu Shizhe Diao Renjie Pi Jipeng Zhang Chi Han Tong Zhang 90 52 0 26 Mar 2024
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Jiawei Zhao Zhenyu Zhang Beidi Chen Zhangyang Wang A. Anandkumar Yuandong Tian 102 228 0 06 Mar 2024
Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning Wenhan Xia Chengwei Qin Elad Hazan 121 61 0 08 Jan 2024
AdaLomo: Low-memory Optimization with Adaptive Learning Rate Kai Lv Hang Yan Qipeng Guo Haijun Lv Xipeng Qiu ODL 76 23 0 16 Oct 2023
ReLoRA: High-Rank Training Through Low-Rank Updates Vladislav Lialin Namrata Shivagunde Sherin Muckatira Anna Rumshisky BDL 88 117 0 11 Jul 2023
ChatGPT: Jack of all trades, master of none Jan Kocoñ Igor Cichecki Oliwier Kaszyca Mateusz Kochanek Dominika Szydło ... Maciej Piasecki Lukasz Radliñski Konrad Wojtasik Stanislaw Wo'zniak Przemyslaw Kazienko AI4MH 117 553 0 21 Feb 2023
Exploring Low Rank Training of Deep Neural Networks Siddhartha Rao Kamalakara Acyr Locatelli Bharat Venkitesh Jimmy Ba Y. Gal Aidan Gomez 84 25 0 27 Sep 2022
FP8 Formats for Deep Learning Paulius Micikevicius Dusan Stosic N. Burgess Marius Cornea Pradeep Dubey ... Naveen Mellempudi S. Oberman Mohammad Shoeybi Michael Siu Hao Wu BDL VLM MQ 145 139 0 12 Sep 2022
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers M. Lewis Younes Belkada Luke Zettlemoyer MQ 110 664 0 15 Aug 2022
On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages Fuxiang Chen F. Fard David Lo T. Bryksin 65 48 0 05 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 208 1,987 0 29 Mar 2022
8-bit Optimizers via Block-wise Quantization Tim Dettmers M. Lewis Sam Shleifer Luke Zettlemoyer MQ 123 302 0 06 Oct 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 502 10,526 0 17 Jun 2021
Towards Unified INT8 Training for Convolutional Neural Network Feng Zhu Ruihao Gong F. Yu Xianglong Liu Yanfei Wang Zhelong Li Xiuqi Yang Junjie Yan MQ 92 152 0 29 Dec 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 490 20,342 0 23 Oct 2019
Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers Yukuan Yang Shuang Wu Lei Deng Tianyi Yan Yuan Xie Guoqi Li MQ 151 112 0 05 Sep 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 689 24,557 0 26 Jul 2019
SWALP : Stochastic Weight Averaging in Low-Precision Training Guandao Yang Tianyi Zhang Polina Kirichenko Junwen Bai A. Wilson Christopher De Sa 68 97 0 26 Apr 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,201 0 20 Apr 2018
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost Noam M. Shazeer Mitchell Stern ODL 84 1,052 0 11 Apr 2018
Training Quantized Nets: A Deeper Understanding Hao Li Soham De Zheng Xu Christoph Studer H. Samet Tom Goldstein MQ 64 211 0 07 Jun 2017
Deep Learning with Limited Numerical Precision Suyog Gupta A. Agrawal K. Gopalakrishnan P. Narayanan HAI 207 2,049 0 09 Feb 2015