Memory-Efficient LLM Training with Online Subspace Descent

Memory-Efficient LLM Training with Online Subspace Descent

23 August 2024

Bo Liu

Papers citing "Memory-Efficient LLM Training with Online Subspace Descent"

6 / 6 papers shown

Title
PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Training Matan Haroush Daniel Soudry 57 0 0 23 May 2025
Cautious Optimizers: Improving Training with One Line of Code Kaizhao Liang Lizhang Chen B. Liu Qiang Liu ODL 135 5 0 25 Nov 2024
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Jiawei Zhao Zhenyu Zhang Beidi Chen Zhangyang Wang A. Anandkumar Yuandong Tian 64 194 0 06 Mar 2024
Symbolic Discovery of Optimization Algorithms Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang ... Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le 102 367 0 13 Feb 2023
8-bit Optimizers via Block-wise Quantization Tim Dettmers M. Lewis Sam Shleifer Luke Zettlemoyer MQ 92 286 0 06 Oct 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 223 9,946 0 17 Jun 2021