Sketchy: Memory-efficient Adaptive Regularization with Frequent
Directions

Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions

7 February 2023

Vladimir Feinberg

Y. Jennifer Sun

Papers citing "Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions"

12 / 12 papers shown

Title
ASGO: Adaptive Structured Gradient Optimization Kang An Yuxing Liu Rui Pan Shiqian Ma D. Goldfarb Tong Zhang ODL 92 2 0 26 Mar 2025
Preconditioned Subspace Langevin Monte Carlo Tyler Maunu Jiayi Yao 90 0 0 18 Dec 2024
Modular Duality in Deep Learning Jeremy Bernstein Laker Newhouse 22 2 0 28 Oct 2024
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization Jui-Nan Yen Si Si Zhao Meng Felix X. Yu Sai Surya Duvvuri Inderjit Dhillon Cho-Jui Hsieh Sanjiv Kumar 27 2 0 27 Oct 2024
Old Optimizer, New Norm: An Anthology Jeremy Bernstein Laker Newhouse ODL 36 12 0 30 Sep 2024
Memory-Efficient LLM Training with Online Subspace Descent Kaizhao Liang Bo Liu Lizhang Chen Qiang Liu 29 7 0 23 Aug 2024
H-Fac: Memory-Efficient Optimization with Factorized Hamiltonian Descent Son Nguyen Lizhang Chen Bo Liu Qiang Liu 25 3 0 14 Jun 2024
4-bit Shampoo for Memory-Efficient Network Training Sike Wang Jia Li Pan Zhou Hua Huang MQ 31 5 0 28 May 2024
Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networks Yongchang Hao Yanshuai Cao Lili Mou ODL 19 1 0 05 Feb 2024
Flora: Low-Rank Adapters Are Secretly Gradient Compressors Yongchang Hao Yanshuai Cao Lili Mou 11 39 0 05 Feb 2024
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 281 2,888 0 15 Sep 2016
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 296 39,194 0 01 Sep 2014