Sketchy Empirical Natural Gradient Methods for Deep Learning

v1v2v3 (latest)

Sketchy Empirical Natural Gradient Methods for Deep Learning

10 June 2020

ArXiv (abs)PDF HTML

Papers citing "Sketchy Empirical Natural Gradient Methods for Deep Learning"

8 / 8 papers shown

Title
MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates Mohammad Mozaffari Sikan Li Zhao Zhang M. Dehnavi 74 4 0 02 Jun 2023
ASDL: A Unified Interface for Gradient Preconditioning in PyTorch Kazuki Osawa Satoki Ishikawa Rio Yokota Shigang Li Torsten Hoefler ODL 92 15 0 08 May 2023
Brand New K-FACs: Speeding up K-FAC with Online Decomposition Updates C. Puiu 32 2 0 16 Oct 2022
Riemannian Natural Gradient Methods Jiang Hu Ruicheng Ao Anthony Man-Cho So Minghan Yang Zaiwen Wen 69 11 0 15 Jul 2022
Randomized K-FACs: Speeding up K-FAC with Randomized Numerical Linear Algebra C. Puiu 65 2 0 30 Jun 2022
Rethinking Exponential Averaging of the Fisher C. Puiu 54 1 0 10 Apr 2022
NG+ : A Multi-Step Matrix-Product Natural Gradient Method for Deep Learning Minghan Yang Dong Xu Qiwen Cui Zaiwen Wen Pengxiang Xu 48 4 0 14 Jun 2021
Eigenvalue-corrected Natural Gradient Based on a New Approximation Kai-Xin Gao Xiaolei Liu Zheng-Hai Huang Min Wang Shuangling Wang Zidong Wang Dachuan Xu F. Yu ODL 48 7 0 27 Nov 2020