Tensor Normal Training for Deep Learning Models

5 June 2021

Papers citing "Tensor Normal Training for Deep Learning Models"

8 / 8 papers shown

Title
Spectral-factorized Positive-definite Curvature Learning for NN Training Wu Lin Felix Dangel Runa Eschenhagen Juhan Bae Richard E. Turner Roger B. Grosse 74 0 0 10 Feb 2025
SOAP: Improving and Stabilizing Shampoo using Adam Nikhil Vyas Depen Morwani Rosie Zhao Itai Shapira David Brandfonbrener Lucas Janson Sham Kakade Sham Kakade 92 28 0 17 Sep 2024
An Improved Empirical Fisher Approximation for Natural Gradient Descent Xiaodong Wu Wenyi Yu Chao Zhang Philip Woodland 47 4 0 10 Jun 2024
A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale Hao-Jun Michael Shi Tsung-Hsien Lee Shintaro Iwasaki Jose Gallego-Posada Zhijing Li Kaushik Rangadurai Dheevatsa Mudigere Michael Rabbat ODL 30 24 0 12 Sep 2023
Eva: A General Vectorized Approximation Framework for Second-order Optimization Lin Zhang Shaoshuai Shi Yue Liu 33 1 0 04 Aug 2023
KrADagrad: Kronecker Approximation-Domination Gradient Preconditioned Stochastic Optimization Jonathan Mei Alexander Moreno Luke Walters ODL 34 1 0 30 May 2023
Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning Achraf Bahamou D. Goldfarb ODL 52 0 0 23 May 2023
ASDL: A Unified Interface for Gradient Preconditioning in PyTorch Kazuki Osawa Satoki Ishikawa Rio Yokota Shigang Li Torsten Hoefler ODL 51 14 0 08 May 2023