A Trace-restricted Kronecker-Factored Approximation to Natural Gradient

21 November 2020

Papers citing "A Trace-restricted Kronecker-Factored Approximation to Natural Gradient"

21 / 21 papers shown

Title
SOAP: Improving and Stabilizing Shampoo using Adam Nikhil Vyas Depen Morwani Rosie Zhao Itai Shapira David Brandfonbrener Lucas Janson Sham Kakade Sham Kakade 118 34 0 17 Sep 2024
Convolutional Neural Network Training with Distributed K-FAC J. G. Pauloski Zhao Zhang Lei Huang Weijia Xu Ian Foster 38 31 0 01 Jul 2020
Practical Quasi-Newton Methods for Training Deep Neural Networks D. Goldfarb Yi Ren Achraf Bahamou ODL 53 105 0 16 Jun 2020
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning Z. Yao A. Gholami Sheng Shen Mustafa Mustafa Kurt Keutzer Michael W. Mahoney ODL 73 281 0 01 Jun 2020
Pathological spectra of the Fisher information metric and its variants in deep neural networks Ryo Karakida S. Akaho S. Amari 45 28 0 14 Oct 2019
Estimation of the Kronecker Covariance Model by Quadratic Form O. Linton Haihan Tang 10 2 0 21 Jun 2019
Eigenvalue Corrected Noisy Natural Gradient Juhan Bae Guodong Zhang Roger C. Grosse 48 16 0 30 Nov 2018
Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks Kazuki Osawa Yohei Tsuji Yuichiro Ueno Akira Naruse Rio Yokota Satoshi Matsuoka ODL 52 95 0 29 Nov 2018
Three Mechanisms of Weight Decay Regularization Guodong Zhang Chaoqi Wang Bowen Xu Roger C. Grosse 51 257 0 29 Oct 2018
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis Thomas George César Laurent Xavier Bouthillier Nicolas Ballas Pascal Vincent ODL 47 151 0 11 Jun 2018
Noisy Natural Gradient as Variational Inference Guodong Zhang Shengyang Sun David Duvenaud Roger C. Grosse ODL 62 211 0 06 Dec 2017
A Kronecker-factored approximate Fisher matrix for convolution layers Roger C. Grosse James Martens ODL 71 260 0 03 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.4K 192,638 0 10 Dec 2015
adaQN: An Adaptive Quasi-Newton Algorithm for Training RNNs N. Keskar A. Berahas ODL 38 35 0 04 Nov 2015
Optimizing Neural Networks with Kronecker-factored Approximate Curvature James Martens Roger C. Grosse ODL 75 999 0 19 Mar 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 866 149,474 0 22 Dec 2014
New insights and perspectives on the natural gradient method James Martens ODL 54 613 0 03 Dec 2014
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 957 99,991 0 04 Sep 2014
Training Neural Networks with Stochastic Hessian-Free Optimization Ryan Kiros BDL 61 48 0 16 Jan 2013
High-dimensional covariance matrix estimation in approximate factor models Jianqing Fan Yuan Liao Martina Mincheva 64 396 0 21 May 2011
Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization Benjamin Recht Maryam Fazel P. Parrilo 256 3,762 0 28 Jun 2007