A Geometric Understanding of Natural Gradient

v1v2v3 (latest)

A Geometric Understanding of Natural Gradient

13 February 2022

ArXiv (abs)PDF HTML

Papers citing "A Geometric Understanding of Natural Gradient"

16 / 16 papers shown

Title
ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning Nilo Schwencke Cyril Furtlehner 137 1 0 14 Dec 2024
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model Guodong Zhang Lala Li Zachary Nado James Martens Sushant Sachdeva George E. Dahl Christopher J. Shallue Roger C. Grosse 93 153 0 09 Jul 2019
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks Guodong Zhang James Martens Roger C. Grosse ODL 88 124 0 27 May 2019
On Exact Computation with an Infinitely Wide Neural Net Sanjeev Arora S. Du Wei Hu Zhiyuan Li Ruslan Salakhutdinov Ruosong Wang 226 925 0 26 Apr 2019
On Lazy Training in Differentiable Programming Lénaïc Chizat Edouard Oyallon Francis R. Bach 111 835 0 19 Dec 2018
Three Mechanisms of Weight Decay Regularization Guodong Zhang Chaoqi Wang Bowen Xu Roger C. Grosse 62 258 0 29 Oct 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 267 3,203 0 20 Jun 2018
Measuring and regularizing networks in function space Ari S. Benjamin David Rolnick Konrad Paul Kording 48 139 0 21 May 2018
Sharp Minima Can Generalize For Deep Nets Laurent Dinh Razvan Pascanu Samy Bengio Yoshua Bengio ODL 116 772 0 15 Mar 2017
A Kronecker-factored approximate Fisher matrix for convolution layers Roger C. Grosse James Martens ODL 105 264 0 03 Feb 2016
Optimizing Neural Networks with Kronecker-factored Approximate Curvature James Martens Roger C. Grosse ODL 104 1,014 0 19 Mar 2015
New insights and perspectives on the natural gradient method James Martens ODL 73 624 0 03 Dec 2014
Non-parametric Stochastic Approximation with Large Step sizes Aymeric Dieuleveut Francis R. Bach 56 170 0 02 Aug 2014
The Information Geometry of Mirror Descent Garvesh Raskutti S. Mukherjee 128 124 0 29 Oct 2013
Riemannian metrics for neural networks I: feedforward networks Yann Ollivier 77 104 0 04 Mar 2013
Revisiting Natural Gradient for Deep Networks Razvan Pascanu Yoshua Bengio ODL 157 389 0 16 Jan 2013