Revisiting Natural Gradient for Deep Networks

16 January 2013

Papers citing "Revisiting Natural Gradient for Deep Networks"

50 / 229 papers shown

Title
Posterior Meta-Replay for Continual Learning Christian Henning Maria R. Cervera Francesco DÁngelo J. Oswald Regina Traber Benjamin Ehret Seijin Kobayashi Benjamin Grewe João Sacramento CLL BDL 51 54 0 01 Mar 2021
A Bayesian Federated Learning Framework with Online Laplace Approximation Liang Liu Xi Jiang Feng Zheng Hong Chen Guo-Jun Qi Heng-Chiao Huang Ling Shao FedML 43 53 0 03 Feb 2021
Are we Forgetting about Compositional Optimisers in Bayesian Optimisation? Antoine Grosnit Alexander I. Cowen-Rivers Rasul Tutunov Ryan-Rhys Griffiths Jun Wang Haitham Bou-Ammar 17 13 0 15 Dec 2020
Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent Kangqiao Liu Liu Ziyin Masakuni Ueda MLT 57 37 0 07 Dec 2020
Natural-gradient learning for spiking neurons Elena Kreutzer Walter Senn Mihai A. Petrovici 11 13 0 23 Nov 2020
Two-Level K-FAC Preconditioning for Deep Learning N. Tselepidis Jonas Köhler Antonio Orvieto ODL 11 2 0 01 Nov 2020
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients Juntang Zhuang Tommy M. Tang Yifan Ding S. Tatikonda Nicha Dvornek X. Papademetris James S. Duncan ODL 6 499 0 15 Oct 2020
Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties Brett Daley Chris Amato ODL 11 4 0 03 Oct 2020
Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks Ryo Karakida Kazuki Osawa 9 25 0 02 Oct 2020
Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra V. Papyan 8 77 0 27 Aug 2020
HEX and Neurodynamic Programming Debangshu Banerjee 8 2 0 11 Aug 2020
Better Fine-Tuning by Reducing Representational Collapse Armen Aghajanyan Akshat Shrivastava Anchit Gupta Naman Goyal Luke Zettlemoyer S. Gupta AAML 29 208 0 06 Aug 2020
CoNES: Convex Natural Evolutionary Strategies Sushant Veer Anirudha Majumdar 13 3 0 16 Jul 2020
Weak error analysis for stochastic gradient descent optimization algorithms A. Bercher Lukas Gonon Arnulf Jentzen Diyora Salimova 24 4 0 03 Jul 2020
Supermasks in Superposition Mitchell Wortsman Vivek Ramanujan Rosanne Liu Aniruddha Kembhavi Mohammad Rastegari J. Yosinski Ali Farhadi SSL CLL 17 279 0 26 Jun 2020
Revisiting Loss Modelling for Unstructured Pruning César Laurent Camille Ballas Thomas George Nicolas Ballas Pascal Vincent 20 14 0 22 Jun 2020
When Does Preconditioning Help or Hurt Generalization? S. Amari Jimmy Ba Roger C. Grosse Xuechen Li Atsushi Nitanda Taiji Suzuki Denny Wu Ji Xu 34 32 0 18 Jun 2020
Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training Diego Granziol S. Zohren Stephen J. Roberts ODL 29 48 0 16 Jun 2020
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry Tomohiro Hayase Ryo Karakida 22 7 0 14 Jun 2020
Unifying Regularisation Methods for Continual Learning Frederik Benzing CLL 8 8 0 11 Jun 2020
Structure preserving deep learning E. Celledoni Matthias Joachim Ehrhardt Christian Etmann R. McLachlan B. Owren Carola-Bibiane Schönlieb Ferdia Sherry AI4CE 13 43 0 05 Jun 2020
Machine Learning for Condensed Matter Physics Edwin Bedolla L. C. Padierna R. Castañeda-Priego AI4CE 15 66 0 28 May 2020
Continual Learning with Extended Kronecker-factored Approximate Curvature Janghyeon Lee H. Hong Donggyu Joo Junmo Kim CLL 9 52 0 16 Apr 2020
Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs Lei Huang Jie Qin Li Liu Fan Zhu Ling Shao AI4CE 20 11 0 25 Feb 2020
Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descent Pu Zhao Pin-Yu Chen Siyue Wang X. Lin AAML 6 36 0 18 Feb 2020
Second-order Information in First-order Optimization Methods Yuzheng Hu Licong Lin Shange Tang ODL 28 2 0 20 Dec 2019
Overcoming Long-term Catastrophic Forgetting through Adversarial Neural Pruning and Synaptic Consolidation Jian-wei Peng Bo Tang Jiang Hao Zhuo Li Yinjie Lei Tao R. Lin Haifeng Li AAML CLL 25 36 0 19 Dec 2019
LOGAN: Latent Optimisation for Generative Adversarial Networks Y. Wu Jeff Donahue David Balduzzi Karen Simonyan Timothy Lillicrap GAN 8 88 0 02 Dec 2019
Automatic Differentiable Monte Carlo: Theory and Application Shi-Xin Zhang Z. Wan H. Yao 6 17 0 20 Nov 2019
Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding Yongqi Zhang Quanming Yao Lei Chen 14 4 0 17 Nov 2019
Searching to Exploit Memorization Effect in Learning from Corrupted Labels Quanming Yao Hansi Yang Bo Han Gang Niu James T. Kwok NoLa 11 16 0 06 Nov 2019
Pathological spectra of the Fisher information metric and its variants in deep neural networks Ryo Karakida S. Akaho S. Amari 17 27 0 14 Oct 2019
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation S. Meng Sharan Vaswani I. Laradji Mark W. Schmidt Simon Lacoste-Julien 11 32 0 11 Oct 2019
The asymptotic spectrum of the Hessian of DNN throughout training Arthur Jacot Franck Gabriel Clément Hongler 11 35 0 01 Oct 2019
A continual learning survey: Defying forgetting in classification tasks Matthias De Lange Rahaf Aljundi Marc Masana Sarah Parisot Xu Jia A. Leonardis Greg Slabaugh Tinne Tuytelaars CLL KELM 12 246 0 18 Sep 2019
Meta-Learning with Warped Gradient Descent Sebastian Flennerhag Andrei A. Rusu Razvan Pascanu Francesco Visin Hujun Yin R. Hadsell 6 209 0 30 Aug 2019
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks Ryo Karakida S. Akaho S. Amari 19 39 0 07 Jun 2019
Understanding Adversarial Behavior of DNNs by Disentangling Non-Robust and Robust Components in Performance Metric Yujun Shi B. Liao Guangyong Chen Yun-Hai Liu Ming-Ming Cheng Jiashi Feng AAML 4 2 0 06 Jun 2019
Limitations of the Empirical Fisher Approximation for Natural Gradient Descent Frederik Kunstner Lukas Balles Philipp Hennig 19 207 0 29 May 2019
Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems Tianle Cai Ruiqi Gao Jikai Hou Siyu Chen Dong Wang Di He Zhihua Zhang Liwei Wang ODL 16 57 0 28 May 2019
A Geometric Modeling of Occam's Razor in Deep Learning Ke Sun Frank Nielsen 9 4 0 27 May 2019
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks Guodong Zhang James Martens Roger C. Grosse ODL 11 124 0 27 May 2019
Adaptive norms for deep learning with regularized Newton methods Jonas Köhler Leonard Adolphs Aurélien Lucchi ODL 9 11 0 22 May 2019
EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis Chaoqi Wang Roger C. Grosse Sanja Fidler Guodong Zhang 21 121 0 15 May 2019
NGO-GM: Natural Gradient Optimization for Graphical Models Eric Benhamou Jamal Atif R. Laraki David Saltiel 8 1 0 14 May 2019
Facilitating Bayesian Continual Learning by Natural Gradients and Stein Gradients Yu Chen Tom Diethe Neil D. Lawrence CLL BDL 6 13 0 24 Apr 2019
Convergence rates for the stochastic gradient descent method for non-convex objective functions Benjamin J. Fehrman Benjamin Gess Arnulf Jentzen 17 101 0 02 Apr 2019
Fisher-Bures Adversary Graph Convolutional Networks Ke Sun Piotr Koniusz Zhen Wang GNN 19 34 0 11 Mar 2019
A Formalization of The Natural Gradient Method for General Similarity Measures Anton Mallasto T. D. Haije Aasa Feragen 11 4 0 24 Feb 2019
Overcoming Multi-Model Forgetting Yassine Benyahia Kaicheng Yu Kamil Bennani-Smires Martin Jaggi A. Davison Mathieu Salzmann C. Musat MoMe KELM CLL 27 37 0 21 Feb 2019