Revisiting Natural Gradient for Deep Networks

16 January 2013

Papers citing "Revisiting Natural Gradient for Deep Networks"

50 / 229 papers shown

Title
Modular Block-diagonal Curvature Approximations for Feedforward Architectures Felix Dangel Stefan Harmeling Philipp Hennig 14 11 0 05 Feb 2019
Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting C. Atkinson B. McCane Lech Szymanski Anthony Robins VLM CLL 4 102 0 06 Dec 2018
Overcoming Catastrophic Forgetting by Soft Parameter Pruning Jian-wei Peng Jiang Hao Zhuo Li Enqiang Guo X. Wan Min Deng Qing Zhu Haifeng Li CLL 15 4 0 04 Dec 2018
Natural Option Critic Saket Tiwari Philip S. Thomas 11 22 0 04 Dec 2018
Transferring Knowledge across Learning Processes Sebastian Flennerhag Pablo G. Moreno Neil D. Lawrence Andreas C. Damianou 19 64 0 03 Dec 2018
First-order and second-order variants of the gradient descent in a unified framework Thomas Pierrot Nicolas Perrin Olivier Sigaud ODL 20 7 0 18 Oct 2018
Combining Natural Gradient with Hessian Free Methods for Sequence Training Adnan Haider P. Woodland ODL 9 4 0 03 Oct 2018
Non-Iterative Knowledge Fusion in Deep Convolutional Neural Networks M. Leontev V. Islenteva S. Sukhov MoMe FedML 11 24 0 25 Sep 2018
A Coordinate-Free Construction of Scalable Natural Gradient Kevin Luk Roger C. Grosse 9 11 0 30 Aug 2018
Fisher Information and Natural Gradient Learning of Random Deep Networks S. Amari Ryo Karakida Masafumi Oizumi 14 34 0 22 Aug 2018
On the Acceleration of L-BFGS with Second-Order Information and Stochastic Batches Jie Liu Yu Rong Martin Takáč Junzhou Huang ODL 22 7 0 14 Jul 2018
Algorithms for solving optimization problems arising from deep neural net models: smooth problems Vyacheslav Kungurtsev Tomás Pevný 16 6 0 30 Jun 2018
The decoupled extended Kalman filter for dynamic exponential-family factorization models C. Gomez-Uribe Brian Karrer 16 6 0 26 Jun 2018
Restricted Boltzmann Machines: Introduction and Review Guido Montúfar AI4CE 12 51 0 19 Jun 2018
Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach Ryo Karakida S. Akaho S. Amari FedML 25 140 0 04 Jun 2018
Bayesian Deep Net GLM and GLMM Minh-Ngoc Tran Nghia Nguyen David J. Nott Robert Kohn BDL 4 73 0 25 May 2018
Measuring and regularizing networks in function space Ari S. Benjamin David Rolnick Konrad Paul Kording 19 137 0 21 May 2018
Small steps and giant leaps: Minimal Newton solvers for Deep Learning João F. Henriques Sébastien Ehrhardt Samuel Albanie Andrea Vedaldi ODL 11 22 0 21 May 2018
Block Mean Approximation for Efficient Second Order Optimization Yao Lu Mehrtash Harandi Richard I. Hartley Razvan Pascanu ODL 9 4 0 16 Apr 2018
Sequence Training of DNN Acoustic Models With Natural Gradient Adnan Haider P. Woodland 18 7 0 06 Apr 2018
A Common Framework for Natural Gradient and Taylor based Optimisation using Manifold Theory Adnan Haider 6 2 0 26 Mar 2018
Improving Optimization for Models With Continuous Symmetry Breaking Robert Bamler Stephan Mandt DRL 20 14 0 08 Mar 2018
Accelerating Natural Gradient with Higher-Order Invariance Yang Song Jiaming Song Stefano Ermon 13 21 0 04 Mar 2018
A DIRT-T Approach to Unsupervised Domain Adaptation Rui Shu Hung Bui Hirokazu Narui Stefano Ermon 19 609 0 23 Feb 2018
Degeneration in VAE: in the Light of Fisher Information Loss Huangjie Zheng Jiangchao Yao Ya-Qin Zhang Ivor W. Tsang DRL 20 17 0 19 Feb 2018
Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces Gellert Weisz Paweł Budzianowski Pei-hao Su Milica Gasic 10 82 0 11 Feb 2018
$$\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space$ $\mathcal{G}$ -SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space Qi Meng Shuxin Zheng Huishuai Zhang Wei-Neng Chen Zhi-Ming Ma Tie-Yan Liu 35 38 0 11 Feb 2018
Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting Xialei Liu Marc Masana Luis Herranz Joost van de Weijer Antonio M. López Andrew D. Bagdanov CLL 26 269 0 08 Feb 2018
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence Arslan Chaudhry P. Dokania Thalaiyasingam Ajanthan Philip H. S. Torr CLL 38 1,110 0 30 Jan 2018
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes Erin Grant Chelsea Finn Sergey Levine Trevor Darrell Thomas L. Griffiths BDL 21 506 0 26 Jan 2018
Block-diagonal Hessian-free Optimization for Training Neural Networks Huishuai Zhang Caiming Xiong James Bradbury R. Socher ODL 6 22 0 20 Dec 2017
Vprop: Variational Inference using RMSprop Mohammad Emtiyaz Khan Zuozhu Liu Voot Tangkaratt Y. Gal BDL 12 17 0 04 Dec 2017
Riemannian approach to batch normalization Minhyung Cho Jaehyung Lee 21 93 0 27 Sep 2017
Neural Optimizer Search with Reinforcement Learning Irwan Bello Barret Zoph Vijay Vasudevan Quoc V. Le ODL 27 382 0 21 Sep 2017
Implicit Regularization in Deep Learning Behnam Neyshabur 17 145 0 06 Sep 2017
Stochastic Gradient Descent: Going As Fast As Possible But Not Faster Alice Schoenauer Sebag Marc Schoenauer Michèle Sebag 12 11 0 05 Sep 2017
Distral: Robust Multitask Reinforcement Learning Yee Whye Teh V. Bapst Wojciech M. Czarnecki John Quan J. Kirkpatrick R. Hadsell N. Heess Razvan Pascanu 25 541 0 13 Jul 2017
Sobolev Training for Neural Networks Wojciech M. Czarnecki Simon Osindero Max Jaderberg G. Swirszcz Razvan Pascanu 19 241 0 15 Jun 2017
YellowFin and the Art of Momentum Tuning Jian Zhang Ioannis Mitliagkas ODL 15 108 0 12 Jun 2017
Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC Yulai Cong Bo Chen Hongwei Liu Mingyuan Zhou BDL 17 66 0 06 Jun 2017
A Neural Network model with Bidirectional Whitening Y. Fujimoto T. Ohira 16 4 0 24 Apr 2017
Overcoming Catastrophic Forgetting by Incremental Moment Matching Sang-Woo Lee Jin-Hwa Kim Jaehyun Jun Jung-Woo Ha Byoung-Tak Zhang CLL 16 666 0 24 Mar 2017
Sharp Minima Can Generalize For Deep Nets Laurent Dinh Razvan Pascanu Samy Bengio Yoshua Bengio ODL 35 754 0 15 Mar 2017
Continual Learning Through Synaptic Intelligence Friedemann Zenke Ben Poole Surya Ganguli 26 51 0 13 Mar 2017
Overcoming catastrophic forgetting in neural networks J. Kirkpatrick Razvan Pascanu Neil C. Rabinowitz J. Veness Guillaume Desjardins ... A. Grabska-Barwinska Demis Hassabis Claudia Clopath D. Kumaran R. Hadsell CLL 22 7,296 0 02 Dec 2016
Combining policy gradient and Q-learning Brendan O'Donoghue Rémi Munos Koray Kavukcuoglu Volodymyr Mnih OffRL OnRL 19 139 0 05 Nov 2016
Loss-aware Binarization of Deep Networks Lu Hou Quanming Yao James T. Kwok MQ 27 220 0 05 Nov 2016
Spectral Inference Methods on Sparse Graphs: Theory and Applications Alaa Saade 13 6 0 14 Oct 2016
Relative Natural Gradient for Learning Large Complex Models Ke Sun Frank Nielsen 29 5 0 20 Jun 2016
Effective Blind Source Separation Based on the Adam Algorithm M. Scarpiniti Simone Scardapane Danilo Comminiello R. Parisi A. Uncini 16 7 0 25 May 2016