Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems

28 May 2019

Tianle Cai

Ruiqi Gao

Papers citing "Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems"

33 / 33 papers shown

Title
ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning Nilo Schwencke Cyril Furtlehner 120 1 0 14 Dec 2024
On the Variance of the Adaptive Learning Rate and Beyond Liyuan Liu Haoming Jiang Pengcheng He Weizhu Chen Xiaodong Liu Jianfeng Gao Jiawei Han ODL 185 1,894 0 08 Aug 2019
Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks Yi Ren D. Goldfarb 48 35 0 05 Jun 2019
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao Quanquan Gu MLT AI4CE 76 384 0 30 May 2019
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks Guodong Zhang James Martens Roger C. Grosse ODL 74 124 0 27 May 2019
Adaptive Gradient Methods with Dynamic Bound of Learning Rate Liangchen Luo Yuanhao Xiong Yan Liu Xu Sun ODL 53 600 0 26 Feb 2019
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent Jaehoon Lee Lechao Xiao S. Schoenholz Yasaman Bahri Roman Novak Jascha Narain Sohl-Dickstein Jeffrey Pennington 166 1,089 0 18 Feb 2019
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei Theodor Misiakiewicz Andrea Montanari MLT 68 277 0 16 Feb 2019
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation Greg Yang 105 286 0 13 Feb 2019
Fixup Initialization: Residual Learning Without Normalization Hongyi Zhang Yann N. Dauphin Tengyu Ma ODL AI4CE 82 348 0 27 Jan 2019
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks Sanjeev Arora S. Du Wei Hu Zhiyuan Li Ruosong Wang MLT 154 966 0 24 Jan 2019
Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits Xialiang Dou Tengyuan Liang MLT 57 42 0 21 Jan 2019
Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path? Samet Oymak Mahdi Soltanolkotabi ODL 40 176 0 25 Dec 2018
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks Difan Zou Yuan Cao Dongruo Zhou Quanquan Gu ODL 146 448 0 21 Nov 2018
Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers Zeyuan Allen-Zhu Yuanzhi Li Yingyu Liang MLT 149 769 0 12 Nov 2018
Gradient Descent Finds Global Minima of Deep Neural Networks S. Du Jason D. Lee Haochuan Li Liwei Wang Masayoshi Tomizuka ODL 156 1,133 0 09 Nov 2018
Gradient Descent Provably Optimizes Over-parameterized Neural Networks S. Du Xiyu Zhai Barnabás Póczós Aarti Singh MLT ODL 168 1,261 0 04 Oct 2018
Just Interpolate: Kernel "Ridgeless" Regression Can Generalize Tengyuan Liang Alexander Rakhlin 56 353 0 01 Aug 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 209 3,160 0 20 Jun 2018
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis Thomas George César Laurent Xavier Bouthillier Nicolas Ballas Pascal Vincent ODL 58 151 0 11 Jun 2018
Revisiting Small Batch Training for Deep Neural Networks Dominic Masters Carlo Luschi ODL 69 665 0 20 Apr 2018
Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent Chi Jin Praneeth Netrapalli Michael I. Jordan ODL 59 261 0 28 Nov 2017
Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation Yuhuai Wu Elman Mansimov Shun Liao Roger C. Grosse Jimmy Ba OffRL 47 624 0 17 Aug 2017
Practical Gauss-Newton Optimisation for Deep Learning Aleksandar Botev H. Ritter David Barber ODL 44 228 0 12 Jun 2017
How to Escape Saddle Points Efficiently Chi Jin Rong Ge Praneeth Netrapalli Sham Kakade Michael I. Jordan ODL 180 834 0 02 Mar 2017
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 380 2,922 0 15 Sep 2016
A Kronecker-factored approximate Fisher matrix for convolution layers Roger C. Grosse James Martens ODL 84 260 0 03 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.6K 192,638 0 10 Dec 2015
Train faster, generalize better: Stability of stochastic gradient descent Moritz Hardt Benjamin Recht Y. Singer 106 1,234 0 03 Sep 2015
Optimizing Neural Networks with Kronecker-factored Approximate Curvature James Martens Roger C. Grosse ODL 92 999 0 19 Mar 2015
Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition Rong Ge Furong Huang Chi Jin Yang Yuan 132 1,056 0 06 Mar 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 1.1K 149,474 0 22 Dec 2014
Revisiting Natural Gradient for Deep Networks Razvan Pascanu Yoshua Bengio ODL 109 388 0 16 Jan 2013