Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

18 February 2019

Jascha Narain Sohl-Dickstein

Jeffrey Pennington

ArXiv PDF HTML

Papers citing "Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent"

50 / 261 papers shown

Title
Review: Deep Learning in Electron Microscopy Jeffrey M. Ede 34 79 0 17 Sep 2020
Predicting Training Time Without Training L. Zancato Alessandro Achille Avinash Ravichandran Rahul Bhotika Stefano Soatto 26 24 0 28 Aug 2020
Deep Networks and the Multiple Manifold Problem Sam Buchanan D. Gilboa John N. Wright 166 39 0 25 Aug 2020
Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization Neha S. Wadia Daniel Duckworth S. Schoenholz Ethan Dyer Jascha Narain Sohl-Dickstein 27 13 0 17 Aug 2020
Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy Zuyue Fu Zhuoran Yang Zhaoran Wang 15 42 0 02 Aug 2020
When and why PINNs fail to train: A neural tangent kernel perspective Sizhuang He Xinling Yu P. Perdikaris 33 878 0 28 Jul 2020
The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training Andrea Montanari Yiqiao Zhong 47 95 0 25 Jul 2020
Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach Luofeng Liao You-Lin Chen Zhuoran Yang Bo Dai Zhaoran Wang Mladen Kolar 27 32 0 02 Jul 2020
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization? Yaniv Blumenfeld D. Gilboa Daniel Soudry ODL 30 13 0 02 Jul 2020
Associative Memory in Iterated Overparameterized Sigmoid Autoencoders Yibo Jiang Cengiz Pehlevan 19 13 0 30 Jun 2020
Tensor Programs II: Neural Tangent Kernel for Any Architecture Greg Yang 58 134 0 25 Jun 2020
Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent Mehdi Abbana Bennani Thang Doan Masashi Sugiyama CLL 50 61 0 21 Jun 2020
An analytic theory of shallow networks dynamics for hinge loss classification Franco Pellegrini Giulio Biroli 35 19 0 19 Jun 2020
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains Matthew Tancik Pratul P. Srinivasan B. Mildenhall Sara Fridovich-Keil N. Raghavan Utkarsh Singhal R. Ramamoorthi Jonathan T. Barron Ren Ng 60 2,344 0 18 Jun 2020
Directional Pruning of Deep Neural Networks Shih-Kang Chao Zhanyu Wang Yue Xing Guang Cheng ODL 21 33 0 16 Jun 2020
Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory Yufeng Zhang Qi Cai Zhuoran Yang Yongxin Chen Zhaoran Wang OOD MLT 105 11 0 08 Jun 2020
Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks Z. Fan Zhichao Wang 44 71 0 25 May 2020
Consistency of Empirical Bayes And Kernel Flow For Hierarchical Parameter Estimation Yifan Chen H. Owhadi Andrew M. Stuart 20 31 0 22 May 2020
Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes Sebastian W. Ober Laurence Aitchison BDL 26 60 0 17 May 2020
Learning the gravitational force law and other analytic functions Atish Agarwala Abhimanyu Das Rina Panigrahy Qiuyi Zhang MLT 16 0 0 15 May 2020
Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond Fanghui Liu Xiaolin Huang Yudong Chen Johan A. K. Suykens BDL 44 172 0 23 Apr 2020
Frequency Bias in Neural Networks for Input of Non-Uniform Density Ronen Basri Meirav Galun Amnon Geifman David Jacobs Yoni Kasten S. Kritchman 39 183 0 10 Mar 2020
The large learning rate phase of deep learning: the catapult mechanism Aitor Lewkowycz Yasaman Bahri Ethan Dyer Jascha Narain Sohl-Dickstein Guy Gur-Ari ODL 159 234 0 04 Mar 2020
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks Chaoyue Liu Libin Zhu M. Belkin ODL 14 247 0 29 Feb 2020
Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows G. Yoo H. Owhadi 22 21 0 19 Feb 2020
Learning Parities with Neural Networks Amit Daniely Eran Malach 24 76 0 18 Feb 2020
Self-Distillation Amplifies Regularization in Hilbert Space H. Mobahi Mehrdad Farajtabar Peter L. Bartlett 19 226 0 13 Feb 2020
Machine Unlearning: Linear Filtration for Logit-based Classifiers Thomas Baumhauer Pascal Schöttle Matthias Zeppelzauer MU 109 130 0 07 Feb 2020
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks Blake Bordelon Abdulkadir Canatar Cengiz Pehlevan 146 201 0 07 Feb 2020
On the infinite width limit of neural networks with a standard parameterization Jascha Narain Sohl-Dickstein Roman Novak S. Schoenholz Jaehoon Lee 32 47 0 21 Jan 2020
Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective S. Amari 27 12 0 20 Jan 2020
Mean field theory for deep dropout networks: digging up gradient backpropagation deeply Wei Huang R. Xu Weitao Du Yutian Zeng Yunce Zhao 22 6 0 19 Dec 2019
Neural Tangents: Fast and Easy Infinite Neural Networks in Python Roman Novak Lechao Xiao Jiri Hron Jaehoon Lee Alexander A. Alemi Jascha Narain Sohl-Dickstein S. Schoenholz 29 224 0 05 Dec 2019
Towards Understanding the Spectral Bias of Deep Learning Yuan Cao Zhiying Fang Yue Wu Ding-Xuan Zhou Quanquan Gu 35 214 0 03 Dec 2019
Information in Infinite Ensembles of Infinitely-Wide Neural Networks Ravid Shwartz-Ziv Alexander A. Alemi 19 21 0 20 Nov 2019
Neural Spectrum Alignment: Empirical Study Dmitry Kopitkov Vadim Indelman 29 14 0 19 Oct 2019
The Local Elasticity of Neural Networks Hangfeng He Weijie J. Su 40 44 0 15 Oct 2019
On the expected behaviour of noise regularised deep neural networks as Gaussian processes Arnu Pretorius Herman Kamper Steve Kroon 16 9 0 12 Oct 2019
Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks Sanjeev Arora S. Du Zhiyuan Li Ruslan Salakhutdinov Ruosong Wang Dingli Yu AAML 19 161 0 03 Oct 2019
Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks Yu Bai J. Lee 24 116 0 03 Oct 2019
Non-Gaussian processes and neural networks at finite widths Sho Yaida 15 88 0 30 Sep 2019
Asymptotics of Wide Networks from Feynman Diagrams Ethan Dyer Guy Gur-Ari 29 113 0 25 Sep 2019
Finite Depth and Width Corrections to the Neural Tangent Kernel Boris Hanin Mihai Nica MDE 27 150 0 13 Sep 2019
Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy Boyi Liu Qi Cai Zhuoran Yang Zhaoran Wang 24 108 0 25 Jun 2019
The Functional Neural Process Christos Louizos Xiahan Shi Klamer Schutte Max Welling BDL 38 77 0 19 Jun 2019
Gradient Descent Maximizes the Margin of Homogeneous Neural Networks Kaifeng Lyu Jian Li 52 322 0 13 Jun 2019
Kernel and Rich Regimes in Overparametrized Models Blake E. Woodworth Suriya Gunasekar Pedro H. P. Savarese E. Moroshko Itay Golan J. Lee Daniel Soudry Nathan Srebro 24 352 0 13 Jun 2019
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks Ryo Karakida S. Akaho S. Amari 27 39 0 07 Jun 2019
Approximate Inference Turns Deep Networks into Gaussian Processes Mohammad Emtiyaz Khan Alexander Immer Ehsan Abedi M. Korzepa UQCV BDL 36 122 0 05 Jun 2019
Generalization bounds for deep convolutional neural networks Philip M. Long Hanie Sedghi MLT 42 89 0 29 May 2019