Batch Normalization Orthogonalizes Representations in Deep Random Networks

7 June 2021

Papers citing "Batch Normalization Orthogonalizes Representations in Deep Random Networks"

27 / 27 papers shown

Title
Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs Michael Scholkemper Xinyi Wu Ali Jadbabaie Michael T. Schaub 126 7 0 05 Jun 2024
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 252 514 0 11 Feb 2021
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs Jonathan Frankle D. Schwab Ari S. Morcos 54 140 0 29 Feb 2020
A Deep Conditioning Treatment of Neural Networks Naman Agarwal Pranjal Awasthi Satyen Kale AI4CE 39 16 0 04 Feb 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 280 42,038 0 03 Dec 2019
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks Ryo Karakida S. Akaho S. Amari 53 41 0 07 Jun 2019
On the Inductive Bias of Neural Tangent Kernels A. Bietti Julien Mairal 51 255 0 29 May 2019
A Mean Field Theory of Batch Normalization Greg Yang Jeffrey Pennington Vinay Rao Jascha Narain Sohl-Dickstein S. Schoenholz 49 178 0 21 Feb 2019
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization Sanjeev Arora Zhiyuan Li Kaifeng Lyu 58 131 0 10 Dec 2018
A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks Sanjeev Arora Nadav Cohen Noah Golowich Wei Hu 100 284 0 04 Oct 2018
Deep Convolutional Networks as shallow Gaussian Processes Adrià Garriga-Alonso C. Rasmussen Laurence Aitchison BDL UQCV 76 269 0 16 Aug 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 188 3,160 0 20 Jun 2018
Understanding Batch Normalization Johan Bjorck Carla P. Gomes B. Selman Kilian Q. Weinberger 95 603 0 01 Jun 2018
How Does Batch Normalization Help Optimization? Shibani Santurkar Dimitris Tsipras Andrew Ilyas Aleksander Madry ODL 73 1,531 0 29 May 2018
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization Jonas Köhler Hadi Daneshmand Aurelien Lucchi M. Zhou K. Neymeyr Thomas Hofmann 44 91 0 27 May 2018
Gaussian Process Behaviour in Wide Deep Neural Networks A. G. Matthews Mark Rowland Jiri Hron Richard Turner Zoubin Ghahramani BDL 122 553 0 30 Apr 2018
The Emergence of Spectral Universality in Deep Networks Jeffrey Pennington S. Schoenholz Surya Ganguli 44 171 0 27 Feb 2018
Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks Peter L. Bartlett D. Helmbold Philip M. Long 69 116 0 16 Feb 2018
Deep Neural Networks as Gaussian Processes Jaehoon Lee Yasaman Bahri Roman Novak S. Schoenholz Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 109 1,086 0 01 Nov 2017
Self-Normalizing Neural Networks Günter Klambauer Thomas Unterthiner Andreas Mayr Sepp Hochreiter 246 2,496 0 08 Jun 2017
Deep Information Propagation S. Schoenholz Justin Gilmer Surya Ganguli Jascha Narain Sohl-Dickstein 64 363 0 04 Nov 2016
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 631 36,599 0 25 Aug 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 268 10,412 0 21 Jul 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.4K 192,638 0 10 Dec 2015
Steps Toward Deep Kernel Methods from Infinite Neural Networks Tamir Hazan Tommi Jaakkola 44 83 0 20 Aug 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 333 43,154 0 11 Feb 2015
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Andrew M. Saxe James L. McClelland Surya Ganguli ODL 128 1,830 0 20 Dec 2013