Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

14 June 2018

Lechao Xiao

Yasaman Bahri

Jascha Narain Sohl-Dickstein

S. Schoenholz

Jeffrey Pennington

ArXiv PDF HTML

Papers citing "Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks"

27 / 77 papers shown

Title
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization? Yaniv Blumenfeld D. Gilboa Daniel Soudry ODL 27 13 0 02 Jul 2020
Deep Isometric Learning for Visual Recognition Haozhi Qi Chong You Xinyu Wang Yi Ma Jitendra Malik VLM 30 53 0 30 Jun 2020
Tensor Programs II: Neural Tangent Kernel for Any Architecture Greg Yang 58 134 0 25 Jun 2020
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry Tomohiro Hayase Ryo Karakida 29 7 0 14 Jun 2020
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks Soham De Samuel L. Smith ODL 14 20 0 24 Feb 2020
On the distance between two neural networks and the stability of learning Jeremy Bernstein Arash Vahdat Yisong Yue Xuan Li ODL 200 57 0 09 Feb 2020
On the infinite width limit of neural networks with a standard parameterization Jascha Narain Sohl-Dickstein Roman Novak S. Schoenholz Jaehoon Lee 26 47 0 21 Jan 2020
A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks Zhaodong Chen Lei Deng Bangyan Wang Guoqi Li Yuan Xie 35 28 0 01 Jan 2020
Towards Efficient Training for Neural Network Quantization Qing Jin Linjie Yang Zhenyu A. Liao MQ 13 42 0 21 Dec 2019
Mean field theory for deep dropout networks: digging up gradient backpropagation deeply Wei Huang R. Xu Weitao Du Yutian Zeng Yunce Zhao 19 6 0 19 Dec 2019
Optimization for deep learning: theory and algorithms Ruoyu Sun ODL 19 168 0 19 Dec 2019
Neural Tangents: Fast and Easy Infinite Neural Networks in Python Roman Novak Lechao Xiao Jiri Hron Jaehoon Lee Alexander A. Alemi Jascha Narain Sohl-Dickstein S. Schoenholz 27 224 0 05 Dec 2019
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes Greg Yang 33 190 0 28 Oct 2019
On the expected behaviour of noise regularised deep neural networks as Gaussian processes Arnu Pretorius Herman Kamper Steve Kroon 16 9 0 12 Oct 2019
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images Denis Yarats Amy Zhang Ilya Kostrikov Brandon Amos Joelle Pineau Rob Fergus DRL 42 436 0 02 Oct 2019
Asymptotics of Wide Networks from Feynman Diagrams Ethan Dyer Guy Gur-Ari 24 113 0 25 Sep 2019
Optimal Machine Intelligence at the Edge of Chaos Ling Feng Lin Zhang C. Lai 27 8 0 11 Sep 2019
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks Ryo Karakida S. Akaho S. Amari 27 39 0 07 Jun 2019
Residual Networks as Nonlinear Systems: Stability Analysis using Linearization Kai Rothauge Z. Yao Zixi Hu Michael W. Mahoney 13 2 0 31 May 2019
Mean-field Analysis of Batch Normalization Ming-Bo Wei J. Stokes D. Schwab MLT 25 8 0 06 Mar 2019
Feedback alignment in deep convolutional networks Theodore H. Moskovitz Ashok Litwin-Kumar L. F. Abbott 19 59 0 12 Dec 2018
On the Convergence Rate of Training Recurrent Neural Networks Zeyuan Allen-Zhu Yuanzhi Li Zhao-quan Song 18 191 0 29 Oct 2018
A general learning system based on neuron bursting and tonic firing H. Lui 15 0 0 22 Oct 2018
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes Roman Novak Lechao Xiao Jaehoon Lee Yasaman Bahri Greg Yang Jiri Hron Daniel A. Abolafia Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 25 306 0 11 Oct 2018
Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach Ryo Karakida S. Akaho S. Amari FedML 47 140 0 04 Jun 2018
Understanding Batch Normalization Johan Bjorck Carla P. Gomes B. Selman Kilian Q. Weinberger 18 593 0 01 Jun 2018
Convolutional Neural Networks for Sentence Classification Yoon Kim AILaw VLM 255 13,364 0 25 Aug 2014