Spectrum concentration in deep residual learning: a free probability
approach

v1v2v3 (latest)

Spectrum concentration in deep residual learning: a free probability approach

31 July 2018

ArXiv (abs)PDF HTML

Papers citing "Spectrum concentration in deep residual learning: a free probability approach"

14 / 14 papers shown

Title
Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function W. Tarnowski P. Warchol Stanislaw Jastrzebski Jacek Tabor M. Nowak 58 38 0 24 Sep 2018
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 304 354 0 14 Jun 2018
On the Spectrum of Random Features Maps of High Dimensional Data Zhenyu Liao Romain Couillet 57 51 0 30 May 2018
The Emergence of Spectral Universality in Deep Networks Jeffrey Pennington S. Schoenholz Surya Ganguli 64 173 0 27 Feb 2018
Mean Field Residual Networks: On the Edge of Chaos Greg Yang S. Schoenholz 71 194 0 24 Dec 2017
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice Jeffrey Pennington S. Schoenholz Surya Ganguli ODL 43 254 0 13 Nov 2017
Identity Matters in Deep Learning Moritz Hardt Tengyu Ma OOD 92 399 0 14 Nov 2016
Deep Information Propagation S. Schoenholz Justin Gilmer Surya Ganguli Jascha Narain Sohl-Dickstein 85 371 0 04 Nov 2016
Identity Mappings in Deep Residual Networks Kaiming He Xinming Zhang Shaoqing Ren Jian Sun 354 10,196 0 16 Mar 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,426 0 10 Dec 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 465 43,341 0 11 Feb 2015
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 341 18,651 0 06 Feb 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.1K 150,312 0 22 Dec 2014
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Andrew M. Saxe James L. McClelland Surya Ganguli ODL 188 1,852 0 20 Dec 2013