Exponential expressivity in deep neural networks through transient chaos

16 June 2016

Ben Poole

Subhaneil Lahiri

M. Raghu

Jascha Narain Sohl-Dickstein

Surya Ganguli

ArXiv PDF HTML

Papers citing "Exponential expressivity in deep neural networks through transient chaos"

49 / 148 papers shown

Title
Interpretable deep Gaussian processes with moments Chi-Ken Lu Scott Cheng-Hsin Yang Xiaoran Hao Patrick Shafto 18 19 0 27 May 2019
Is Deeper Better only when Shallow is Good? Eran Malach Shai Shalev-Shwartz 28 45 0 08 Mar 2019
Mean-field Analysis of Batch Normalization Ming-Bo Wei J. Stokes D. Schwab MLT 33 8 0 06 Mar 2019
On the Impact of the Activation Function on Deep Neural Networks Training Soufiane Hayou Arnaud Doucet Judith Rousseau ODL 25 195 0 19 Feb 2019
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent Jaehoon Lee Lechao Xiao S. Schoenholz Yasaman Bahri Roman Novak Jascha Narain Sohl-Dickstein Jeffrey Pennington 34 1,077 0 18 Feb 2019
Capacity allocation analysis of neural networks: A tool for principled architecture design Jonathan Donier 22 4 0 12 Feb 2019
On the security relevance of weights in deep learning Kathrin Grosse T. A. Trost Marius Mosbach Michael Backes Dietrich Klakow AAML 32 6 0 08 Feb 2019
Stiffness: A New Perspective on Generalization in Neural Networks Stanislav Fort Pawel Krzysztof Nowak Stanislaw Jastrzebski S. Narayanan 21 94 0 28 Jan 2019
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs D. Gilboa B. Chang Minmin Chen Greg Yang S. Schoenholz Ed H. Chi Jeffrey Pennington 34 40 0 25 Jan 2019
On the Convergence Rate of Training Recurrent Neural Networks Zeyuan Allen-Zhu Yuanzhi Li Zhao Song 23 191 0 29 Oct 2018
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes Roman Novak Lechao Xiao Jaehoon Lee Yasaman Bahri Greg Yang Jiri Hron Daniel A. Abolafia Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 25 307 0 11 Oct 2018
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning Charles H. Martin Michael W. Mahoney AI4CE 47 192 0 02 Oct 2018
A theoretical framework for deep locally connected ReLU network Yuandong Tian PINN 25 10 0 28 Sep 2018
Fisher Information and Natural Gradient Learning of Random Deep Networks S. Amari Ryo Karakida Masafumi Oizumi 19 34 0 22 Aug 2018
Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization G. Wang G. Giannakis Jie Chen MLT 24 131 0 14 Aug 2018
Deep Generative Modeling for Scene Synthesis via Hybrid Representations Zaiwei Zhang Zhenpei Yang Chongyang Ma Linjie Luo Alexander G. Huth E. Vouga Qi-Xing Huang GAN 3DPC 21 125 0 06 Aug 2018
On the Spectral Bias of Neural Networks Nasim Rahaman A. Baratin Devansh Arpit Felix Dräxler Min Lin Fred Hamprecht Yoshua Bengio Aaron Courville 57 1,395 0 22 Jun 2018
Learning One-hidden-layer ReLU Networks via Gradient Descent Xiao Zhang Yaodong Yu Lingxiao Wang Quanquan Gu MLT 30 134 0 20 Jun 2018
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 244 349 0 14 Jun 2018
Variational Implicit Processes Chao Ma Yingzhen Li José Miguel Hernández-Lobato BDL 24 68 0 06 Jun 2018
Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach Ryo Karakida S. Akaho S. Amari FedML 47 140 0 04 Jun 2018
Deep learning generalizes because the parameter-function map is biased towards simple functions Guillermo Valle Pérez Chico Q. Camargo A. Louis MLT AI4CE 18 226 0 22 May 2018
Tropical Geometry of Deep Neural Networks Liwen Zhang Gregory Naitzat Lek-Heng Lim 40 137 0 18 May 2018
Gaussian Process Behaviour in Wide Deep Neural Networks A. G. Matthews Mark Rowland Jiri Hron Richard Turner Zoubin Ghahramani BDL 35 550 0 30 Apr 2018
Analysis on the Nonlinear Dynamics of Deep Neural Networks: Topological Entropy and Chaos Husheng Li 22 11 0 03 Apr 2018
The Emergence of Spectral Universality in Deep Networks Jeffrey Pennington S. Schoenholz Surya Ganguli 21 170 0 27 Feb 2018
On Characterizing the Capacity of Neural Networks using Algebraic Topology William H. Guss Ruslan Salakhutdinov 35 89 0 13 Feb 2018
Deep Learning Works in Practice. But Does it Work in Theory? L. Hoang R. Guerraoui PINN 35 3 0 31 Jan 2018
Mean Field Residual Networks: On the Edge of Chaos Greg Yang S. Schoenholz 9 189 0 24 Dec 2017
The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions George Philipp D. Song J. Carbonell ODL 35 46 0 15 Dec 2017
Invariance of Weight Distributions in Rectified MLPs Russell Tsuchida Farbod Roosta-Khorasani M. Gallagher MLT 32 35 0 24 Nov 2017
Intriguing Properties of Adversarial Examples E. D. Cubuk Barret Zoph S. Schoenholz Quoc V. Le AAML 31 84 0 08 Nov 2017
An efficient quantum algorithm for generative machine learning Xun Gao Zhengyu Zhang L. Duan 17 25 0 06 Nov 2017
Deep Neural Networks as Gaussian Processes Jaehoon Lee Yasaman Bahri Roman Novak S. Schoenholz Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 19 1,074 0 01 Nov 2017
Approximating Continuous Functions by ReLU Nets of Minimal Width Boris Hanin Mark Sellke 41 229 0 31 Oct 2017
How deep learning works --The geometry of deep learning Xiao Dong Jiasong Wu Ling Zhou GNN 35 8 0 30 Oct 2017
A Correspondence Between Random Neural Networks and Statistical Field Theory S. Schoenholz Jeffrey Pennington Jascha Narain Sohl-Dickstein 19 20 0 18 Oct 2017
Classification and Geometry of General Perceptual Manifolds SueYeon Chung Daniel D. Lee H. Sompolinsky 14 151 0 17 Oct 2017
Do Neural Nets Learn Statistical Laws behind Natural Language? Shuntaro Takahashi Kumiko Tanaka-Ishii 41 27 0 16 Jul 2017
Deep Convolutional Framelets: A General Deep Learning Framework for Inverse Problems J. C. Ye Yoseob Han Eunju Cha 36 16 0 03 Jul 2017
Recovery Guarantees for One-hidden-layer Neural Networks Kai Zhong Zhao Song Prateek Jain Peter L. Bartlett Inderjit S. Dhillon MLT 34 336 0 10 Jun 2017
Robustness of classifiers to universal perturbations: a geometric perspective Seyed-Mohsen Moosavi-Dezfooli Alhussein Fawzi Omar Fawzi P. Frossard Stefano Soatto AAML 24 118 0 26 May 2017
The power of deeper networks for expressing natural functions David Rolnick Max Tegmark 36 174 0 16 May 2017
Survey of Expressivity in Deep Neural Networks M. Raghu Ben Poole Jon M. Kleinberg Surya Ganguli Jascha Narain Sohl-Dickstein 30 15 0 24 Nov 2016
Deep Residual Learning for Compressed Sensing CT Reconstruction via Persistent Homology Analysis Yoseob Han J. Yoo J. C. Ye 32 203 0 19 Nov 2016
Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification Woong Bae J. Yoo J. C. Ye SupR 24 177 0 19 Nov 2016
Why does deep and cheap learning work so well? Henry W. Lin Max Tegmark David Rolnick 40 602 0 29 Aug 2016
On the Expressive Power of Deep Neural Networks M. Raghu Ben Poole Jon M. Kleinberg Surya Ganguli Jascha Narain Sohl-Dickstein 29 777 0 16 Jun 2016
Quantifying the probable approximation error of probabilistic inference programs Marco F. Cusumano-Towner Vikash K. Mansinghka 33 7 0 31 May 2016