Exponential expressivity in deep neural networks through transient chaos

16 June 2016

Ben Poole

Subhaneil Lahiri

M. Raghu

Jascha Narain Sohl-Dickstein

Surya Ganguli

ArXiv PDF HTML

Papers citing "Exponential expressivity in deep neural networks through transient chaos"

50 / 148 papers shown

Title
Bayesian neural network unit priors and generalized Weibull-tail property M. Vladimirova Julyan Arbel Stéphane Girard BDL 54 9 0 06 Oct 2021
On the Impact of Stable Ranks in Deep Nets B. Georgiev L. Franken Mayukh Mukherjee Georgios Arvanitidis 21 3 0 05 Oct 2021
Deep Neural Networks and Tabular Data: A Survey V. Borisov Tobias Leemann Kathrin Seßler Johannes Haug Martin Pawelczyk Gjergji Kasneci LMTD 47 648 0 05 Oct 2021
Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks Zhichao Wang Yizhe Zhu 35 18 0 20 Sep 2021
Neural Operator: Learning Maps Between Function Spaces Nikola B. Kovachki Zong-Yi Li Burigede Liu Kamyar Azizzadenesheli K. Bhattacharya Andrew M. Stuart Anima Anandkumar AI4CE 52 440 0 19 Aug 2021
Convergence of Deep ReLU Networks Yuesheng Xu Haizhang Zhang 37 27 0 27 Jul 2021
Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group J. Erdmenger Kevin T. Grosvenor R. Jefferson 54 17 0 14 Jul 2021
Precise characterization of the prior predictive distribution of deep ReLU networks Lorenzo Noci Gregor Bachmann Kevin Roth Sebastian Nowozin Thomas Hofmann BDL UQCV 29 32 0 11 Jun 2021
The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective Geoff Pleiss John P. Cunningham 28 24 0 11 Jun 2021
Activation function design for deep networks: linearity and effective initialisation Michael Murray V. Abrol Jared Tanner ODL LLMSV 29 18 0 17 May 2021
Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes James Lucas Juhan Bae Michael Ruogu Zhang Stanislav Fort R. Zemel Roger C. Grosse MoMe 164 28 0 22 Apr 2021
Deep ReLU Networks Preserve Expected Length Boris Hanin Ryan Jeong David Rolnick 29 14 0 21 Feb 2021
Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks Quynh N. Nguyen Marco Mondelli Guido Montúfar 25 81 0 21 Dec 2020
On 1/n neural representation and robustness Josue Nassar Piotr A. Sokól SueYeon Chung K. Harris Il Memming Park AAML OOD 24 23 0 08 Dec 2020
On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case L. Pastur V. Slavin CML 24 12 0 20 Nov 2020
Chaos and Complexity from Quantum Neural Network: A study with Diffusion Metric in Machine Learning S. Choudhury Ankan Dutta Debisree Ray 22 21 0 16 Nov 2020
Stable ResNet Soufiane Hayou Eugenio Clerico Bo He George Deligiannidis Arnaud Doucet Judith Rousseau ODL SSeg 46 51 0 24 Oct 2020
Tensor Programs III: Neural Matrix Laws Greg Yang 14 44 0 22 Sep 2020
Review: Deep Learning in Electron Microscopy Jeffrey M. Ede 36 79 0 17 Sep 2020
Explicit Regularisation in Gaussian Noise Injections A. Camuto M. Willetts Umut Simsekli Stephen J. Roberts Chris Holmes 25 55 0 14 Jul 2020
Reformulation of the No-Free-Lunch Theorem for Entangled Data Sets Kunal Sharma M. Cerezo Zoë Holmes L. Cincio A. Sornborger Patrick J. Coles 24 48 0 09 Jul 2020
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization? Yaniv Blumenfeld D. Gilboa Daniel Soudry ODL 30 13 0 02 Jul 2020
Deep Isometric Learning for Visual Recognition Haozhi Qi Chong You Xueliang Wang Yi Ma Jitendra Malik VLM 35 54 0 30 Jun 2020
Associative Memory in Iterated Overparameterized Sigmoid Autoencoders Yibo Jiang Cengiz Pehlevan 19 13 0 30 Jun 2020
Tensor Programs II: Neural Tangent Kernel for Any Architecture Greg Yang 58 135 0 25 Jun 2020
Deep Residual Mixture Models Perttu Hämäläinen Martin Trapp Tuure Saloheimo Arno Solin 36 8 0 22 Jun 2020
Neural Anisotropy Directions Guillermo Ortiz-Jiménez Apostolos Modas Seyed-Mohsen Moosavi-Dezfooli P. Frossard 34 16 0 17 Jun 2020
Is deeper better? It depends on locality of relevant features Takashi Mori Masahito Ueda OOD 25 4 0 26 May 2020
Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks Z. Fan Zhichao Wang 44 71 0 25 May 2020
Learning the gravitational force law and other analytic functions Atish Agarwala Abhimanyu Das Rina Panigrahy Qiuyi Zhang MLT 16 0 0 15 May 2020
ReZero is All You Need: Fast Convergence at Large Depth Thomas C. Bachlechner Bodhisattwa Prasad Majumder H. H. Mao G. Cottrell Julian McAuley AI4CE 21 276 0 10 Mar 2020
Untangling in Invariant Speech Recognition Cory Stephenson J. Feather Suchismita Padhy Oguz H. Elibol Hanlin Tang Josh H. McDermott SueYeon Chung SSL 30 30 0 03 Mar 2020
Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective S. Amari 27 12 0 20 Jan 2020
On Random Matrices Arising in Deep Neural Networks. Gaussian Case L. Pastur 21 23 0 17 Jan 2020
Deep Learning-Based Solvability of Underdetermined Inverse Problems in Medical Imaging Chang Min Hyun Seong Hyeon Baek M. Lee S. Lee J.K. Seo 27 39 0 06 Jan 2020
Empirical Studies on the Properties of Linear Regions in Deep Neural Networks Xiao Zhang Dongrui Wu 21 38 0 04 Jan 2020
A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks Zhaodong Chen Lei Deng Bangyan Wang Guoqi Li Yuan Xie 35 28 0 01 Jan 2020
Towards Efficient Training for Neural Network Quantization Qing Jin Linjie Yang Zhenyu A. Liao MQ 19 42 0 21 Dec 2019
Mean field theory for deep dropout networks: digging up gradient backpropagation deeply Wei Huang R. Xu Weitao Du Yutian Zeng Yunce Zhao 25 6 0 19 Dec 2019
Optimization for deep learning: theory and algorithms Ruoyu Sun ODL 27 168 0 19 Dec 2019
Neural Tangents: Fast and Easy Infinite Neural Networks in Python Roman Novak Lechao Xiao Jiri Hron Jaehoon Lee Alexander A. Alemi Jascha Narain Sohl-Dickstein S. Schoenholz 38 225 0 05 Dec 2019
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes Greg Yang 33 193 0 28 Oct 2019
From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks H. Yoshino 22 6 0 22 Oct 2019
On the expected behaviour of noise regularised deep neural networks as Gaussian processes Arnu Pretorius Herman Kamper Steve Kroon 27 9 0 12 Oct 2019
Neural networks are a priori biased towards Boolean functions with low entropy Chris Mingard Joar Skalse Guillermo Valle Pérez David Martínez-Rubio Vladimir Mikulik A. Louis FAtt AI4CE 24 37 0 25 Sep 2019
Optimal Machine Intelligence at the Edge of Chaos Ling Feng Lin Zhang C. Lai 33 8 0 11 Sep 2019
Optimal Function Approximation with Relu Neural Networks Bo Liu Yi Liang 25 33 0 09 Sep 2019
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks Ryo Karakida S. Akaho S. Amari 27 40 0 07 Jun 2019
Infinitely deep neural networks as diffusion processes Stefano Peluchetti Stefano Favaro ODL 14 31 0 27 May 2019
Interpretable deep Gaussian processes with moments Chi-Ken Lu Scott Cheng-Hsin Yang Xiaoran Hao Patrick Shafto 18 19 0 27 May 2019