Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels

4 December 2022

Papers citing "Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels"

40 / 40 papers shown

Title
Neural Tangent Kernel: A Survey Eugene Golikov Eduard Pokonechnyy Vladimir Korviakov 48 14 0 29 Aug 2022
Nondeterminism and Instability in Neural Network Optimization Cecilia Summers M. Dinneen 52 40 0 08 Mar 2021
Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization Qianyi Li H. Sompolinsky 126 72 0 07 Dec 2020
Learning from Noisy Labels with Deep Neural Networks: A Survey Hwanjun Song Minseok Kim Dongmin Park Yooju Shin Jae-Gil Lee NoLa 92 985 0 16 Jul 2020
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes Greg Yang 105 198 0 28 Oct 2019
On Mutual Information Maximization for Representation Learning Michael Tschannen Josip Djolonga Paul Kishan Rubenstein Sylvain Gelly Mario Lucic SSL 154 493 0 31 Jul 2019
Contrastive Multiview Coding Yonglong Tian Dilip Krishnan Phillip Isola SSL 151 2,395 0 13 Jun 2019
Data-Efficient Image Recognition with Contrastive Predictive Coding Olivier J. Hénaff A. Srinivas J. Fauw Ali Razavi Carl Doersch S. M. Ali Eslami Aaron van den Oord SSL 106 1,427 0 22 May 2019
On Variational Bounds of Mutual Information Ben Poole Sherjil Ozair Aaron van den Oord Alexander A. Alemi George Tucker SSL 90 806 0 16 May 2019
Nonparametric Density Estimation for High-Dimensional Data - Algorithms and Applications Zhipeng Wang D. W. Scott 54 69 0 30 Mar 2019
Mean Field Analysis of Deep Neural Networks Justin A. Sirignano K. Spiliopoulos 54 82 0 11 Mar 2019
A Theoretical Analysis of Contrastive Unsupervised Representation Learning Sanjeev Arora H. Khandeparkar M. Khodak Orestis Plevrakis Nikunj Saunshi SSL 96 776 0 25 Feb 2019
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent Jaehoon Lee Lechao Xiao S. Schoenholz Yasaman Bahri Roman Novak Jascha Narain Sohl-Dickstein Jeffrey Pennington 178 1,097 0 18 Feb 2019
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei Theodor Misiakiewicz Andrea Montanari MLT 68 278 0 16 Feb 2019
Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks Phan-Minh Nguyen AI4CE 44 72 0 07 Feb 2019
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs D. Gilboa B. Chang Minmin Chen Greg Yang S. Schoenholz Ed H. Chi Jeffrey Pennington 68 42 0 25 Jan 2019
On Lazy Training in Differentiable Programming Lénaïc Chizat Edouard Oyallon Francis R. Bach 100 833 0 19 Dec 2018
Formal Limitations on the Measurement of Mutual Information David A. McAllester K. Stratos SSL 63 275 0 10 Nov 2018
Deep Metric Learning with Hierarchical Triplet Loss Weifeng Ge Weilin Huang Dengke Dong Matthew R. Scott 144 413 0 16 Oct 2018
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes Roman Novak Lechao Xiao Jaehoon Lee Yasaman Bahri Greg Yang Jiri Hron Daniel A. Abolafia Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 54 308 0 11 Oct 2018
Learning deep representations by mutual information estimation and maximization R. Devon Hjelm A. Fedorov Samuel Lavoie-Marchildon Karan Grewal Phil Bachman Adam Trischler Yoshua Bengio SSL DRL 284 2,661 0 20 Aug 2018
Deep Convolutional Networks as shallow Gaussian Processes Adrià Garriga-Alonso C. Rasmussen Laurence Aitchison BDL UQCV 83 270 0 16 Aug 2018
Representation Learning with Contrastive Predictive Coding Aaron van den Oord Yazhe Li Oriol Vinyals DRL SSL 280 10,253 0 10 Jul 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 234 3,191 0 20 Jun 2018
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks Minmin Chen Jeffrey Pennington S. Schoenholz SyDa AI4CE 52 116 0 14 Jun 2018
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 298 353 0 14 Jun 2018
Gaussian Process Behaviour in Wide Deep Neural Networks A. G. Matthews Mark Rowland Jiri Hron Richard Turner Zoubin Ghahramani BDL 141 557 0 30 Apr 2018
The Emergence of Spectral Universality in Deep Networks Jeffrey Pennington S. Schoenholz Surya Ganguli 56 173 0 27 Feb 2018
State Representation Learning for Control: An Overview Timothée Lesort Natalia Díaz Rodríguez Jean-François Goudou David Filliat OffRL 96 320 0 12 Feb 2018
Mean Field Residual Networks: On the Edge of Chaos Greg Yang S. Schoenholz 53 191 0 24 Dec 2017
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice Jeffrey Pennington S. Schoenholz Surya Ganguli ODL 43 252 0 13 Nov 2017
Gaussian Lower Bound for the Information Bottleneck Limit Amichai Painsky Naftali Tishby 28 15 0 07 Nov 2017
Deep Neural Networks as Gaussian Processes Jaehoon Lee Yasaman Bahri Roman Novak S. Schoenholz Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 113 1,091 0 01 Nov 2017
Representation Learning on Graphs: Methods and Applications William L. Hamilton Rex Ying J. Leskovec GNN 147 1,971 0 17 Sep 2017
Deep Variational Information Bottleneck Alexander A. Alemi Ian S. Fischer Joshua V. Dillon Kevin Patrick Murphy 98 1,714 0 01 Dec 2016
Deep Information Propagation S. Schoenholz Justin Gilmer Surya Ganguli Jascha Narain Sohl-Dickstein 77 367 0 04 Nov 2016
Exponential expressivity in deep neural networks through transient chaos Ben Poole Subhaneil Lahiri M. Raghu Jascha Narain Sohl-Dickstein Surya Ganguli 88 591 0 16 Jun 2016
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Alec Radford Luke Metz Soumith Chintala GAN OOD 243 13,989 0 19 Nov 2015
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Andrew M. Saxe James L. McClelland Surya Ganguli ODL 162 1,844 0 20 Dec 2013
Representation Learning: A Review and New Perspectives Yoshua Bengio Aaron Courville Pascal Vincent OOD SSL 220 12,422 0 24 Jun 2012