An analytic theory of generalization dynamics and transfer learning in deep linear networks

27 September 2018

Papers citing "An analytic theory of generalization dynamics and transfer learning in deep linear networks"

19 / 19 papers shown

Title
Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking) Yoonsoo Nam Seok Hyeong Lee Clementine Domine Yea Chan Park Charles London Wonyl Choi Niclas Goring Seungjai Lee AI4CE 163 1 0 28 Feb 2025
From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks Clémentine Dominé Nicolas Anguita A. Proca Lukas Braun D. Kunin P. Mediano Andrew M. Saxe 92 5 0 22 Sep 2024
Optimal Regularization Can Mitigate Double Descent Preetum Nakkiran Prayaag Venkat Sham Kakade Tengyu Ma 81 133 0 04 Mar 2020
Generalization in multitask deep neural classifiers: a statistical physics approach Tyler Lee A. Ndirango AI4CE 135 20 0 30 Oct 2019
The Emergence of Spectral Universality in Deep Networks Jeffrey Pennington S. Schoenholz Surya Ganguli 58 173 0 27 Feb 2018
Stronger generalization bounds for deep nets via a compression approach Sanjeev Arora Rong Ge Behnam Neyshabur Yi Zhang MLT AI4CE 84 640 0 14 Feb 2018
Size-Independent Sample Complexity of Neural Networks Noah Golowich Alexander Rakhlin Ohad Shamir 151 547 0 18 Dec 2017
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice Jeffrey Pennington S. Schoenholz Surya Ganguli ODL 43 252 0 13 Nov 2017
High-dimensional dynamics of generalization error in neural networks Madhu S. Advani Andrew M. Saxe AI4CE 134 469 0 10 Oct 2017
A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks Behnam Neyshabur Srinadh Bhojanapalli Nathan Srebro 80 606 0 29 Jul 2017
Spectrally-normalized margin bounds for neural networks Peter L. Bartlett Dylan J. Foster Matus Telgarsky ODL 202 1,217 0 26 Jun 2017
A Closer Look at Memorization in Deep Networks Devansh Arpit Stanislaw Jastrzebski Nicolas Ballas David M. Krueger Emmanuel Bengio ... Tegan Maharaj Asja Fischer Aaron Courville Yoshua Bengio Simon Lacoste-Julien TDI 122 1,816 0 16 Jun 2017
Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data Gintare Karolina Dziugaite Daniel M. Roy 106 813 0 31 Mar 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 339 4,626 0 10 Nov 2016
Deep Information Propagation S. Schoenholz Justin Gilmer Surya Ganguli Jascha Narain Sohl-Dickstein 79 367 0 04 Nov 2016
Policy Distillation Andrei A. Rusu Sergio Gomez Colmenarejo Çağlar Gülçehre Guillaume Desjardins J. Kirkpatrick Razvan Pascanu Volodymyr Mnih Koray Kavukcuoglu R. Hadsell 81 692 0 19 Nov 2015
Multi-task Sequence to Sequence Learning Minh-Thang Luong Quoc V. Le Ilya Sutskever Oriol Vinyals Lukasz Kaiser AIMat 107 807 0 19 Nov 2015
Norm-Based Capacity Control in Neural Networks Behnam Neyshabur Ryota Tomioka Nathan Srebro 290 587 0 27 Feb 2015
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Andrew M. Saxe James L. McClelland Surya Ganguli ODL 165 1,844 0 20 Dec 2013