Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

1 February 2022

Lenka Zdeborová

Papers citing "Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks"

30 / 30 papers shown

Title
Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning Andrew Bond Zafer Dogan 28 0 0 01 Nov 2024
A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities Yatin Dandi Luca Pesce Hugo Cui Florent Krzakala Yue M. Lu Bruno Loureiro MLT 39 1 0 24 Oct 2024
A theoretical perspective on mode collapse in variational inference Roman Soletskyi Marylou Gabrié Bruno Loureiro DRL 37 2 0 17 Oct 2024
Optimal Protocols for Continual Learning via Statistical Physics and Control Theory Francesco Mori Stefano Sarao Mannelli Francesca Mignacco 38 3 0 26 Sep 2024
Training Dynamics of Nonlinear Contrastive Learning Model in the High Dimensional Limit Lineghuan Meng Chuang Wang 28 1 0 11 Jun 2024
Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs Luca Arnaboldi Yatin Dandi Florent Krzakala Bruno Loureiro Luca Pesce Ludovic Stephan 49 1 0 04 Jun 2024
Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features Rodrigo Veiga Anastasia Remizova Nicolas Macris 40 0 0 12 Feb 2024
Should Under-parameterized Student Networks Copy or Average Teacher Weights? Berfin Simsek Amire Bendjeddou W. Gerstner Johanni Brea 32 7 0 03 Nov 2023
Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing Yuma Ichikawa Koji Hukushima 29 5 0 24 Oct 2023
On the different regimes of Stochastic Gradient Descent Antonio Sclocchi M. Wyart 31 18 0 19 Sep 2023
Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape Persia Jana Kamali Pierfrancesco Urbani 25 6 0 09 Sep 2023
Minibatch training of neural network ensembles via trajectory sampling Jamie F. Mair Luke Causer J. P. Garrahan 21 0 0 23 Jun 2023
The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions Nishil Patel Sebastian Lee Stefano Sarao Mannelli Sebastian Goldt Adrew Saxe OffRL 36 3 0 17 Jun 2023
Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD Luca Arnaboldi Florent Krzakala Bruno Loureiro Ludovic Stephan MLT 37 3 0 29 May 2023
Phase transitions in the mini-batch size for sparse and dense two-layer neural networks Raffaele Marino F. Ricci-Tersenghi 30 14 0 10 May 2023
Leveraging the two timescale regime to demonstrate convergence of neural networks Pierre Marion Raphael Berthier 38 5 0 19 Apr 2023
Mapping of attention mechanisms to a generalized Potts model Riccardo Rende Federica Gerace Alessandro Laio Sebastian Goldt 21 22 0 14 Apr 2023
Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks Blake Bordelon Cengiz Pehlevan MLT 40 29 0 06 Apr 2023
High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance Krishnakumar Balasubramanian Promit Ghosal Ye He 40 5 0 03 Apr 2023
From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks Luca Arnaboldi Ludovic Stephan Florent Krzakala Bruno Loureiro MLT 38 31 0 12 Feb 2023
Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization R. Srinivasan Francesca Mignacco M. Sorbaro Maria Refinetti A. Cooper Gabriel Kreiman Giorgia Dellaferrera 32 15 0 10 Feb 2023
Learning Single-Index Models with Shallow Neural Networks A. Bietti Joan Bruna Clayton Sanford M. Song 170 68 0 27 Oct 2022
Rigorous dynamical mean field theory for stochastic gradient descent methods Cédric Gerbelot Emanuele Troiani Francesca Mignacco Florent Krzakala Lenka Zdeborova 38 26 0 12 Oct 2022
Neural Networks Efficiently Learn Low-Dimensional Representations with SGD Alireza Mousavi-Hosseini Sejun Park M. Girotti Ioannis Mitliagkas Murat A. Erdogdu MLT 324 48 0 29 Sep 2022
Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energy Michael Celentano 46 20 0 19 Aug 2022
High-dimensional limit theorems for SGD: Effective dynamics and critical scaling Gerard Ben Arous Reza Gheissari Aukosh Jagannath 62 58 0 08 Jun 2022
High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation Jimmy Ba Murat A. Erdogdu Taiji Suzuki Zhichao Wang Denny Wu Greg Yang MLT 42 121 0 03 May 2022
Data-driven emergence of convolutional structure in neural networks Alessandro Ingrosso Sebastian Goldt 61 38 0 01 Feb 2022
The dynamics of representation learning in shallow, non-linear autoencoders Maria Refinetti Sebastian Goldt AI4CE 22 17 0 06 Jan 2022
Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach Grant M. Rotskoff Eric Vanden-Eijnden 68 118 0 02 May 2018