Rigorous dynamical mean field theory for stochastic gradient descent
methods

Rigorous dynamical mean field theory for stochastic gradient descent methods

12 October 2022

Cédric Gerbelot

Emanuele Troiani

Francesca Mignacco

Florent Krzakala

Lenka Zdeborova

Papers citing "Rigorous dynamical mean field theory for stochastic gradient descent methods"

18 / 18 papers shown

Title
Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer Blake Bordelon C. Pehlevan AI4CE 64 1 0 04 Feb 2025
Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression Kai Tan Pierre C. Bellec 26 0 0 03 Oct 2024
How Feature Learning Can Improve Neural Scaling Laws Blake Bordelon Alexander B. Atanasov C. Pehlevan 57 12 0 26 Sep 2024
The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms Elizabeth Collins-Woodfin Inbar Seroussi Begona García Malaxechebarría Andrew W. Mackenzie Elliot Paquette Courtney Paquette 30 1 0 30 May 2024
Infinite Limits of Multi-head Transformer Dynamics Blake Bordelon Hamza Tahir Chaudhry C. Pehlevan AI4CE 47 9 0 24 May 2024
Linear Operator Approximate Message Passing (OpAMP) Riccardo Rossetti B. Nazer Galen Reeves 24 2 0 13 May 2024
Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features Rodrigo Veiga Anastasia Remizova Nicolas Macris 40 0 0 12 Feb 2024
The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents Yatin Dandi Emanuele Troiani Luca Arnaboldi Luca Pesce Lenka Zdeborová Florent Krzakala MLT 66 26 0 05 Feb 2024
A Dynamical Model of Neural Scaling Laws Blake Bordelon Alexander B. Atanasov C. Pehlevan 51 36 0 02 Feb 2024
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit Blake Bordelon Lorenzo Noci Mufan Bill Li Boris Hanin C. Pehlevan 32 23 0 28 Sep 2023
Loss Dynamics of Temporal Difference Reinforcement Learning Blake Bordelon P. Masset Henry Kuo C. Pehlevan AI4CE 21 0 0 10 Jul 2023
Batches Stabilize the Minimum Norm Risk in High Dimensional Overparameterized Linear Regression Shahar Stein Ioushua Inbar Hasidim O. Shayevitz M. Feder 19 0 0 14 Jun 2023
Phase transitions in the mini-batch size for sparse and dense two-layer neural networks Raffaele Marino F. Ricci-Tersenghi 30 14 0 10 May 2023
High-dimensional limit of one-pass SGD on least squares Elizabeth Collins-Woodfin Elliot Paquette 30 3 0 13 Apr 2023
High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance Krishnakumar Balasubramanian Promit Ghosal Ye He 38 5 0 03 Apr 2023
Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regression Bhavya Agrawalla Krishnakumar Balasubramanian Promit Ghosal 25 2 0 20 Feb 2023
The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks Blake Bordelon C. Pehlevan 41 22 0 05 Oct 2022
Dynamic mean field programming G. Stamatescu 19 0 0 10 Jun 2022