v1v2 (latest)

Fluctuation-dissipation relations for stochastic gradient descent

28 September 2018

Papers citing "Fluctuation-dissipation relations for stochastic gradient descent"

39 / 39 papers shown

Title
SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training Ildus Sadrtdinov Ivan Klimov E. Lobacheva Dmitry Vetrov 28 0 0 29 May 2025
AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent Nikola Surjanovic Alexandre Bouchard-Côté Trevor Campbell 32 0 0 27 May 2025
Formation of Representations in Neural Networks Liu Ziyin Isaac Chuang Tomer Galanti T. Poggio 247 7 0 03 Oct 2024
Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron Christian Schmid James M. Murray 91 0 0 05 Sep 2024
How Neural Networks Learn the Support is an Implicit Regularization Effect of SGD Pierfrancesco Beneventano Andrea Pinto Tomaso A. Poggio MLT 58 1 0 17 Jun 2024
Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems Maria Chiara Angelini A. Cavaliere Raffaele Marino F. Ricci-Tersenghi 124 5 0 11 Sep 2023
Machine learning in and out of equilibrium Shishir Adhikari Alkan Kabakcciouglu A. Strang Deniz Yuret M. Hinczewski 65 5 0 06 Jun 2023
Climate Intervention Analysis using AI Model Guided by Statistical Physics Principles S. K. Kim Kalai Ramea Salva Rühling Cachay H. Hirasawa Subhashis Hazarika D. Hingmire Peetak Mitra P. Rasch Hansi K. A. Singh AI4CE 54 0 0 07 Feb 2023
Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learning Haibo Yang Pei-Yuan Qiu Jia Liu FedML 74 12 0 03 Oct 2022
Implicit regularization of dropout Zhongwang Zhang Zhi-Qin John Xu 70 29 0 13 Jul 2022
Studying Generalization Through Data Averaging C. Gomez-Uribe FedML 133 0 0 28 Jun 2022
Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization G. Luca E. Silverstein 73 11 0 26 Jan 2022
Optimization Planning for 3D ConvNets Zhaofan Qiu Ting Yao Chong-Wah Ngo Tao Mei 3DPC 3DH 86 9 0 11 Jan 2022
The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion D. Kunin Javier Sagastuy-Breña Lauren Gillespie Eshed Margalit Hidenori Tanaka Surya Ganguli Daniel L. K. Yamins 93 20 0 19 Jul 2021
Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms A. Camuto George Deligiannidis Murat A. Erdogdu Mert Gurbuzbalaban Umut cSimcsekli Lingjiong Zhu 80 29 0 09 Jun 2021
Fluctuation-dissipation Type Theorem in Stochastic Linear Learning Manhyung Han J. Park Taewoong Lee J. Han 41 6 0 04 Jun 2021
How to decay your learning rate Aitor Lewkowycz 114 24 0 23 Mar 2021
SVRG Meets AdaGrad: Painless Variance Reduction Benjamin Dubois-Taine Sharan Vaswani Reza Babanezhad Mark Schmidt Simon Lacoste-Julien 61 18 0 18 Feb 2021
Strength of Minibatch Noise in SGD Liu Ziyin Kangqiao Liu Takashi Mori Masakuni Ueda ODL MLT 64 35 0 10 Feb 2021
SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality Courtney Paquette Kiwon Lee Fabian Pedregosa Elliot Paquette 59 35 0 08 Feb 2021
On the Origin of Implicit Regularization in Stochastic Gradient Descent Samuel L. Smith Benoit Dherin David Barrett Soham De MLT 62 204 0 28 Jan 2021
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics D. Kunin Javier Sagastuy-Breña Surya Ganguli Daniel L. K. Yamins Hidenori Tanaka 167 80 0 08 Dec 2020
Robust, Accurate Stochastic Optimization for Variational Inference Akash Kumar Dhaka Alejandro Catalina Michael Riis Andersen Maans Magnusson Jonathan H. Huggins Aki Vehtari 71 34 0 01 Sep 2020
Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum Jerry Chee Ping Li 43 12 0 27 Aug 2020
Active Importance Sampling for Variational Objectives Dominated by Rare Events: Consequences for Optimization and Generalization Grant M. Rotskoff Andrew R. Mitchell Eric Vanden-Eijnden 51 13 0 11 Aug 2020
Shape Matters: Understanding the Implicit Bias of the Noise Covariance Jeff Z. HaoChen Colin Wei Jason D. Lee Tengyu Ma 213 95 0 15 Jun 2020
The Implicit and Explicit Regularization Effects of Dropout Colin Wei Sham Kakade Tengyu Ma 123 118 0 28 Feb 2020
Statistical Adaptive Stochastic Gradient Methods Pengchuan Zhang Hunter Lang Qiang Liu Lin Xiao ODL 74 11 0 25 Feb 2020
The Early Phase of Neural Network Training Jonathan Frankle D. Schwab Ari S. Morcos 96 174 0 24 Feb 2020
On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks Umut Simsekli Mert Gurbuzbalaban T. H. Nguyen G. Richard Levent Sagun 88 59 0 29 Nov 2019
Understanding the Role of Momentum in Stochastic Gradient Methods Igor Gitman Hunter Lang Pengchuan Zhang Lin Xiao 77 95 0 30 Oct 2019
From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks H. Yoshino 44 7 0 22 Oct 2019
How noise affects the Hessian spectrum in overparameterized neural networks Ming-Bo Wei D. Schwab 85 28 0 01 Oct 2019
Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective Guan-Horng Liu Evangelos A. Theodorou AI4CE 121 72 0 28 Aug 2019
First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise T. H. Nguyen Umut Simsekli Mert Gurbuzbalaban G. Richard 79 65 0 21 Jun 2019
On the interplay between noise and curvature and its effect on optimization and generalization Valentin Thomas Fabian Pedregosa B. V. Merrienboer Pierre-Antoine Mangazol Yoshua Bengio Nicolas Le Roux 64 61 0 18 Jun 2019
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study Daniel S. Park Jascha Narain Sohl-Dickstein Quoc V. Le Samuel L. Smith 96 57 0 09 May 2019
Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians Vardan Papyan 82 88 0 24 Jan 2019
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks Umut Simsekli Levent Sagun Mert Gurbuzbalaban 114 252 0 18 Jan 2019