What Happens after SGD Reaches Zero Loss? --A Mathematical Framework

13 October 2021

Tianhao Wang

Papers citing "What Happens after SGD Reaches Zero Loss? --A Mathematical Framework"

22 / 22 papers shown

Title
Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training Zhanpeng Zhou Mingze Wang Yuchen Mao Bingrui Li Junchi Yan AAML 71 1 0 14 Oct 2024
Reparameterization invariance in approximate Bayesian inference Hrittik Roy M. Miani Carl Henrik Ek Philipp Hennig Marvin Pfortner Lukas Tatzel Søren Hauberg BDL 72 9 0 05 Jun 2024
Does SGD really happen in tiny subspaces? Minhak Song Kwangjun Ahn Chulhee Yun 90 6 1 25 May 2024
Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity Scott Pesme Loucas Pillaud-Vivien Nicolas Flammarion 43 100 0 17 Jun 2021
On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent Shahar Azulay E. Moroshko Mor Shpigel Nacson Blake E. Woodworth Nathan Srebro Amir Globerson Daniel Soudry AI4CE 53 73 0 19 Feb 2021
Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets? Zhiyuan Li Yi Zhang Sanjeev Arora BDL MLT 71 39 0 16 Oct 2020
Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate Zhiyuan Li Kaifeng Lyu Sanjeev Arora 66 74 0 06 Oct 2020
Shape Matters: Understanding the Implicit Bias of the Noise Covariance Jeff Z. HaoChen Colin Wei Jason D. Lee Tengyu Ma 106 94 0 15 Jun 2020
Implicit Regularization for Optimal Sparse Recovery Tomas Vaskevicius Varun Kanade Patrick Rebeschini 35 100 0 11 Sep 2019
Gradient Descent Maximizes the Margin of Homogeneous Neural Networks Kaifeng Lyu Jian Li 68 332 0 13 Jun 2019
Kernel and Rich Regimes in Overparametrized Models Blake E. Woodworth Suriya Gunasekar Pedro H. P. Savarese E. Moroshko Itay Golan Jason D. Lee Daniel Soudry Nathan Srebro 66 358 0 13 Jun 2019
Implicit Regularization in Deep Matrix Factorization Sanjeev Arora Nadav Cohen Wei Hu Yuping Luo AI4CE 66 500 0 31 May 2019
Convergence rates for the stochastic gradient descent method for non-convex objective functions Benjamin J. Fehrman Benjamin Gess Arnulf Jentzen 56 101 0 02 Apr 2019
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation Greg Yang 98 286 0 13 Feb 2019
Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers Zeyuan Allen-Zhu Yuanzhi Li Yingyu Liang MLT 140 769 0 12 Nov 2018
Gradient Descent Provably Optimizes Over-parameterized Neural Networks S. Du Xiyu Zhai Barnabás Póczós Aarti Singh MLT ODL 159 1,261 0 04 Oct 2018
Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data Yuanzhi Li Yingyu Liang MLT 149 652 0 03 Aug 2018
Essentially No Barriers in Neural Network Energy Landscape Felix Dräxler K. Veschgini M. Salmhofer Fred Hamprecht MoMe 97 430 0 02 Mar 2018
Three Factors Influencing Minima in SGD Stanislaw Jastrzebski Zachary Kenton Devansh Arpit Nicolas Ballas Asja Fischer Yoshua Bengio Amos Storkey 70 459 0 13 Nov 2017
Topology and Geometry of Half-Rectified Network Optimization C. Freeman Joan Bruna 142 235 0 04 Nov 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 369 2,922 0 15 Sep 2016
One weird trick for parallelizing convolutional neural networks A. Krizhevsky GNN 86 1,297 0 23 Apr 2014