Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions

13 December 2021

Papers citing "Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions"

39 / 39 papers shown

Title
Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation Simon Eberle Arnulf Jentzen Adrian Riekert G. Weiss 54 12 0 18 Aug 2021
A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions Arnulf Jentzen Adrian Riekert 56 13 0 10 Aug 2021
Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation Arnulf Jentzen Adrian Riekert 45 23 0 09 Jul 2021
Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis Stephan Wojtowytsch 45 51 0 04 May 2021
A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions Arnulf Jentzen Adrian Riekert MLT 66 13 0 01 Apr 2021
Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions Patrick Cheridito Arnulf Jentzen Florian Rossmannek 46 10 0 19 Mar 2021
Convergence rates for gradient descent in the training of overparameterized artificial neural networks with biases Arnulf Jentzen T. Kröger ODL 56 7 0 23 Feb 2021
A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions Patrick Cheridito Arnulf Jentzen Adrian Riekert Florian Rossmannek 40 24 0 19 Feb 2021
Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes Steffen Dereich Sebastian Kassing 58 27 0 16 Feb 2021
Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't E. Weinan Chao Ma Stephan Wojtowytsch Lei Wu AI4CE 63 134 0 22 Sep 2020
A Dynamical Central Limit Theorem for Shallow Neural Networks Zhengdao Chen Grant M. Rotskoff Joan Bruna Eric Vanden-Eijnden 53 30 0 21 Aug 2020
Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms A. Lovas Iosif Lytras Miklós Rásonyi Sotirios Sabanis 53 25 0 25 Jun 2020
Non-convergence of stochastic gradient descent in the training of deep neural networks Patrick Cheridito Arnulf Jentzen Florian Rossmannek 46 37 0 12 Jun 2020
Stopping Criteria for, and Strong Convergence of, Stochastic Gradient Descent on Bottou-Curtis-Nocedal Functions V. Patel 46 23 0 01 Apr 2020
Nonasymptotic analysis of Stochastic Gradient Hamiltonian Monte Carlo under local conditions for nonconvex optimization Ömer Deniz Akyildiz Sotirios Sabanis 61 17 0 13 Feb 2020
Full error analysis for the training of deep neural networks C. Beck Arnulf Jentzen Benno Kuckuck 34 47 0 30 Sep 2019
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks Guodong Zhang James Martens Roger C. Grosse ODL 74 124 0 27 May 2019
The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent Karthik A. Sankararaman Soham De Zheng Xu Wenjie Huang Tom Goldstein ODL 57 103 0 15 Apr 2019
A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics E. Weinan Chao Ma Lei Wu MLT 53 123 0 08 Apr 2019
Convergence rates for the stochastic gradient descent method for non-convex objective functions Benjamin J. Fehrman Benjamin Gess Arnulf Jentzen 56 101 0 02 Apr 2019
Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions Yunwen Lei Ting Hu Guiying Li K. Tang MLT 60 116 0 03 Feb 2019
Non-asymptotic Analysis of Biased Stochastic Approximation Scheme Belhal Karimi B. Miasojedow Eric Moulines Hoi-To Wai 42 90 0 02 Feb 2019
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks Sanjeev Arora S. Du Wei Hu Zhiyuan Li Ruosong Wang MLT 146 966 0 24 Jan 2019
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks Difan Zou Yuan Cao Dongruo Zhou Quanquan Gu ODL 137 448 0 21 Nov 2018
Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers Zeyuan Allen-Zhu Yuanzhi Li Yingyu Liang MLT 140 769 0 12 Nov 2018
A Convergence Theory for Deep Learning via Over-Parameterization Zeyuan Allen-Zhu Yuanzhi Li Zhao Song AI4CE ODL 194 1,457 0 09 Nov 2018
Gradient Descent Finds Global Minima of Deep Neural Networks S. Du Jason D. Lee Haochuan Li Liwei Wang Masayoshi Tomizuka ODL 147 1,133 0 09 Nov 2018
Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks Ohad Shamir 47 46 0 23 Sep 2018
Collapse of Deep and Narrow Neural Nets Lu Lu Yanhui Su George Karniadakis ODL 52 155 0 15 Aug 2018
Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data Yuanzhi Li Yingyu Liang MLT 149 652 0 03 Aug 2018
Solving the Kolmogorov PDE by means of deep learning C. Beck S. Becker Philipp Grohs Nor Jaafari Arnulf Jentzen 46 94 0 01 Jun 2018
Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning rates Arnulf Jentzen Philippe von Wurstemberger 79 31 0 22 Mar 2018
An overview of gradient descent optimization algorithms Sebastian Ruder ODL 180 6,170 0 15 Sep 2016
Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition Hamed Karimi J. Nutini Mark Schmidt 225 1,208 0 16 Aug 2016
Optimization Methods for Large-Scale Machine Learning Léon Bottou Frank E. Curtis J. Nocedal 177 3,198 0 15 Jun 2016
Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions Ioannis Panageas Georgios Piliouras 41 141 0 02 May 2016
Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition Rong Ge Furong Huang Chi Jin Yang Yuan 127 1,056 0 06 Mar 2015
Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n) Francis R. Bach Eric Moulines 81 404 0 10 Jun 2013
Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization Alexander Rakhlin Ohad Shamir Karthik Sridharan 114 764 0 26 Sep 2011