Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis

28 October 2022

Papers citing "Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis"

32 / 32 papers shown

Title
Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs Fanchen Bu D. Chang 44 6 0 12 May 2022
Robust Training of Neural Networks Using Scale Invariant Architectures Zhiyuan Li Srinadh Bhojanapalli Manzil Zaheer Sashank J. Reddi Surinder Kumar 53 28 0 02 Feb 2022
What Happens after SGD Reaches Zero Loss? --A Mathematical Framework Zhiyuan Li Tianhao Wang Sanjeev Arora MLT 95 100 0 13 Oct 2021
Continuous vs. Discrete Optimization of Deep Neural Networks Omer Elkabetz Nadav Cohen 77 44 0 14 Jul 2021
Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis Stephan Wojtowytsch 61 33 0 04 Jun 2021
Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks Hidenori Tanaka D. Kunin 64 28 0 06 May 2021
On the Origin of Implicit Regularization in Stochastic Gradient Descent Samuel L. Smith Benoit Dherin David Barrett Soham De MLT 30 202 0 28 Jan 2021
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics D. Kunin Javier Sagastuy-Breña Surya Ganguli Daniel L. K. Yamins Hidenori Tanaka 136 79 0 08 Dec 2020
Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate Zhiyuan Li Kaifeng Lyu Sanjeev Arora 75 74 0 06 Oct 2020
Implicit Gradient Regularization David Barrett Benoit Dherin 67 149 0 23 Sep 2020
Spherical Perspective on Learning with Normalization Layers Simon Roburin Yann de Mont-Marin Andrei Bursuc Renaud Marlet P. Pérez Mathieu Aubry 31 6 0 23 Jun 2020
Array Programming with NumPy Charles R. Harris K. Millman S. Walt R. Gommers Pauli Virtanen ... Tyler Reddy Warren Weckesser Hameer Abbasi C. Gohlke T. Oliphant 100 14,757 0 18 Jun 2020
The Implicit Regularization of Stochastic Gradient Flow for Least Squares Alnur Ali Yan Sun Robert Tibshirani 61 77 0 17 Mar 2020
An Exponential Learning Rate Schedule for Deep Learning Zhiyuan Li Sanjeev Arora 42 214 0 16 Oct 2019
Continuous Time Analysis of Momentum Methods Nikola B. Kovachki Andrew M. Stuart 94 34 0 10 Jun 2019
Online Normalization for Training Neural Networks Vitaliy Chiley I. Sharapov Atli Kosson Urs Koster R. Reece S. D. L. Fuente Vishal Subbiah Michael James OnRL 43 55 0 15 May 2019
Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximation Yuanyuan Feng Tingran Gao Lei Li Jian‐Guo Liu Yulong Lu 34 25 0 02 Feb 2019
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization Sanjeev Arora Zhiyuan Li Kaifeng Lyu 61 131 0 10 Dec 2018
Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations Qianxiao Li Cheng Tai E. Weinan 95 149 0 05 Nov 2018
Three Mechanisms of Weight Decay Regularization Guodong Zhang Chaoqi Wang Bowen Xu Roger C. Grosse 54 257 0 29 Oct 2018
Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimization Anas Barakat Pascal Bianchi 48 75 0 04 Oct 2018
Norm matters: efficient and accurate normalization schemes in deep networks Elad Hoffer Ron Banner Itay Golan Daniel Soudry OffRL 53 179 0 05 Mar 2018
L2 Regularization versus Batch and Weight Normalization Twan van Laarhoven 43 295 0 16 Jun 2017
Stein Variational Gradient Descent as Gradient Flow Qiang Liu OT 62 274 0 25 Apr 2017
Identity Mappings in Deep Residual Networks Kaiming He Xinming Zhang Shaoqing Ren Jian Sun 309 10,149 0 16 Mar 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.6K 192,638 0 10 Dec 2015
Stochastic modified equations and adaptive stochastic gradient algorithms Qianxiao Li Cheng Tai E. Weinan 59 282 0 19 Nov 2015
A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights Weijie Su Stephen P. Boyd Emmanuel J. Candes 151 1,161 0 04 Mar 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 386 43,154 0 11 Feb 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 1.2K 149,474 0 22 Dec 2014
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.2K 39,383 0 01 Sep 2014
Microsoft COCO: Common Objects in Context Nayeon Lee Michael Maire Serge J. Belongie Lubomir Bourdev Ross B. Girshick James Hays Pietro Perona Deva Ramanan C. L. Zitnick Piotr Dollár ObjD 340 43,290 0 01 May 2014