Understanding the Role of Momentum in Stochastic Gradient Methods

Understanding the Role of Momentum in Stochastic Gradient Methods

30 October 2019

Pengchuan Zhang

Papers citing "Understanding the Role of Momentum in Stochastic Gradient Methods"

16 / 16 papers shown

Title
Stochastic Gradient Descent in Non-Convex Problems: Asymptotic Convergence with Relaxed Step-Size via Stopping Time Methods Ruinan Jin Difei Cheng Hong Qiao Xin Shi Shaodong Liu Bo Zhang 28 0 0 17 Apr 2025
SOREL: A Stochastic Algorithm for Spectral Risks Minimization Yuze Ge Rujun Jiang 38 0 0 19 Jul 2024
Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance Dimitris Oikonomou Nicolas Loizou 55 4 0 06 Jun 2024
Information-Theoretic Generalization Bounds for Deep Neural Networks Haiyun He Christina Lee Yu 35 4 0 04 Apr 2024
Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normality Kejie Tang Weidong Liu Yichen Zhang Xi Chen 16 2 0 28 May 2023
Tradeoffs between convergence rate and noise amplification for momentum-based accelerated optimization algorithms Hesameddin Mohammadi Meisam Razaviyayn Mihailo R. Jovanović 13 7 0 24 Sep 2022
On the fast convergence of minibatch heavy ball momentum Raghu Bollapragada Tyler Chen Rachel A. Ward 24 17 0 15 Jun 2022
Gradient Temporal Difference with Momentum: Stability and Convergence Rohan Deb S. Bhatnagar 16 5 0 22 Nov 2021
An Asymptotic Analysis of Minibatch-Based Momentum Methods for Linear Regression Models Yuan Gao Xuening Zhu Haobo Qi Guodong Li Riquan Zhang Hansheng Wang 15 3 0 02 Nov 2021
Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization Tao Sun Huaming Ling Zuoqiang Shi Dongsheng Li Bao Wang ODL 22 13 0 18 Oct 2021
Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization Peiyuan Zhang Antonio Orvieto Hadi Daneshmand Thomas Hofmann Roy S. Smith 24 9 0 23 Feb 2021
Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent Kangqiao Liu Liu Ziyin Masakuni Ueda MLT 61 37 0 07 Dec 2020
Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM Chaouki Ben Issaid Anis Elgabli Jihong Park M. Bennis Mérouane Debbah FedML 31 13 0 14 Sep 2020
On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings Mahmoud Assran Michael G. Rabbat 8 59 0 27 Feb 2020
Statistical Adaptive Stochastic Gradient Methods Pengchuan Zhang Hunter Lang Qiang Liu Lin Xiao ODL 15 11 0 25 Feb 2020
Quasi-hyperbolic momentum and Adam for deep learning Jerry Ma Denis Yarats ODL 84 129 0 16 Oct 2018