The Role of Momentum Parameters in the Optimal Convergence of Adaptive
Polyak's Heavy-ball Methods

The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods

15 February 2021

Papers citing "The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods"

17 / 17 papers shown

Title
Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ball Othmane Sebbouh Robert Mansel Gower Aaron Defazio 43 22 0 14 Jun 2020
The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization Wei Tao Zhisong Pan Gao-wei Wu Qing Tao 23 19 0 08 Jun 2020
A new regret analysis for Adam-type algorithms Ahmet Alacaoglu Yura Malitsky P. Mertikopoulos Volkan Cevher ODL 56 41 0 21 Mar 2020
Understanding the Role of Momentum in Stochastic Gradient Methods Igor Gitman Hunter Lang Pengchuan Zhang Lin Xiao 57 95 0 30 Oct 2019
Heavy-ball Algorithms Always Escape Saddle Points Tao Sun Dongsheng Li Zhe Quan Hao Jiang Shengguo Li Y. Dou ODL 44 21 0 23 Jul 2019
The Role of Memory in Stochastic Optimization Antonio Orvieto Jonas Köhler Aurelien Lucchi 68 30 0 02 Jul 2019
On the Convergence of Adam and Beyond Sashank J. Reddi Satyen Kale Surinder Kumar 93 2,499 0 19 Apr 2019
Tight Analyses for Non-Smooth Stochastic Gradient Descent Nicholas J. A. Harvey Christopher Liaw Y. Plan Sikander Randhawa 45 138 0 13 Dec 2018
Non-ergodic Convergence Analysis of Heavy-Ball Algorithms Tao Sun Penghang Yin Dongsheng Li Chun Huang Lei Guan Hao Jiang 36 46 0 05 Nov 2018
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization Xiangyi Chen Sijia Liu Ruoyu Sun Mingyi Hong 55 323 0 08 Aug 2018
The Unusual Effectiveness of Averaging in GAN Training Yasin Yazici Chuan-Sheng Foo Stefan Winkler Kim-Hui Yap Georgios Piliouras V. Chandrasekhar 106 175 0 12 Jun 2018
Variants of RMSProp and Adagrad with Logarithmic Regret Bounds Mahesh Chandra Mukkamala Matthias Hein ODL 54 258 0 17 Jun 2017
The Marginal Value of Adaptive Gradient Methods in Machine Learning Ashia Wilson Rebecca Roelofs Mitchell Stern Nathan Srebro Benjamin Recht ODL 62 1,030 0 23 May 2017
An overview of gradient descent optimization algorithms Sebastian Ruder ODL 204 6,184 0 15 Sep 2016
Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization Tianbao Yang Qihang Lin Zhe Li 61 122 0 12 Apr 2016
iPiano: Inertial Proximal Algorithm for Non-Convex Optimization Peter Ochs Yunjin Chen Thomas Brox Thomas Pock 73 433 0 18 Apr 2014
Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization Alexander Rakhlin Ohad Shamir Karthik Sridharan 164 768 0 26 Sep 2011