v1v2 (latest)

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

12 April 2016

Papers citing "Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization"

48 / 48 papers shown

Title
Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum H. Cai Sulaiman A. Alghunaim Ali H.Sayed 115 1 0 18 Jun 2024
Almost sure convergence rates of stochastic gradient methods under gradient domination Simon Weissmann Sara Klein Waïss Azizian Leif Döring 93 3 0 22 May 2024
AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks Hao Sun Li Shen Qihuang Zhong Liang Ding Shi-Yong Chen Jingwei Sun Jing Li Guangzhong Sun Dacheng Tao 98 34 0 01 Mar 2023
On Almost Sure Convergence Rates of Stochastic Gradient Methods Jun Liu Ye Yuan 97 38 0 09 Feb 2022
A Novel Convergence Analysis for Algorithms of the Adam Family Zhishuai Guo Yi Tian Xu W. Yin Rong Jin Tianbao Yang 88 49 0 07 Dec 2021
Training Generative Adversarial Networks with Adaptive Composite Gradient Huiqing Qi Fang Li Shengli Tan Xiangyun Zhang GAN 51 4 0 10 Nov 2021
EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedback Ilyas Fatkhullin Igor Sokolov Eduard A. Gorbunov Zhize Li Peter Richtárik 129 47 0 07 Oct 2021
Accelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentum Guojing Cong Tianyi Liu 110 0 0 01 Oct 2021
Momentum Accelerates the Convergence of Stochastic AUPRC Maximization Guanghui Wang Minghao Yang Lijun Zhang Tianbao Yang 85 22 0 02 Jul 2021
Escaping Saddle Points Faster with Stochastic Momentum Jun-Kun Wang Chi-Heng Lin Jacob D. Abernethy ODL 77 22 0 05 Jun 2021
Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems Stefano Sarao Mannelli Pierfrancesco Urbani 67 10 0 23 Feb 2021
The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods Wei Tao Sheng Long Gao-wei Wu Qing Tao 43 14 0 15 Feb 2021
Accelerating Training of Batch Normalization: A Manifold Perspective Mingyang Yi 47 3 0 08 Jan 2021
Adaptive Gradient Quantization for Data-Parallel SGD Fartash Faghri Iman Tabrizian I. Markov Dan Alistarh Daniel M. Roy Ali Ramezani-Kebrya MQ 63 83 0 23 Oct 2020
Quickly Finding a Benign Region via Heavy Ball Momentum in Non-Convex Optimization Jun-Kun Wang Jacob D. Abernethy 110 7 0 04 Oct 2020
Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimization Aaron Defazio 100 23 0 01 Oct 2020
Federated Learning with Nesterov Accelerated Gradient Zhengjie Yang Wei Bao Dong Yuan Nguyen H. Tran Albert Y. Zomaya FedML 68 32 0 18 Sep 2020
Online Algorithms for Estimating Change Rates of Web Pages Konstantin Avrachenkov Kishor P. Patil Gugan Thoppe 56 16 0 17 Sep 2020
Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data Qianqian Tong Guannan Liang J. Bi FedML 108 28 0 14 Sep 2020
Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum Jerry Chee Ping Li 43 12 0 27 Aug 2020
Differentially Private Accelerated Optimization Algorithms Nurdan Kuru cS. .Ilker Birbil Mert Gurbuzbalaban S. Yıldırım 76 23 0 05 Aug 2020
A High Probability Analysis of Adaptive SGD with Momentum Xiaoyun Li Francesco Orabona 142 69 0 28 Jul 2020
A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning Samuel Horváth Peter Richtárik 79 60 0 19 Jun 2020
Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ball Othmane Sebbouh Robert Mansel Gower Aaron Defazio 67 23 0 14 Jun 2020
An Analysis of the Adaptation Speed of Causal Models Rémi Le Priol Reza Babanezhad Harikandeh Yoshua Bengio Simon Lacoste-Julien CML 58 14 0 18 May 2020
MixML: A Unified Analysis of Weakly Consistent Parallel Learning Yucheng Lu J. Nash Christopher De Sa FedML 91 12 0 14 May 2020
A Simple Convergence Proof of Adam and Adagrad Alexandre Défossez Léon Bottou Francis R. Bach Nicolas Usunier 140 159 0 05 Mar 2020
On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings Mahmoud Assran Michael G. Rabbat 78 59 0 27 Feb 2020
Faster On-Device Training Using New Federated Momentum Algorithm Zhouyuan Huo Qian Yang Bin Gu Heng-Chiao Huang FedML 195 47 0 06 Feb 2020
A Rule for Gradient Estimator Selection, with an Application to Variational Inference Tomas Geffner Justin Domke 66 6 0 05 Nov 2019
Understanding the Role of Momentum in Stochastic Gradient Methods Igor Gitman Hunter Lang Pengchuan Zhang Lin Xiao 77 95 0 30 Oct 2019
On the Acceleration of Deep Learning Model Parallelism with Staleness An Xu Zhouyuan Huo Heng-Chiao Huang 65 38 0 05 Sep 2019
The Role of Memory in Stochastic Optimization Antonio Orvieto Jonas Köhler Aurelien Lucchi 94 31 0 02 Jul 2019
Continuous Time Analysis of Momentum Methods Nikola B. Kovachki Andrew M. Stuart 131 36 0 10 Jun 2019
Provable Smoothness Guarantees for Black-Box Variational Inference Justin Domke 74 36 0 24 Jan 2019
Accelerated Linear Convergence of Stochastic Momentum Methods in Wasserstein Distances Bugra Can Mert Gurbuzbalaban Lingjiong Zhu 102 45 0 22 Jan 2019
Stagewise Training Accelerates Convergence of Testing Error Over SGD Zhuoning Yuan Yan Yan Rong Jin Tianbao Yang 111 11 0 10 Dec 2018
Weakly-Convex Concave Min-Max Optimization: Provable Algorithms and Applications in Machine Learning Hassan Rafique Mingrui Liu Qihang Lin Tianbao Yang 126 111 0 04 Oct 2018
On the Generalization of Stochastic Gradient Descent with Momentum Ali Ramezani-Kebrya Kimon Antonakopoulos Volkan Cevher Ashish Khisti Ben Liang MLT 73 26 0 12 Sep 2018
On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization Dongruo Zhou Yiqi Tang Yuan Cao Ziyan Yang Quanquan Gu 100 150 0 16 Aug 2018
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization Xiangyi Chen Sijia Liu Ruoyu Sun Mingyi Hong 95 324 0 08 Aug 2018
Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration Soham De Anirbit Mukherjee Enayat Ullah 79 101 0 18 Jul 2018
Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methods Nicolas Loizou Peter Richtárik 80 204 0 27 Dec 2017
NEON+: Accelerated Gradient Methods for Extracting Negative Curvature for Non-Convex Optimization Yi Tian Xu Rong Jin Tianbao Yang 97 26 0 04 Dec 2017
Linearly convergent stochastic heavy ball method for minimizing generalization error Nicolas Loizou Peter Richtárik 131 45 0 30 Oct 2017
Convergence Analysis of Optimization Algorithms Hyoungseok Kim Jihoon Kang Woo-Myoung Park SukHyun Ko Yoon-Ho Choi Daesung Yu YoungSook Song JungWon Choi 18 8 0 06 Jul 2017
Stochastic Heavy Ball S. Gadat Fabien Panloup Sofiane Saadane 132 105 0 14 Sep 2016
RSG: Beating Subgradient Method without Smoothness and Strong Convexity Tianbao Yang Qihang Lin 172 85 0 09 Dec 2015