On the Convergence of Adam under Non-uniform Smoothness: Separability
from SGDM and Beyond

On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond

22 March 2024

ArXiv (abs)PDF HTML

Papers citing "On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond"

11 / 11 papers shown

Title
Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance Qi Zhang Yi Zhou Shaofeng Zou 109 7 0 01 Apr 2024
Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case Meixuan He Yuqing Liang Jinlan Liu Dongpo Xu 61 9 0 20 Jul 2023
Convex and Non-convex Optimization Under Generalized Smoothness Haochuan Li Jian Qian Yi Tian Alexander Rakhlin Ali Jadbabaie 70 44 0 02 Jun 2023
Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods Junchi Yang Xiang Li Ilyas Fatkhullin Niao He 74 17 0 21 May 2023
The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance Matthew Faw Isidoros Tziotis Constantine Caramanis Aryan Mokhtari Sanjay Shakkottai Rachel A. Ward 67 61 0 11 Feb 2022
Improved Analysis of Clipping Algorithms for Non-convex Optimization Bohang Zhang Jikai Jin Cong Fang Liwei Wang 104 92 0 05 Oct 2020
A Simple Convergence Proof of Adam and Adagrad Alexandre Défossez Léon Bottou Francis R. Bach Nicolas Usunier 123 159 0 05 Mar 2020
Lower Bounds for Non-Convex Stochastic Optimization Yossi Arjevani Y. Carmon John C. Duchi Dylan J. Foster Nathan Srebro Blake E. Woodworth 93 362 0 05 Dec 2019
Why gradient clipping accelerates training: A theoretical justification for adaptivity J.N. Zhang Tianxing He S. Sra Ali Jadbabaie 80 467 0 28 May 2019
A Sufficient Condition for Convergences of Adam and RMSProp Fangyu Zou Li Shen Zequn Jie Weizhong Zhang Wei Liu 63 372 0 23 Nov 2018
Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration Soham De Anirbit Mukherjee Enayat Ullah 71 101 0 18 Jul 2018