On the Convergence of Adam under Non-uniform Smoothness: Separability
from SGDM and Beyond

On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond

22 March 2024

ArXiv (abs)PDF HTML

Papers citing "On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond"

11 / 11 papers shown

Title
Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance Qi Zhang Yi Zhou Shaofeng Zou 109 7 0 01 Apr 2024
Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case Meixuan He Yuqing Liang Jinlan Liu Dongpo Xu 61 9 0 20 Jul 2023
Convex and Non-convex Optimization Under Generalized Smoothness Haochuan Li Jian Qian Yi Tian Alexander Rakhlin Ali Jadbabaie 70 44 0 02 Jun 2023
Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods Junchi Yang Xiang Li Ilyas Fatkhullin Niao He 74 17 0 21 May 2023
The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance Matthew Faw Isidoros Tziotis Constantine Caramanis Aryan Mokhtari Sanjay Shakkottai Rachel A. Ward 67 61 0 11 Feb 2022
Improved Analysis of Clipping Algorithms for Non-convex Optimization Bohang Zhang Jikai Jin Cong Fang Liwei Wang 104 92 0 05 Oct 2020
A Simple Convergence Proof of Adam and Adagrad Alexandre Défossez Léon Bottou Francis R. Bach Nicolas Usunier 123 159 0 05 Mar 2020
Lower Bounds for Non-Convex Stochastic Optimization Yossi Arjevani Y. Carmon John C. Duchi Dylan J. Foster Nathan Srebro Blake E. Woodworth 93 362 0 05 Dec 2019
Why gradient clipping accelerates training: A theoretical justification for adaptivity J.N. Zhang Tianxing He S. Sra Ali Jadbabaie 80 467 0 28 May 2019
A Sufficient Condition for Convergences of Adam and RMSProp Fangyu Zou Li Shen Zequn Jie Weizhong Zhang Wei Liu 63 372 0 23 Nov 2018
Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration Soham De Anirbit Mukherjee Enayat Ullah 71 101 0 18 Jul 2018

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.