Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions

29 May 2023

Papers citing "Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions"

36 / 36 papers shown

Title
$On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm$ On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm Huan Li Yiming Dong Zhouchen Lin 0 0 0 17 May 2025
Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization Dmitry Kovalev 57 0 0 16 Mar 2025
On the Convergence of Adam-Type Algorithm for Bilevel Optimization under Unbounded Smoothness Xiaochuan Gong Jie Hao Mingrui Liu 58 0 0 05 Mar 2025
Sparklen: A Statistical Learning Toolkit for High-Dimensional Hawkes Processes in Python Romain Edmond Lacoste GP 58 0 0 26 Feb 2025
A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness Xiaochuan Gong Jie Hao Mingrui Liu 52 3 0 31 Dec 2024
Convergence Rate Analysis of LION Yiming Dong Huan Li Zhouchen Lin 39 0 0 12 Nov 2024
Efficient Adaptive Optimization via Subset-Norm and Subspace-Momentum: Fast, Memory-Reduced Training with Convergence Guarantees T. Nguyen Huy Le Nguyen ODL 33 0 0 11 Nov 2024
An Energy-Based Self-Adaptive Learning Rate for Stochastic Gradient Descent: Enhancing Unconstrained Optimization with VAV method Jiahao Zhang Christian Moya Guang Lin 43 0 0 10 Nov 2024
An Attention-Based Algorithm for Gravity Adaptation Zone Calibration Chen Yu 19 0 0 06 Oct 2024
An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness Xiaochuan Gong Jie Hao Mingrui Liu 46 2 0 28 Sep 2024
ERM-Lasso classification algorithm for Multivariate Hawkes Processes paths Charlotte Dion‐Blanc Christophe Denis Laure Sansonnet Romain E Lacoste 44 1 0 16 Jul 2024
An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes Antonio Orvieto Lin Xiao 42 2 0 05 Jul 2024
Empirical Tests of Optimization Assumptions in Deep Learning Hoang Tran Qinzi Zhang Ashok Cutkosky 41 1 0 01 Jul 2024
Large Batch Analysis for Adagrad Under Anisotropic Smoothness Yuxing Liu Rui Pan Tong Zhang 26 5 0 21 Jun 2024
Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions Devyani Maladkar Ruichen Jiang Aryan Mokhtari 38 6 0 07 Jun 2024
The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms Elizabeth Collins-Woodfin Inbar Seroussi Begona García Malaxechebarría Andrew W. Mackenzie Elliot Paquette Courtney Paquette 30 1 0 30 May 2024
Towards Stability of Parameter-free Optimization Yijiang Pang Shuyang Yu Hoang Bao Jiayu Zhou 34 1 0 07 May 2024
CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion Oswaldo Ludwig 21 0 0 29 Apr 2024
Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance Qi Zhang Yi Zhou Shaofeng Zou 42 3 0 01 Apr 2024
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond Bohan Wang Huishuai Zhang Qi Meng Ruoyu Sun Zhi-Ming Ma Wei Chen 37 7 0 22 Mar 2024
Why Transformers Need Adam: A Hessian Perspective Yushun Zhang Congliang Chen Tian Ding Ziniu Li Ruoyu Sun Zhimin Luo 37 43 0 26 Feb 2024
Revisiting Convergence of AdaGrad with Relaxed Assumptions Yusu Hong Junhong Lin 28 12 0 21 Feb 2024
AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods Tim Tsz-Kit Lau Han Liu Mladen Kolar ODL 24 6 0 17 Feb 2024
Towards Quantifying the Preconditioning Effect of Adam Rudrajit Das Naman Agarwal Sujay Sanghavi Inderjit S. Dhillon 20 5 0 11 Feb 2024
On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions Yusu Hong Junhong Lin 46 10 0 06 Feb 2024
Stochastic Two Points Method for Deep Model Zeroth-order Optimization Yijiang Pang Jiayu Zhou 22 0 0 02 Feb 2024
Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis Jie Hao Xiaochuan Gong Mingrui Liu 30 7 0 17 Jan 2024
Parameter-Agnostic Optimization under Relaxed Smoothness Florian Hübler Junchi Yang Xiang Li Niao He 31 12 0 06 Nov 2023
Signal Processing Meets SGD: From Momentum to Filter Zhipeng Yao Guisong Chang Jiaqi Zhang Qi Zhang Dazhou Li Yu Zhang ODL 29 0 0 06 Nov 2023
High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noise Yusu Hong Junhong Lin 25 7 0 03 Nov 2023
Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity Bohan Wang Jingwen Fu Huishuai Zhang Nanning Zheng Wei Chen 15 17 0 27 Oct 2023
Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case Meixuan He Yuqing Liang Jinlan Liu Dongpo Xu 19 8 0 20 Jul 2023
Convergence of Adam Under Relaxed Assumptions Haochuan Li Alexander Rakhlin Ali Jadbabaie 37 54 0 27 Apr 2023
Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGD Matthew Faw Litu Rout C. Caramanis Sanjay Shakkottai 16 37 0 13 Feb 2023
A High Probability Analysis of Adaptive SGD with Momentum Xiaoyun Li Francesco Orabona 92 65 0 28 Jul 2020
A Simple Convergence Proof of Adam and Adagrad Alexandre Défossez Léon Bottou Francis R. Bach Nicolas Usunier 56 143 0 05 Mar 2020