Robustness to Unbounded Smoothness of Generalized SignSGD

Robustness to Unbounded Smoothness of Generalized SignSGD

23 August 2022

Francesco Orabona

Wei Zhang

Papers citing "Robustness to Unbounded Smoothness of Generalized SignSGD"

16 / 16 papers shown

Title
Understanding Why Adam Outperforms SGD: Gradient Heterogeneity in Transformers Akiyoshi Tomihari Issei Sato ODL 61 1 0 31 Jan 2025
From Gradient Clipping to Normalization for Heavy Tailed SGD Florian Hübler Ilyas Fatkhullin Niao He 40 5 0 17 Oct 2024
Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees Aleksandar Armacki Shuhua Yu Pranay Sharma Gauri Joshi Dragana Bajović D. Jakovetić S. Kar 57 2 0 17 Oct 2024
Extended convexity and smoothness and their applications in deep learning Binchuan Qi Wei Gong Li Li 61 0 0 08 Oct 2024
An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness Xiaochuan Gong Jie Hao Mingrui Liu 46 2 0 28 Sep 2024
Does SGD really happen in tiny subspaces? Minhak Song Kwangjun Ahn Chulhee Yun 71 4 1 25 May 2024
$Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization$ Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization Shuo Xie Zhiyuan Li OffRL 47 13 0 05 Apr 2024
Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance Qi Zhang Yi Zhou Shaofeng Zou 42 3 0 01 Apr 2024
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models Frederik Kunstner Robin Yadav Alan Milligan Mark Schmidt Alberto Bietti 39 26 0 29 Feb 2024
On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions Yusu Hong Junhong Lin 46 10 0 06 Feb 2024
Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis Jie Hao Xiaochuan Gong Mingrui Liu 30 7 0 17 Jan 2024
Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization Zhen Qin Zhishuai Liu Pan Xu 20 1 0 24 Oct 2023
Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts Lizhang Chen Bo Liu Kaizhao Liang Qian Liu ODL 21 15 0 09 Oct 2023
Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions Bo Wang Huishuai Zhang Zhirui Ma Wei Chen 34 49 0 29 May 2023
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training Hong Liu Zhiyuan Li David Leo Wright Hall Percy Liang Tengyu Ma VLM 55 130 0 23 May 2023
A Simple Convergence Proof of Adam and Adagrad Alexandre Défossez Léon Bottou Francis R. Bach Nicolas Usunier 56 143 0 05 Mar 2020