Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness

12 February 2021

Papers citing "Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness"

21 / 21 papers shown

Title
Error Feedback under $(L_0,L_1)$ -Smoothness: Normalization and Momentum Sarit Khirirat Abdurakhmon Sadiev Artem Riabinin Eduard A. Gorbunov Peter Richtárik 27 0 0 22 Oct 2024
From Gradient Clipping to Normalization for Heavy Tailed SGD Florian Hübler Ilyas Fatkhullin Niao He 40 5 0 17 Oct 2024
Private and Communication-Efficient Federated Learning based on Differentially Private Sketches Meifan Zhang Zhanhong Xie Lihua Yin FedML 29 1 0 08 Oct 2024
Contextual Bandits for Unbounded Context Distributions Puning Zhao Jiafei Wu Zhe Liu Huiwen Wu Q. Zhang Zong Ke Tianhang Zheng 65 3 0 19 Aug 2024
Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum H. Cai Sulaiman A. Alghunaim Ali H.Sayed 43 1 0 18 Jun 2024
Stochastic Weakly Convex Optimization Beyond Lipschitz Continuity Wenzhi Gao Qi Deng 22 1 0 25 Jan 2024
Rethinking SIGN Training: Provable Nonconvex Acceleration without First- and Second-Order Gradient Lipschitz Tao Sun Congliang Chen Peng Qiao Li Shen Xinwang Liu Dongsheng Li 34 3 0 23 Oct 2023
Understanding Fairness Surrogate Functions in Algorithmic Fairness Wei Yao Zhanke Zhou Zhicong Li Bo Han Yong Liu 29 3 0 17 Oct 2023
High Probability Analysis for Non-Convex Stochastic Optimization with Clipping Shaojie Li Yong Liu 35 2 0 25 Jul 2023
Clip21: Error Feedback for Gradient Clipping Sarit Khirirat Eduard A. Gorbunov Samuel Horváth Rustem Islamov Fakhri Karray Peter Richtárik 32 10 0 30 May 2023
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training Hong Liu Zhiyuan Li David Leo Wright Hall Percy Liang Tengyu Ma VLM 46 128 0 23 May 2023
Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees Nachuan Xiao Xiaoyin Hu Xin Liu Kim-Chuan Toh 16 15 0 06 May 2023
Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees Anastasia Koloskova Hadrien Hendrikx Sebastian U. Stich 104 49 0 02 May 2023
Unified analysis of SGD-type methods Eduard A. Gorbunov 24 2 0 29 Mar 2023
EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Data M. Crawshaw Yajie Bao Mingrui Liu FedML 21 8 0 14 Feb 2023
U-Clip: On-Average Unbiased Stochastic Gradient Clipping Bryn Elesedy Marcus Hutter 13 1 0 06 Feb 2023
Robustness to Unbounded Smoothness of Generalized SignSGD M. Crawshaw Mingrui Liu Francesco Orabona Wei Zhang Zhenxun Zhuang AAML 28 63 0 23 Aug 2022
Clipped Stochastic Methods for Variational Inequalities with Heavy-Tailed Noise Eduard A. Gorbunov Marina Danilova David Dobre Pavel Dvurechensky Alexander Gasnikov Gauthier Gidel 24 24 0 02 Jun 2022
A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks Mingrui Liu Zhenxun Zhuang Yunwei Lei Chunyang Liao 30 16 0 10 May 2022
CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU Zangwei Zheng Peng Xu Xuan Zou Da Tang Zhen Li ... Xiangzhuo Ding Fuzhao Xue Ziheng Qing Youlong Cheng Yang You VLM 44 7 0 13 Apr 2022
Improved Learning Rates for Stochastic Optimization: Two Theoretical Viewpoints Shaojie Li Yong Liu 20 13 0 19 Jul 2021