Momentum-Based Variance Reduction in Non-Convex SGD

24 May 2019

Papers citing "Momentum-Based Variance Reduction in Non-Convex SGD"

50 / 94 papers shown

Title
Second-Order Convergence in Private Stochastic Non-Convex Optimization Youming Tao Zuyuan Zhang Dongxiao Yu Xiuzhen Cheng Falko Dressler Di Wang 7 0 0 21 May 2025
Efficient Curvature-Aware Hypergradient Approximation for Bilevel Optimization Youran Dong Junfeng Yang Wei-Ting Yao Jin Zhang 207 0 0 04 May 2025
Variance Reduction Methods Do Not Need to Compute Full Gradients: Improved Efficiency through Shuffling Daniil Medyakov Gleb Molodtsov S. Chezhegov Alexey Rebrikov Aleksandr Beznosikov 105 0 0 21 Feb 2025
FedTLU: Federated Learning with Targeted Layer Updates Jong-Ik Park Carlee Joe-Wong FedML 89 0 0 28 Jan 2025
Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML T. Dahan Kfir Y. Levy 72 0 0 17 Jan 2025
On the Performance Analysis of Momentum Method: A Frequency Domain Perspective Xianliang Li Jun Luo Zhiwei Zheng Hanxiao Wang Li Luo Lingkun Wen Linlong Wu Sheng Xu 74 0 0 29 Nov 2024
Extended convexity and smoothness and their applications in deep learning Binchuan Qi Wei Gong Li Li 63 0 0 08 Oct 2024
DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction Xinwei Zhang Zhiqi Bu Borja Balle Mingyi Hong Meisam Razaviyayn Vahab Mirrokni 78 2 0 04 Oct 2024
An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness Xiaochuan Gong Jie Hao Mingrui Liu 52 2 0 28 Sep 2024
On the Implicit Relation Between Low-Rank Adaptation and Differential Privacy Saber Malekmohammadi G. Farnadi 32 2 0 26 Sep 2024
Gradient-Free Method for Heavily Constrained Nonconvex Optimization Wanli Shi Hongchang Gao Bin Gu 21 5 0 31 Aug 2024
SOREL: A Stochastic Algorithm for Spectral Risks Minimization Yuze Ge Rujun Jiang 40 0 0 19 Jul 2024
Private Zeroth-Order Nonsmooth Nonconvex Optimization Qinzi Zhang Hoang Tran Ashok Cutkosky 40 4 0 27 Jun 2024
Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum H. Cai Sulaiman A. Alghunaim Ali H.Sayed 52 1 0 18 Jun 2024
Random Scaling and Momentum for Non-smooth Non-convex Optimization Qinzi Zhang Ashok Cutkosky 43 4 0 16 May 2024
Robust Decentralized Learning with Local Updates and Gradient Tracking Sajjad Ghiasvand Amirhossein Reisizadeh Mahnoosh Alizadeh Ramtin Pedarsani 42 3 0 02 May 2024
Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPs Swetha Ganesh Washim Uddin Mondal Vaneet Aggarwal 49 3 0 02 Apr 2024
Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries Swetha Ganesh Jiayu Chen Gugan Thoppe Vaneet Aggarwal FedML 68 1 0 15 Mar 2024
Non-Convex Stochastic Composite Optimization with Polyak Momentum Yuan Gao Anton Rodomanov Sebastian U. Stich 39 6 0 05 Mar 2024
Flora: Low-Rank Adapters Are Secretly Gradient Compressors Yongchang Hao Yanshuai Cao Lili Mou 16 41 0 05 Feb 2024
Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis Jie Hao Xiaochuan Gong Mingrui Liu 33 7 0 17 Jan 2024
FedDRO: Federated Compositional Optimization for Distributionally Robust Learning Prashant Khanduri Chengyin Li Rafi Ibn Sultan Yao Qiang Joerg Kliewer Dongxiao Zhu 45 0 0 21 Nov 2023
A Coefficient Makes SVRG Effective Yida Yin Zhiqiu Xu Zhiyuan Li Trevor Darrell Zhuang Liu 36 1 0 09 Nov 2023
Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization Zhen Qin Zhishuai Liu Pan Xu 28 1 0 24 Oct 2023
Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Rates Ahmad Rammal Kaja Gruntkowska Nikita Fedin Eduard A. Gorbunov Peter Richtárik 45 5 0 15 Oct 2023
Variance-reduced accelerated methods for decentralized stochastic double-regularized nonconvex strongly-concave minimax problems Gabriel Mancino-Ball Yangyang Xu 20 8 0 14 Jul 2023
Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality Ziyang Wei Wanrong Zhu Wei Biao Wu 27 3 0 13 Jul 2023
Federated Multi-Sequence Stochastic Approximation with Local Hypergradient Estimation Davoud Ataee Tarzanagh Mingchen Li Pranay Sharma Samet Oymak 34 0 0 02 Jun 2023
Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features Aleksandr Beznosikov David Dobre Gauthier Gidel 25 5 0 23 Apr 2023
On Momentum-Based Gradient Methods for Bilevel Optimization with Nonconvex Lower-Level Feihu Huang 32 18 0 07 Mar 2023
Enhanced Adaptive Gradient Algorithms for Nonconvex-PL Minimax Optimization Feihu Huang Chunyu Xuan Xinrui Wang Siqi Zhang Songcan Chen 28 7 0 07 Mar 2023
A One-Sample Decentralized Proximal Algorithm for Non-Convex Stochastic Composite Optimization Tesi Xiao Xuxing Chen Krishnakumar Balasubramanian Saeed Ghadimi 36 10 0 20 Feb 2023
Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Risks Arun Ganesh Daogao Liu Sewoong Oh Abhradeep Thakurta ODL 27 13 0 20 Feb 2023
Breaking the Lower Bound with (Little) Structure: Acceleration in Non-Convex Stochastic Optimization with Heavy-Tailed Noise Zijian Liu Jiawei Zhang Zhengyuan Zhou 34 12 0 14 Feb 2023
Near-Optimal Non-Convex Stochastic Optimization under Generalized Smoothness Zijian Liu Srikanth Jagabathula Zhengyuan Zhou 24 5 0 13 Feb 2023
Stochastic Dimension-reduced Second-order Methods for Policy Optimization Jinsong Liu Chen Xie Qinwen Deng Dongdong Ge Yi-Li Ye 32 1 0 28 Jan 2023
Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimization Le‐Yu Chen Jing Xu Luo Luo 31 15 0 16 Jan 2023
Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction Bowen Lei Dongkuan Xu Ruqi Zhang Shuren He Bani Mallick 42 6 0 09 Jan 2023
Stochastic Inexact Augmented Lagrangian Method for Nonconvex Expectation Constrained Optimization Zichong Li Pinzhuo Chen Sijia Liu Songtao Lu Yangyang Xu 35 17 0 19 Dec 2022
Gradient Descent-Type Methods: Background and Simple Unified Convergence Analysis Quoc Tran-Dinh Marten van Dijk 34 0 0 19 Dec 2022
Variance-Reduced Conservative Policy Iteration Naman Agarwal Brian Bullins Karan Singh 32 3 0 12 Dec 2022
Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimization Xu Cai Chaobing Song Stephen J. Wright Jelena Diakonikolas 36 14 0 09 Dec 2022
Decentralized Stochastic Gradient Descent Ascent for Finite-Sum Minimax Problems Hongchang Gao 24 16 0 06 Dec 2022
An Efficient Stochastic Algorithm for Decentralized Nonconvex-Strongly-Concave Minimax Optimization Le‐Yu Chen Haishan Ye Luo Luo 73 5 0 05 Dec 2022
Adaptive Stochastic Optimisation of Nonconvex Composite Objectives Weijia Shao F. Sivrikaya S. Albayrak 21 0 0 21 Nov 2022
Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization Ali Kavis Stratis Skoulakis Kimon Antonakopoulos L. Dadi V. Cevher 24 15 0 03 Nov 2022
Fast Adaptive Federated Bilevel Optimization Feihu Huang FedML 22 7 0 02 Nov 2022
Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models Shujian Zhang Chengyue Gong Xingchao Liu RALM 51 6 0 02 Nov 2022
Momentum Aggregation for Private Non-convex ERM Hoang Tran Ashok Cutkosky 26 14 0 12 Oct 2022
SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum Cocoercive Variational Inequalities Aleksandr Beznosikov Alexander Gasnikov 40 2 0 12 Oct 2022