SGD: General Analysis and Improved Rates

27 January 2019

Papers citing "SGD: General Analysis and Improved Rates"

50 / 80 papers shown

Title
Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization Dmitry Kovalev 57 0 0 16 Mar 2025
SAPPHIRE: Preconditioned Stochastic Variance Reduction for Faster Large-Scale Statistical Learning Jingruo Sun Zachary Frangella Madeleine Udell 36 0 0 28 Jan 2025
Random Reshuffling for Stochastic Gradient Langevin Dynamics Luke Shaw Peter A. Whalley 93 3 0 28 Jan 2025
Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance Dimitris Oikonomou Nicolas Loizou 55 4 0 06 Jun 2024
Demystifying SGD with Doubly Stochastic Gradients Kyurae Kim Joohwan Ko Yian Ma Jacob R. Gardner 53 0 0 03 Jun 2024
An Inexact Halpern Iteration with Application to Distributionally Robust Optimization Ling Liang Kim-Chuan Toh Jia Jie Zhu 32 4 0 08 Feb 2024
AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size P. Ostroukhov Aigerim Zhumabayeva Chulu Xiang Alexander Gasnikov Martin Takáč Dmitry Kamzolov ODL 46 2 0 07 Feb 2024
Provably Scalable Black-Box Variational Inference with Structured Variational Families Joohwan Ko Kyurae Kim W. Kim Jacob R. Gardner BDL 33 2 0 19 Jan 2024
Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications Jie Hu Vishwaraj Doshi Do Young Eun 38 4 0 17 Jan 2024
On Adaptive Stochastic Optimization for Streaming Data: A Newton's Method with O(dN) Operations Antoine Godichon-Baggioni Nicklas Werge ODL 37 3 0 29 Nov 2023
Demystifying the Myths and Legends of Nonconvex Convergence of SGD Aritra Dutta El Houcine Bergou Soumia Boucherouite Nicklas Werge M. Kandemir Xin Li 26 0 0 19 Oct 2023
PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning Mingjia Shi Yuhao Zhou Kai Wang Huaizheng Zhang Shudong Huang Qing Ye Jiangcheng Lv 29 9 0 13 Oct 2023
When MiniBatch SGD Meets SplitFed Learning:Convergence Analysis and Performance Evaluation Chao Huang Geng Tian Ming Tang FedML 34 4 0 23 Aug 2023
Provable convergence guarantees for black-box variational inference Justin Domke Guillaume Garrigos Robert Mansel Gower 18 18 0 04 Jun 2023
Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications Antoine Godichon-Baggioni Pierre Tarrago 24 5 0 01 Mar 2023
Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditions S. Choudhury Eduard A. Gorbunov Nicolas Loizou 27 13 0 27 Feb 2023
Statistical Inference with Stochastic Gradient Methods under $φ$ -mixing Data Ruiqi Liu Xinyu Chen Zuofeng Shang FedML 19 6 0 24 Feb 2023
Federated Gradient Matching Pursuit Halyun Jeong Deanna Needell Jing Qin FedML 37 1 0 20 Feb 2023
Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize Mert Gurbuzbalaban Yuanhan Hu Umut Simsekli Lingjiong Zhu LRM 23 1 0 10 Feb 2023
Distributed Stochastic Optimization under a General Variance Condition Kun-Yen Huang Xiao Li Shin-Yi Pu FedML 40 6 0 30 Jan 2023
Convergence of First-Order Algorithms for Meta-Learning with Moreau Envelopes Konstantin Mishchenko Slavomír Hanzely Peter Richtárik FedML 32 5 0 17 Jan 2023
GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity A. Maranjyan M. Safaryan Peter Richtárik 34 13 0 28 Oct 2022
Neural Networks Efficiently Learn Low-Dimensional Representations with SGD Alireza Mousavi-Hosseini Sejun Park M. Girotti Ioannis Mitliagkas Murat A. Erdogdu MLT 324 48 0 29 Sep 2022
Efficiency Ordering of Stochastic Gradient Descent Jie Hu Vishwaraj Doshi Do Young Eun 31 6 0 15 Sep 2022
Smooth Monotone Stochastic Variational Inequalities and Saddle Point Problems: A Survey Aleksandr Beznosikov Boris Polyak Eduard A. Gorbunov D. Kovalev Alexander Gasnikov 42 31 0 29 Aug 2022
Accelerating SGD for Highly Ill-Conditioned Huge-Scale Online Matrix Completion G. Zhang Hong-Ming Chiu Richard Y. Zhang 24 10 0 24 Aug 2022
Empirical Study of Overfitting in Deep FNN Prediction Models for Breast Cancer Metastasis Chuhan Xu Pablo Coen-Pirani Xia Jiang AI4CE 21 1 0 03 Aug 2022
Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learning S. Akintoye Liangxiu Han H. Lloyd Xin Zhang Darren Dancey Haoming Chen Daoqiang Zhang FedML 34 5 0 22 Jul 2022
Towards Understanding Sharpness-Aware Minimization Maksym Andriushchenko Nicolas Flammarion AAML 35 134 0 13 Jun 2022
FedControl: When Control Theory Meets Federated Learning Adnane Mansour Gaia Carenini Alexandre Duplessis D. Naccache FedML 21 2 0 27 May 2022
FedAvg with Fine Tuning: Local Updates Lead to Representation Learning Liam Collins Hamed Hassani Aryan Mokhtari Sanjay Shakkottai FedML 32 75 0 27 May 2022
Learning from time-dependent streaming data with online stochastic algorithms Antoine Godichon-Baggioni Nicklas Werge Olivier Wintenberger 22 3 0 25 May 2022
Federated Random Reshuffling with Compression and Variance Reduction Grigory Malinovsky Peter Richtárik FedML 27 10 0 08 May 2022
Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD Konstantinos E. Nikolakakis Farzin Haddadpour Amin Karbasi Dionysios S. Kalogerias 43 17 0 26 Apr 2022
Deep Unlearning via Randomized Conditionally Independent Hessians Ronak R. Mehta Sourav Pal Vikas Singh Sathya Ravi MU 27 81 0 15 Apr 2022
BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling Cheng Wan Youjie Li Ang Li Namjae Kim Yingyan Lin GNN 34 75 0 21 Mar 2022
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally! Konstantin Mishchenko Grigory Malinovsky Sebastian U. Stich Peter Richtárik 11 151 0 18 Feb 2022
Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods Aleksandr Beznosikov Eduard A. Gorbunov Hugo Berard Nicolas Loizou 19 48 0 15 Feb 2022
Towards Federated Learning on Time-Evolving Heterogeneous Data Yongxin Guo Tao R. Lin Xiaoying Tang FedML 22 30 0 25 Dec 2021
Differentially Private Federated Learning on Heterogeneous Data Maxence Noble A. Bellet Aymeric Dieuleveut FedML 13 102 0 17 Nov 2021
Stochastic Extragradient: General Analysis and Improved Rates Eduard A. Gorbunov Hugo Berard Gauthier Gidel Nicolas Loizou 22 40 0 16 Nov 2021
Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentum J. Kim Panos Toulis Anastasios Kyrillidis 24 8 0 11 Nov 2021
Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize Ryan DÓrazio Nicolas Loizou I. Laradji Ioannis Mitliagkas 34 30 0 28 Oct 2021
Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descent Sharan Vaswani Benjamin Dubois-Taine Reza Babanezhad 51 11 0 21 Oct 2021
Stochastic Primal-Dual Deep Unrolling Junqi Tang Subhadip Mukherjee Carola-Bibiane Schönlieb 24 4 0 19 Oct 2021
Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Streaming Data Antoine Godichon-Baggioni Nicklas Werge Olivier Wintenberger 19 7 0 15 Sep 2021
Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data Zhipeng Cai Ozan Sener V. Koltun CLL 33 83 0 20 Aug 2021
Decentralized Composite Optimization with Compression Yao Li Xiaorui Liu Jiliang Tang Ming Yan Kun Yuan 27 9 0 10 Aug 2021
Towards Biologically Plausible Convolutional Networks Roman Pogodin Yash Mehta Timothy Lillicrap P. Latham 26 22 0 22 Jun 2021
Stochastic Polyak Stepsize with a Moving Target Robert Mansel Gower Aaron Defazio Michael G. Rabbat 29 17 0 22 Jun 2021