A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent

27 May 2019

Papers citing "A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent"

35 / 35 papers shown

Title
A stochastic gradient descent algorithm with random search directions Eméric Gbaguidi ODL 80 0 0 25 Mar 2025
Demystifying SGD with Doubly Stochastic Gradients Kyurae Kim Joohwan Ko Yian Ma Jacob R. Gardner 53 0 0 03 Jun 2024
LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression Laurent Condat A. Maranjyan Peter Richtárik 49 4 0 07 Mar 2024
AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size P. Ostroukhov Aigerim Zhumabayeva Chulu Xiang Alexander Gasnikov Martin Takáč Dmitry Kamzolov ODL 46 2 0 07 Feb 2024
Provably Scalable Black-Box Variational Inference with Structured Variational Families Joohwan Ko Kyurae Kim W. Kim Jacob R. Gardner BDL 33 2 0 19 Jan 2024
Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updates Siqi Zhang S. Choudhury Sebastian U. Stich Nicolas Loizou FedML 19 3 0 08 Jun 2023
Provable convergence guarantees for black-box variational inference Justin Domke Guillaume Garrigos Robert Mansel Gower 23 18 0 04 Jun 2023
Error Feedback Shines when Features are Rare Peter Richtárik Elnur Gasanov Konstantin Burlachenko 38 2 0 24 May 2023
Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Space Michael Diao Krishnakumar Balasubramanian Sinho Chewi Adil Salim BDL 32 21 0 10 Apr 2023
Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction Bowen Lei Dongkuan Xu Ruqi Zhang Shuren He Bani Mallick 37 6 0 09 Jan 2023
Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimization Xu Cai Chaobing Song Stephen J. Wright Jelena Diakonikolas 36 14 0 09 Dec 2022
BALPA: A Balanced Primal-Dual Algorithm for Nonsmooth Optimization with Application to Distributed Optimization Luyao Guo Jinde Cao Xinli Shi Shaofu Yang 15 0 0 06 Dec 2022
On the effectiveness of partial variance reduction in federated learning with heterogeneous data Bo-wen Li Mikkel N. Schmidt T. S. Alstrøm Sebastian U. Stich FedML 37 9 0 05 Dec 2022
GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity A. Maranjyan M. Safaryan Peter Richtárik 34 13 0 28 Oct 2022
SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum Cocoercive Variational Inequalities Aleksandr Beznosikov Alexander Gasnikov 40 2 0 12 Oct 2022
Smooth Monotone Stochastic Variational Inequalities and Saddle Point Problems: A Survey Aleksandr Beznosikov Boris Polyak Eduard A. Gorbunov D. Kovalev Alexander Gasnikov 42 31 0 29 Aug 2022
Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topology Yan Huang Ying Sun Zehan Zhu Changzhi Yan Jinming Xu FedML 30 15 0 08 Jul 2022
Federated Optimization Algorithms with Random Reshuffling and Gradient Compression Abdurakhmon Sadiev Grigory Malinovsky Eduard A. Gorbunov Igor Sokolov Ahmed Khaled Konstantin Burlachenko Peter Richtárik FedML 16 21 0 14 Jun 2022
Federated Random Reshuffling with Compression and Variance Reduction Grigory Malinovsky Peter Richtárik FedML 27 10 0 08 May 2022
Deep Unlearning via Randomized Conditionally Independent Hessians Ronak R. Mehta Sourav Pal Vikas Singh Sathya Ravi MU 27 81 0 15 Apr 2022
Nonlinear gradient mappings and stochastic optimization: A general framework with applications to heavy-tail noise D. Jakovetić Dragana Bajović Anit Kumar Sahu S. Kar Nemanja Milošević Dusan Stamenkovic 19 12 0 06 Apr 2022
Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods Aleksandr Beznosikov Eduard A. Gorbunov Hugo Berard Nicolas Loizou 19 49 0 15 Feb 2022
Stochastic Extragradient: General Analysis and Improved Rates Eduard A. Gorbunov Hugo Berard Gauthier Gidel Nicolas Loizou 22 40 0 16 Nov 2021
Federated Expectation Maximization with heterogeneity mitigation and variance reduction Aymeric Dieuleveut G. Fort Eric Moulines Geneviève Robin FedML 31 5 0 03 Nov 2021
FedNL: Making Newton-Type Methods Applicable to Federated Learning M. Safaryan Rustem Islamov Xun Qian Peter Richtárik FedML 33 78 0 05 Jun 2021
PMGT-VR: A decentralized proximal-gradient algorithmic framework with variance reduction Haishan Ye Wei Xiong Tong Zhang 16 16 0 30 Dec 2020
Convergence Properties of Stochastic Hypergradients Riccardo Grazzi Massimiliano Pontil Saverio Salzo 24 26 0 13 Nov 2020
Local SGD: Unified Theory and New Efficient Methods Eduard A. Gorbunov Filip Hanzely Peter Richtárik FedML 37 109 0 03 Nov 2020
Variance-Reduced Methods for Machine Learning Robert Mansel Gower Mark W. Schmidt Francis R. Bach Peter Richtárik 19 111 0 02 Oct 2020
Optimization for Supervised Machine Learning: Randomized Algorithms for Data and Parameters Filip Hanzely 37 0 0 26 Aug 2020
On stochastic mirror descent with interacting particles: convergence properties and variance reduction Anastasia Borovykh N. Kantas P. Parpas G. Pavliotis 28 12 0 15 Jul 2020
Stochastic Hamiltonian Gradient Methods for Smooth Games Nicolas Loizou Hugo Berard Alexia Jolicoeur-Martineau Pascal Vincent Simon Lacoste-Julien Ioannis Mitliagkas 39 50 0 08 Jul 2020
DeltaGrad: Rapid retraining of machine learning models Yinjun Wu Yan Sun S. Davidson MU 25 196 0 26 Jun 2020
Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization Samuel Horváth Lihua Lei Peter Richtárik Michael I. Jordan 57 30 0 13 Feb 2020
Better Theory for SGD in the Nonconvex World Ahmed Khaled Peter Richtárik 13 179 0 09 Feb 2020