On the convergence properties of a $K$ -step averaging stochastic gradient descent algorithm for nonconvex optimization

3 August 2017

Papers citing "On the convergence properties of a $K$-step averaging stochastic gradient descent algorithm for nonconvex optimization"

41 / 141 papers shown

Title
Minibatch vs Local SGD for Heterogeneous Distributed Learning Blake E. Woodworth Kumar Kshitij Patel Nathan Srebro FedML 22 199 0 08 Jun 2020
Local SGD With a Communication Overhead Depending Only on the Number of Workers Artin Spiridonoff Alexander Olshevsky I. Paschalidis FedML 25 19 0 03 Jun 2020
Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks Zhishuai Guo Mingrui Liu Zhuoning Yuan Li Shen Wei Liu Tianbao Yang 33 42 0 05 May 2020
A Hybrid-Order Distributed SGD Method for Non-Convex Optimization to Balance Communication Overhead, Computational Complexity, and Convergence Rate Naeimeh Omidvar M. Maddah-ali Hamed Mahdavi ODL 19 3 0 27 Mar 2020
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates Anastasia Koloskova Nicolas Loizou Sadra Boreiri Martin Jaggi Sebastian U. Stich FedML 41 493 0 23 Mar 2020
Communication optimization strategies for distributed deep neural network training: A survey Shuo Ouyang Dezun Dong Yemao Xu Liquan Xiao 30 12 0 06 Mar 2020
Device Heterogeneity in Federated Learning: A Superquantile Approach Yassine Laguel Krishna Pillutla J. Malick Zaïd Harchaoui FedML 40 22 0 25 Feb 2020
Communication-Efficient Edge AI: Algorithms and Systems Yuanming Shi Kai Yang Tao Jiang Jun Zhang Khaled B. Letaief GNN 17 326 0 22 Feb 2020
Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD Jianyu Wang Hao Liang Gauri Joshi 14 33 0 21 Feb 2020
Dynamic Federated Learning Elsa Rizk Stefan Vlaski Ali H. Sayed FedML 19 25 0 20 Feb 2020
Communication-Efficient Distributed SVD via Local Power Iterations Xiang Li Shusen Wang Kun Chen Zhihua Zhang 35 21 0 19 Feb 2020
Personalized Federated Learning: A Meta-Learning Approach Alireza Fallah Aryan Mokhtari Asuman Ozdaglar FedML 36 561 0 19 Feb 2020
Is Local SGD Better than Minibatch SGD? Blake E. Woodworth Kumar Kshitij Patel Sebastian U. Stich Zhen Dai Brian Bullins H. B. McMahan Ohad Shamir Nathan Srebro FedML 34 253 0 18 Feb 2020
Faster On-Device Training Using New Federated Momentum Algorithm Zhouyuan Huo Qian Yang Bin Gu Heng-Chiao Huang FedML 19 47 0 06 Feb 2020
FedDANE: A Federated Newton-Type Method Tian Li Anit Kumar Sahu Manzil Zaheer Maziar Sanjabi Ameet Talwalkar Virginia Smith FedML 18 155 0 07 Jan 2020
Variance Reduced Local SGD with Lower Communication Complexity Xian-Feng Liang Shuheng Shen Jingchang Liu Zhen Pan Enhong Chen Yifei Cheng FedML 24 152 0 30 Dec 2019
On the Convergence of Local Descent Methods in Federated Learning Farzin Haddadpour M. Mahdavi FedML 19 266 0 31 Oct 2019
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization Farzin Haddadpour Mohammad Mahdi Kamani M. Mahdavi V. Cadambe FedML 33 199 0 30 Oct 2019
Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation Canh T. Dinh N. H. Tran Minh N. H. Nguyen Choong Seon Hong Wei Bao Albert Y. Zomaya Vincent Gramoli FedML 17 329 0 29 Oct 2019
Communication-Efficient Local Decentralized SGD Methods Xiang Li Wenhao Yang Shusen Wang Zhihua Zhang 30 53 0 21 Oct 2019
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum Jianyu Wang Vinayak Tantia Nicolas Ballas Michael G. Rabbat 12 200 0 01 Oct 2019
Matrix Sketching for Secure Collaborative Machine Learning Mengjiao Zhang Shusen Wang FedML 16 14 0 24 Sep 2019
Tighter Theory for Local SGD on Identical and Heterogeneous Data Ahmed Khaled Konstantin Mishchenko Peter Richtárik 29 426 0 10 Sep 2019
First Analysis of Local GD on Heterogeneous Data Ahmed Khaled Konstantin Mishchenko Peter Richtárik FedML 13 172 0 10 Sep 2019
Federated Learning: Challenges, Methods, and Future Directions Tian Li Anit Kumar Sahu Ameet Talwalkar Virginia Smith FedML 39 4,417 0 21 Aug 2019
On the Convergence of FedAvg on Non-IID Data Xiang Li Kaixuan Huang Wenhao Yang Shusen Wang Zhihua Zhang FedML 61 2,283 0 04 Jul 2019
Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent Shuheng Shen Linli Xu Jingchang Liu Xianfeng Liang Yifei Cheng ODL FedML 21 24 0 28 Jun 2019
Distributed Optimization for Over-Parameterized Learning Chi Zhang Qianxiao Li 16 4 0 14 Jun 2019
Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms Xiangyi Chen Tiancong Chen Haoran Sun Zhiwei Steven Wu Mingyi Hong FedML 16 73 0 04 Jun 2019
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization Hao Yu R. L. Jin Sen Yang FedML 40 379 0 09 May 2019
A Distributed Hierarchical SGD Algorithm with Sparse Global Reduction Fan Zhou Guojing Cong 11 8 0 12 Mar 2019
Federated Optimization in Heterogeneous Networks Tian Li Anit Kumar Sahu Manzil Zaheer Maziar Sanjabi Ameet Talwalkar Virginia Smith FedML 15 5,017 0 14 Dec 2018
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD Jianyu Wang Gauri Joshi FedML 27 231 0 19 Oct 2018
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms Jianyu Wang Gauri Joshi 30 348 0 22 Aug 2018
Don't Use Large Mini-Batches, Use Local SGD Tao R. Lin Sebastian U. Stich Kumar Kshitij Patel Martin Jaggi 57 429 0 22 Aug 2018
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning Hao Yu Sen Yang Shenghuo Zhu MoMe FedML 29 597 0 17 Jul 2018
Local SGD Converges Fast and Communicates Little Sebastian U. Stich FedML 73 1,043 0 24 May 2018
Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD Sanghamitra Dutta Gauri Joshi Soumyadip Ghosh Parijat Dube P. Nagpurkar 22 193 0 03 Mar 2018
Avoiding Communication in Proximal Methods for Convex Optimization Problems Saeed Soori Aditya Devarakonda J. Demmel Mert Gurbuzbalaban M. Dehnavi 19 7 0 24 Oct 2017
Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes Ohad Shamir Tong Zhang 101 570 0 08 Dec 2012
Optimal Distributed Online Prediction using Mini-Batches O. Dekel Ran Gilad-Bachrach Ohad Shamir Lin Xiao 177 683 0 07 Dec 2010