On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data

9 June 2022

Tong Zhang

Papers citing "On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data"

33 / 33 papers shown

Title
Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspective Masoud Kavian Romain Chor Milad Sefidgaran Abdellatif Zaidi FedML 102 1 0 03 Mar 2025
Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimization Dun Zeng Zheshun Wu Shiyu Liu Yu Pan Xiaoying Tang Zenglin Xu MLT FedML 143 1 0 25 Nov 2024
Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective Margalit Glasgow Honglin Yuan Tengyu Ma FedML 55 45 0 05 Nov 2021
Iterated Vector Fields and Conservatism, with Applications to Federated Learning Zachary B. Charles Keith Rush 65 6 0 08 Sep 2021
A Field Guide to Federated Optimization Jianyu Wang Zachary B. Charles Zheng Xu Gauri Joshi H. B. McMahan ... Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu FedML 273 421 0 14 Jul 2021
On Large-Cohort Training for Federated Learning Zachary B. Charles Zachary Garrett Zhouyuan Huo Sergei Shmulyian Virginia Smith FedML 66 112 0 15 Jun 2021
Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning Zachary B. Charles Jakub Konecný FedML 98 64 0 08 Mar 2021
Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning Haibo Yang Minghong Fang Jia Liu FedML 69 261 0 27 Jan 2021
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization Jianyu Wang Qinghua Liu Hao Liang Gauri Joshi H. Vincent Poor MoMe FedML 70 1,348 0 15 Jul 2020
Minibatch vs Local SGD for Heterogeneous Distributed Learning Blake E. Woodworth Kumar Kshitij Patel Nathan Srebro FedML 121 204 0 08 Jun 2020
From Local SGD to Local Fixed-Point Methods for Federated Learning Grigory Malinovsky D. Kovalev Elnur Gasanov Laurent Condat Peter Richtárik FedML 126 117 0 03 Apr 2020
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates Anastasia Koloskova Nicolas Loizou Sadra Boreiri Martin Jaggi Sebastian U. Stich FedML 85 514 0 23 Mar 2020
Adaptive Federated Optimization Sashank J. Reddi Zachary B. Charles Manzil Zaheer Zachary Garrett Keith Rush Jakub Konecný Sanjiv Kumar H. B. McMahan FedML 183 1,452 0 29 Feb 2020
Is Local SGD Better than Minibatch SGD? Blake E. Woodworth Kumar Kshitij Patel Sebastian U. Stich Zhen Dai Brian Bullins H. B. McMahan Ohad Shamir Nathan Srebro FedML 74 254 0 18 Feb 2020
Advances and Open Problems in Federated Learning Peter Kairouz H. B. McMahan Brendan Avent A. Bellet M. Bennis ... Zheng Xu Qiang Yang Felix X. Yu Han Yu Sen Zhao FedML AI4CE 268 6,285 0 10 Dec 2019
On the Convergence of Local Descent Methods in Federated Learning Farzin Haddadpour M. Mahdavi FedML 87 274 0 31 Oct 2019
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning Sai Praneeth Karimireddy Satyen Kale M. Mohri Sashank J. Reddi Sebastian U. Stich A. Suresh FedML 67 347 0 14 Oct 2019
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum Jianyu Wang Vinayak Tantia Nicolas Ballas Michael G. Rabbat 76 201 0 01 Oct 2019
Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification T. Hsu Qi Matthew Brown FedML 145 1,163 0 13 Sep 2019
Tighter Theory for Local SGD on Identical and Heterogeneous Data Ahmed Khaled Konstantin Mishchenko Peter Richtárik 76 435 0 10 Sep 2019
Unified Optimal Analysis of the (Stochastic) Gradient Method Sebastian U. Stich 54 113 0 09 Jul 2019
On the Convergence of FedAvg on Non-IID Data Xiang Li Kaixuan Huang Wenhao Yang Shusen Wang Zhihua Zhang FedML 164 2,348 0 04 Jul 2019
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization Hao Yu Rong Jin Sen Yang FedML 94 387 0 09 May 2019
On the Convergence of Adam and Beyond Sashank J. Reddi Satyen Kale Surinder Kumar 106 2,505 0 19 Apr 2019
SGD: General Analysis and Improved Rates Robert Mansel Gower Nicolas Loizou Xun Qian Alibek Sailanbayev Egor Shulgin Peter Richtárik 88 380 0 27 Jan 2019
Federated Optimization in Heterogeneous Networks Tian Li Anit Kumar Sahu Manzil Zaheer Maziar Sanjabi Ameet Talwalkar Virginia Smith FedML 190 5,220 0 14 Dec 2018
Stochastic Gradient Push for Distributed Deep Learning Mahmoud Assran Nicolas Loizou Nicolas Ballas Michael G. Rabbat 91 348 0 27 Nov 2018
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms Jianyu Wang Gauri Joshi 172 349 0 22 Aug 2018
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning Hao Yu Sen Yang Shenghuo Zhu MoMe FedML 79 608 0 17 Jul 2018
Local SGD Converges Fast and Communicates Little Sebastian U. Stich FedML 187 1,069 0 24 May 2018
On the convergence properties of a $K$ -step averaging stochastic gradient descent algorithm for nonconvex optimization Fan Zhou Guojing Cong 149 236 0 03 Aug 2017
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent Xiangru Lian Ce Zhang Huan Zhang Cho-Jui Hsieh Wei Zhang Ji Liu 50 1,235 0 25 May 2017
Communication-Efficient Learning of Deep Networks from Decentralized Data H. B. McMahan Eider Moore Daniel Ramage S. Hampson Blaise Agüera y Arcas FedML 408 17,593 0 17 Feb 2016