SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum

1 October 2019

Papers citing "SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum"

50 / 50 papers shown

Title
Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training Hiroki Naganuma Xinzhi Zhang Man-Chung Yue Ioannis Mitliagkas Philipp A. Witte Russell J. Hewett Yin Tat Lee 65 0 0 25 Apr 2025
EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models Jialiang Cheng Ning Gao Yun Yue Zhiling Ye Jiadi Jiang Jian Sha OffRL 92 0 0 10 Dec 2024
Distributed Sign Momentum with Local Steps for Training Transformers Shuhua Yu Ding Zhou Cong Xie An Xu Zhi-Li Zhang Xin Liu S. Kar 79 0 0 26 Nov 2024
A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needs Yan Sun Li Shen Dacheng Tao FedML 35 0 0 27 Sep 2024
On the Convergence of a Federated Expectation-Maximization Algorithm Zhixu Tao Rajita Chandak Sanjeev R. Kulkarni FedML 40 0 0 11 Aug 2024
Ordered Momentum for Asynchronous SGD Chang-Wei Shi Yi-Rui Yang Wu-Jun Li ODL 67 0 0 27 Jul 2024
A New Theoretical Perspective on Data Heterogeneity in Federated Optimization Jiayi Wang Shiqiang Wang Rong-Rong Chen Mingyue Ji FedML 35 1 0 22 Jul 2024
Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey Feng Liang Zhen Zhang Haifeng Lu Victor C. M. Leung Yanyi Guo Xiping Hu GNN 39 6 0 09 Apr 2024
Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization Xinyu Zhang Weiyu Sun Ying-Cong Chen FedML 46 5 0 13 Sep 2023
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup Yan Sun Li Shen Hao Sun Liang Ding Dacheng Tao FedML 29 17 0 30 Jul 2023
DropCompute: simple and more robust distributed synchronous training via compute variance reduction Niv Giladi Shahar Gottlieb Moran Shkolnik A. Karnieli Ron Banner Elad Hoffer Kfir Y. Levy Daniel Soudry 45 2 0 18 Jun 2023
Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape Yan Sun Li Shen Shi-Yong Chen Liang Ding Dacheng Tao FedML 42 33 0 19 May 2023
A Unified Momentum-based Paradigm of Decentralized SGD for Non-Convex Models and Heterogeneous Data Haizhou Du Chengdong Ni 22 1 0 01 Mar 2023
On the Convergence of Federated Averaging with Cyclic Client Participation Yae Jee Cho Pranay Sharma Gauri Joshi Zheng Xu Satyen Kale Tong Zhang FedML 46 27 0 06 Feb 2023
Hierarchical Federated Learning with Momentum Acceleration in Multi-Tier Networks Zhengjie Yang Sen Fu Wei Bao Dong Yuan Albert Y. Zomaya FedML 49 6 0 26 Oct 2022
On the Stability Analysis of Open Federated Learning Systems Youbang Sun H. Fernando Tianyi Chen Shahin Shahrampour FedML 31 1 0 25 Sep 2022
FedFOR: Stateless Heterogeneous Federated Learning with First-Order Regularization Junjiao Tian James Smith Z. Kira 27 3 0 21 Sep 2022
On Generalization of Decentralized Learning with Separable Data Hossein Taheri Christos Thrampoulidis FedML 47 11 0 15 Sep 2022
Byzantines can also Learn from History: Fall of Centered Clipping in Federated Learning Kerem Ozfatura Emre Ozfatura Alptekin Kupcu Deniz Gunduz AAML FedML 41 13 0 21 Aug 2022
Accelerated Federated Learning with Decoupled Adaptive Optimization Jiayin Jin Jiaxiang Ren Yang Zhou Lingjuan Lyu Ji Liu Dejing Dou AI4CE FedML 24 52 0 14 Jul 2022
Distributed Adversarial Training to Robustify Deep Neural Networks at Scale Gaoyuan Zhang Songtao Lu Yihua Zhang Xiangyi Chen Pin-Yu Chen Quanfu Fan Lee Martie L. Horesh Min-Fong Hong Sijia Liu OOD 35 12 0 13 Jun 2022
Generalized Federated Learning via Sharpness Aware Minimization Zhe Qu Xingyu Li Rui Duan Yaojiang Liu Bo Tang Zhuo Lu FedML 53 132 0 06 Jun 2022
Communication-Efficient Adaptive Federated Learning Yujia Wang Lu Lin Jinghui Chen FedML 27 71 0 05 May 2022
Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction Junyi Li Feihu Huang Heng-Chiao Huang FedML 30 27 0 03 May 2022
Robust Federated Learning with Connectivity Failures: A Semi-Decentralized Framework with Collaborative Relaying M. Yemini R. Saha Emre Ozfatura Deniz Gündüz Andrea J. Goldsmith FedML 55 8 0 24 Feb 2022
Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank? Sheikh Shams Azam Seyyedali Hosseinalipour Qiang Qiu Christopher G. Brinton FedML 38 20 0 01 Feb 2022
Collaborative Learning over Wireless Networks: An Introductory Overview Emre Ozfatura Deniz Gunduz H. Vincent Poor 32 11 0 07 Dec 2021
What Do We Mean by Generalization in Federated Learning? Honglin Yuan Warren Morningstar Lin Ning K. Singhal OOD FedML 46 71 0 27 Oct 2021
Exponential Graph is Provably Efficient for Decentralized Deep Training Bicheng Ying Kun Yuan Yiming Chen Hanbin Hu Pan Pan W. Yin FedML 44 83 0 26 Oct 2021
Acceleration in Distributed Optimization under Similarity Helena Lofstrom G. Scutari Tianyue Cao Alexander Gasnikov 32 26 0 24 Oct 2021
Trade-offs of Local SGD at Scale: An Empirical Study Jose Javier Gonzalez Ortiz Jonathan Frankle Michael G. Rabbat Ari S. Morcos Nicolas Ballas FedML 43 19 0 15 Oct 2021
FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learning Charlie Hou K. K. Thekumparampil Giulia Fanti Sewoong Oh FedML 39 14 0 16 Aug 2021
ErrorCompensatedX: error compensation for variance reduced algorithms Hanlin Tang Yao Li Ji Liu Ming Yan 34 10 0 04 Aug 2021
A Field Guide to Federated Optimization Jianyu Wang Zachary B. Charles Zheng Xu Gauri Joshi H. B. McMahan ... Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu FedML 187 412 0 14 Jul 2021
Secure Distributed Training at Scale Eduard A. Gorbunov Alexander Borzunov Michael Diskin Max Ryabinin FedML 28 15 0 21 Jun 2021
FedCM: Federated Learning with Client-level Momentum Jing Xu Sen Wang Liwei Wang Andrew Chi-Chih Yao FedML 37 94 0 21 Jun 2021
Local Adaptivity in Federated Learning: Convergence and Consistency Jianyu Wang Zheng Xu Zachary Garrett Zachary B. Charles Luyang Liu Gauri Joshi FedML 37 39 0 04 Jun 2021
Consensus Control for Decentralized Deep Learning Lingjing Kong Tao R. Lin Anastasia Koloskova Martin Jaggi Sebastian U. Stich 19 76 0 09 Feb 2021
Time-Correlated Sparsification for Communication-Efficient Federated Learning Emre Ozfatura Kerem Ozfatura Deniz Gunduz FedML 43 47 0 21 Jan 2021
CADA: Communication-Adaptive Distributed Adam Tianyi Chen Ziye Guo Yuejiao Sun W. Yin ODL 14 24 0 31 Dec 2020
FedADC: Accelerated Federated Learning with Drift Control Emre Ozfatura Kerem Ozfatura Deniz Gunduz FedML 53 37 0 16 Dec 2020
Faster Non-Convex Federated Learning via Global and Local Momentum Rudrajit Das Anish Acharya Abolfazl Hashemi Sujay Sanghavi Inderjit S. Dhillon Ufuk Topcu FedML 40 82 0 07 Dec 2020
Federated Composite Optimization Honglin Yuan Manzil Zaheer Sashank J. Reddi FedML 37 58 0 17 Nov 2020
Throughput-Optimal Topology Design for Cross-Silo Federated Learning Othmane Marfoq Chuan Xu Giovanni Neglia Richard Vidal FedML 67 85 0 23 Oct 2020
Federated Learning with Nesterov Accelerated Gradient Zhengjie Yang Wei Bao Dong Yuan Nguyen H. Tran Albert Y. Zomaya FedML 24 31 0 18 Sep 2020
An improved convergence analysis for decentralized online stochastic non-convex optimization Ran Xin U. Khan S. Kar 44 100 0 10 Aug 2020
FetchSGD: Communication-Efficient Federated Learning with Sketching D. Rothchild Ashwinee Panda Enayat Ullah Nikita Ivkin Ion Stoica Vladimir Braverman Joseph E. Gonzalez Raman Arora FedML 33 361 0 15 Jul 2020
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization Jianyu Wang Qinghua Liu Hao Liang Gauri Joshi H. Vincent Poor MoMe FedML 23 1,304 0 15 Jul 2020
Federated Accelerated Stochastic Gradient Descent Honglin Yuan Tengyu Ma FedML 30 172 0 16 Jun 2020
Quasi-hyperbolic momentum and Adam for deep learning Jerry Ma Denis Yarats ODL 86 129 0 16 Oct 2018