Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data

9 February 2021

Tao R. Lin

Sai Praneeth Karimireddy

Papers citing "Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data"

37 / 37 papers shown

Title
PDSL: Privacy-Preserved Decentralized Stochastic Learning with Heterogeneous Data Distribution Lina Wang Yunsheng Yuan Chunxiao Wang Feng Li FedML 43 0 0 31 Mar 2025
A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration Yuchen Hu Xi Chen Weidong Liu Xiaojun Mao 59 0 0 31 Jan 2025
Scalable Decentralized Learning with Teleportation Yuki Takezawa Sebastian U. Stich 72 1 0 25 Jan 2025
Ordered Momentum for Asynchronous SGD Chang-Wei Shi Yi-Rui Yang Wu-Jun Li ODL 62 0 0 27 Jul 2024
SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Data Sakshi Choudhary Sai Aparna Aketi Kaushik Roy FedML 45 0 0 22 May 2024
DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models Nastaran Saadati Minh Pham Nasla Saleem Joshua R. Waite Aditya Balu Zhanhong Jiang Chinmay Hegde Soumik Sarkar MoMe 49 1 0 11 Apr 2024
Convergence of Decentralized Stochastic Subgradient-based Methods for Nonsmooth Nonconvex functions Siyuan Zhang Nachuan Xiao Xin Liu 61 1 0 18 Mar 2024
Averaging Rate Scheduler for Decentralized Learning on Heterogeneous Data Sai Aparna Aketi Sakshi Choudhary Kaushik Roy 32 1 0 05 Mar 2024
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup Yan Sun Li Shen Hao Sun Liang Ding Dacheng Tao FedML 29 17 0 30 Jul 2023
Achieving Linear Speedup in Decentralized Stochastic Compositional Minimax Optimization Hongchang Gao 42 1 0 25 Jul 2023
Structured Cooperative Learning with Graphical Model Priors Shuang-Yang Li Dinesh Manocha Xinmei Tian Dacheng Tao 38 0 0 16 Jun 2023
Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergence Yuki Takezawa Ryoma Sato Han Bao Kenta Niwa M. Yamada 39 9 0 19 May 2023
Homogenizing Non-IID datasets via In-Distribution Knowledge Distillation for Decentralized Learning Deepak Ravikumar Gobinda Saha Sai Aparna Aketi Kaushik Roy 21 2 0 09 Apr 2023
CoDeC: Communication-Efficient Decentralized Continual Learning Sakshi Choudhary Sai Aparna Aketi Gobinda Saha Kaushik Roy CLL 53 3 0 27 Mar 2023
A Unified Momentum-based Paradigm of Decentralized SGD for Non-Convex Models and Heterogeneous Data Haizhou Du Chengdong Ni 22 1 0 01 Mar 2023
Beyond spectral gap (extended): The role of the topology in decentralized learning Thijs Vogels Hadrien Hendrikx Martin Jaggi 29 3 0 05 Jan 2023
Aggregation in the Mirror Space (AIMS): Fast, Accurate Distributed Machine Learning in Military Settings Ryan Yang Haizhou Du Andre Wibisono Patrick Baker 16 1 0 28 Oct 2022
On Generalization of Decentralized Learning with Separable Data Hossein Taheri Christos Thrampoulidis FedML 42 11 0 15 Sep 2022
Beyond spectral gap: The role of the topology in decentralized learning Thijs Vogels Hadrien Hendrikx Martin Jaggi FedML 11 27 0 07 Jun 2022
Data-heterogeneity-aware Mixing for Decentralized Learning Yatin Dandi Anastasia Koloskova Martin Jaggi Sebastian U. Stich 38 18 0 13 Apr 2022
Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching Zexi Li Jiaxun Lu Shuang Luo Didi Zhu Yunfeng Shao Yinchuan Li Zhimeng Zhang Yongheng Wang Chao-Xiang Wu FedML 40 36 0 23 Mar 2022
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally! Konstantin Mishchenko Grigory Malinovsky Sebastian U. Stich Peter Richtárik 11 152 0 18 Feb 2022
An Improved Analysis of Gradient Tracking for Decentralized Machine Learning Anastasia Koloskova Tao R. Lin Sebastian U. Stich 22 106 0 08 Feb 2022
Personalized Federated Learning with Adaptive Batchnorm for Healthcare Wang Lu Jindong Wang Yiqiang Chen Xin Qin Renjun Xu Dimitrios Dimitriadis Tao Qin FedML OOD 24 61 0 01 Dec 2021
Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications Khaled B. Letaief Yuanming Shi Jianmin Lu Jianhua Lu 45 417 0 24 Nov 2021
BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning Bicheng Ying Kun Yuan Hanbin Hu Yiming Chen W. Yin FedML 39 27 0 08 Nov 2021
Exponential Graph is Provably Efficient for Decentralized Deep Training Bicheng Ying Kun Yuan Yiming Chen Hanbin Hu Pan Pan W. Yin FedML 39 83 0 26 Oct 2021
RelaySum for Decentralized Deep Learning on Heterogeneous Data Thijs Vogels Lie He Anastasia Koloskova Tao R. Lin Sai Praneeth Karimireddy Sebastian U. Stich Martin Jaggi FedML MoE 11 61 0 08 Oct 2021
FedCM: Federated Learning with Client-level Momentum Jing Xu Sen Wang Liwei Wang Andrew Chi-Chih Yao FedML 32 94 0 21 Jun 2021
Decentralized Local Stochastic Extra-Gradient for Variational Inequalities Aleksandr Beznosikov Pavel Dvurechensky Anastasia Koloskova V. Samokhin Sebastian U. Stich Alexander Gasnikov 32 43 0 15 Jun 2021
D-Cliques: Compensating for Data Heterogeneity with Topology in Decentralized Federated Learning A. Bellet Anne-Marie Kermarrec Erick Lavoie FedML 28 21 0 15 Apr 2021
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization Xiaoxiao Li Meirui Jiang Xiaofei Zhang Michael Kamp Qi Dou OOD FedML 168 790 0 15 Feb 2021
Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training Shen-Yi Zhao Chang-Wei Shi Yin-Peng Xie Wu-Jun Li ODL 26 8 0 28 Jul 2020
FedML: A Research Library and Benchmark for Federated Machine Learning Chaoyang He Songze Li Jinhyun So Xiao Zeng Mi Zhang ... Yang Liu Ramesh Raskar Qiang Yang M. Annavaram Salman Avestimehr FedML 168 566 0 27 Jul 2020
Bag of Tricks for Image Classification with Convolutional Neural Networks Tong He Zhi-Li Zhang Hang Zhang Zhongyue Zhang Junyuan Xie Mu Li 224 1,400 0 04 Dec 2018
Quasi-hyperbolic momentum and Adam for deep learning Jerry Ma Denis Yarats ODL 84 129 0 16 Oct 2018
Optimal Distributed Online Prediction using Mini-Batches O. Dekel Ran Gilad-Bachrach Ohad Shamir Lin Xiao 182 683 0 07 Dec 2010