Stochastic Gradient Push for Distributed Deep Learning

27 November 2018

Papers citing "Stochastic Gradient Push for Distributed Deep Learning"

50 / 65 papers shown

Title
Dyn-D $^2$ P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guarantee Zehan Zhu Yan Huang Xin Wang Shouling Ji Jinming Xu 26 0 0 10 May 2025
Hubs and Spokes Learning: Efficient and Scalable Collaborative Machine Learning Atul Sharma Kavindu Herath Saurabh Bagchi Chaoyue Liu Somali Chaterji FedML 42 0 0 29 Apr 2025
From Interpretation to Correction: A Decentralized Optimization Framework for Exact Convergence in Federated Learning Bicheng Ying Zhe Li Haibo Yang FedML 70 0 0 25 Mar 2025
Scalable Decentralized Learning with Teleportation Yuki Takezawa Sebastian U. Stich 64 1 0 25 Jan 2025
Distributed Sign Momentum with Local Steps for Training Transformers Shuhua Yu Ding Zhou Cong Xie An Xu Zhi-Li Zhang Xin Liu S. Kar 69 0 0 26 Nov 2024
Fair Decentralized Learning Sayan Biswas Anne-Marie Kermarrec Rishi Sharma Thibaud Trinca M. Vos FedML 32 0 0 03 Oct 2024
Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance Dimitris Oikonomou Nicolas Loizou 55 4 0 06 Jun 2024
SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Data Sakshi Choudhary Sai Aparna Aketi Kaushik Roy FedML 45 0 0 22 May 2024
Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors Chun-Yin Huang Kartik Srinivas Xin Zhang Xiaoxiao Li DD 58 6 0 19 May 2024
PrivSGP-VR: Differentially Private Variance-Reduced Stochastic Gradient Push with Tight Utility Bounds Zehan Zhu Yan Huang Xin Wang Jinming Xu 43 0 0 04 May 2024
Sequence-to-sequence models in peer-to-peer learning: A practical application Robert Šajina Ivo Ipšić 46 0 0 02 May 2024
DropCompute: simple and more robust distributed synchronous training via compute variance reduction Niv Giladi Shahar Gottlieb Moran Shkolnik A. Karnieli Ron Banner Elad Hoffer Kfir Y. Levy Daniel Soudry 33 2 0 18 Jun 2023
Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updates Siqi Zhang S. Choudhury Sebastian U. Stich Nicolas Loizou FedML 19 3 0 08 Jun 2023
Collaborative Learning via Prediction Consensus Dongyang Fan Celestine Mendler-Dünner Martin Jaggi FedML 29 7 0 29 May 2023
Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergence Yuki Takezawa Ryoma Sato Han Bao Kenta Niwa M. Yamada 31 9 0 19 May 2023
Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compression Boyue Li Yuejie Chi 21 12 0 17 May 2023
Homogenizing Non-IID datasets via In-Distribution Knowledge Distillation for Decentralized Learning Deepak Ravikumar Gobinda Saha Sai Aparna Aketi Kaushik Roy 21 1 0 09 Apr 2023
CoDeC: Communication-Efficient Decentralized Continual Learning Sakshi Choudhary Sai Aparna Aketi Gobinda Saha Kaushik Roy CLL 50 3 0 27 Mar 2023
Beyond spectral gap (extended): The role of the topology in decentralized learning Thijs Vogels Hadrien Hendrikx Martin Jaggi 29 3 0 05 Jan 2023
Addressing Data Heterogeneity in Decentralized Learning via Topological Pre-processing Waqwoya Abebe Ali Jannesari 25 0 0 16 Dec 2022
On Generalization of Decentralized Learning with Separable Data Hossein Taheri Christos Thrampoulidis FedML 27 10 0 15 Sep 2022
Beyond spectral gap: The role of the topology in decentralized learning Thijs Vogels Hadrien Hendrikx Martin Jaggi FedML 11 27 0 07 Jun 2022
FedAvg with Fine Tuning: Local Updates Lead to Representation Learning Liam Collins Hamed Hassani Aryan Mokhtari Sanjay Shakkottai FedML 32 75 0 27 May 2022
Semi-Decentralized Federated Learning with Collaborative Relaying M. Yemini R. Saha Emre Ozfatura Deniz Gündüz Andrea J. Goldsmith FedML 42 32 0 23 May 2022
Data-heterogeneity-aware Mixing for Decentralized Learning Yatin Dandi Anastasia Koloskova Martin Jaggi Sebastian U. Stich 38 18 0 13 Apr 2022
CFL: Cluster Federated Learning in Large-scale Peer-to-Peer Networks Qian Chen Zilong Wang Yilin Zhou Jiawei Chen Danyang Xiao Xiaodong Lin FedML 17 6 0 08 Apr 2022
Robust Federated Learning with Connectivity Failures: A Semi-Decentralized Framework with Collaborative Relaying M. Yemini R. Saha Emre Ozfatura Deniz Gündüz Andrea J. Goldsmith FedML 45 8 0 24 Feb 2022
Variance-Reduced Stochastic Quasi-Newton Methods for Decentralized Learning: Part I Jiaojiao Zhang Huikang Liu Anthony Man-Cho So Qing Ling 24 14 0 19 Jan 2022
Distributed Random Reshuffling over Networks Kun-Yen Huang Xiao Li Andre Milzarek Shi Pu Junwen Qiu 39 11 0 31 Dec 2021
Personalized On-Device E-health Analytics with Decentralized Block Coordinate Descent Guanhua Ye Hongzhi Yin Tong Chen Miao Xu Quoc Viet Hung Nguyen Jiangning Song 41 9 0 17 Dec 2021
BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning Bicheng Ying Kun Yuan Hanbin Hu Yiming Chen W. Yin FedML 39 27 0 08 Nov 2021
Exponential Graph is Provably Efficient for Decentralized Deep Training Bicheng Ying Kun Yuan Yiming Chen Hanbin Hu Pan Pan W. Yin FedML 39 83 0 26 Oct 2021
A Unified and Refined Convergence Analysis for Non-Convex Decentralized Learning Sulaiman A. Alghunaim Kun Yuan 25 57 0 19 Oct 2021
Trade-offs of Local SGD at Scale: An Empirical Study Jose Javier Gonzalez Ortiz Jonathan Frankle Michael G. Rabbat Ari S. Morcos Nicolas Ballas FedML 43 19 0 15 Oct 2021
RelaySum for Decentralized Deep Learning on Heterogeneous Data Thijs Vogels Lie He Anastasia Koloskova Tao R. Lin Sai Praneeth Karimireddy Sebastian U. Stich Martin Jaggi FedML MoE 11 61 0 08 Oct 2021
On the Convergence of Decentralized Adaptive Gradient Methods Xiangyi Chen Belhal Karimi Weijie Zhao Ping Li 23 21 0 07 Sep 2021
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines Shigang Li Torsten Hoefler GNN AI4CE LRM 80 131 0 14 Jul 2021
Secure Distributed Training at Scale Eduard A. Gorbunov Alexander Borzunov Michael Diskin Max Ryabinin FedML 21 15 0 21 Jun 2021
Decentralized Local Stochastic Extra-Gradient for Variational Inequalities Aleksandr Beznosikov Pavel Dvurechensky Anastasia Koloskova V. Samokhin Sebastian U. Stich Alexander Gasnikov 32 43 0 15 Jun 2021
From Distributed Machine Learning to Federated Learning: A Survey Ji Liu Jizhou Huang Yang Zhou Xuhong Li Shilei Ji Haoyi Xiong Dejing Dou FedML OOD 51 244 0 29 Apr 2021
Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices Max Ryabinin Eduard A. Gorbunov Vsevolod Plokhotnyuk Gennady Pekhimenko 35 32 0 04 Mar 2021
Sparse-Push: Communication- & Energy-Efficient Decentralized Distributed Learning over Directed & Time-Varying Graphs with non-IID Datasets Sai Aparna Aketi Amandeep Singh J. Rabaey 29 10 0 10 Feb 2021
Consensus Control for Decentralized Deep Learning Lingjing Kong Tao R. Lin Anastasia Koloskova Martin Jaggi Sebastian U. Stich 19 76 0 09 Feb 2021
Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications Shuyan Hu Xiaojing Chen Wei Ni Ekram Hossain Xin Wang AI4CE 47 111 0 02 Dec 2020
On the Benefits of Multiple Gossip Steps in Communication-Constrained Decentralized Optimization Abolfazl Hashemi Anish Acharya Rudrajit Das H. Vikalo Sujay Sanghavi Inderjit Dhillon 20 7 0 20 Nov 2020
CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery Kiwan Maeng Shivam Bharuka Isabel Gao M. C. Jeffrey V. Saraph ... Caroline Trippel Jiyan Yang Michael G. Rabbat Brandon Lucia Carole-Jean Wu OffRL 24 31 0 05 Nov 2020
FederBoost: Private Federated Learning for GBDT Zhihua Tian Rui Zhang Xiaoyang Hou Jian-wei Liu K. Ren Jian Liu Kui Ren FedML AI4CE 47 66 0 05 Nov 2020
Throughput-Optimal Topology Design for Cross-Silo Federated Learning Othmane Marfoq Chuan Xu Giovanni Neglia Richard Vidal FedML 67 85 0 23 Oct 2020
Periodic Stochastic Gradient Descent with Momentum for Decentralized Training Hongchang Gao Heng-Chiao Huang 23 25 0 24 Aug 2020
An improved convergence analysis for decentralized online stochastic non-convex optimization Ran Xin U. Khan S. Kar 18 100 0 10 Aug 2020