RelaySum for Decentralized Deep Learning on Heterogeneous Data

8 October 2021

Thijs Vogels

Lie He

Anastasia Koloskova

Tao R. Lin

Sai Praneeth Karimireddy

Papers citing "RelaySum for Decentralized Deep Learning on Heterogeneous Data"

23 / 23 papers shown

Title
A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration Yuchen Hu Xi Chen Weidong Liu Xiaojun Mao 84 0 0 31 Jan 2025
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training Kun Yuan Yiming Chen Xinmeng Huang Yingya Zhang Pan Pan Yinghui Xu W. Yin MoE 81 64 0 24 Apr 2021
Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data Tao R. Lin Sai Praneeth Karimireddy Sebastian U. Stich Martin Jaggi FedML 73 101 0 09 Feb 2021
Optimal Complexity in Decentralized Training Yucheng Lu Christopher De Sa 70 75 0 15 Jun 2020
Evolving Normalization-Activation Layers Hanxiao Liu Andrew Brock Karen Simonyan Quoc V. Le 84 80 0 06 Apr 2020
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates Anastasia Koloskova Nicolas Loizou Sadra Boreiri Martin Jaggi Sebastian U. Stich FedML 78 505 0 23 Mar 2020
Advances and Open Problems in Federated Learning Peter Kairouz H. B. McMahan Brendan Avent A. Bellet M. Bennis ... Zheng Xu Qiang Yang Felix X. Yu Han Yu Sen Zhao FedML AI4CE 203 6,229 0 10 Dec 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 204 7,481 0 02 Oct 2019
Unified Optimal Analysis of the (Stochastic) Gradient Method Sebastian U. Stich 54 113 0 09 Jul 2019
Bayesian Nonparametric Federated Learning of Neural Networks Mikhail Yurochkin Mayank Agarwal S. Ghosh Kristjan Greenewald T. Hoang Y. Khazaeni FedML 124 728 0 28 May 2019
Stochastic Gradient Push for Distributed Deep Learning Mahmoud Assran Nicolas Loizou Nicolas Ballas Michael G. Rabbat 73 344 0 27 Nov 2018
Push-Pull Gradient Methods for Distributed Optimization in Networks Shi Pu Wei Shi Jinming Xu A. Nedić 32 320 0 15 Oct 2018
Distributed Stochastic Gradient Tracking Methods Shi Pu A. Nedić 62 291 0 25 May 2018
D $^2$ : Decentralized Training over Decentralized Data Hanlin Tang Xiangru Lian Ming Yan Ce Zhang Ji Liu 31 350 0 19 Mar 2018
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent Xiangru Lian Ce Zhang Huan Zhang Cho-Jui Hsieh Wei Zhang Ji Liu 48 1,226 0 25 May 2017
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 346 10,467 0 21 Jul 2016
Communication-Efficient Learning of Deep Networks from Decentralized Data H. B. McMahan Eider Moore Daniel Ramage S. Hampson Blaise Agüera y Arcas FedML 380 17,437 0 17 Feb 2016
NEXT: In-Network Nonconvex Optimization P. Lorenzo G. Scutari 91 508 0 01 Feb 2016
Character-level Convolutional Networks for Text Classification Xiang Zhang Jiaqi Zhao Yann LeCun 252 6,101 0 04 Sep 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 430 43,234 0 11 Feb 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 1.5K 149,842 0 22 Dec 2014
Graph diameter, eigenvalues, and minimum-time consensus Julien Hendrickx Raphaël M. Jungers Alexander Olshevsky Guillaume Vankeerberghen 79 71 0 27 Nov 2012
Distributed Delayed Stochastic Optimization Alekh Agarwal John C. Duchi 123 626 0 28 Apr 2011