Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD

3 March 2018

Papers citing "Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD"

50 / 101 papers shown

Title
Orthogonal Calibration for Asynchronous Federated Learning Jiayun Zhang Shuheng Li Haiyu Huang Xiaofan Yu Rajesh K. Gupta Jingbo Shang FedML 65 0 0 21 Feb 2025
ATA: Adaptive Task Allocation for Efficient Resource Management in Distributed Machine Learning A. Maranjyan El Mehdi Saad Peter Richtárik Francesco Orabona 57 0 0 02 Feb 2025
Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexity A. Maranjyan A. Tyurin Peter Richtárik 39 2 0 28 Jan 2025
MindFlayer: Efficient Asynchronous Parallel SGD in the Presence of Heterogeneous and Random Worker Compute Times A. Maranjyan Omar Shaikh Omar Peter Richtárik 26 3 0 05 Oct 2024
Faster Stochastic Optimization with Arbitrary Delays via Asynchronous Mini-Batching Amit Attia Ofir Gaash Tomer Koren 40 0 0 14 Aug 2024
Finite-Time Analysis of Asynchronous Multi-Agent TD Learning Nicolò Dal Fabbro Arman Adibi Aritra Mitra George J. Pappas 42 1 0 29 Jul 2024
Ordered Momentum for Asynchronous SGD Chang-Wei Shi Yi-Rui Yang Wu-Jun Li ODL 56 0 0 27 Jul 2024
Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Framework Siyuan Yu Wei Chen H. V. Poor 32 0 0 17 Jun 2024
Buffered Asynchronous Secure Aggregation for Cross-Device Federated Learning Kun Wang Yi-Rui Yang Wu-Jun Li 42 0 0 05 Jun 2024
ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training Adel Nabli Louis Fournier Pierre Erbacher Louis Serrano Eugene Belilovsky Edouard Oyallon FedML 46 1 0 03 Jun 2024
FedAST: Federated Asynchronous Simultaneous Training Baris Askin Pranay Sharma Carlee Joe-Wong Gauri Joshi 49 1 0 01 Jun 2024
Communication-Efficient Distributed Deep Learning via Federated Dynamic Averaging Michail Theologitis Georgios Frangias Georgios Anestis V. Samoladas Antonios Deligiannakis FedML 40 0 0 31 May 2024
Dual-Delayed Asynchronous SGD for Arbitrarily Heterogeneous Data Xiaolu Wang Yuchang Sun Hoi-To Wai Jun Zhang 49 0 0 27 May 2024
Freya PAGE: First Optimal Time Complexity for Large-Scale Nonconvex Finite-Sum Optimization with Heterogeneous Asynchronous Computations A. Tyurin Kaja Gruntkowska Peter Richtárik 42 3 0 24 May 2024
Asynchronous Federated Stochastic Optimization for Heterogeneous Objectives Under Arbitrary Delays Charikleia Iakovidou Kibaek Kim FedML 35 2 0 16 May 2024
AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodes Youshao Xiao Lin Ju Zhenglei Zhou Siyuan Li Zhaoxin Huan ... Rujie Jiang Lin Wang Xiaolu Zhang Lei Liang Jun Zhou 32 1 0 15 Apr 2024
Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling Arman Adibi Nicolò Dal Fabbro Luca Schenato Sanjeev R. Kulkarni H. Vincent Poor George J. Pappas Hamed Hassani A. Mitra 35 8 0 19 Feb 2024
Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clients Xiaolu Wang Zijian Li Shi Jin Jun Zhang FedML 23 3 0 17 Feb 2024
Momentum Approximation in Asynchronous Private Federated Learning Tao Yu Congzheng Song Jianyu Wang Mona Chitnis FedML 37 1 0 14 Feb 2024
CO2: Efficient Distributed Training with Full Communication-Computation Overlap Weigao Sun Zhen Qin Weixuan Sun Shidi Li Dong Li Xuyang Shen Yu Qiao Yiran Zhong OffRL 61 10 0 29 Jan 2024
Ravnest: Decentralized Asynchronous Training on Heterogeneous Devices A. Menon Unnikrishnan Menon Kailash Ahirwar 21 1 0 03 Jan 2024
DIGEST: Fast and Communication Efficient Decentralized Learning with Local Updates Peyman Gholami H. Seferoglu FedML 18 11 0 14 Jul 2023
The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey Jana Vatter R. Mayer Hans-Arno Jacobsen GNN AI4TS AI4CE 45 23 0 23 May 2023
Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching S. Tyagi Prateek Sharma 16 22 0 20 May 2023
Fast and Straggler-Tolerant Distributed SGD with Reduced Computation Load Maximilian Egger Serge Kas Hanna Rawad Bitar FedML 24 0 0 17 Apr 2023
Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacy Anastasia Koloskova Ryan McKenna Zachary B. Charles Keith Rush Brendan McMahan 27 8 0 02 Feb 2023
ABS: Adaptive Bounded Staleness Converges Faster and Communicates Less Qiao Tan Feng Zhu Jingjing Zhang 49 0 0 21 Jan 2023
Accelerating Parallel Stochastic Gradient Descent via Non-blocking Mini-batches Haoze He Parijat Dube 6 3 0 02 Nov 2022
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities Brian Bartoldson B. Kailkhura Davis W. Blalock 31 47 0 13 Oct 2022
STSyn: Speeding Up Local SGD with Straggler-Tolerant Synchronization Feng Zhu Jingjing Zhang Xin Wang 33 3 0 06 Oct 2022
Adaptive Stochastic Gradient Descent for Fast and Communication-Efficient Distributed Learning Serge Kas Hanna Rawad Bitar Parimal Parag Venkateswara Dasari S. E. Rouayheb 22 2 0 04 Aug 2022
Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning Anastasia Koloskova Sebastian U. Stich Martin Jaggi FedML 24 77 0 16 Jun 2022
AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregation Qiyuan Wang Qianqian Yang Shibo He Zhiguo Shi Jiming Chen FedML 37 25 0 27 May 2022
GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Model Wenbo Su Yuanxing Zhang Yufeng Cai Kaixu Ren Pengjie Wang ... Jing Chen Hongbo Deng Jian Xu Lin Qu Bo Zheng 28 4 0 23 May 2022
CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU Zangwei Zheng Peng Xu Xuan Zou Da Tang Zhen Li ... Xiangzhuo Ding Fuzhao Xue Ziheng Qing Youlong Cheng Yang You VLM 44 7 0 13 Apr 2022
DeFTA: A Plug-and-Play Decentralized Replacement for FedAvg Yuhao Zhou M. Shi Yuxin Tian Qing Ye Jiancheng Lv FedML 19 2 0 06 Apr 2022
FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulations Mirian Hipolito Garcia Andre Manoel Daniel Madrigal Diaz Fatemehsadat Mireshghallah Robert Sim Dimitrios Dimitriadis FedML 27 57 0 25 Mar 2022
Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data Zihao Zhou Yanan Li Xuebin Ren Shusen Yang 20 29 0 02 Mar 2022
Cost-Efficient Distributed Learning via Combinatorial Multi-Armed Bandits Maximilian Egger Rawad Bitar Antonia Wachter-Zeh Deniz Gunduz FedML 22 2 0 16 Feb 2022
Asynchronous Decentralized Learning over Unreliable Wireless Networks Eunjeong Jeong Matteo Zecchin Marios Kountouris 19 16 0 02 Feb 2022
Lightweight Projective Derivative Codes for Compressed Asynchronous Gradient Descent Pedro Soto Ilia Ilmer Haibin Guan Jun Li 27 3 0 31 Jan 2022
Privacy-Preserving Serverless Edge Learning with Decentralized Small Data Shih-Chun Lin Chia-Hung Lin 21 4 0 29 Nov 2021
DSAG: A mixed synchronous-asynchronous iterative method for straggler-resilient learning A. Severinson E. Rosnes S. E. Rouayheb Alexandre Graell i Amat 14 2 0 27 Nov 2021
Trade-offs of Local SGD at Scale: An Empirical Study Jose Javier Gonzalez Ortiz Jonathan Frankle Michael G. Rabbat Ari S. Morcos Nicolas Ballas FedML 37 19 0 15 Oct 2021
Distributed Optimization using Heterogeneous Compute Systems S. Vineeth 22 0 0 03 Oct 2021
Coding for Straggler Mitigation in Federated Learning Siddhartha Kumar Reent Schlegel E. Rosnes Alexandre Graell i Amat FedML 21 11 0 30 Sep 2021
On the Future of Cloud Engineering David Bermbach A. Chandra C. Krintz A. Gokhale Aleksander Slominski L. Thamsen Everton Cavalcante Tian Guo Ivona Brandić R. Wolski 35 23 0 19 Aug 2021
A Field Guide to Federated Optimization Jianyu Wang Zachary B. Charles Zheng Xu Gauri Joshi H. B. McMahan ... Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu FedML 187 412 0 14 Jul 2021
Learning Under Delayed Feedback: Implicitly Adapting to Gradient Delays R. Aviv Ido Hakimi Assaf Schuster Kfir Y. Levy 30 9 0 23 Jun 2021
Federated Learning with Buffered Asynchronous Aggregation John Nguyen Kshitiz Malik Hongyuan Zhan Ashkan Yousefpour Michael G. Rabbat Mani Malek Dzmitry Huba FedML 33 288 0 11 Jun 2021