Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

3 March 2021

Papers citing "Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates"

11 / 11 papers shown

Title
Ordered Momentum for Asynchronous SGD Chang-Wei Shi Yi-Rui Yang Wu-Jun Li ODL 67 0 0 27 Jul 2024
Asynchronous Federated Stochastic Optimization for Heterogeneous Objectives Under Arbitrary Delays Charikleia Iakovidou Kibaek Kim FedML 40 2 0 16 May 2024
Asynchronous Federated Learning with Bidirectional Quantized Communications and Buffered Aggregation Tomàs Ortega Hamid Jafarkhani FedML 41 6 0 01 Aug 2023
PersA-FL: Personalized Asynchronous Federated Learning Taha Toghani Soomin Lee César A. Uribe FedML 62 6 0 03 Oct 2022
Characterizing & Finding Good Data Orderings for Fast Convergence of Sequential Gradient Methods Amirkeivan Mohtashami Sebastian U. Stich Martin Jaggi 26 13 0 03 Feb 2022
Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimization Kaiwen Zhou Anthony Man-Cho So James Cheng 32 1 0 30 Sep 2021
Layered gradient accumulation and modular pipeline parallelism: fast and efficient training of large language models J. Lamy-Poirier MoE 29 8 0 04 Jun 2021
Consensus Control for Decentralized Deep Learning Lingjing Kong Tao R. Lin Anastasia Koloskova Martin Jaggi Sebastian U. Stich 19 76 0 09 Feb 2021
New Convergence Aspects of Stochastic Gradient Algorithms Lam M. Nguyen Phuong Ha Nguyen Peter Richtárik K. Scheinberg Martin Takáč Marten van Dijk 33 66 0 10 Nov 2018
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 312 2,896 0 15 Sep 2016
Optimal Distributed Online Prediction using Mini-Batches O. Dekel Ran Gilad-Bachrach Ohad Shamir Lin Xiao 186 683 0 07 Dec 2010