Fully Decoupled Neural Network Learning Using Delayed Gradients

Fully Decoupled Neural Network Learning Using Delayed Gradients

21 June 2019

Yi Wang

Zhiping Lin

Papers citing "Fully Decoupled Neural Network Learning Using Delayed Gradients"

11 / 11 papers shown

Title
PETRA: Parallel End-to-end Training with Reversible Architectures Stéphane Rivaud Louis Fournier Thomas Pumir Eugene Belilovsky Michael Eickenberg Edouard Oyallon 25 0 0 04 Jun 2024
ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training Adel Nabli Louis Fournier Pierre Erbacher Louis Serrano Eugene Belilovsky Edouard Oyallon FedML 48 1 0 03 Jun 2024
A Survey From Distributed Machine Learning to Distributed Deep Learning Mohammad Dehghani Zahra Yazdanparast 23 0 0 11 Jul 2023
ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction Vahid Janfaza Shantanu Mandal Farabi Mahmud A. Muzahid 19 2 0 22 May 2023
Asynchronous Training Schemes in Distributed Learning with Time Delay Haoxiang Wang Zhanhong Jiang Chao Liu S. Sarkar D. Jiang Young M. Lee 25 2 0 28 Aug 2022
Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learning S. Akintoye Liangxiu Han H. Lloyd Xin Zhang Darren Dancey Haoming Chen Daoqiang Zhang FedML 34 5 0 22 Jul 2022
Cortico-cerebellar networks as decoupling neural interfaces J. Pemberton E. Boven Richard Apps Rui Ponte Costa 30 6 0 21 Oct 2021
Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework Junxiang Wang Hongyi Li Zheng Chai Yongchao Wang Yue Cheng Liang Zhao MQ 21 3 0 20 May 2021
A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning S. Akintoye Liangxiu Han Xin Zhang Haoming Chen Daoqiang Zhang 22 15 0 11 Apr 2021
Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization Huiping Zhuang Zhiping Lin Kar-Ann Toh 26 4 0 03 Dec 2020
Pipelined Backpropagation at Scale: Training Large Models without Batches Atli Kosson Vitaliy Chiley Abhinav Venigalla Joel Hestness Urs Koster 35 33 0 25 Mar 2020