PipeMare: Asynchronous Pipeline Parallel DNN Training

PipeMare: Asynchronous Pipeline Parallel DNN Training

9 October 2019

Christopher Ré

Christopher R. Aberger

Christopher De Sa

Papers citing "PipeMare: Asynchronous Pipeline Parallel DNN Training"

10 / 60 papers shown

Title
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan M. Shoeybi Jared Casper P. LeGresley M. Patwary ... Prethvi Kashinkunti J. Bernauer Bryan Catanzaro Amar Phanishayee Matei A. Zaharia MoE 37 656 0 09 Apr 2021
Parareal Neural Networks Emulating a Parallel-in-time Algorithm Zhanyu Ma Jiyang Xie Jingyi Yu AI4CE 33 9 0 16 Mar 2021
Parallel Training of Deep Networks with Local Updates Michael Laskin Luke Metz Seth Nabarrao Mark Saroufim Badreddine Noune Carlo Luschi Jascha Narain Sohl-Dickstein Pieter Abbeel FedML 32 26 0 07 Dec 2020
Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism Vipul Gupta Dhruv Choudhary P. T. P. Tang Xiaohan Wei Xing Wang Yuzhen Huang A. Kejariwal Kannan Ramchandran Michael W. Mahoney 21 33 0 18 Oct 2020
Auto-MAP: A DQN Framework for Exploring Distributed Execution Plans for DNN Workloads Siyu Wang Yi Rong Shiqing Fan Zhen Zheng Lansong Diao Guoping Long Jun Yang Xiaoyong Liu Wei Lin 15 9 0 08 Jul 2020
Adaptive Braking for Mitigating Gradient Delay Abhinav Venigalla Atli Kosson Vitaliy Chiley Urs Koster ODL 8 4 0 02 Jul 2020
DAPPLE: A Pipelined Data Parallel Approach for Training Large Models Shiqing Fan Yi Rong Chen Meng Zongyan Cao Siyu Wang ... Jun Yang Lixue Xia Lansong Diao Xiaoyong Liu Wei Lin 21 233 0 02 Jul 2020
Memory-Efficient Pipeline-Parallel DNN Training Deepak Narayanan Amar Phanishayee Kaiyu Shi Xie Chen Matei A. Zaharia MoE 45 212 0 16 Jun 2020
Pipelined Backpropagation at Scale: Training Large Models without Batches Atli Kosson Vitaliy Chiley Abhinav Venigalla Joel Hestness Urs Koster 35 33 0 25 Mar 2020
Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts Max Ryabinin Anton I. Gusev FedML 27 48 0 10 Feb 2020