XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training

v1v2v3 (latest)

XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training

24 October 2019

Dongsheng Li

ArXiv (abs)PDF HTML

Papers citing "XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training"

19 / 19 papers shown

Title
Nesterov Method for Asynchronous Pipeline Parallel Optimization Thalaiyasingam Ajanthan Sameera Ramasinghe Yan Zuo Gil Avraham Alexander Long 81 0 0 02 May 2025
TiMePReSt: Time and Memory Efficient Pipeline Parallel DNN Training with Removed Staleness Ankita Dutta Nabendu Chaki Rajat K. De 73 0 0 18 Oct 2024
Activations and Gradients Compression for Model-Parallel Training Mikhail Rudakov Aleksandr Beznosikov Yaroslav Kholodov Alexander Gasnikov 89 2 0 15 Jan 2024
Ravnest: Decentralized Asynchronous Training on Heterogeneous Devices A. Menon Unnikrishnan Menon Kailash Ahirwar 60 1 0 03 Jan 2024
PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction Lei Guan Dongsheng Li Jiye Liang Wenjian Wang Wenjian Wang Xicheng Lu 152 1 0 01 Dec 2023
AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on AdamW Basis Lei Guan ODL 118 4 0 05 Sep 2023
XGrad: Boosting Gradient-Based Optimizers With Weight Prediction Lei Guan Dongsheng Li Yanqi Shi Jian Meng ODL 96 2 0 26 May 2023
DISCO: Distributed Inference with Sparse Communications Minghai Qin Chaowen Sun Jaco A. Hofmann D. Vučinić FedML 54 1 0 22 Feb 2023
Weight Prediction Boosts the Convergence of AdamW Lei Guan 106 19 0 01 Feb 2023
LOFT: Finding Lottery Tickets through Filter-wise Training Qihan Wang Chen Dun Fangshuo Liao C. Jermaine Anastasios Kyrillidis 69 3 0 28 Oct 2022
PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks Enrico Meloni Lapo Faggi Simone Marullo Alessandro Betti Matteo Tiezzi Marco Gori S. Melacci GNN AI4TS 29 1 0 17 Oct 2022
DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices Xueyu Hou Yongjie Guan Tao Han Ning Zhang 107 42 0 03 Feb 2022
FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Framework for Heterogeneous Edge Devices Yuhao Chen Qianqian Yang Shibo He Zhiguo Shi Jiming Chen 55 3 0 06 Oct 2021
ResIST: Layer-Wise Decomposition of ResNets for Distributed Training Chen Dun Cameron R. Wolfe C. Jermaine Anastasios Kyrillidis 87 21 0 02 Jul 2021
Automatic Graph Partitioning for Very Large-scale Deep Learning Masahiro Tanaka Kenjiro Taura T. Hanawa Kentaro Torisawa GNN AI4CE 54 21 0 30 Mar 2021
Parallel Training of Deep Networks with Local Updates Michael Laskin Luke Metz Seth Nabarrao Mark Saroufim Badreddine Noune Carlo Luschi Jascha Narain Sohl-Dickstein Pieter Abbeel FedML 120 27 0 07 Dec 2020
DAPPLE: A Pipelined Data Parallel Approach for Training Large Models Shiqing Fan Yi Rong Chen Meng Zongyan Cao Siyu Wang ... Jun Yang Lixue Xia Lansong Diao Xiaoyong Liu Wei Lin 96 241 0 02 Jul 2020
torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models Chiheon Kim Heungsub Lee Myungryong Jeong Woonhyuk Baek Boogeon Yoon Ildoo Kim Sungbin Lim Sungwoong Kim MoE AI4CE 51 54 0 21 Apr 2020
Taming Momentum in a Distributed Asynchronous Environment Ido Hakimi Saar Barkai Moshe Gabel Assaf Schuster 93 23 0 26 Jul 2019