torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

21 April 2020

Myungryong Jeong

Papers citing "torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models"

11 / 11 papers shown

Title
ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling Xiao Wang Jong Youl Choi Takuya Kurihaya Isaac Lyngaas Hong-Jun Yoon ... Dali Wang Peter Thornton Prasanna Balaprakash M. Ashfaq Dan Lu 30 0 0 07 May 2025
FreeRide: Harvesting Bubbles in Pipeline Parallelism Jiashu Zhang Zihan Pan Molly Xu Khuzaima S. Daudjee 90 0 0 11 Sep 2024
PETRA: Parallel End-to-end Training with Reversible Architectures Stéphane Rivaud Louis Fournier Thomas Pumir Eugene Belilovsky Michael Eickenberg Edouard Oyallon 25 0 0 04 Jun 2024
Linear Attention Sequence Parallelism Weigao Sun Zhen Qin Dong Li Xuyang Shen Yu Qiao Yiran Zhong 76 2 0 03 Apr 2024
A Survey From Distributed Machine Learning to Distributed Deep Learning Mohammad Dehghani Zahra Yazdanparast 26 0 0 11 Jul 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 31 47 0 02 Feb 2023
Systems for Parallel and Distributed Large-Model Deep Learning Training Kabir Nagrecha GNN VLM MoE 34 7 0 06 Jan 2023
PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks Enrico Meloni Lapo Faggi Simone Marullo Alessandro Betti Matteo Tiezzi Marco Gori S. Melacci GNN AI4TS 24 1 0 17 Oct 2022
Hydra: A System for Large Multi-Model Deep Learning Kabir Nagrecha Arun Kumar MoE AI4CE 38 5 0 16 Oct 2021
An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks A. Kahira Truong Thao Nguyen L. Bautista-Gomez Ryousei Takano Rosa M. Badia Mohamed Wahib 18 9 0 19 Apr 2021
DAPPLE: A Pipelined Data Parallel Approach for Training Large Models Shiqing Fan Yi Rong Chen Meng Zongyan Cao Siyu Wang ... Jun Yang Lixue Xia Lansong Diao Xiaoyong Liu Wei Lin 21 233 0 02 Jul 2020