Optimizing Multi-GPU Parallelization Strategies for Deep Learning
Training

Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training

30 July 2019

Papers citing "Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training"

8 / 8 papers shown

Title
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training Jared Fernandez Luca Wehrstedt Leonid Shamis Mostafa Elhoushi Kalyan Saladi Yonatan Bisk Emma Strubell Jacob Kahn 195 3 0 20 Nov 2024
Expediting Distributed DNN Training with Device Topology-Aware Graph Deployment Shiwei Zhang Xiaodong Yi Lansong Diao Chuan Wu Siyu Wang W. Lin GNN 11 5 0 13 Feb 2023
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression Jaeyong Song Jinkyu Yim Jaewon Jung Hongsun Jang H. Kim Youngsok Kim Jinho Lee GNN 16 25 0 24 Jan 2023
Dataloader Parameter Tuner: An Automated Dataloader Parameter Tuner for Deep Learning Models Jooyoung Park DoangJoo Synn XinYu Piao Jong-Kook Kim 11 0 0 11 Oct 2022
DAPPLE: A Pipelined Data Parallel Approach for Training Large Models Shiqing Fan Yi Rong Chen Meng Zongyan Cao Siyu Wang ... Jun Yang Lixue Xia Lansong Diao Xiaoyong Liu Wei Lin 21 232 0 02 Jul 2020
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism Jay H. Park Gyeongchan Yun Chang Yi N. T. Nguyen Seungmin Lee Jaesik Choi S. Noh Young-ri Choi MoE 25 128 0 28 May 2020
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,743 0 26 Sep 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 281 2,889 0 15 Sep 2016