Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash

Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash

13 November 2018

Hisahiro Suganuma

Pongsakorn U-chupala

Yoshiki Tanaka

Yuichi Kageyama

Papers citing "Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash"

14 / 14 papers shown

Title
Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models Saeed Rashidi William Won Sudarshan Srinivasan Srinivas Sridharan T. Krishna GNN 30 30 0 09 Oct 2021
GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent Heesu Kim Hanmin Park Taehyun Kim Kwanheum Cho Eojin Lee Soojung Ryu Hyuk-Jae Lee Kiyoung Choi Jinho Lee 24 36 0 15 Feb 2021
A Comprehensive Survey on Hardware-Aware Neural Architecture Search Hadjer Benmeziane Kaoutar El Maghraoui Hamza Ouarnoughi Smail Niar Martin Wistuba Naigang Wang 34 98 0 22 Jan 2021
Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms Saeed Rashidi Matthew Denton Srinivas Sridharan Sudarshan Srinivasan Amoghavarsha Suresh Jade Nie T. Krishna 34 45 0 30 Jun 2020
The Limit of the Batch Size Yang You Yuhui Wang Huan Zhang Zhao-jie Zhang J. Demmel Cho-Jui Hsieh 16 15 0 15 Jun 2020
Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures Dhiraj D. Kalamkar E. Georganas Sudarshan Srinivasan Jianping Chen Mikhail Shiryaev A. Heinecke 56 48 0 10 May 2020
Communication optimization strategies for distributed deep neural network training: A survey Shuo Ouyang Dezun Dong Yemao Xu Liquan Xiao 30 12 0 06 Mar 2020
Taming Momentum in a Distributed Asynchronous Environment Ido Hakimi Saar Barkai Moshe Gabel Assaf Schuster 19 23 0 26 Jul 2019
Database Meets Deep Learning: Challenges and Opportunities Wei Wang Meihui Zhang Gang Chen H. V. Jagadish Beng Chin Ooi K. Tan 18 147 0 21 Jun 2019
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes Yang You Jing Li Sashank J. Reddi Jonathan Hseu Sanjiv Kumar Srinadh Bhojanapalli Xiaodan Song J. Demmel Kurt Keutzer Cho-Jui Hsieh ODL 28 985 0 01 Apr 2019
Augment your batch: better training with larger batches Elad Hoffer Tal Ben-Nun Itay Hubara Niv Giladi Torsten Hoefler Daniel Soudry ODL 30 72 0 27 Jan 2019
Large batch size training of neural networks with adversarial training and second-order information Z. Yao A. Gholami Daiyaan Arfeen Richard Liaw Joseph E. Gonzalez Kurt Keutzer Michael W. Mahoney ODL 14 42 0 02 Oct 2018
Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis Tal Ben-Nun Torsten Hoefler GNN 33 704 0 26 Feb 2018
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 310 2,896 0 15 Sep 2016