COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Training

30 November 2022

Papers citing "COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Training"

35 / 35 papers shown

Title
DistSim: A performance model of large-scale hybrid distributed DNN training Guandong Lu Run Chen Yakai Wang Yangjie Zhou Rui Zhang ... Yanming Miao Zhifang Cai Li-Wei Li Jingwen Leng Minyi Guo 74 11 0 14 Jun 2023
TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings N. Jouppi George Kurian Sheng Li Peter C. Ma R. Nagarajan ... Brian Towles C. Young Xiaoping Zhou Zongwei Zhou David A. Patterson BDL VLM 114 357 0 04 Apr 2023
ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale William Won Taekyung Heo Saeed Rashidi Srinivas Sridharan Sudarshan Srinivasan T. Krishna 44 47 0 24 Mar 2023
Auto-Parallelizing Large Models with Rhino: A Systematic Approach on Production AI Platform Shiwei Zhang Lansong Diao Siyu Wang Zongyan Cao Yiliang Gu Chang Si Ziji Shi Zhen Zheng Chuan Wu W. Lin AI4CE 47 4 0 16 Feb 2023
Ridgeline: A 2D Roofline Model for Distributed Systems Fabio Checconi Jesmin Jahan Tithi Fabrizio Petrini 12 5 0 03 Sep 2022
Impact of RoCE Congestion Control Policies on Distributed Training of DNNs Tarannum Khan Saeed Rashidi Srinivas Sridharan Pallavi Shurpali Aditya Akella T. Krishna OOD 57 11 0 22 Jul 2022
TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory H. Maruf Hao Wang A. Dhanotia Johannes Weiner Niket Agarwal Pallab Bhattacharya Chris Petersen Mosharaf Chowdhury Shobhit O. Kanaujia Prakash Chauhan 39 190 0 06 Jun 2022
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model Shaden Smith M. Patwary Brandon Norick P. LeGresley Samyam Rajbhandari ... Mohammad Shoeybi Yuxiong He Michael Houston Saurabh Tiwary Bryan Catanzaro MoE 145 740 0 28 Jan 2022
LaMDA: Language Models for Dialog Applications R. Thoppilan Daniel De Freitas Jamie Hall Noam M. Shazeer Apoorv Kulshreshtha ... Blaise Aguera-Arcas Claire Cui M. Croak Ed H. Chi Quoc Le ALM 129 1,593 0 20 Jan 2022
Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models Saeed Rashidi William Won Sudarshan Srinivasan Srinivas Sridharan T. Krishna GNN 47 31 0 09 Oct 2021
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining Junyang Lin An Yang Jinze Bai Chang Zhou Le Jiang ... Jie Zhang Yong Li Wei Lin Jingren Zhou Hongxia Yang MoE 113 44 0 08 Oct 2021
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning Samyam Rajbhandari Olatunji Ruwase Jeff Rasley Shaden Smith Yuxiong He GNN 83 383 0 16 Apr 2021
Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models Dheevatsa Mudigere Y. Hao Jianyu Huang Zhihao Jia Andrew Tulloch ... Ajit Mathews Lin Qiao M. Smelyanskiy Bill Jia Vijay Rao 75 152 0 12 Apr 2021
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan Mohammad Shoeybi Jared Casper P. LeGresley M. Patwary ... Prethvi Kashinkunti J. Bernauer Bryan Catanzaro Amar Phanishayee Matei A. Zaharia MoE 105 688 0 09 Apr 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 244 428 0 18 Jan 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity W. Fedus Barret Zoph Noam M. Shazeer MoE 85 2,181 0 11 Jan 2021
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale Bilge Acun Matthew Murphy Xiaodong Wang Jade Nie Carole-Jean Wu K. Hazelwood 68 112 0 11 Nov 2020
Hierarchical Roofline Performance Analysis for Deep Learning Applications Charlene Yang Yunsong Wang S. Farrell Thorsten Kurth Samuel Williams 44 15 0 11 Sep 2020
Time-Based Roofline for Deep Learning Performance Analysis Yunsong Wang Charlene Yang S. Farrell Yan Zhang Thorsten Kurth Samuel Williams 54 17 0 09 Sep 2020
8 Steps to 3.7 TFLOP/s on NVIDIA V100 GPU: Roofline Analysis and Other Tricks Charlene Yang 40 10 0 26 Aug 2020
Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms Saeed Rashidi Matthew Denton Srinivas Sridharan Sudarshan Srinivasan Amoghavarsha Suresh Jade Nie T. Krishna 61 47 0 30 Jun 2020
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT Prakhar Ganesh Yao Chen Xin Lou Mohammad Ali Khan Yifan Yang Hassan Sajjad Preslav Nakov Deming Chen Marianne Winslett AI4CE 93 201 0 27 Feb 2020
Characterizing Deep Learning Training Workloads on Alibaba-PAI Mengdi Wang Chen Meng Guoping Long Chuan Wu Jun Yang Wei Lin Yangqing Jia 55 54 0 14 Oct 2019
Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization Paras Jain Ajay Jain Aniruddha Nrusimha A. Gholami Pieter Abbeel Kurt Keutzer Ion Stoica Joseph E. Gonzalez 56 193 0 07 Oct 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Mohammad Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 326 1,899 0 17 Sep 2019
The Architectural Implications of Facebook's DNN-based Personalized Recommendation Udit Gupta Carole-Jean Wu Xiaodong Wang Maxim Naumov Brandon Reagen ... Andrey Malevich Dheevatsa Mudigere M. Smelyanskiy Liang Xiong Xuan Zhang GNN 73 290 0 06 Jun 2019
Deep Learning Recommendation Model for Personalization and Recommendation Systems Maxim Naumov Dheevatsa Mudigere Hao-Jun Michael Shi Jianyu Huang Narayanan Sundaraman ... Wenlin Chen Vijay Rao Bill Jia Liang Xiong M. Smelyanskiy 88 733 0 31 May 2019
Performance Analysis of Deep Learning Workloads on Leading-edge Systems Yihui Ren Shinjae Yoo A. Hoisie ELM 23 22 0 21 May 2019
Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads Myeongjae Jeon Shivaram Venkataraman Amar Phanishayee Junjie Qian Wencong Xiao Fan Yang GNN 65 353 0 17 Jan 2019
An Empirical Model of Large-Batch Training Sam McCandlish Jared Kaplan Dario Amodei OpenAI Dota Team 65 277 0 14 Dec 2018
Mesh-TensorFlow: Deep Learning for Supercomputers Noam M. Shazeer Youlong Cheng Niki Parmar Dustin Tran Ashish Vaswani ... HyoukJoong Lee O. Milenkovic C. Young Ryan Sepassi Blake Hechtman GNN MoE AI4CE 84 389 0 05 Nov 2018
Deep Interest Evolution Network for Click-Through Rate Prediction Guorui Zhou Na Mou Ying Fan Qi Pi Weijie Bian Chang Zhou Xiaoqiang Zhu Kun Gai 84 1,062 0 11 Sep 2018
Beyond Data and Model Parallelism for Deep Neural Networks Zhihao Jia Matei A. Zaharia A. Aiken GNN AI4CE 56 505 0 14 Jul 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 687 131,526 0 12 Jun 2017
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper ... Chong-Jun Wang Bo Xiao Dani Yogatama J. Zhan Zhenyao Zhu 120 2,972 0 08 Dec 2015