Supporting Very Large Models using Automatic Dataflow Graph Partitioning

24 July 2018

Papers citing "Supporting Very Large Models using Automatic Dataflow Graph Partitioning"

13 / 63 papers shown

Title
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models Zhuohan Li Siyuan Zhuang Shiyuan Guo Danyang Zhuo Hao Zhang D. Song Ion Stoica MoE 19 121 0 16 Feb 2021
Scaling Distributed Deep Learning Workloads beyond the Memory Capacity with KARMA M. Wahib Haoyu Zhang Truong Thao Nguyen Aleksandr Drozd Jens Domke Lingqi Zhang Ryousei Takano Satoshi Matsuoka OODD 34 23 0 26 Aug 2020
A Computational-Graph Partitioning Method for Training Memory-Constrained DNNs Fareed Qararyah M. Wahib Douga Dikbayir M. E. Belviranli D. Unat 28 8 0 19 Aug 2020
DAPPLE: A Pipelined Data Parallel Approach for Training Large Models Shiqing Fan Yi Rong Chen Meng Zongyan Cao Siyu Wang ... Jun Yang Lixue Xia Lansong Diao Xiaoyong Liu Wei Lin 21 232 0 02 Jul 2020
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism Jay H. Park Gyeongchan Yun Chang Yi N. T. Nguyen Seungmin Lee Jaesik Choi S. Noh Young-ri Choi MoE 25 128 0 28 May 2020
TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism Zhenkun Cai Kaihao Ma Xiao Yan Yidi Wu Yuzhen Huang James Cheng Teng Su F. Yu 19 42 0 16 Apr 2020
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Samyam Rajbhandari Jeff Rasley Olatunji Ruwase Yuxiong He ALM AI4CE 17 807 0 04 Oct 2019
Heterogeneity-Aware Asynchronous Decentralized Training Qinyi Luo Jiaao He Youwei Zhuo Xuehai Qian 19 8 0 17 Sep 2019
Training on the Edge: The why and the how Navjot Kukreja Alena Shilova Olivier Beaumont Jan Huckelheim N. Ferrier P. Hovland Gerard Gorman 16 33 0 13 Feb 2019
HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array Linghao Song Jiachen Mao Youwei Zhuo Xuehai Qian Hai Helen Li Yiran Chen 24 97 0 07 Jan 2019
Wireless Network Intelligence at the Edge Jihong Park S. Samarakoon M. Bennis Mérouane Debbah 21 518 0 07 Dec 2018
Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks Soojeong Kim Gyeong-In Yu Hojin Park Sungwoo Cho Eunji Jeong Hyeonmin Ha Sanha Lee Joo Seong Jeong Byung-Gon Chun 23 73 0 08 Aug 2018
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,746 0 26 Sep 2016