TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism

16 April 2020

Papers citing "TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism"

22 / 22 papers shown

Title
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training Jared Fernandez Luca Wehrstedt Leonid Shamis Mostafa Elhoushi Kalyan Saladi Yonatan Bisk Emma Strubell Jacob Kahn 488 3 0 20 Nov 2024
UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming Hao Lin Ke Wu Jie Li Jun Yu Li Wu-Jun Li 61 2 0 31 Jul 2023
Mesh-TensorFlow: Deep Learning for Supercomputers Noam M. Shazeer Youlong Cheng Niki Parmar Dustin Tran Ashish Vaswani ... HyoukJoong Lee O. Milenkovic C. Young Ryan Sepassi Blake Hechtman GNN MoE AI4CE 84 389 0 05 Nov 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 94,891 0 11 Oct 2018
Supporting Very Large Models using Automatic Dataflow Graph Partitioning Minjie Wang Chien-chin Huang Jinyang Li 91 155 0 24 Jul 2018
Beyond Data and Model Parallelism for Deep Neural Networks Zhihao Jia Matei A. Zaharia A. Aiken GNN AI4CE 64 504 0 14 Jul 2018
Horovod: fast and easy distributed deep learning in TensorFlow Alexander Sergeev Mike Del Balso 100 1,220 0 15 Feb 2018
Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks Zhihao Jia Sina Lin C. Qi A. Aiken 59 117 0 14 Feb 2018
Regularized Evolution for Image Classifier Architecture Search Esteban Real A. Aggarwal Yanping Huang Quoc V. Le 160 3,031 0 05 Feb 2018
Device Placement Optimization with Reinforcement Learning Azalia Mirhoseini Hieu H. Pham Quoc V. Le Benoit Steiner Rasmus Larsen Yuefeng Zhou Naveen Kumar Mohammad Norouzi Samy Bengio J. Dean 85 440 0 13 Jun 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 704 131,652 0 12 Jun 2017
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Zhiwen Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 903 6,790 0 26 Sep 2016
Memory-Efficient Backpropagation Through Time A. Gruslys Rémi Munos Ivo Danihelka Marc Lanctot Alex Graves 63 229 0 10 Jun 2016
TensorFlow: A system for large-scale machine learning Martín Abadi P. Barham Jianmin Chen Zhiwen Chen Andy Davis ... Vijay Vasudevan Pete Warden Martin Wicke Yuan Yu Xiaoqiang Zhang GNN AI4CE 433 18,361 0 27 May 2016
Wide Residual Networks Sergey Zagoruyko N. Komodakis 349 7,985 0 23 May 2016
Training Deep Nets with Sublinear Memory Cost Tianqi Chen Bing Xu Chiyuan Zhang Carlos Guestrin 103 1,167 0 21 Apr 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,020 0 10 Dec 2015
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems Tianqi Chen Mu Li Yutian Li Min Lin Naiyan Wang Minjie Wang Tianjun Xiao Bing Xu Chiyuan Zhang Zheng Zhang 196 2,247 0 03 Dec 2015
Rethinking the Inception Architecture for Computer Vision Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Z. Wojna 3DV BDL 883 27,373 0 02 Dec 2015
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 1.6K 100,386 0 04 Sep 2014
Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio AIMat 560 27,311 0 01 Sep 2014
One weird trick for parallelizing convolutional neural networks A. Krizhevsky GNN 90 1,303 0 23 Apr 2014