v1v2 (latest)

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

26 February 2018

Papers citing "Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis"

50 / 154 papers shown

Title
Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training Yijie Zheng Bangjun Xiao Lei Shi Xiaoyang Li Faming Wu Tianyu Li Xuefeng Xiao Yanzhe Zhang Yansen Wang Shouda Liu MLLM MoE 127 1 0 31 Mar 2025
Mixtera: A Data Plane for Foundation Model Training Maximilian Böther Xiaozhe Yao Tolga Kerimoglu Ana Klimovic Viktor Gsteiger Ana Klimovic MoE 180 0 0 27 Feb 2025
PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction Lei Guan Dongsheng Li Jiye Liang Wenjian Wang Wenjian Wang Xicheng Lu 127 1 0 01 Dec 2023
Federated Transfer Learning with Dynamic Gradient Aggregation Dimitrios Dimitriadis K. Kumatani R. Gmyr Yashesh Gaur Sefik Emre Eskimez FedML 96 15 0 06 Aug 2020
Communication optimization strategies for distributed deep neural network training: A survey Shuo Ouyang Dezun Dong Yemao Xu Liquan Xiao 110 12 0 06 Mar 2020
Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelism Nikoli Dryden N. Maruyama Tom Benson Tim Moon M. Snir B. Van Essen 69 49 0 15 Mar 2019
Large-Batch Training for LSTM and Beyond Yang You Jonathan Hseu Chris Ying J. Demmel Kurt Keutzer Cho-Jui Hsieh 54 91 0 24 Jan 2019
Image Classification at Supercomputer Scale Chris Ying Sameer Kumar Dehao Chen Tao Wang Youlong Cheng VLM 61 123 0 16 Nov 2018
Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash Hiroaki Mikami Hisahiro Suganuma Pongsakorn U-chupala Yoshiki Tanaka Yuichi Kageyama 67 77 0 13 Nov 2018
Measuring the Effects of Data Parallelism on Neural Network Training Christopher J. Shallue Jaehoon Lee J. Antognini J. Mamou J. Ketterling Yao Wang 100 409 0 08 Nov 2018
Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks Xiaodong Cui Wei Zhang Zoltán Tüske M. Picheny ODL 90 91 0 16 Oct 2018
Neural Architecture Optimization Renqian Luo Fei Tian Tao Qin Enhong Chen Tie-Yan Liu 3DV 102 658 0 22 Aug 2018
Large Scale Language Modeling: Converging on 40GB of Text in Four Hours Raul Puri Robert M. Kirby Nikolai Yakovenko Bryan Catanzaro 69 29 0 03 Aug 2018
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes Xianyan Jia Shutao Song W. He Yangzihao Wang Haidong Rong ... Li Yu Tiegang Chen Guangxiao Hu Shaoshuai Shi Xiaowen Chu 90 384 0 30 Jul 2018
DARTS: Differentiable Architecture Search Hanxiao Liu Karen Simonyan Yiming Yang 208 4,375 0 24 Jun 2018
Scaling Neural Machine Translation Myle Ott Sergey Edunov David Grangier Michael Auli AIMat 194 615 0 01 Jun 2018
GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent J. Daily Abhinav Vishnu Charles Siegel T. Warfel Vinay C. Amatya 52 95 0 15 Mar 2018
SparCML: High-Performance Sparse Communication for Machine Learning Cédric Renggli Saleh Ashkboos Mehdi Aghagolzadeh Dan Alistarh Torsten Hoefler 85 127 0 22 Feb 2018
Efficient Sparse-Winograd Convolutional Neural Networks Xingyu Liu Jeff Pool Song Han W. Dally 56 122 0 18 Feb 2018
Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions Nicolas Vasilache O. Zinenko Theodoros Theodoridis Priya Goyal Zach DeVito William S. Moses Sven Verdoolaege Andrew Adams Albert Cohen 88 436 0 13 Feb 2018
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Tianqi Chen T. Moreau Ziheng Jiang Lianmin Zheng Eddie Q. Yan ... Leyuan Wang Yuwei Hu Luis Ceze Carlos Guestrin Arvind Krishnamurthy 214 374 0 12 Feb 2018
Neural Architecture Search with Bayesian Optimisation and Optimal Transport Kirthevasan Kandasamy Willie Neiswanger J. Schneider Barnabás Póczós Eric Xing 108 609 0 11 Feb 2018
Efficient Neural Architecture Search via Parameter Sharing Hieu H. Pham M. Guan Barret Zoph Quoc V. Le J. Dean 117 2,773 0 09 Feb 2018
Regularized Evolution for Image Classifier Architecture Search Esteban Real A. Aggarwal Yanping Huang Quoc V. Le 187 3,039 0 05 Feb 2018
Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning F. Such Vashisht Madhavan Edoardo Conti Joel Lehman Kenneth O. Stanley Jeff Clune 118 695 0 18 Dec 2017
Integrated Model, Batch and Domain Parallelism in Training Neural Networks A. Gholami A. Azad Peter H. Jin Kurt Keutzer A. Buluç 81 84 0 12 Dec 2017
Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks Shankar Krishnan Ying Xiao Rif A. Saurous ODL 45 20 0 08 Dec 2017
AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training Chia-Yu Chen Jungwook Choi D. Brand A. Agrawal Wei Zhang K. Gopalakrishnan ODL 54 174 0 07 Dec 2017
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training Chengyue Wu Song Han Huizi Mao Yu Wang W. Dally 155 1,410 0 05 Dec 2017
Progressive Neural Architecture Search Chenxi Liu Barret Zoph Maxim Neumann Jonathon Shlens Wei Hua Li Li Li Fei-Fei Alan Yuille Jonathan Huang Kevin Patrick Murphy 117 1,997 0 02 Dec 2017
Population Based Training of Neural Networks Max Jaderberg Valentin Dalibard Simon Osindero Wojciech M. Czarnecki Jeff Donahue ... Tim Green Iain Dunning Karen Simonyan Chrisantha Fernando Koray Kavukcuoglu 96 744 0 27 Nov 2017
Simple And Efficient Architecture Search for Convolutional Neural Networks T. Elsken J. H. Metzen Frank Hutter 90 232 0 13 Nov 2017
Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks Urs Koster T. Webb Xin Eric Wang Marcel Nassar Arjun K. Bansal ... Luke Hornof A. Khosrowshahi Carey Kloss Ruby J. Pai N. Rao MQ 51 262 0 06 Nov 2017
Don't Decay the Learning Rate, Increase the Batch Size Samuel L. Smith Pieter-Jan Kindermans Chris Ying Quoc V. Le ODL 122 996 0 01 Nov 2017
Hierarchical Representations for Efficient Architecture Search Hanxiao Liu Karen Simonyan Oriol Vinyals Chrisantha Fernando Koray Kavukcuoglu 3DV 114 929 0 01 Nov 2017
Progressive Growing of GANs for Improved Quality, Stability, and Variation Tero Karras Timo Aila S. Laine J. Lehtinen GAN 184 7,380 0 27 Oct 2017
Meta Learning Shared Hierarchies Kevin Frans Jonathan Ho Xi Chen Pieter Abbeel John Schulman 68 355 0 26 Oct 2017
Asynchronous Decentralized Parallel Stochastic Gradient Descent Xiangru Lian Wei Zhang Ce Zhang Ji Liu ODL 58 500 0 18 Oct 2017
ImageNet Training in Minutes Yang You Zhao-jie Zhang Cho-Jui Hsieh J. Demmel Kurt Keutzer VLM LRM 109 57 0 14 Sep 2017
Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads Tian Li Jie Zhong Ji Liu Wentao Wu Ce Zhang 51 70 0 24 Aug 2017
Practical Block-wise Neural Network Architecture Generation Zhaobai Zhong Junjie Yan Wei Wu Jing Shao Cheng-Lin Liu 87 126 0 18 Aug 2017
SMASH: One-Shot Model Architecture Search through HyperNetworks Andrew Brock Theodore Lim J. Ritchie Nick Weston 169 765 0 17 Aug 2017
Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data Thorsten Kurth Jian Zhang N. Satish Ioannis Mitliagkas Evan Racah ... J. Deslippe Mikhail Shiryaev Srinivas Sridharan P. Prabhat Pradeep Dubey 68 83 0 17 Aug 2017
DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices Dawei Li Xiaolong Wang Deguang Kong 63 99 0 16 Aug 2017
Large Batch Training of Convolutional Networks Yang You Igor Gitman Boris Ginsburg ODL 157 853 0 13 Aug 2017
Scaling Deep Learning on GPU and Knights Landing clusters Yang You A. Buluç J. Demmel GNN 58 75 0 09 Aug 2017
Learning Transferable Architectures for Scalable Image Recognition Barret Zoph Vijay Vasudevan Jonathon Shlens Quoc V. Le 208 5,612 0 21 Jul 2017
Dual Path Networks Yunpeng Chen Jianan Li Huaxin Xiao Xiaojie Jin Shuicheng Yan Jiashi Feng 93 830 0 06 Jul 2017
One Model To Learn Them All Lukasz Kaiser Aidan Gomez Noam M. Shazeer Ashish Vaswani Niki Parmar Llion Jones Jakob Uszkoreit VLM ViT 82 334 0 16 Jun 2017
YellowFin and the Art of Momentum Tuning Jian Zhang Ioannis Mitliagkas ODL 91 108 0 12 Jun 2017