Horovod: fast and easy distributed deep learning in TensorFlow

15 February 2018

Papers citing "Horovod: fast and easy distributed deep learning in TensorFlow"

50 / 174 papers shown

Title
Prediction-Assisted Online Distributed Deep Learning Workload Scheduling in GPU Clusters Ziyue Luo Jia-Wei Liu Myungjin Lee Ness B. Shroff 44 0 0 09 Jan 2025
Training Through Failure: Effects of Data Consistency in Parallel Machine Learning Training Ray Cao Sherry Luo Steve Gan Sujeeth Jinesh 23 0 0 08 Jun 2024
I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey Noah Lewis J. L. Bez Suren Byna 62 0 0 16 Apr 2024
On the Efficiency of Privacy Attacks in Federated Learning Nawrin Tabassum Ka-Ho Chow Xuyu Wang Wenbin Zhang Yanzhao Wu FedML 42 1 0 15 Apr 2024
On the Burstiness of Distributed Machine Learning Traffic Natchanon Luangsomboon Fahimeh Fazel Jorg Liebeherr A. Sobhani Shichao Guan Xingjun Chu 38 1 0 30 Dec 2023
Moirai: Towards Optimal Placement for Distributed Inference on Heterogeneous Devices Beibei Zhang Hongwei Zhu Feng Gao Zhihui Yang Xiaoyang Sean Wang 29 1 0 07 Dec 2023
Eva: A General Vectorized Approximation Framework for Second-order Optimization Lin Zhang Shaoshuai Shi Bo Li 33 1 0 04 Aug 2023
Multi-GPU Approach for Training of Graph ML Models on large CFD Meshes Sebastian Strönisch Maximilian Sander A. Knüpfer M. Meyer AI4CE 30 8 0 25 Jul 2023
Robust Fully-Asynchronous Methods for Distributed Training over General Architecture Zehan Zhu Ye Tian Yan Huang Jinming Xu Shibo He OOD 32 2 0 21 Jul 2023
A Survey From Distributed Machine Learning to Distributed Deep Learning Mohammad Dehghani Zahra Yazdanparast 31 0 0 11 Jul 2023
Automated Tensor Model Parallelism with Overlapped Communication for Efficient Foundation Model Training Shengwei Li Zhiquan Lai Yanqi Hao Weijie Liu Ke-shi Ge Xiaoge Deng Dongsheng Li KaiCheng Lu 21 10 0 25 May 2023
A Survey on Class Imbalance in Federated Learning Jing Zhang Chuanwen Li Jianzgong Qi Jiayuan He FedML 47 13 0 21 Mar 2023
Cloudless-Training: A Framework to Improve Efficiency of Geo-Distributed ML Training W. Tan Xiao Shi Cunchi Lv Xiaofang Zhao FedML 36 1 0 09 Mar 2023
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT Yihan Cao Siyu Li Yixin Liu Zhiling Yan Yutong Dai Philip S. Yu Lichao Sun 38 509 0 07 Mar 2023
Learning Electron Bunch Distribution along a FEL Beamline by Normalising Flows Anna Willmann J. C. Cabadağ Yen-Yu Chang R. Pausch Amin Ghaith A. Debus A. Irman Michael Bussmann U. Schramm Nico Hoffmann 11 0 0 27 Feb 2023
FLINT: A Platform for Federated Learning Integration Ewen N. Wang Ajaykumar Kannan Yuefeng Liang Boyi Chen Mosharaf Chowdhury 40 24 0 24 Feb 2023
h-analysis and data-parallel physics-informed neural networks Paul Escapil-Inchauspé G. A. Ruz PINN AI4CE 31 2 0 17 Feb 2023
Auto-Parallelizing Large Models with Rhino: A Systematic Approach on Production AI Platform Shiwei Zhang Lansong Diao Siyu Wang Zongyan Cao Yiliang Gu Chang Si Ziji Shi Zhen Zheng Chuan Wu W. Lin AI4CE 32 4 0 16 Feb 2023
Expediting Distributed DNN Training with Device Topology-Aware Graph Deployment Shiwei Zhang Xiaodong Yi Lansong Diao Chuan Wu Siyu Wang W. Lin GNN 22 5 0 13 Feb 2023
Landscape of High-performance Python to Develop Data Science and Machine Learning Applications Oscar Castro P. Bruneau Jean-Sébastien Sottet D. Torregrossa 29 7 0 07 Feb 2023
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models Yuliang Liu Shenggui Li Jiarui Fang Yan Shao Boyuan Yao Yang You OffRL 27 7 0 06 Feb 2023
Systems for Parallel and Distributed Large-Model Deep Learning Training Kabir Nagrecha GNN VLM MoE 34 7 0 06 Jan 2023
Does compressing activations help model parallel training? S. Bian Dacheng Li Hongyi Wang Eric P. Xing Shivaram Venkataraman 37 5 0 06 Jan 2023
Containerisation for High Performance Computing Systems: Survey and Prospects Naweiluo Zhou Huan Zhou Dennis Hoppe 38 25 0 16 Dec 2022
Multiscale Graph Neural Networks for Protein Residue Contact Map Prediction Kuang Liu R. Kalia Xinlian Liu A. Nakano K. Nomura P. Vashishta R. Zamora-Resendiz 25 2 0 02 Dec 2022
Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox Qiyue Yin Tongtong Yu S. Shen Jun Yang Meijing Zhao Kaiqi Huang Bin Liang Liangsheng Wang OffRL 33 13 0 01 Dec 2022
Aspects of scaling and scalability for flow-based sampling of lattice QCD Ryan Abbott M. S. Albergo Aleksandar Botev D. Boyda Kyle Cranmer ... Ali Razavi Danilo Jimenez Rezende F. Romero-López P. Shanahan Julian M. Urban 37 33 0 14 Nov 2022
A Deep Double Ritz Method (D $^2$ RM) for solving Partial Differential Equations using Neural Networks C. Uriarte David Pardo I. Muga J. Muñoz‐Matute 49 18 0 07 Nov 2022
SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogates Baixi Sun Xiaodong Yu Chengming Zhang Jiannan Tian Sian Jin K. Iskra Tao Zhou Tekin Bicer Pete Beckman Dingwen Tao 32 1 0 01 Nov 2022
Management of Machine Learning Lifecycle Artifacts: A Survey Marius Schlegel K. Sattler 30 35 0 21 Oct 2022
AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness Dacheng Li Hongyi Wang Eric P. Xing Haotong Zhang MoE 22 21 0 13 Oct 2022
Cloud Classification with Unsupervised Deep Learning Takuya Kurihana Ian Foster Rebecca Willett S. Jenkins Kathryn Koenig Ruby Werman Ricardo Barros Lourenço Casper Neo Elisabeth Moyer 24 8 0 30 Sep 2022
Optimizing DNN Compilation for Distributed Training with Joint OP and Tensor Fusion Xiaodong Yi Shiwei Zhang Lansong Diao Chuan Wu Zhen Zheng Shiqing Fan Siyu Wang Jun Yang W. Lin 46 4 0 26 Sep 2022
Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment Daegun Yoon Sangyoon Oh 26 0 0 18 Sep 2022
Neural Nets with a Newton Conjugate Gradient Method on Multiple GPUs Severin Reiz T. Neckel H. Bungartz ODL 33 1 0 03 Aug 2022
Towards Efficient Communications in Federated Learning: A Contemporary Survey Zihao Zhao Yuzhu Mao Yang Liu Linqi Song Ouyang Ye Xinlei Chen Wenbo Ding FedML 66 60 0 02 Aug 2022
Large-scale Knowledge Distillation with Elastic Heterogeneous Computing Resources Ji Liu Daxiang Dong Xi Wang An Qin Xingjian Li P. Valduriez Dejing Dou Dianhai Yu 36 6 0 14 Jul 2022
Emerging Patterns in the Continuum Representation of Protein-Lipid Fingerprints Konstantia Georgouli Helgi I. Ingólfsson Fikret Aydin Mark Heimann F. Lightstone P. Bremer H. Bhatia 13 0 0 09 Jul 2022
High-Performance GPU-to-CPU Transpilation and Optimization via High-Level Parallel Constructs William S. Moses Ivan R. Ivanov Jens Domke Toshio Endo J. Doerfert O. Zinenko 14 15 0 01 Jul 2022
Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning Lin Zhang Shaoshuai Shi Wei Wang Bo Li 38 10 0 30 Jun 2022
GMI-DRL: Empowering Multi-GPU Deep Reinforcement Learning with GPU Spatial Multiplexing Yuke Wang Boyuan Feng Ziyi Wang Tong Geng Ang Li Yufei Ding AI4CE 49 0 0 16 Jun 2022
Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models Zhiquan Lai Shengwei Li Xudong Tang Ke-shi Ge Weijie Liu Yabo Duan Linbo Qiao Dongsheng Li 35 41 0 10 Jun 2022
Tutel: Adaptive Mixture-of-Experts at Scale Changho Hwang Wei Cui Yifan Xiong Ziyue Yang Ze Liu ... Joe Chau Peng Cheng Fan Yang Mao Yang Y. Xiong MoE 118 112 0 07 Jun 2022
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees Jue Wang Binhang Yuan Luka Rimanic Yongjun He Tri Dao Beidi Chen Christopher Ré Ce Zhang AI4CE 31 11 0 02 Jun 2022
Decentralized Training of Foundation Models in Heterogeneous Environments Binhang Yuan Yongjun He Jared Davis Tianyi Zhang Tri Dao Beidi Chen Percy Liang Christopher Ré Ce Zhang 38 91 0 02 Jun 2022
Scalable algorithms for physics-informed neural and graph networks K. Shukla Mengjia Xu N. Trask George Karniadakis PINN AI4CE 75 40 0 16 May 2022
SMLT: A Serverless Framework for Scalable and Adaptive Machine Learning Design and Training Ahsan Ali Syed Zawad Paarijaat Aditya Istemi Ekin Akkus Ruichuan Chen Feng Yan 34 9 0 04 May 2022
MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud Zhen Zhang Shuai Zheng Yida Wang Justin Chiu George Karypis Trishul Chilimbi Mu Li Xin Jin 28 39 0 30 Apr 2022
Distributed intelligence on the Edge-to-Cloud Continuum: A systematic literature review Daniel Rosendo Alexandru Costan P. Valduriez Gabriel Antoniu 19 80 0 29 Apr 2022
Analysing the Influence of Attack Configurations on the Reconstruction of Medical Images in Federated Learning M. Dahlgaard Morten Wehlast Jorgensen N. Fuglsang Hiba Nassar FedML AAML 38 2 0 25 Apr 2022