How to scale distributed deep learning?

14 November 2016

Papers citing "How to scale distributed deep learning?"

50 / 60 papers shown

Title
From Centralized to Decentralized Federated Learning: Theoretical Insights, Privacy Preservation, and Robustness Challenges Qiongxiu Li Wenrui Yu Yufei Xia Jun Pang FedML 60 1 0 10 Mar 2025
Distributed Sign Momentum with Local Steps for Training Transformers Shuhua Yu Ding Zhou Cong Xie An Xu Zhi-Li Zhang Xin Liu S. Kar 69 0 0 26 Nov 2024
Provable Privacy Advantages of Decentralized Federated Learning via Distributed Optimization Wenrui Yu Qiongxiu Li Milan Lopuhaä-Zwakenberg Mads Græsbøll Christensen Richard Heusdens FedML 30 3 0 12 Jul 2024
AI-coupled HPC Workflow Applications, Middleware and Performance Wes Brewer Ana Gainaru Frédéric Suter Feiyi Wang M. Emani S. Jha 30 10 0 20 Jun 2024
Adjacent Leader Decentralized Stochastic Gradient Descent Haoze He Jing Wang A. Choromańska 30 0 0 18 May 2024
Sequence-to-sequence models in peer-to-peer learning: A practical application Robert Šajina Ivo Ipšić 43 0 0 02 May 2024
Scale-Robust Timely Asynchronous Decentralized Learning Purbesh Mitra S. Ulukus 19 1 0 30 Apr 2024
Topology-Dependent Privacy Bound For Decentralized Federated Learning Qiongxiu Li Wenrui Yu Changlong Ji Richard Heusdens 24 3 0 13 Dec 2023
Tram-FL: Routing-based Model Training for Decentralized Federated Learning Kota Maejima Takayuki Nishio Asato Yamazaki Yuko Hara-Azumi 31 3 0 09 Aug 2023
The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey Jana Vatter R. Mayer Hans-Arno Jacobsen GNN AI4TS AI4CE 45 23 0 23 May 2023
Age-Aware Gossiping in Network Topologies Purbesh Mitra S. Ulukus 19 11 0 06 Apr 2023
Accuracy is not the only Metric that matters: Estimating the Energy Consumption of Deep Learning Models Johannes Getzner Bertrand Charpentier Stephan Günnemann 21 5 0 03 Apr 2023
Accelerating Parallel Stochastic Gradient Descent via Non-blocking Mini-batches Haoze He Parijat Dube 6 3 0 02 Nov 2022
RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning Haoze He Parijat Dube 15 1 0 02 Nov 2022
Asynchronous Fully-Decentralized SGD in the Cluster-Based Model Hagit Attiya N. Schiller FedML 20 0 0 22 Feb 2022
SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks Chaoyang He Emir Ceyani Keshav Balasubramanian M. Annavaram Salman Avestimehr FedML 25 50 0 04 Jun 2021
Partitioning sparse deep neural networks for scalable training and inference G. Demirci Hakan Ferhatosmanoglu 18 11 0 23 Apr 2021
EventGraD: Event-Triggered Communication in Parallel Machine Learning Soumyadip Ghosh B. Aquino V. Gupta FedML 21 8 0 12 Mar 2021
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale Bilge Acun Matthew Murphy Xiaodong Wang Jade Nie Carole-Jean Wu K. Hazelwood 23 109 0 11 Nov 2020
A Distributed Training Algorithm of Generative Adversarial Networks with Quantized Gradients Xiaojun Chen Shu Yang Liyan Shen Xuanrong Pang 9 4 0 26 Oct 2020
Decentralized Deep Learning using Momentum-Accelerated Consensus Aditya Balu Zhanhong Jiang Sin Yong Tan Chinmay Hedge Young M. Lee S. Sarkar FedML 24 22 0 21 Oct 2020
A Low Complexity Decentralized Neural Net with Centralized Equivalence using Layer-wise Learning Xinyue Liang Alireza M. Javid Mikael Skoglund S. Chatterjee FedML 17 4 0 29 Sep 2020
Asynchronous Distributed Optimization with Stochastic Delays Margalit Glasgow Mary Wootters 12 3 0 22 Sep 2020
Communication-efficient Decentralized Machine Learning over Heterogeneous Networks Pan Zhou Qian Lin Dumitrel Loghin Beng Chin Ooi Yuncheng Wu Hongfang Yu 13 35 0 12 Sep 2020
Synergy between Machine/Deep Learning and Software Engineering: How Far Are We? Simin Wang LiGuo Huang Jidong Ge Tengfei Zhang Haitao Feng Ming Li He Zhang Vincent Ng AI4CE 7 7 0 12 Aug 2020
Multi-Level Local SGD for Heterogeneous Hierarchical Networks Timothy Castiglia Anirban Das S. Patterson 20 13 0 27 Jul 2020
Contextualizing Enhances Gradient Based Meta Learning Evan Vogelbaum Rumen Dangovski L. Jing Marin Soljacic 34 3 0 17 Jul 2020
Convolutional Neural Network Training with Distributed K-FAC J. G. Pauloski Zhao Zhang Lei Huang Weijia Xu Ian Foster 15 30 0 01 Jul 2020
Machine Learning Systems for Intelligent Services in the IoT: A Survey Wiebke Toussaint Aaron Yi Ding LRM 27 0 0 29 May 2020
Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging Shigang Li Tal Ben-Nun Giorgi Nadiradze Salvatore Di Girolamo Nikoli Dryden Dan Alistarh Torsten Hoefler 21 14 0 30 Apr 2020
A Review of Privacy-preserving Federated Learning for the Internet-of-Things Christopher Briggs Zhong Fan Péter András 23 14 0 24 Apr 2020
The Future of Digital Health with Federated Learning Nicola Rieke Jonny Hancox Wenqi Li Fausto Milletari H. Roth ... Ronald M. Summers Andrew Trask Daguang Xu Maximilian Baust M. Jorge Cardoso OOD 174 1,707 0 18 Mar 2020
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey Zhenheng Tang S. Shi Wei Wang Bo-wen Li Xiaowen Chu 21 48 0 10 Mar 2020
Communication-Efficient Edge AI: Algorithms and Systems Yuanming Shi Kai Yang Tao Jiang Jun Zhang Khaled B. Letaief GNN 17 326 0 22 Feb 2020
A Random Gossip BMUF Process for Neural Language Modeling Yiheng Huang Jinchuan Tian Lei Han Guangsen Wang Xingcheng Song Dan Su Dong Yu 9 3 0 19 Sep 2019
Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations Shigang Li Tal Ben-Nun Salvatore Di Girolamo Dan Alistarh Torsten Hoefler 11 58 0 12 Aug 2019
Decentralized Bayesian Learning over Graphs Anusha Lalitha Xinghan Wang O. Kilinc Y. Lu T. Javidi F. Koushanfar FedML 28 25 0 24 May 2019
Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing Zhi Zhou Xu Chen En Li Liekang Zeng Ke Luo Junshan Zhang 21 1,420 0 24 May 2019
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression Hanlin Tang Xiangru Lian Chen Yu Tong Zhang Ji Liu 11 216 0 15 May 2019
Distributed Byzantine Tolerant Stochastic Gradient Descent in the Era of Big Data Richeng Jin Xiaofan He H. Dai FedML 11 13 0 27 Feb 2019
CodedReduce: A Fast and Robust Framework for Gradient Aggregation in Distributed Learning Amirhossein Reisizadeh Saurav Prakash Ramtin Pedarsani A. Avestimehr 20 23 0 06 Feb 2019
Bandwidth Reduction using Importance Weighted Pruning on Ring AllReduce Zehua Cheng Zhenghua Xu 14 8 0 06 Jan 2019
Wireless Network Intelligence at the Edge Jihong Park S. Samarakoon M. Bennis Mérouane Debbah 21 518 0 07 Dec 2018
Elastic Gossip: Distributing Neural Network Training Using Gossip-like Protocols Siddharth Pramod FedML 9 2 0 06 Dec 2018
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent Noah Golmant N. Vemuri Z. Yao Vladimir Feinberg A. Gholami Kai Rothauge Michael W. Mahoney Joseph E. Gonzalez 13 72 0 30 Nov 2018
Stochastic Gradient Push for Distributed Deep Learning Mahmoud Assran Nicolas Loizou Nicolas Ballas Michael G. Rabbat 22 343 0 27 Nov 2018
Hydra: A Peer to Peer Distributed Training & Data Collection Framework Vaibhav Mathur K. Chahal OffRL 11 2 0 24 Nov 2018
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks K. Chahal Manraj Singh Grover Kuntal Dey 3DH OOD 4 53 0 28 Oct 2018
Language Modeling at Scale Md. Mostofa Ali Patwary Milind Chabbi Heewoo Jun Jiaji Huang G. Diamos Kenneth Ward Church ALM 20 5 0 23 Oct 2018
Distributed Learning over Unreliable Networks Chen Yu Hanlin Tang Cédric Renggli S. Kassing Ankit Singla Dan Alistarh Ce Zhang Ji Liu OOD 19 60 0 17 Oct 2018