Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

5 December 2017

Song Han

Papers citing "Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training"

50 / 616 papers shown

Title
Priority-based Parameter Propagation for Distributed DNN Training Anand Jayarajan Jinliang Wei Garth A. Gibson Alexandra Fedorova Gennady Pekhimenko AI4CE 22 178 0 10 May 2019
Realizing Petabyte Scale Acoustic Modeling S. Parthasarathi Nitin Sivakrishnan Pranav Ladkat N. Strom 16 11 0 24 Apr 2019
Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools R. Mayer Hans-Arno Jacobsen GNN 27 186 0 27 Mar 2019
Communication-efficient distributed SGD with Sketching Nikita Ivkin D. Rothchild Enayat Ullah Vladimir Braverman Ion Stoica R. Arora FedML 14 198 0 12 Mar 2019
Robust and Communication-Efficient Federated Learning from Non-IID Data Felix Sattler Simon Wiedemann K. Müller Wojciech Samek FedML 24 1,335 0 07 Mar 2019
Speeding up Deep Learning with Transient Servers Shijian Li R. Walls Lijie Xu Tian Guo 30 12 0 28 Feb 2019
Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training Chengjie Li Ruixuan Li Yining Qi Yuhua Li Pan Zhou Song Guo Keqin Li 27 15 0 21 Feb 2019
Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes Peng Sun Wansen Feng Ruobing Han Shengen Yan Yonggang Wen AI4CE 24 70 0 19 Feb 2019
Federated Machine Learning: Concept and Applications Qiang Yang Yang Liu Tianjian Chen Yongxin Tong FedML 28 2,276 0 13 Feb 2019
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication Anastasia Koloskova Sebastian U. Stich Martin Jaggi FedML 25 503 0 01 Feb 2019
Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM Hongxu Yin Guoyang Chen Yingmin Li Shuai Che Weifeng Zhang N. Jha 36 10 0 30 Jan 2019
Error Feedback Fixes SignSGD and other Gradient Compression Schemes Sai Praneeth Karimireddy Quentin Rebjock Sebastian U. Stich Martin Jaggi 27 493 0 28 Jan 2019
99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it Konstantin Mishchenko Filip Hanzely Peter Richtárik 16 13 0 27 Jan 2019
Information-Theoretic Understanding of Population Risk Improvement with Model Compression Yuheng Bu Weihao Gao Shaofeng Zou V. Veeravalli MedIm 13 15 0 27 Jan 2019
A Distributed Synchronous SGD Algorithm with Global Top- $k$ Sparsification for Low Bandwidth Networks S. Shi Qiang-qiang Wang Kaiyong Zhao Zhenheng Tang Yuxin Wang Xiang Huang Xiaowen Chu 40 135 0 14 Jan 2019
Quantized Epoch-SGD for Communication-Efficient Distributed Learning Shen-Yi Zhao Hao Gao Wu-Jun Li FedML 22 3 0 10 Jan 2019
Bandwidth Reduction using Importance Weighted Pruning on Ring AllReduce Zehua Cheng Zhenghua Xu 19 8 0 06 Jan 2019
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air Mohammad Mohammadi Amiri Deniz Gunduz 30 53 0 03 Jan 2019
Federated Learning via Over-the-Air Computation Kai Yang Tao Jiang Yuanming Shi Z. Ding FedML 26 873 0 31 Dec 2018
Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version) Guangxu Zhu Yong Wang Kaibin Huang FedML 38 638 0 30 Dec 2018
Stanza: Layer Separation for Distributed Training in Deep Learning Xiaorui Wu Hongao Xu Bo Li Y. Xiong MoE 22 9 0 27 Dec 2018
Distributed Learning with Sparse Communications by Identification Dmitry Grishchenko F. Iutzeler J. Malick Massih-Reza Amini 19 19 0 10 Dec 2018
No Peek: A Survey of private distributed deep learning Praneeth Vepakomma Tristan Swedish Ramesh Raskar O. Gupta Abhimanyu Dubey SyDa FedML 30 100 0 08 Dec 2018
Split learning for health: Distributed deep learning without sharing raw patient data Praneeth Vepakomma O. Gupta Tristan Swedish Ramesh Raskar FedML 63 692 0 03 Dec 2018
MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms S. Shi Xiaowen Chu Bo Li FedML 24 89 0 27 Nov 2018
Hydra: A Peer to Peer Distributed Training & Data Collection Framework Vaibhav Mathur K. Chahal OffRL 24 2 0 24 Nov 2018
SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks Linnan Wang Wei Wu Junyu Zhang Hang Liu G. Bosilca Maurice Herlihy Rodrigo Fonseca GNN 18 5 0 21 Nov 2018
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training Youjie Li Hang Qiu Songze Li A. Avestimehr N. Kim Alex Schwing FedML 21 104 0 08 Nov 2018
GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training Timo C. Wunderlich Zhifeng Lin S. A. Aamir Andreas Grübl Youjie Li David Stöckel Alex Schwing M. Annavaram A. Avestimehr MQ 11 64 0 08 Nov 2018
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks K. Chahal Manraj Singh Grover Kuntal Dey 3DH OOD 6 53 0 28 Oct 2018
Batch Normalization Sampling Zhaodong Chen Lei Deng Guoqi Li Jiawei Sun Xing Hu Xin Ma Yuan Xie 21 0 0 25 Oct 2018
Computation Scheduling for Distributed Machine Learning with Straggling Workers Mohammad Mohammadi Amiri Deniz Gunduz FedML 16 2 0 23 Oct 2018
Collaborative Deep Learning Across Multiple Data Centers Kele Xu Haibo Mi Dawei Feng Huaimin Wang Chuan Chen Zibin Zheng Xu Lan FedML 134 18 0 16 Oct 2018
signSGD with Majority Vote is Communication Efficient And Fault Tolerant Jeremy Bernstein Jiawei Zhao Kamyar Azizzadenesheli Anima Anandkumar FedML 31 46 0 11 Oct 2018
Dynamic Sparse Graph for Efficient Deep Learning L. Liu Lei Deng Xing Hu Maohua Zhu Guoqi Li Yufei Ding Yuan Xie GNN 40 42 0 01 Oct 2018
The Convergence of Sparsified Gradient Methods Dan Alistarh Torsten Hoefler M. Johansson Sarit Khirirat Nikola Konstantinov Cédric Renggli 11 489 0 27 Sep 2018
Sparsified SGD with Memory Sebastian U. Stich Jean-Baptiste Cordonnier Martin Jaggi 41 740 0 20 Sep 2018
Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform Chi-Chung Chen Chia-Lin Yang Hsiang-Yun Cheng 33 100 0 08 Sep 2018
Towards an Intelligent Edge: Wireless Communication Meets Machine Learning Guangxu Zhu Dongzhu Liu Yuqing Du Changsheng You Jun Zhang Kaibin Huang 9 505 0 02 Sep 2018
Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation Nikolay Bogoychev Marcin Junczys-Dowmunt Kenneth Heafield Alham Fikri Aji ODL 27 17 0 27 Aug 2018
Sparsity in Deep Neural Networks - An Empirical Investigation with TensorQuant D. Loroch Franz-Josef Pfreundt Norbert Wehn J. Keuper 31 5 0 27 Aug 2018
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms Jianyu Wang Gauri Joshi 33 348 0 22 Aug 2018
Don't Use Large Mini-Batches, Use Local SGD Tao R. Lin Sebastian U. Stich Kumar Kshitij Patel Martin Jaggi 57 429 0 22 Aug 2018
A study on speech enhancement using exponent-only floating point quantized neural network (EOFP-QNN) Y. Hsu Yu-Chen Lin Szu-Wei Fu Yu Tsao Tei-Wei Kuo MQ 22 15 0 17 Aug 2018
RedSync : Reducing Synchronization Traffic for Distributed Deep Learning Jiarui Fang Haohuan Fu Guangwen Yang Cho-Jui Hsieh GNN 20 25 0 13 Aug 2018
Pushing the boundaries of parallel Deep Learning -- A practical approach Paolo Viviani M. Drocco Marco Aldinucci OOD 36 0 0 25 Jun 2018
Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization Jiaxiang Wu Weidong Huang Junzhou Huang Tong Zhang 24 235 0 21 Jun 2018
ATOMO: Communication-efficient Learning via Atomic Sparsification Hongyi Wang Scott Sievert Zachary B. Charles Shengchao Liu S. Wright Dimitris Papailiopoulos 20 351 0 11 Jun 2018
The Effect of Network Width on the Performance of Large-batch Training Lingjiao Chen Hongyi Wang Jinman Zhao Dimitris Papailiopoulos Paraschos Koutris 21 22 0 11 Jun 2018
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark Cody Coleman Daniel Kang Deepak Narayanan Luigi Nardi Tian Zhao Jian Zhang Peter Bailis K. Olukotun Christopher Ré Matei A. Zaharia 13 117 0 04 Jun 2018