v1v2v3 (latest)

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

5 December 2017

Song Han

ArXiv (abs)PDF HTML Github (222★)

Papers citing "Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training"

50 / 625 papers shown

Title
Global Momentum Compression for Sparse Communication in Distributed Learning Chang-Wei Shi Shen-Yi Zhao Yin-Peng Xie Hao Gao Wu-Jun Li 80 1 0 30 May 2019
Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data Shicong Cen Huishuai Zhang Yuejie Chi Wei-neng Chen Tie-Yan Liu FedML 108 27 0 29 May 2019
Accelerated Sparsified SGD with Error Feedback Tomoya Murata Taiji Suzuki 53 2 0 29 May 2019
Natural Compression for Distributed Deep Learning Samuel Horváth Chen-Yu Ho L. Horvath Atal Narayan Sahu Marco Canini Peter Richtárik 93 151 0 27 May 2019
Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks Xiaoliang Dai Hongxu Yin N. Jha 83 32 0 27 May 2019
Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback Shuai Zheng Ziyue Huang James T. Kwok 56 115 0 27 May 2019
Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing Zhi Zhou Xu Chen En Li Liekang Zeng Ke Luo Junshan Zhang 111 1,450 0 24 May 2019
Accelerating DNN Training in Wireless Federated Edge Learning Systems Jinke Ren Guanding Yu Guangyao Ding FedML 85 175 0 23 May 2019
Decentralized Learning of Generative Adversarial Networks from Non-iid Data Ryo Yonetani Tomohiro Takahashi Atsushi Hashimoto Yoshitaka Ushiku 92 24 0 23 May 2019
Priority-based Parameter Propagation for Distributed DNN Training Anand Jayarajan Jinliang Wei Garth A. Gibson Alexandra Fedorova Gennady Pekhimenko AI4CE 62 182 0 10 May 2019
Realizing Petabyte Scale Acoustic Modeling S. Parthasarathi Nitin Sivakrishnan Pranav Ladkat N. Strom 60 11 0 24 Apr 2019
Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools R. Mayer Hans-Arno Jacobsen GNN 81 193 0 27 Mar 2019
Communication-efficient distributed SGD with Sketching Nikita Ivkin D. Rothchild Enayat Ullah Vladimir Braverman Ion Stoica R. Arora FedML 93 201 0 12 Mar 2019
Robust and Communication-Efficient Federated Learning from Non-IID Data Felix Sattler Simon Wiedemann K. Müller Wojciech Samek FedML 79 1,367 0 07 Mar 2019
Speeding up Deep Learning with Transient Servers Shijian Li R. Walls Lijie Xu Tian Guo 54 12 0 28 Feb 2019
Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training Chengjie Li Ruixuan Li Yining Qi Yuhua Li Pan Zhou Song Guo Keqin Li 88 16 0 21 Feb 2019
Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes Peng Sun Wansen Feng Ruobing Han Shengen Yan Yonggang Wen AI4CE 100 70 0 19 Feb 2019
Federated Machine Learning: Concept and Applications Qiang Yang Yang Liu Tianjian Chen Yongxin Tong FedML 95 2,352 0 13 Feb 2019
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication Anastasia Koloskova Sebastian U. Stich Martin Jaggi FedML 87 510 0 01 Feb 2019
Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM Hongxu Yin Guoyang Chen Yingmin Li Shuai Che Weifeng Zhang N. Jha 58 10 0 30 Jan 2019
Error Feedback Fixes SignSGD and other Gradient Compression Schemes Sai Praneeth Karimireddy Quentin Rebjock Sebastian U. Stich Martin Jaggi 113 503 0 28 Jan 2019
99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it Konstantin Mishchenko Filip Hanzely Peter Richtárik 66 13 0 27 Jan 2019
Information-Theoretic Understanding of Population Risk Improvement with Model Compression Yuheng Bu Weihao Gao Shaofeng Zou Venugopal V. Veeravalli MedIm 48 15 0 27 Jan 2019
A Distributed Synchronous SGD Algorithm with Global Top- $k$ Sparsification for Low Bandwidth Networks Shaoshuai Shi Qiang-qiang Wang Kaiyong Zhao Zhenheng Tang Yuxin Wang Xiang Huang Xiaowen Chu 79 137 0 14 Jan 2019
Quantized Epoch-SGD for Communication-Efficient Distributed Learning Shen-Yi Zhao Hao Gao Wu-Jun Li FedML 56 3 0 10 Jan 2019
Bandwidth Reduction using Importance Weighted Pruning on Ring AllReduce Zehua Cheng Zhenghua Xu 50 8 0 06 Jan 2019
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air Mohammad Mohammadi Amiri Deniz Gunduz 87 53 0 03 Jan 2019
Federated Learning via Over-the-Air Computation Kai Yang Tao Jiang Yuanming Shi Z. Ding FedML 102 882 0 31 Dec 2018
Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version) Guangxu Zhu Yong Wang Kaibin Huang FedML 103 650 0 30 Dec 2018
Stanza: Layer Separation for Distributed Training in Deep Learning Xiaorui Wu Hongao Xu Bo Li Y. Xiong MoE 54 9 0 27 Dec 2018
Distributed Learning with Sparse Communications by Identification Dmitry Grishchenko F. Iutzeler J. Malick Massih-Reza Amini 62 19 0 10 Dec 2018
No Peek: A Survey of private distributed deep learning Praneeth Vepakomma Tristan Swedish Ramesh Raskar O. Gupta Abhimanyu Dubey SyDa FedML 77 100 0 08 Dec 2018
Split learning for health: Distributed deep learning without sharing raw patient data Praneeth Vepakomma O. Gupta Tristan Swedish Ramesh Raskar FedML 125 713 0 03 Dec 2018
MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms Shaoshuai Shi Xiaowen Chu Bo Li FedML 91 90 0 27 Nov 2018
Hydra: A Peer to Peer Distributed Training & Data Collection Framework Vaibhav Mathur K. Chahal OffRL 35 2 0 24 Nov 2018
SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks Linnan Wang Wei Wu Junyu Zhang Hang Liu G. Bosilca Maurice Herlihy Rodrigo Fonseca GNN 47 5 0 21 Nov 2018
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training Youjie Li Hang Qiu Songze Li A. Avestimehr Nam Sung Kim Alex Schwing FedML 118 104 0 08 Nov 2018
GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training Timo C. Wunderlich Zhifeng Lin S. A. Aamir Andreas Grübl Youjie Li David Stöckel Alex Schwing M. Annavaram A. Avestimehr MQ 47 64 0 08 Nov 2018
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks K. Chahal Manraj Singh Grover Kuntal Dey 3DH OOD 88 54 0 28 Oct 2018
Batch Normalization Sampling Zhaodong Chen Lei Deng Guoqi Li Jiawei Sun Xing Hu Xin Ma Yuan Xie 41 0 0 25 Oct 2018
Computation Scheduling for Distributed Machine Learning with Straggling Workers Mohammad Mohammadi Amiri Deniz Gunduz FedML 82 3 0 23 Oct 2018
Collaborative Deep Learning Across Multiple Data Centers Kele Xu Haibo Mi Dawei Feng Huaimin Wang Chuan Chen Zibin Zheng Xu Lan FedML 344 18 0 16 Oct 2018
signSGD with Majority Vote is Communication Efficient And Fault Tolerant Jeremy Bernstein Jiawei Zhao Kamyar Azizzadenesheli Anima Anandkumar FedML 68 47 0 11 Oct 2018
Dynamic Sparse Graph for Efficient Deep Learning Liu Liu Lei Deng Xing Hu Maohua Zhu Guoqi Li Yufei Ding Yuan Xie GNN 90 42 0 01 Oct 2018
The Convergence of Sparsified Gradient Methods Dan Alistarh Torsten Hoefler M. Johansson Sarit Khirirat Nikola Konstantinov Cédric Renggli 181 493 0 27 Sep 2018
Sparsified SGD with Memory Sebastian U. Stich Jean-Baptiste Cordonnier Martin Jaggi 106 753 0 20 Sep 2018
Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform Chi-Chung Chen Chia-Lin Yang Hsiang-Yun Cheng 94 101 0 08 Sep 2018
Towards an Intelligent Edge: Wireless Communication Meets Machine Learning Guangxu Zhu Dongzhu Liu Yuqing Du Changsheng You Jun Zhang Kaibin Huang 85 507 0 02 Sep 2018
Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation Nikolay Bogoychev Marcin Junczys-Dowmunt Kenneth Heafield Alham Fikri Aji ODL 49 17 0 27 Aug 2018
Sparsity in Deep Neural Networks - An Empirical Investigation with TensorQuant D. Loroch Franz-Josef Pfreundt Norbert Wehn J. Keuper 46 5 0 27 Aug 2018