Compressed Communication for Distributed Training: Adaptive Methods and System

17 May 2021

Papers citing "Compressed Communication for Distributed Training: Adaptive Methods and System"

27 / 27 papers shown

Title
Efficient-Adam: Communication-Efficient Distributed Adam Congliang Chen Li Shen Wei Liu Zhi-Quan Luo 56 20 0 28 May 2022
An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems A. Abdelmoniem Ahmed Elzanaty Mohamed-Slim Alouini Marco Canini 106 76 0 26 Jan 2021
APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm Hanlin Tang Shaoduo Gan Samyam Rajbhandari Xiangru Lian Ji Liu Yuxiong He Ce Zhang 61 8 0 26 Aug 2020
CSER: Communication-efficient SGD with Error Reset Cong Xie Shuai Zheng Oluwasanmi Koyejo Indranil Gupta Mu Li Yanghua Peng 71 41 0 26 Jul 2020
Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes Shuai Zheng Yanghua Peng Sheng Zha Mu Li ODL 53 21 0 24 Jun 2020
A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning Samuel Horváth Peter Richtárik 68 61 0 19 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 826 42,332 0 28 May 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 608 4,893 0 23 Jan 2020
GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing Jian Guo He He Tong He Leonard Lausen Mu Li ... Hang Zhang Zhi-Li Zhang Zhongyue Zhang Shuai Zheng Yi Zhu VLM BDL 74 197 0 09 Jul 2019
Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations Debraj Basu Deepesh Data C. Karakuş Suhas Diggavi MQ 60 406 0 06 Jun 2019
Natural Compression for Distributed Deep Learning Samuel Horváth Chen-Yu Ho L. Horvath Atal Narayan Sahu Marco Canini Peter Richtárik 68 152 0 27 May 2019
Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback Shuai Zheng Ziyue Huang James T. Kwok 54 115 0 27 May 2019
On the Convergence of Adam and Beyond Sashank J. Reddi Satyen Kale Surinder Kumar 99 2,504 0 19 Apr 2019
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes Yang You Jing Li Sashank J. Reddi Jonathan Hseu Sanjiv Kumar Srinadh Bhojanapalli Xiaodan Song J. Demmel Kurt Keutzer Cho-Jui Hsieh ODL 251 999 0 01 Apr 2019
Error Feedback Fixes SignSGD and other Gradient Compression Schemes Sai Praneeth Karimireddy Quentin Rebjock Sebastian U. Stich Martin Jaggi 66 502 0 28 Jan 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,114 0 11 Oct 2018
Sparsified SGD with Memory Sebastian U. Stich Jean-Baptiste Cordonnier Martin Jaggi 74 754 0 20 Sep 2018
Horovod: fast and easy distributed deep learning in TensorFlow Alexander Sergeev Mike Del Balso 100 1,221 0 15 Feb 2018
signSGD: Compressed Optimisation for Non-Convex Problems Jeremy Bernstein Yu Wang Kamyar Azizzadenesheli Anima Anandkumar FedML ODL 101 1,050 0 13 Feb 2018
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training Chengyue Wu Song Han Huizi Mao Yu Wang W. Dally 138 1,407 0 05 Dec 2017
Gradient Sparsification for Communication-Efficient Distributed Optimization Jianqiao Wangni Jialei Wang Ji Liu Tong Zhang 90 528 0 26 Oct 2017
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning W. Wen Cong Xu Feng Yan Chunpeng Wu Yandan Wang Yiran Chen Hai Helen Li 140 990 0 22 May 2017
Sparse Communication for Distributed Gradient Descent Alham Fikri Aji Kenneth Heafield 66 741 0 17 Apr 2017
Identity Mappings in Deep Residual Networks Kaiming He Xinming Zhang Shaoqing Ren Jian Sun 354 10,192 0 16 Mar 2016
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 1.9K 150,260 0 22 Dec 2014
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 1.7K 100,479 0 04 Sep 2014
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,590 0 01 Sep 2014