DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression

15 May 2019

Tong Zhang

Papers citing "DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression"

38 / 38 papers shown

Title
Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression Yutong He Xinmeng Huang Yiming Chen W. Yin Kun Yuan 59 7 0 12 May 2023
Error Feedback Fixes SignSGD and other Gradient Compression Schemes Sai Praneeth Karimireddy Quentin Rebjock Sebastian U. Stich Martin Jaggi 54 502 0 28 Jan 2019
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training Youjie Li Hang Qiu Songze Li A. Avestimehr Nam Sung Kim Alex Schwing FedML 59 104 0 08 Nov 2018
Communication Efficient Parallel Algorithms for Optimization on Manifolds B. Saparbayeva M. Zhang Lizhen Lin 23 4 0 26 Oct 2018
signSGD with Majority Vote is Communication Efficient And Fault Tolerant Jeremy Bernstein Jiawei Zhao Kamyar Azizzadenesheli Anima Anandkumar FedML 53 46 0 11 Oct 2018
The Convergence of Sparsified Gradient Methods Dan Alistarh Torsten Hoefler M. Johansson Sarit Khirirat Nikola Konstantinov Cédric Renggli 165 493 0 27 Sep 2018
Sparsified SGD with Memory Sebastian U. Stich Jean-Baptiste Cordonnier Martin Jaggi 71 749 0 20 Sep 2018
COLA: Decentralized Linear Learning Lie He An Bian Martin Jaggi 80 120 0 13 Aug 2018
Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization Jiaxiang Wu Weidong Huang Junzhou Huang Tong Zhang 71 236 0 21 Jun 2018
ATOMO: Communication-efficient Learning via Atomic Sparsification Hongyi Wang Scott Sievert Zachary B. Charles Shengchao Liu S. Wright Dimitris Papailiopoulos 63 353 0 11 Jun 2018
Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms Xueru Zhang Mohammad Mahdi Khalili M. Liu FedML 88 90 0 06 Jun 2018
cpSGD: Communication-efficient and differentially-private distributed SGD Naman Agarwal A. Suresh Felix X. Yu Sanjiv Kumar H. B. McMahan FedML 118 490 0 27 May 2018
Towards More Efficient Stochastic Decentralized Learning: Faster Convergence and Sparse Communication Zebang Shen Aryan Mokhtari Tengfei Zhou P. Zhao Hui Qian 93 56 0 25 May 2018
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning Tianyi Chen G. Giannakis Tao Sun W. Yin 53 298 0 25 May 2018
Decoupled Parallel Backpropagation with Convergence Guarantee Zhouyuan Huo Bin Gu Qian Yang Heng-Chiao Huang 62 97 0 27 Apr 2018
D $^2$ : Decentralized Training over Decentralized Data Hanlin Tang Xiangru Lian Ming Yan Ce Zhang Ji Liu 31 350 0 19 Mar 2018
Communication Compression for Decentralized Training Hanlin Tang Shaoduo Gan Ce Zhang Tong Zhang Ji Liu 53 273 0 17 Mar 2018
SparCML: High-Performance Sparse Communication for Machine Learning Cédric Renggli Saleh Ashkboos Mehdi Aghagolzadeh Dan Alistarh Torsten Hoefler 58 126 0 22 Feb 2018
Distributed Stochastic Optimization via Adaptive SGD Ashok Cutkosky R. Busa-Fekete FedML 65 21 0 16 Feb 2018
signSGD: Compressed Optimisation for Non-Convex Problems Jeremy Bernstein Yu Wang Kamyar Azizzadenesheli Anima Anandkumar FedML ODL 87 1,042 0 13 Feb 2018
AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training Chia-Yu Chen Jungwook Choi D. Brand A. Agrawal Wei Zhang K. Gopalakrishnan ODL 49 174 0 07 Dec 2017
Gradient Sparsification for Communication-Efficient Distributed Optimization Jianqiao Wangni Jialei Wang Ji Liu Tong Zhang 74 525 0 26 Oct 2017
Asynchronous Decentralized Parallel Stochastic Gradient Descent Xiangru Lian Wei Zhang Ce Zhang Ji Liu ODL 46 500 0 18 Oct 2017
Adaptive Consensus ADMM for Distributed Optimization Zheng Xu Gavin Taylor Hao Li Mário A. T. Figueiredo Xiaoming Yuan Tom Goldstein 39 62 0 09 Jun 2017
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent Xiangru Lian Ce Zhang Huan Zhang Cho-Jui Hsieh Wei Zhang Ji Liu 50 1,227 0 25 May 2017
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning W. Wen Cong Xu Feng Yan Chunpeng Wu Yandan Wang Yiran Chen Hai Helen Li 140 987 0 22 May 2017
Communication-efficient Algorithms for Distributed Stochastic Principal Component Analysis Dan Garber Ohad Shamir Nathan Srebro 47 43 0 27 Feb 2017
Decentralized Consensus Optimization with Asynchrony and Delays Tianyu Wu Kun Yuan Qing Ling W. Yin Ali H. Sayed 34 10 0 01 Dec 2016
How to scale distributed deep learning? Peter H. Jin Qiaochu Yuan F. Iandola Kurt Keutzer 3DH 51 137 0 14 Nov 2016
Distributed Mean Estimation with Limited Communication A. Suresh Felix X. Yu Sanjiv Kumar H. B. McMahan FedML 97 364 0 02 Nov 2016
TensorFlow: A system for large-scale machine learning Martín Abadi P. Barham Jianmin Chen Zhiwen Chen Andy Davis ... Vijay Vasudevan Pete Warden Martin Wicke Yuan Yu Xiaoqiang Zhang GNN AI4CE 427 18,346 0 27 May 2016
Efficient Distributed Learning with Sparsity Jialei Wang Mladen Kolar Nathan Srebro Tong Zhang FedML 59 152 0 25 May 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 193,814 0 10 Dec 2015
Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization Xiangru Lian Yijun Huang Y. Li Ji Liu 135 499 0 27 Jun 2015
An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimization Hamid Reza Feyzmahdavian Arda Aytekin M. Johansson 51 117 0 18 May 2015
Distributed optimization over time-varying directed graphs A. Nedić Alexander Olshevsky 59 997 0 10 Mar 2013
HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent Feng Niu Benjamin Recht Christopher Ré Stephen J. Wright 191 2,273 0 28 Jun 2011
Distributed Delayed Stochastic Optimization Alekh Agarwal John C. Duchi 123 626 0 28 Apr 2011