Sparse Communication for Distributed Gradient Descent

We make distributed stochastic gradient descent faster by exchanging 99% sparse updates instead of dense updates. In data-parallel training, nodes pull updated values of the parameters from a sharded server, compute gradients, push their gradients to the server, and repeat. These push and pull updates strain the network. However, most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices. Even simple coordinate and value encoding achieves 50x reduction in bandwidth. Our experiment with a neural machine translation on 4 GPUs achieved a 22% speed boost without impacting BLEU score.