On Nonconvex Decentralized Gradient Descent

Consensus optimization has received considerable attention in recent years. A number of decentralized algorithms have been proposed for convex consensus optimization. However, on consensus optimization with \emph{nonconvex} objective functions, our understanding to the behavior of these algorithms is limited. When we lose convexity, we cannot hope for obtaining globally optimal solutions (though we still do sometimes). Somewhat surprisingly, we retain most other properties from the convex setting for the decentralized consensus algorithms DGD and Prox-DGD, such as convergence to a (neighborhood of) consensus stationary solution and the rates of convergence when diminishing (constant) step sizes are used. It is worth noting that the Prox-DGD algorithm can handle nonconvex nonsmooth functions assume that their proximal operators can be computed. To establish some of these properties, the existing proofs from the convex setting need changes, and some results require a completely different line of analysis.