On the Powerball Method

We propose a new method to accelerate the convergence of optimization algorithms. This method adds a power coefficient $\gamma\in(0,1)$ to the gradient during optimization. We call this the Powerball method after the well-known Heavy-ball method \cite{heavyball}. We prove that the Powerball method can achieve $\epsilon$ accuracy for strongly convex functions by using $O\left((1-\gamma)^{-1}\epsilon^{\gamma-1}\right)$ iterations. We also demonstrate that the Powerball method provides a $10$-fold speed up of the convergence of both gradient descent and L-BFGS on multiple real datasets.