Non-asymptotic Superlinear Convergence of Standard Quasi-Newton Methods

In this paper, we study and prove the non-asymptotic superlinear convergence rate of the Broyden class of quasi-Newton methods including Davidon--Fletcher--Powell (DFP) method and Broyden--Fletcher--Goldfarb--Shanno (BFGS) method. The asymptotic superlinear convergence rate of these quasi-Newton methods has been extensively studied, but their explicit finite time local convergence rate is not fully investigated. In this paper, we provide a finite time (non-asymptotic) convergence analysis for BFGS and DFP methods under the assumptions that the objective function is strongly convex, its gradient is Lipschitz continuous, and its Hessian is Lipschitz continuous only in the direction of the optimal solution. We show that in a local neighborhood of the optimal solution, the iterates generated by both DFP and BFGS converge to the optimal solution at a superlinear rate of $(1/k)^{k/2}$, where $k$ is the number of iterations. We also prove the same local superlinear convergence rate in the case that the objective function is self-concordant. Numerical experiments on different objective functions confirm our explicit convergence rates. Our theoretical guarantee is one of the first results that provide a non-asymptotic superlinear convergence rate for DFP and BFGS quasi-Newton methods.