TAC for Sparse Learning: Simultaneous Control of Algorithmic Complexity and Statistical Error

We propose a computational framework named tightening after contraction (TAC) to simultaneously control algorithmic and statistical error when fitting high dimensional models. TAC is a two-stage algorithmic implementation of the local linear approximation to a family of folded concave penalized quasi-likelihood. The first (contraction) stage solves a convex program with a crude precision tolerance to obtain an initial estimator, which is further refined in a second (tightening) stage by iteratively solving a sequence of convex programs with smaller precision tolerances. Theoretically, we establish a phase transition phenomenon: the first stage has a sublinear iteration complexity, while the second stage achieves an improved linear iteration complexity. Though this framework is completely algorithmic, it provides solutions with optimal statistical errors and controlled algorithmic complexity for a large family of nonconvex statistical optimization problems. The iteration effects on statistical errors are clearly demonstrated via a contraction property. Our theory relies on localized versions of sparse eigenvalue and restricted eigenvalue conditions, which allow us to analyze a large family of loss and penalty functions and provide the strongest guarantees under the weakest assumptions. Moreover, TAC requires much weaker minimal signal strength than other procedures. Thorough numerical results are provided to back up the obtained theory.