The Variational Garrote

In this paper, I present a new solution method for sparse regression using L0 regularization. The model introduces a sparseness mechanism in the likelihood, instead of in the prior, as is done in the spike and slab model. The posterior probability is computed in the variational approximation. The variational parameters appear in the approximate model in a way that is similar to Breiman's Garrote model. I refer to this method as the variational Garrote (VG). The VG is compared numerically with the Lasso method and with ridge regression. Numerical results on synthetic data show that the VG yields more accurate predictions and more accurately reconstructs the true model than the other methods. The naive implementation of the VG scales cubic with the number of features. By introducing Lagrange multipliers we obtain a dual formulation of the problem that scales cubic in the number of samples, but close to linear in the number of features.