The Binary Garrote

In this paper, I present a new model and solution method for sparse regression. The model introduces binary selector variables $s_i$ for the features $i$ in a way that is similar to Breiman's Garrote model. I refer to this method as the binary Garrote (BG). The posterior probability for $s_i$ is computed in the variational approximation. The BG is compared numerically with the Lasso method and with ridge regression. Numerical results on synthetic data show that the BG yields more accurate predictions and more accurately reconstructs the true model than the other methods. The naive implementation of the BG requires the inversion of a modified covariance matrix which scales cubic in the number of features. We indicate how for sparse problem the solution can be computed linear in the number of features.