Scalable MCMC for Bayes Shrinkage Priors

Gaussian scale mixture priors are frequently employed in Bayesian analysis of high-dimensional models, and a theoretical literature exists showing optimal risk properties of several members of this family in $p \gg n$ settings when the truth is sparse. However, while implementations of frequentist methods such as the Lasso can scale to dimension in the hundreds of thousands, corresponding Bayesian methods that use MCMC for computation are often limited to problems at least an order of magnitude smaller. This is in large part due to convergence toward unity of the spectral gap of the associated Markov kernel as the dimension grows. Here we propose an MCMC algorithm for computation in these models that combines blocked Gibbs, Metropolis-Hastings, and slice sampling. Our algorithm has computational cost per step comparable to the best existing alternatives, but superior convergence properties, giving effective sample sizes of 50 to 100 fold larger for identical computation time. Moreover, the convergence rate of our algorithm deteriorates much more slowly than alternatives as the dimension grows. We illustrate the scalability of the algorithm in simulations with up to 20,000 predictors.