Variable-at-a-time Implementations of Metropolis-Hastings

It is common practice in Markov chain Monte Carlo to update a high-dimensional chain one variable (or sub-block of variables) at a time, rather than to conduct a single block update. While this modification can make the choice of proposal easier, the theoretical properties of the associated Markov chain have received limited attention. We present conditions under which the chain converges uniformly to its stationary distribution at a geometric rate. Also, we develop a recipe for performing regenerative simulation in this setting and demonstrate its application for estimating Markov chain Monte Carlo standard errors. In both our investigation of convergence rates and in Monte Carlo standard error estimation we pay particular attention to the case with state-independent component-wise proposals. We illustrate our results in two examples, a toy Bayesian inference problem and a practically relevant example involving maximum likelihood estimation for a generalized linear mixed model.