Markov chain Monte Carlo (MCMC) lies at the core of modern Bayesian methodology, much of which would be impossible without it. Thus, the convergence properties of Markov chains have received significant attention, and in particular, proving (geometric) ergodicity is of critical interest. In the current era of "Big Data," a question unique to the Bayesian paradigm is how the ability to sample from the posterior via MCMC changes in a high-dimensional context. Many of the current methods for ascertaining the behavior of Markov chains typically proceed as if the dimension of the parameter, p, and the sample size, n, are fixed, but what happens as these grow, i.e., what is the convergence complexity? We demonstrate theoretically the precise nature and severity of the convergence problems that occur in some of the commonly used Markov chains when they are implemented in high dimensions. We show that these convergence problems effectively eliminate the apparent safeguard of geometric ergodicity. We then set forth a formal framework for understanding and diagnosing such convergence problems. In particular, we theoretically characterize phase transitions in the convergence behavior of popular MCMC schemes in various n and p regimes. Moreover, we also show that certain standard diagnostic tests can be misleading in high-dimensional settings. We then proceed to demonstrate theoretical principles by which MCMCs can be constructed and analyzed to yield bounded geometric convergence rates even as the dimension p grows without bound, effectively recovering geometric ergodicity. We also show a universality result for the convergence rate of MCMCs across an entire spectrum of models. Additionally, we propose a diagnostic tool for establishing convergence (or the lack thereof) for high-dimensional MCMC schemes.
View on arXiv