Asynchronous Gibbs Sampling

Gibbs sampling is a widely used Markov Chain Monte Carlo (MCMC) method for numerically approximating integrals of interest in Bayesian statistics and other mathematical sciences. It is widely believed that MCMC methods do not extend easily to parallel implementations, as their inherently sequential nature incurs a large synchronization cost. This means that new solutions are needed to bring Bayesian analysis fully into the era of large-scale computation.In this paper, we present a novel scheme - Asynchronous Gibbs sampling - that allows us to perform MCMC in a parallel fashion with no synchronization or locking, avoiding the typical performance bottlenecks of parallel algorithms. Our method is especially attractive in settings, such as hierarchical random-effects modeling in which each observation has its own random effect, where the problem dimension grows with the sample size. We present two variants: an exact algorithm, and an approximate algorithm with better scaling properties. We prove convergence of the exact algorithm under some basic regularity conditions, and discuss the proof for similar parallelization schemes for other iterative algorithms. We provide three examples that illustrate some of the algorithm's properties with respect to scaling, and one example that compares the exact and approximate algorithms. Because our hardware resources are bounded, we have not yet found a limit to the algorithm's scaling, and thus its true capabilities remain unknown.