Asynchronous Gibbs Sampling

Gibbs sampling is a widely used Markov Chain Monte Carlo (MCMC) method for numerically approximating integrals of interest in Bayesian statistics and other mathematical sciences. It is widely believed that MCMC methods do not extend readily to parallel implementations needed in big data settings, as their inherently sequential nature incurs a large synchronization cost. In this paper, we present a novel scheme - Asynchronous Gibbs sampling - that allows us to perform MCMC in a parallel fashion with no synchronization or locking, avoiding the typical performance bottlenecks of parallel algorithms. Our method is especially attractive in settings, such as hierarchical random-effects modeling in which each observation has its own random effect, where the problem dimension grows with the sample size. We present two variants: an exact algorithm, and an approximate algorithm that does not require transmitting data. We prove convergence of the exact algorithm. We provide examples that illustrate some of the algorithm's properties with respect to scaling, and an example that compares the exact and approximate algorithms.