Distributed Gaussian Processes

We propose the generalised Bayesian Committee Machine (gBMC), a practical and scalable hierarchical Gaussian process model for large-scale distributed non-parametric regression. The gBCM is a family of product-of-experts models that hierarchically recombines independent computations to form an approximation of a full Gaussian process. The gBCM includes classical product-of-experts models and the Bayesian Committee Machine as special cases, while it addresses their respective shortcomings. Closed-form computations allow for efficient and straightforward parallelisation and distributed computing with a small memory footprint, but without an explicit sparse approximation. Since training and predicting is independent of the computational graph our model can be used on heterogeneous computing infrastructures, ranging from laptops to large clusters. We provide strong experimental evidence that the gBCM works well on large data sets.