Tractable Approximate Gaussian Inference for Bayesian Neural Networks

In this paper, we propose an analytical method allowing for tractable approximate Gaussian inference (TAGI) in Bayesian neural networks. The method enables: (1) the analytical inference of the posterior mean vector and diagonal covariance matrix for weights and bias, (2) the end-to-end treatment of uncertainty from the input layer to the output, and (3) the online inference of model parameters using a single observation at a time. The method proposed has a computational complexity of O(n) with respect to the number of parameters n, and the tests performed on regression and classification benchmarks confirm that, for a same network architecture, it matches the performance of existing methods relying on gradient backpropagation.