Efficient Gradient-Based Inference through Transformations between Bayes Nets and Neural Nets

Hierarchical Bayesian networks and neural networks with stochastic hidden units are commonly perceived as two separate types of models. We show that either of these types of models can be transformed into an instance of the other, by switching between centered and differentiable non-centered parameterizations (CP vs DNCP) of the latent variables. We derive rules for deciding when such parameterizations are beneficial for gradient-based inference in terms of decreased posterior correlations, and show that in the DNCP, a Monte Carlo estimator of the marginal likelihood can be used for learning the parameters. Theoretical results are validated in experiments.