Mixed Hamiltonian Monte Carlo for Mixed Discrete and Continuous Variables

Hamiltonian Monte Carlo (HMC) has emerged as a powerful Markov Chain Monte Carlo (MCMC) method to sample from complex continuous distributions. However, a fundamental limitation of HMC is that it can't be applied to distributions with mixed discrete and continuous variables. In this paper, we propose mixed HMC as a general framework to address this limitation. Mixed HMC is a novel family of MCMC algorithms that evolves the discrete and continuous variables in tandem in a way analogous to HMC, allowing more frequent updates of discrete variables while maintaining HMC's ability to suppress random-walk behavior. We establish mixed HMC's theoretical properties, and present an efficient implementation with Laplace momentum that can be incorporated into existing HMC implementations with minimal overhead. The superior performances of mixed HMC over existing methods are demonstrated with numerical experiments on Gaussian mixture models (GMMs), variable selection in Bayesian logistic regression (BLR), and correlated topic models (CTMs).