Separating high-dimensional data like images into independent latent factors remains an open research problem. Here we develop a method that jointly learns a linear independent component analysis (ICA) model with non-linear bijective feature maps. By combining these two methods, we scale ICA methods to learn interpretable latent structures for high-dimensional images. Given the complexities of training such a hybrid model, we introduce novel theory that constrains linear ICA to lie close to the manifold of decorrelating matrices, the Stiefel manifold. By doing so we create models that converge quickly and are easy to train. Our hybrid model achieves better unsupervised latent factor discovery than flow-based models and linear ICA on large image datasets.
View on arXiv