Style Feature Extraction Using Contrastive Conditioned Variational Autoencoders with Mutual Information Constraints

It is crucial to extract fine-grained features such as styles from unlabeled data in data analysis. Unsupervised methods, such as variational autoencoders (VAEs), can extract styles, but the extracted styles are usually mixed with other features. We can isolate the styles using VAEs conditioned by class labels, known as conditional VAEs (CVAEs). However, methods to extract only styles using unlabeled data are not established. In this paper, we construct a CVAE-based method that extracts style features using only unlabeled data. The proposed model roughly consists of two parallel parts; a contrastive learning (CL) part that extracts style-independent features and a CVAE part that extracts style features. CL models generally learn representations independent of data augmentation, which can be seen as a perturbation in styles, in a self-supervised way. Taking the style-independent features as a condition, the CVAE learns to extract only styles. In the training procedure, a CL model is trained beforehand, and then the CVAE is trained while the CL model is fixed. Additionally, to prevent the CVAE from learning to ignore the condition and failing to extract only styles, we introduce a constraint based on mutual information between the CL features and the VAE features. Experiments on two simple datasets, MNIST and an original dataset based on Google Fonts, show that the proposed method efficiently extracts style features. Further experiments using real-world natural image datasets also show the method's extendability.