Stacked What-Where Auto-encoders

We present a novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides an unified approach to supervised, semi-supervised and unsupervised learning without requiring sampling. An instantiation of SWWAE is essentially a convolutional net (Convnet) (LeCun et al. (1998)) coupled with a deconvolutional net (Deconvnet) (Zeiler et al. (2010)). The objective function includes reconstruction terms that penalize the hidden states in the Deconvnet for being different from the hidden state of the Convnet. Each pooling layer is seen producing two sets of variables: the "what" which are fed to the next layer, and its complementary variable "where" that are fed to the corresponding layer in the generative decoder.