Denoising autoencoder with modulated lateral connections learns invariant representations of natural images

This paper demonstrates that suitable lateral connections between encoder and decoder allow higher layers of a deep denoising autoencoder to focus on representing invariant features. Without the lateral connections, the deep autoencoder has to carry detailed information through the highest layers. The lateral connections to the decoder mapping carry information about details that are needed to translate abstract invariant features to detailed reconstructions. This translation was found to be efficient when invariant features were allowed to modulate the strength of the lateral connection. Denoising autoencoder with modulated and additive lateral connections, and without lateral connections were compared in experiments using real-world images. The experiments verify that adding modulated lateral connections to the model 1) improves the accuracy of the probability model for inputs, as measured by denoising performance; 2) needs less parameters in the model; and 3) results in representations whose degree of invariance grows faster towards the higher layers.