Limitation of capsule networks

A recent development in deep learning groups multiple neurons to capsules such that each capsule represent an object or part of an object. Routing algorithms route the output of capsules from lower level layers to upper level layers. As we show in this paper, these routing procedures are unfit to learn some concrete but simple problems. Specifically, we provide a formal proof that the EM-routing and the routing-by-agreement algorithms cannot distinguish vectors from their negative counterpart. Therefore, a capsule network can only learn problems where the negative input represents the same class as the input itself. We support this theoretical work experimentally and show that a capsule network cannot classify the sign of a scalar with accuracy higher than chance. Methods that prevent this issue from happening are also presented in this paper. As we will show and reason, avoiding this drawback has a positive effect for the training of capsule networks.