Learning to Execute

Recurrent Neural Networks (RNNs) with Long-Short Term Memory units (LSTM) are widely used because they are expressive and are easy to train. Our interest lies in empirically evaluating the expressiveness and the learnability of LSTMs by training them to evaluate short computer programs, a problem that has traditionally been viewed as too complex for neural networks. We consider a simple class of programs that can be evaluated with a single left-to-right pass using constant memory. Our main result is that LSTMs can learn to map the character-level representations of such programs to their correct outputs. Notably, it was necessary to use curriculum learning, and while conventional curriculum learning proved ineffective, we developed an new variant of curriculum learning that improved our networks' performance in all experimental conditions.