Learning to Execute
Wojciech Zaremba, Ilya Sutskever · arXiv (Cornell University) · 2014
Recurrent Neural Networks (RNNs) with Long Short-Term Memory units (LSTM) are widely used because they are expressive and are easy to train. Our interest lies in empirically evaluating the expressiveness and the learnability of LSTMs in the sequence-to-sequence regime by training them to evaluate short computer programs, a domain that has traditionally been seen as too complex for neural networks. We consider a simple class of programs that can be evaluated with a single left-to-right pass using constant memory. Our main result is that LSTMs can learn to map the character-level representations of such programs to their correct outputs. Notably, it was necessary to use curriculum learning, and while conventional curriculum learning proved ineffective, we developed a new variant of curriculum learning that improved our networks' performance in all experimental conditions. The improved curriculum had a dramatic impact on an addition problem, making it possible to train an LSTM to add two 9-digit numbers with 99% accuracy.
Cited by 283 works. doi:10.48550/arxiv.1410.4615 Open-access PDF
Authors
References
- Long Short-Term Memory
- Sequence to Sequence Learning with Neural Networks
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
- Speech recognition with deep recurrent neural networks
- Generating Sequences With Recurrent Neural Networks
- Recurrent neural network based language model
- Deep learning via Hessian-free optimization
- Curriculum learning
- Recurrent Neural Network Regularization
- Advances in optimizing recurrent networks