Compressive Transformers for Long-Range Sequence Modelling
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar et al. · arXiv (Cornell University) · 2019
We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.
Cited by 43 works. doi:10.48550/arxiv.1911.05507 Open-access PDF
Authors
References
- Long Short-Term Memory
- Adam: A Method for Stochastic Optimization
- Learning representations by back-propagating errors
- Latent dirichlet allocation
- Attention Is All You Need
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Generating Sequences With Recurrent Neural Networks
- Recurrent neural network based language model
- WaveNet: A Generative Model for Raw Audio
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books