Context dependent recurrent neural network language model
Tomáš Mikolov, Geoffrey Zweig · IEEE Spoken Language Technology Workshop (SLT) · 2012
Recurrent neural network language models (RNNLMs) have recently demonstrated state-of-the-art performance across a variety of tasks. In this paper, we improve their performance by providing a contextual real-valued input vector in association with each word. This vector is used to convey contextual information about the sentence being modeled. By performing Latent Dirichlet Allocation using a block of preceding text, we achieve a topic-conditioned RNNLM. This approach has the key advantage of avoiding the data fragmentation associated with building multiple topic models on different data subsets. We report perplexity results on the Penn Treebank data, where we achieve a new state-of-the-art. We further apply the model to the Wall Street Journal speech recognition task, where we observe improvements in word-error-rate.
Cited by 560 works. doi:10.1109/slt.2012.6424228
Authors
References
- Long Short-Term Memory
- Indexing by latent semantic analysis
- Latent dirichlet allocation
- Building a Large Annotated Corpus of English: The Penn Treebank
- Learning long-term dependencies with gradient descent is difficult
- Recurrent neural network based language model
- 10.1162/153244303322533223
- Kaldi Speech Recognition Toolkit
- An empirical study of smoothing techniques for language modeling
- 10.1162/jmlr.2003.3.4-5.993