Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre et al. · arXiv (Cornell University) · 2014
In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder-Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.
Cited by 3,293 works. doi:10.48550/arxiv.1406.1078 Open-access PDF
Authors
- Kyunghyun Cho
- Bart van Merriënboer
- Çağlar Gülçehre
- Dzmitry Bahdanau
- Fethi Bougares
- Holger Schwenk
- Yoshua Bengio
References
- ImageNet classification with deep convolutional neural networks
- Long Short-Term Memory
- Distributed Representations of Words and Phrases and their Compositionality
- Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition
- ADADELTA: An Adaptive Learning Rate Method
- 10.1162/153244303322533223
- Statistical phrase-based translation
- Deep Sparse Rectifier Neural Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Recurrent Continuous Translation Models