Minimum Word Error Rate Training for Attention-Based Sequence-to-Sequence Models
Rohit Prabhavalkar, Tara N. Sainath, Yonghui Wu et al. · IEEE International Conference on Acoustics Speech and Signal Processing · 2018
Sequence-to-sequence models, such as attention-based models in automatic speech recognition (ASR), are typically trained to optimize the cross-entropy criterion which corresponds to improving the log-likelihood of the data. However, system performance is usually measured in terms of word error rate (WER), not log-likelihood. Traditional ASR systems benefit from discriminative sequence training which optimizes criteria such as the state-level minimum Bayes risk (sMBR) which are more closely related to WER. In the present work, we explore techniques to train attention-based models to directly minimize expected word error rate. We consider two loss functions which approximate the expected number of word errors: either by sampling from the model, or by using N-best lists of decoded hypotheses, which we find to be more effective than the sampling-based method. In experimental evaluations, we find that the proposed training procedure improves performance by up to 8.2% relative to the baseline system. This allows us to train grapheme-based, uni-directional attention-based models which match the performance of a traditional, state-of-the-art, discriminative sequence-trained system on a mobile voice-search task.
Cited by 142 works. doi:10.1109/icassp.2018.8461809 Open-access PDF
Authors
References
- Long Short-Term Memory
- AutoResearch Lite: a reproducible paper-inspired experiment runner (Adam mechanism reproduction and convergence-speed comparison)
- BLEU
- Attention Is All You Need
- Sequence to Sequence Learning with Neural Networks
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Bidirectional recurrent neural networks
- Simple statistical gradient-following algorithms for connectionist reinforcement learning
- Connectionist temporal classification
- Sequence Transduction with Recurrent Neural Networks