On rectified linear units for speech processing
Matthew D. Zeiler, M. Ranzato, Rajat Monga et al. · IEEE International Conference on Acoustics Speech and Signal Processing · 2013
Deep neural networks have recently become the gold standard for acoustic modeling in speech recognition systems. The key computational unit of a deep network is a linear projection followed by a point-wise non-linearity, which is typically a logistic function. In this work, we show that we can improve generalization and make training of deep networks faster and simpler by substituting the logistic units with rectified linear units. These units are linear when their input is positive and zero otherwise. In a supervised setting, we can successfully train very deep nets from random initialization on a large vocabulary speech recognition task achieving lower word error rates than using a logistic network with the same topology. Similarly in an unsupervised setting, we show how we can learn sparse features that can be useful for discriminative tasks. All our experiments are executed in a distributed environment using several hundred machines and several hundred hours of speech data.
Cited by 420 works. doi:10.1109/icassp.2013.6638312
Authors
- Matthew D. Zeiler
- M. Ranzato
- Rajat Monga
- M. Mao
- Ke Yang
- Quoc Viet Le
- P. Nguyen
- Andrew Senior
- Vincent Vanhoucke
- Geoffrey E. Hinton
References
- ImageNet classification with deep convolutional neural networks
- Rectified Linear Units Improve Restricted Boltzmann Machines
- Large Scale Distributed Deep Networks
- Training Products of Experts by Minimizing Contrastive Divergence
- Deep Sparse Rectifier Neural Networks
- Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition
- Application of pretrained deep neural networks to large vocabulary speech recognition
- Learning Fast Approximations of Sparse Coding
- Scalable minimum Bayes risk training of deep neural network acoustic models using distributed hessian-free optimization
- Auto-encoder bottleneck features using deep belief networks