Why Does Unsupervised Pre-training Help Deep Learning?
Dumitru Erhan, Yoshua Bengio, Aaron C. Courville et al. · 2010
Much recent research has been devoted to learning algorithms for deep architectures such as Deep Belief Networks and stacks of autoencoder variants with impressive results being obtained in several areas, mostly on vision and language datasets. The best results obtained on supervised learning tasks often involve an unsupervised learning component, usually in an unsupervised pre-training phase. The main question investigated here is the following: why does unsupervised pre-training work so well? Through extensive experimentation, we explore several possible explanations discussed in the literature including its action as a regularizer (Erhan et al., 2009b) and as an aid to optimization (Bengio et al., 2007). Our results build on the work of Erhan et al. (2009b), showing that unsupervised pre-training appears to play predominantly a regularization role in subsequent supervised training. However our results in an online setting, with a virtually unlimited data stream, point to a somewhat more nuanced interpretation of the roles of optimization and regularization in the unsupervised pre-training effect.
Cited by 2,038 works.
Authors
References
- Gradient-based learning applied to document recognition
- A Fast Learning Algorithm for Deep Belief Nets
- Reducing the Dimensionality of Data with Neural Networks
- Visualizing Data using t-SNE
- Training Products of Experts by Minimizing Contrastive Divergence
- A Global Geometric Framework for Nonlinear Dimensionality Reduction
- Greedy Layer-Wise Training of Deep Networks
- Learning Deep Architectures for AI
- Extracting and composing robust features with denoising autoencoders
- A unified architecture for natural language processing