Fast dropout training
Sida Wang, Christopher D. Manning · 2013
Preventing feature co-adaptation by encour-aging independent contributions from differ-ent features often improves classification and regression performance. Dropout training (Hinton et al., 2012) does this by randomly dropping out (zeroing) hidden units and in-put features during training of neural net-works. However, repeatedly sampling a ran-dom subset of input features makes training much slower. Based on an examination of the implied objective function of dropout train-ing, we show how to do fast dropout training by sampling from or integrating a Gaussian approximation, instead of doing Monte Carlo optimization of this objective. This approx-imation, justified by the central limit theo-rem and empirical evidence, gives an order of magnitude speedup and more stability. We show how to do fast dropout training for clas-sification, regression, and multilayer neural networks. Beyond dropout, our technique is extended to integrate out other types of noise and small image transformations. 1.
Cited by 243 works.
Authors
References
- ImageNet classification with deep convolutional neural networks
- Improving neural networks by preventing co-adaptation of feature detectors
- Learning Word Vectors for Sentiment Analysis
- Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
- The Evidence Framework Applied to Classification Networks
- On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes
- Training with Noise is Equivalent to Tikhonov Regularization
- Baselines and Bigrams: Simple, Good Sentiment and Topic Classification
- Learning with Marginalized Corrupted Features
- Dependency Tree-based Sentiment Classification using CRFs with Hidden Variables