A Fast Learning Algorithm for Deep Belief Nets
Geoffrey E. Hinton, Simon Osindero, Yee‐Whye Teh · Neural Computation · 2006
We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.
Cited by 16,282 works. doi:10.1162/neco.2006.18.7.1527
Authors
References
- Gradient-based learning applied to document recognition
- Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference
- Training Products of Experts by Minimizing Contrastive Divergence
- Best practices for convolutional neural networks applied to visual document analysis
- Shape matching and object recognition using shape contexts
- Projection Pursuit Regression
- A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
- The "Wake-Sleep" Algorithm for Unsupervised Neural Networks
- Boosting a Weak Learning Algorithm by Majority
- Exponential Family Harmoniums with an Application to Information Retrieval