Building high-level features using large scale unsupervised learning
Marc’Aurelio Ranzato, Rajat Monga, Matthieu Devin et al. · International Conference on Machine Learning · 2012
We consider the problem of building high-level, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images using unlabeled images? To answer this, we train a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200×200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and out-of-plane rotation. We also find that the same network is sensitive to other high-level concepts such as cat faces and human bodies. Starting with these learned features, we trained our network to obtain 15.8% accuracy in recognizing 20,000 object categories from ImageNet, a leap of 70% relative improvement over the previous state-of-the-art.
Cited by 632 works.
Authors
References
- Gradient-based learning applied to document recognition
- ImageNet: A large-scale hierarchical image database
- A Fast Learning Algorithm for Deep Belief Nets
- Reducing the Dimensionality of Data with Neural Networks
- Learning Multiple Layers of Features from Tiny Images
- Receptive fields of single neurones in the cat's striate cortex
- What is the best multi-stage architecture for object recognition?
- Greedy Layer-Wise Training of Deep Networks
- Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations
- Emergence of simple-cell receptive field properties by learning a sparse code for natural images