DeViSE: A Deep Visual-Semantic Embedding Model
Andrea Frome, Greg S. Corrado, Jon Shlens et al. · 2013
Modern visual recognition systems are often limited in their ability to scale to large numbers of object categories. This limitation is in part due to the increasing difficulty of acquiring sufficient training data in the form of labeled images as the number of object categories grows. One remedy is to leverage data from other sources – such as text data – both to train visual models and to constrain their pre-dictions. In this paper we present a new deep visual-semantic embedding model trained to identify visual objects using both labeled image data as well as seman-tic information gleaned from unannotated text. We demonstrate that this model matches state-of-the-art performance on the 1000-class ImageNet object recogni-tion challenge while making more semantically reasonable errors, and also show that the semantic information can be exploited to make predictions about tens of thousands of image labels not observed during training. Semantic knowledge improves such zero-shot predictions achieving hit rates of up to 18 % across thou-sands of novel labels never seen by the visual model. 1
Cited by 1,864 works.
Authors
References
- ImageNet classification with deep convolutional neural networks
- ImageNet: A large-scale hierarchical image database
- Improving neural networks by preventing co-adaptation of feature detectors
- Efficient Estimation of Word Representations in Vector Space
- Large Scale Distributed Deep Networks
- Visualizing Data using t-SNE
- Distributed Representations of Words and Phrases and their Compositionality
- Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
- 10.1162/153244303322533223
- Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost