Image retrieval using scene graphs
Justin M. Johnson, Ranjay Krishna, Michael M. Stark et al. · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · 2015
This paper develops a novel framework for semantic image retrieval based on the notion of a scene graph. Our scene graphs represent objects (“man”, “boat”), attributes of objects (“boat is white”) and relationships between objects (“man standing on boat”). We use these scene graphs as queries to retrieve semantically related images. To this end, we design a conditional random field model that reasons about possible groundings of scene graphs to test images. The likelihoods of these groundings are used as ranking scores for retrieval. We introduce a novel dataset of 5,000 human-generated scene graphs grounded to images and use this dataset to evaluate our method for image retrieval. In particular, we evaluate retrieval using full scene graphs and small scene subgraphs, and show that our method outperforms retrieval methods that use only objects or low-level image features. In addition, we show that our full model can be used to improve object localization compared to baseline methods.
Cited by 1,076 works. doi:10.1109/cvpr.2015.7298990
Authors
- Justin M. Johnson
- Ranjay Krishna
- Michael M. Stark
- Li-Jia Li
- David A. Shamma
- Michael S. Bernstein
- Li Fei-Fei
References
- ImageNet classification with deep convolutional neural networks
- ImageNet: A large-scale hierarchical image database
- Distinctive Image Features from Scale-Invariant Keypoints
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
- ImageNet Large Scale Visual Recognition Challenge
- Microsoft COCO: Common Objects in Context
- The Pascal Visual Object Classes (VOC) Challenge
- Caffe
- Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
- Selective Search for Object Recognition