Real-Time Visual Concept Classification
Jasper Uijlings, Arnold W. M. Smeulders, Remko J. H. Scha · IEEE Transactions on Multimedia · 2010
As datasets grow increasingly large in content-based image and video retrieval, computational efficiency of concept classification is important. This paper reviews techniques to accelerate concept classification, where we show the trade-off between computational efficiency and accuracy. As a basis, we use the Bag-of-Words algorithm that in the 2008 benchmarks of TRECVID and PASCAL lead to the best performance scores. We divide the evaluation in three steps: 1) Descriptor Extraction, where we evaluate SIFT, SURF, DAISY, and Semantic Textons. 2) Visual Word Assignment, where we compare a k-means visual vocabulary with a Random Forest and evaluate subsampling, dimension reduction with PCA, and division strategies of the Spatial Pyramid. 3) Classification, where we evaluate the χ2, RBF, and Fast Histogram Intersection kernel for the SVM. Apart from the evaluation, we accelerate the calculation of densely sampled SIFT and SURF, accelerate nearest neighbor assignment, and improve accuracy of the Histogram Intersection kernel. We conclude by discussing whether further acceleration of the Bag-of-Words pipeline is possible. Our results lead to a 7-fold speed increase without accuracy loss, and a 70-fold speed increase with 3% accuracy loss. The latter system does classification in real-time, which opens up new applications for automatic concept classification. For example, this system permits five standard desktop PCs to automatically tag for 20 classes all images that are currently uploaded to Flickr.
Cited by 133 works. doi:10.1109/tmm.2010.2052027
Authors
References
- Distinctive Image Features from Scale-Invariant Keypoints
- LIBSVM
- Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
- Video Google: a text retrieval approach to object matching in videos
- Libsvm : A library for support vector machines
- Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
- A performance evaluation of local descriptors
- Scalable Recognition with a Vocabulary Tree
- Visual categorization with bags of keypoints
- Speeded-Up Robust Features (SURF)