Crowdsourcing annotations for visual object detection
Hao Su, Jia Hao Deng, Li Fei-Fei · 2012
A large number of images with ground truth object bounding boxes are critical for learning object detectors, which is a fundamental task in compute vision. In this paper, we study strategies to crowd-source bounding box annotations. The core challenge of building such a system is to effectively control the data quality with minimal cost. Our key observation is that drawing a bounding box is significantly more difficult and time consuming than giving answers to multiple choice questions. Thus quality control through additional verification tasks is more cost effective than consensus based algorithms. In particular, we present a system that consists of three simple sub-tasks — a drawing task, a quality verification task and a coverage verification task. Experimental results demonstrate that our system is scalable, accurate, and cost-effective. 1
Cited by 186 works.
Authors
References
- ImageNet: A large-scale hierarchical image database
- Object Detection with Discriminatively Trained Part-Based Models
- Robust Real-Time Face Detection
- Utility data annotation with Amazon Mechanical Turk
- The multidimensional wisdom of crowds
- Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise
- Online crowdsourcing: Rating annotators and obtaining cost-effective labels
- Strong supervision from weak annotation: Interactive training of deformable part models