Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data
Michael Danielczuk, Matthew Matl, Saurabh Gupta et al. · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · 2019
The ability to segment unknown objects in depth images has potential to enhance robot skills in grasping and object tracking. Recent computer vision research has demonstrated that Mask R-CNN can be trained to segment specific categories of objects in RGB images when massive hand-labeled datasets are available. As generating these datasets is time-consuming, we instead train with synthetic depth images. Many robots now use depth sensors, and recent results suggest training on synthetic depth data can transfer successfully to the real world. We present a method for automated dataset generation and rapidly generate a synthetic training dataset of 50,000 depth images and 320,000 object masks using simulated heaps of 3D CAD models. We train a variant of Mask R-CNN with domain randomization on the generated dataset to perform category-agnostic instance segmentation without any hand-labeled data and we evaluate the trained network, which we refer to as Synthetic Depth (SD) Mask R-CNN, on a set of real, high-resolution depth images of challenging, densely-cluttered bins containing objects with highly-varied geometry. SD Mask R-CNN outperforms point cloud clustering baselines by an absolute 15% in Average Precision and 20% in Average Recall on COCO benchmarks, and achieves performance levels similar to a Mask R-CNN trained on a massive, hand-labeled RGB dataset and fine-tuned on real images from the experimental setup. We deploy the model in an instance-specific grasping pipeline to demonstrate its usefulness in a robotics application. Code, the synthetic training dataset, and supplementary material are available at https://bit.ly/2letCuE.
Cited by 178 works. doi:10.1109/icra.2019.8793744 Open-access PDF
Authors
References
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Fully convolutional networks for semantic segmentation
- Microsoft COCO: Common Objects in Context
- Hypercolumns for object segmentation and fine-grained localization
- Segmentation as selective search for object recognition
- Multiscale Combinatorial Grouping
- Simultaneous Detection and Segmentation
- Semantic Segmentation with Second-Order Pooling
- PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation