Scene Understanding from RGB-D

June 2, 2015, Society for Information Display Symposium, San Jose, CA—Jitenda Malik from UC Berkeley presented details on current work in information extraction from color plus depth images. The addition of the depth information eases much of the effort needed to extract features within the images.

When we assume the sensing is done with and RGB-D imager array, we can extract 3-D understanding with a typical recognition pipeline to reorganize and recognize the extracted content. See figure.

The basic flow for image processing

Normally, the process starts with boundary defects or breaks and continues with semantic of the object to identify the edge defects and the object.

The edge defects include color changes, depth, convex and concave gradients and local gradients for contour detection. The inclusion of depth information reduces distortion and increases precision to make a more complete understanding of the features within an image. The tasks are simplified when the discontinuities are labeled with RGB-D information.

A bottom-up flow started with 500 candidates for regional proposal generation for potential object detection. Adding a neural network and machine learning algorithms allows the software to export a region for convolutional neural network classification. This object detection algorithm is now getting about 50 percent accuracy. The tools are able to use semantic segmentation for surfaces and other continuous objects, even with color changes within the object. A bounding box for the convolutional neural network for objects uses pose estimation to find the continuation of objects. This CNN has been trained on normal images and gets as high as 90 percent accuracy.

The work is moving to 3-D models and finds an object, get its pose, and renders model information into an existing scene. So far, all the work has been on static images. For future work, they need more data and realistic CAD models.