One-sentence Summary

Researchers from New York University, University of Illinois at Urbana-Champaign, and Microsoft Research Cambridge present an RGBD scene parsing method that exploits 3D cues to decompose typical, often messy indoor scenes into floor, walls, supporting surfaces, and object regions, employs a novel integer programming formulation to infer physical support relationships, and introduces a dataset of 1449 RGBD images capturing 464 diverse indoor scenes, demonstrating that the 3D scene cues and inferred support significantly improve object segmentation.

Key Contributions

• A scene parsing method recovers floor, walls, supporting surfaces, and object regions from a single RGBD image of cluttered indoor scenes, using room-aligned 3D cues, structural classes, and surface fitting to handle occlusion and missing surfaces.
• A novel integer programming formulation infers physical support relations between surfaces and objects, enabling global reasoning across image regions and robust inference even when supporting regions are invisible.
• A new dataset of 1449 RGBD images capturing 464 diverse indoor scenes provides detailed object and support annotations, and experiments show inferred support relations and 3D scene cues improve object segmentation.

Introduction

The authors tackle the problem of producing a physical scene parse for robotics and scene understanding, where merely labeling objects is insufficient. Tasks like grasping, navigation, and interaction require partitioning the scene into surfaces and objects and reasoning about how they support one another. Prior work often relied on 2D heuristics, required manual annotations, or concentrated on outdoor settings, and existing RGB-D methods stopped at object recognition without modeling support relations. The authors introduce a fully automatic approach that jointly estimates 3D room geometry, segments objects, and infers support relationships by combining depth-based plane fitting, structural object classes (ground, permanent structures, furniture, and props), and learned priors over physical constraints. They also contribute a large, annotated RGB-D dataset of cluttered indoor scenes, enabling robust support inference even when supporting surfaces are invisible or heavily occluded.

Dataset

The authors introduce a large Kinect RGBD dataset for indoor scene understanding, designed to be more diverse and comprehensive than previous datasets.

• Composition and sources The dataset contains 1,449 RGBD images captured in 464 indoor scenes across 26 scene classes. Data was collected from a wide range of commercial and residential buildings in three different US cities, ensuring varied real-world settings.

• Key details for each subset

  • Images: 1,449 RGBD frames.
  • Scenes: 464 unique indoor scenes.
  • Labels: Dense per-pixel semantic labeling was obtained via Amazon Mechanical Turk.
  • Instance-level annotations: If multiple instances of a class appear in a scene, each receives a unique instance label (e.g., “cup 1” and “cup 2”).
  • Object statistics: 35,064 distinct objects spanning 894 different classes.
  • Support annotations: Each image is provided with manually added support tuples of the form $[R_i, R_j, \text{type}]$[Ri​,Rj​,type], where $R_i$Ri​ is the supported region, $R_j$Rj​ is the supporting region, and type indicates “from below” (e.g., cup on table) or “from behind” (e.g., picture on a wall). No explicit filtering or cropping beyond the original capture is described.

• How the paper uses the data The dataset serves as the primary experimental resource for training and evaluating models on indoor scene understanding tasks, including semantic segmentation, instance labeling, and support relationship inference. The provided description does not specify a particular training/validation split or mixture ratios; the dataset is used as a whole to benchmark the proposed approach.

• Processing and metadata construction Beyond the raw RGBD capture, the main processing is the manual annotation pipeline: per-pixel class labels, instance IDs, and support triplets. No image cropping or other augmentation strategies are reported in the dataset construction details.

Method

The authors leverage a comprehensive pipeline for understanding indoor scenes, starting with structural modeling, followed by object segmentation, and finally inferring support relationships.

Modeling the Structure of Indoor Scenes The initial inference of scene surfaces is treated as an alignment and segmentation problem. Given registered RGB and depth images, the authors first compute 3D surface normals at each pixel. To align the 3D measurements to room coordinates, they rely on the Manhattan world assumption, extracting straight lines from the RGB image and computing mean-shift modes of surface normals to find three dominant orthogonal directions. Candidates for principal directions are scored based on their alignment with surface normals and line directions:

The set of candidates with the largest score is selected to form a rotation matrix that aligns the scene. This alignment procedure significantly improves the vertical orientation of scene floors.

Following alignment, the authors generate potential wall, floor, support, and ceiling planes using a RANSAC procedure. To determine pixel correspondence for each plane, they solve a segmentation problem using graph cuts with alpha expansion. The energy function minimizes unary terms encoding 3D position and normal matching, weighted by depth measurement confidence, alongside a pairwise term enforcing gradient-sensitive smoothing.

Segmentation To classify objects and interpret relations, the image is segmented into regions corresponding to individual object or surface instances. An initial oversegmentation is created using the watershed algorithm applied to Pb boundaries, forced to be consistent with the 3D plane regions. For hierarchical segmentation, pairs of regions are iteratively merged based on learned similarities. A boosted decision tree classifier predicts boundary strengths to determine whether two regions belong to the same object instance, utilizing both standard 2D features and 3D features derived from depth and surface orientation.

Modeling Support Relationships The authors model physical support relations among segmented regions. Each region is assigned a support relation variable $S_i$Si​ (supported by another region, a hidden object, or the ground), a support type $T_i$Ti​ (from below or behind), and a structure class $M_i$Mi​ (Ground, Furniture, Prop, or Structure). The goal is to infer the most probable joint assignment by minimizing an energy function that combines likelihood terms from trained classifiers and prior terms encoding transition probabilities, support consistency, and ground consistency.

The maximum a posteriori inference problem is formulated as an integer program using boolean indicator variables for the unobserved variables. The objective function encodes the support likelihood, structure class likelihood, and transition priors. Constraints ensure each region is assigned a single support, type, and structure label, while maintaining consistency and marginalization. The integrality constraints are relaxed to solve the problem as a linear program, with fractional solutions resolved by selecting the most likely assignments.

Local classifiers are trained to compute the likelihood terms. A logistic regression classifier predicts support relations using geometrical, shape, and location features. Another logistic regression classifier predicts structure classes using SIFT features, histograms of surface normals, bounding box dimensions, and color histograms.

Experiment

The experiments evaluate segmentation, support inference, and structure class prediction. Segmentation accuracy improves when fusing RGB and depth features and incorporating intermediate support and structure cues. Support inference benefits from a global energy-based model that outperforms rule-based baselines by reasoning about support types and non-local context, though floor and structure misclassifications still cause cascading errors. Structure class prediction sees only a slight gain from global optimization, with performance limited by noisy depth and segmentation inaccuracies.

Combining RGB and depth features yields substantially better segmentation overlap than using either modality alone, with area-weighted scores rising by about 10 and 7 percentage points over RGB-only and depth-only, respectively. Adding support classifier features gives a further small boost, and incorporating structure class features provides an even more modest additional gain. The unweighted scores follow the same trend but remain lower than the area-weighted scores. RGBD features improve area-weighted segmentation accuracy by roughly 10 percentage points over RGB only and 7 percentage points over depth only. Adding support features to RGBD segmentation yields a slight gain of 0.7 weighted points, while further adding structure class features adds only 0.5 points. Unweighted overlap scores are consistently lower than weighted scores, but the relative ordering of feature combinations is the same.

Simple rule-based methods achieve reasonable support inference accuracy on ground truth regions but degrade sharply on segmented regions, highlighting their sensitivity to segmentation errors. The energy minimization approach consistently outperforms all baselines, particularly in type-aware accuracy, showing the value of combining global reasoning with discriminative inference. Correctly identifying the support type (below vs. behind) remains difficult for all methods, with type-aware scores consistently lower than type-agnostic ones. Image Plane Rules drop from 63.9% type-agnostic accuracy on ground truth regions to 22.1% on segmented regions, illustrating heavy reliance on perfect segmentation. Structure Class Rules outperform the Support Classifier on ground truth type-agnostic accuracy (72.0% vs. 70.1%), but the Support Classifier is more type-aware on ground truth (63.4% vs. 57.7%). Energy Min (LP) achieves the highest type-aware accuracy on both ground truth (72.6%) and segmented regions (54.5%), a large margin over the next best method on each. The type-aware accuracy gap between ground truth and segmented regions is smallest for Energy Min (18.1 percentage points) and largest for Image Plane Rules (31.3 points), demonstrating better robustness to segmentation quality. All methods show a notable drop from type-agnostic to type-aware metrics, confirming that distinguishing support types is a harder task than inferring the existence of a support relation.

The segmentation experiments demonstrate that fusing RGB and depth modalities substantially improves region overlap over either modality alone, while adding support and structure class features provides only modest further gains. The support inference evaluation reveals that simple rule-based methods achieve reasonable accuracy on ground truth regions but collapse on automatically segmented regions, highlighting acute sensitivity to segmentation errors. An energy minimization approach that incorporates global reasoning consistently outperforms baselines, particularly on the harder task of distinguishing support types, and exhibits the smallest drop in accuracy when moving from ground truth to segmented inputs, indicating greater robustness to imperfect segmentation.