One-sentence Summary

The authors introduce SceneFun3D, a large-scale dataset of over 14.8k fine-grained interaction annotations on 710 high-resolution real-world 3D indoor scenes, along with motion parameters and task descriptions, to enable three novel tasks—functionality segmentation, task-driven affordance grounding, and 3D motion estimation—where existing methods underperform, demonstrating the challenge of functional scene understanding.

Key Contributions

• SceneFun3D provides a large-scale dataset of 710 high-resolution real-world indoor scenes captured via laser scans, comprising over 14.8k fine-grained interaction annotations, motion parameter information, and natural language task descriptions for functional interactive elements.
• Three novel tasks are introduced: functionality segmentation, task-driven affordance grounding, and 3D motion estimation, shifting the focus from object-level understanding to interaction-centric analysis of functional elements.
• State-of-the-art closed-set and open-set 3D scene understanding methods are adapted to these tasks, and experiments demonstrate that accurate functionality reasoning in real 3D scenes remains challenging despite recent progress.

Introduction

The perception of functional interactive elements like knobs, handles, and buttons is crucial for embodied AI, robotics, and mixed reality, yet it remains underexplored because commodity RGB‑D reconstructions typically fail to capture the fine geometry of such small details. Earlier 3D affordance datasets annotate object parts on CAD models but lack real‑world scene‑level annotations, and Gibsonian affordances alone do not convey the purpose or telic function of an interactive element. The authors present SceneFun3D, the first large‑scale dataset that uses high‑resolution laser scans to provide 14.8k detailed masks and affordance labels for functional elements in real indoor scenes, complemented by task‑driven natural language descriptions and interaction‑centric motion parameters. They further introduce three novel benchmarks—functionality segmentation, task‑driven affordance grounding, and 3D motion estimation—to consistently evaluate fine‑grained scene understanding.

Dataset

The authors build the SceneFun3D dataset on top of ARKitScenes, combining high-resolution laser scans with posed RGB video to capture functional interactive elements in indoor environments.

• Data sources and composition ARKitScenes provides multiple Faro Focus S70 laser scans per scene (about four) and iPad Pro video streams (Wide/Ultra Wide RGB, LiDAR depth, ARKit meshes), but initially in different coordinate frames. After cleaning and registration, the dataset retains 710 indoor scenes. Scenes are discarded when the laser scanner poses are wrong, the scene is too small and lacks interaction spots, or high-resolution RGB frames are unavailable.

• Point cloud processing Laser scans from each scene are merged using their supplied scanner poses and then downsampled with a 5 mm voxel to keep fine geometry (buttons, knobs, handles) while making the point clouds tractable for models. Before aligning with the camera poses, transparent-surface artefacts are removed via DBSCAN clustering.

• Registration to camera poses To bring the laser scans into the iPad’s coordinate system, the authors first reconstruct a high-resolution point cloud from the RGB-D frames with interpolated camera poses. This proxy cloud is aligned to the cleaned laser scan using Predator followed by Multi-Scale ICP refinement. Visual inspection of the projected colour flags failures, which are resolved with manual correspondences.

• Annotation schema A web-based tool lets annotators directly label the dense point cloud with single-point precision while viewing scene videos. Each functional interactive element receives:

• An instance mask and one of 9 Gibsonian affordance labels (e.g., “tip push”, “grasp”).

• A 3D motion annotation (translation or rotation, including axis origin and direction).

• A natural language task description (e.g., “Turn on the ceiling light”). These are later augmented by ChatGPT rephrasing, expanding 10,913 original descriptions to 17,133. Instances whose geometry is poorly captured (reflective materials) are tagged as “exclude” and omitted from evaluation.

• Dataset splits The test set is the official ARKitScenes validation split (85 scenes). The remaining scenes are randomly divided into training (545 scenes) and validation (80 scenes).

• Scale and statistics

• 14,867 total functionality annotations.

• 14,279 motion annotations (8,325 translational, 6,542 rotational).

• 17,133 language task descriptions (original + rephrased).

• Usage in the paper The splits are used for training and benchmarking models on functional interactive element detection, affordance classification, motion parameter estimation, and task description generation.

Method

The authors address three novel 3D scene understanding tasks centered around functional interactive elements. Task 1 is functionality segmentation. Given an input point cloud $\mathcal{P} = \{(p_i, f_i)\}$P={(pi​,fi​)}, where $p_i \in \mathbb{R}^3$pi​∈R3 are coordinates and $f_i$fi​ are features such as RGB color and normals, the model predicts instance masks $\{m_i\}_{i=1}^K${mi​}i=1K​ and associated affordance labels $\{\ell_i\}_{i=1}^K${ℓi​}i=1K​ for $K$K instances. This requires understanding visual affordances in an object-class-agnostic manner. Task 2 is task-driven affordance grounding. Given a point cloud $P$P and a free-form task description $D$D, the goal is to predict the instance mask $m$m and affordance label $\ell$ℓ of the referred functional element, requiring an understanding of telic affordance. Task 3 is motion estimation. Given $P$P, the model identifies motion parameters $\{\phi_i\}_{i=1}^K${ϕi​}i=1K​ describing the required interaction. Each parameter is represented as $\phi_i = \{t_i, a_i, o_i\}$ϕi​={ti​,ai​,oi​}, where $t_i \in \{rotation, translation\}$ti​∈{rotation,translation} is the motion type, $a_i \in \mathbb{R}^3$ai​∈R3 is the motion axis direction, and $o_i \in \mathbb{R}^3$oi​∈R3 is the motion origin.

To support these tasks, the authors construct the SceneFun3D dataset through a multi-stage pipeline. High-resolution point clouds are acquired using Faro Focus S70 laser scanners. Multiple scans per scene are combined and downsampled with a 5mm voxel size to preserve fine geometric details of small interactive elements, which are often lost in commodity RGB-D reconstructions.

As shown in the figure below, laser scans capture a higher level of detail required for the geometrically fine-grained annotation of small interactive elements compared to RGB-D reconstructions.

Next, the laser scans are registered to the coordinate system of the camera poses. A high-resolution proxy point cloud is reconstructed from RGB-D frames. The DBSCAN clustering algorithm removes extraneous points from transparent surfaces. The laser scan is then aligned to the proxy using Predator and refined with Multi-Scale Iterative Closest Point (ICP).

For semantic annotation, a lightweight web-based tool is developed to annotate high-resolution point clouds directly, achieving single-point accuracy. An accelerated ray-casting algorithm based on Bounding Volume Hierarchies (BVH) speeds up spatial queries. Annotators detect functionalities, assign one of nine Gibsonian-inspired affordance labels, and annotate the instance mask.

As shown in the figure below, examples of functional interactive element annotations illustrate the nine affordance categories.

To capture telic affordance, natural language task descriptions are collected for each functionality. Annotators provide descriptions of tasks uniquely involving the displayed element, which are then augmented using ChatGPT for diversity.

As shown in the figure below, examples of collected natural language task descriptions demonstrate the telic affordance of functionalities in the scene context.

Finally, 3D motion annotations are collected. Annotators observe the point cloud and scene videos to infer the required interaction. They select the motion type, specify the motion axis origin by selecting a point, and define the axis direction using helper tools.

As shown in the figure below, examples of 3D motion annotations show the motion type, axis origin, and direction.

Experiment

The SceneFun3D dataset benchmarks three novel tasks: functionality segmentation, task-driven affordance grounding, and 3D motion estimation. For segmentation, Mask3D-F outperforms SoftGroup-F and LERF, although all methods struggle with very small interactive elements unless a coarse-to-fine curriculum training is employed. In affordance grounding, OpenMask3D-F significantly surpasses LERF by leveraging fine-grained mask proposals, and motion estimation benefits from incorporating surface normal information.

The affordance taxonomy defines seven interaction categories for functional elements in indoor scenes, each characterized by a specific hand or foot movement and contact type. The labels distinguish between pushing, pulling, rotating, and pressing actions based on how a person would engage with the element, for example using fingertips, a hooked finger, a pinch, or a foot. Pulling actions are split into hook_pull (hooking fingers, e.g., fridge handle) and pinch_pull (pinching, e.g., drawer knob), capturing different grip types. Rotate and hook_turn both involve turning, but rotate applies to rotary knobs while hook_turn uses hooked fingers on a lever-like surface like a door handle. A dedicated foot_push label covers pedals operated by foot, such as a trash can pedal, complementing the hand-based interactions.

Mask3D-F attains the best overall segmentation performance among the evaluated methods, substantially surpassing both the open-vocabulary LERF model and the adapted SoftGroup-F across all AP thresholds. The results demonstrate that tailoring instance segmentation methods to fine-grained functional elements with region-based losses yields marked improvements over zero-shot language-driven approaches. Mask3D-F achieves the highest AP, outperforming LERF by over 60% and more than doubling the AP of SoftGroup-F. LERF’s zero-shot language embeddings produce reasonable but notably lower scores than the fine-tuned Mask3D-F, especially at strict IoU thresholds.

OpenMask3D-F achieves the highest scores on task-driven affordance grounding, with 8.0 AP50 and 17.5 AP25, outperforming LERF by a substantial margin. The original OpenMask3D, which relies on object-level masks, scores zero on both metrics, demonstrating that fine-grained functional element masks are essential for identifying how a task can be performed. The results highlight the importance of segmentation granularity for language-guided functional element retrieval. OpenMask3D-F surpasses LERF on AP50 (8.0 vs. 4.9) and AP25 (17.5 vs. 11.3), establishing a clear quantitative edge. The original OpenMask3D obtains 0.0 AP50 and 0.0 AP25 because its object-level mask proposals cannot capture functional elements like buttons and handles.

Adjusting the crop-expansion ratio in OpenMask3D-F has a clear impact on task-driven affordance grounding. The smallest ratio (0.1) yields the lowest scores across both metrics, while moderate ratios (0.5–1.0) substantially improve performance. Beyond 1.0, AP_25 declines, indicating that overly large expansions hurt localization accuracy without further benefit to AP_50. Increasing the expansion ratio from 0.1 to 0.5 raises AP_50 from 4.5 to 8.3, nearly doubling the score. The best AP_25 (17.5) is obtained at a ratio of 1.0, but extending to 2.0 causes AP_25 to drop to 16.5.

Adding estimated surface normal information to the RGB input helps the model predict per-instance motion parameters more accurately, as seen in consistent gains across motion type, axis, and origin accuracy. The overall instance segmentation performance (AP25) remains nearly unchanged, suggesting that the geometric cue specifically benefits motion reasoning without harming mask quality. Including normals improves all motion parameter metrics: +M (motion type), +MA (type and axis), and +MAO (type, axis, and origin) all increase slightly compared to the RGB-only variant. AP25 for instance segmentation stays essentially flat (26.6 vs 26.5), indicating that the added normal information aids motion estimation without degrading mask prediction.

The evaluation setup analyzes fine-grained functional elements in indoor scenes across instance segmentation, task-driven affordance grounding, and motion prediction. Findings demonstrate that methods tailored with region-based losses drastically outperform zero-shot language-driven models, and that segmentation granularity at the level of individual functional elements is essential for successful affordance retrieval, as object-level masks completely fail. Adding surface normal cues improves motion parameter estimation without degrading mask quality, while moderate context expansion yields the best grounding accuracy.