One-sentence Summary

Researchers from Peking University and Tianjin University introduce SpatialScout, a benchmark for evaluating vision-language model (VLM) agents on active object-level 3D mapping across 80 indoor scenes with 3,140 target objects, where agents control a camera to estimate object centers in a shared coordinate frame and are evaluated on localization accuracy and target coverage; it reveals that the strongest evaluated model, Claude Opus 5.5, achieves a mean overlap of $41.71\%$41.71%, with $47.23\%$47.23% of targets exceeding an IoU threshold of $0.5$0.5 and $3.25\%$3.25% exceeding $0.9$0.9, highlighting gaps in precise metric mapping.

Key Contributions

• SpatialScout is a benchmark for evaluating vision-language model (VLM) agents through active object-level 3D mapping, with 80 indoor scenes, 3,140 target objects, object inventories, RGB observations, camera calibration, and camera poses.
• The benchmark requires agents to select viewpoints, associate observations across views, estimate object centers in a shared coordinate frame, and revise map estimates under bounded interaction budgets; evaluation uses object-scale overlap scores that jointly measure localization accuracy and target coverage while penalizing missing predictions.
• Experiments with frontier vision-language models show that the strongest evaluated model, Claude Opus 5.5, achieves a mean overlap of 41.71%, with 47.23% of targets exceeding an IoU threshold of 0.5 and 3.25% exceeding 0.9; code is available at https://github.com/JesseZZZZZ/SpatialScout.

Introduction

Spatial intelligence is critical for embodied agents that must actively observe 3D scenes, resolve occlusion and perspective ambiguity, and build consistent metric spatial representations. Prior spatial benchmarks rely heavily on image- and video-based question answering, where finite answer choices permit guessing or language-prior shortcuts, and even active-interaction benchmarks remain tied to QA or discrete cognitive maps rather than scoring continuous 3D reconstruction. The authors introduce SpatialScout, a benchmark that evaluates vision-language agents through active object-level 3D mapping: agents start from an initial view and target inventory, select subsequent viewpoints, update object-center estimates in a common coordinate frame, and are scored on localization accuracy, target coverage, and final map quality. SpatialScout contains 80 indoor scenes and 3,140 targets, and initial evaluations show that frontier VLMs recover partial scene layouts but still struggle with comprehensive coverage and precise metric localization.

Dataset

Sources and composition The authors state that SpatialScout assembles 80 indoor scenes rendered with SAPIEN through ManiSkill3. The reported source mix is 32 iTHOR, 12 RoboTHOR, 10 ArchitecTHOR, 6 Replica-CAD, and 30 ProcTHOR scenes. Each source scene appears once, and source layouts and target placements are retained.

Scale and statistics The benchmark contains 3,410 target instances. Scenes contribute 30 to 40 targets: 12 scenes have 35 targets, and 68 scenes have 40 targets. The target-region radius ranges from 1.95 m to 9.23 m. Across the catalog, 1,489 targets (43.3%) belong to equivalence groups with multiple instances in the same scene, which increases association difficulty.

Inventory and annotations Each scene includes a public inventory E = {(i, c_i, g_i)} for N targets, with instance identifier i, category c_i, and equivalence group g_i. During an episode, the agent maintains center estimates for submitted identifiers. Evaluation compares predicted centers within each equivalence group. Each target center is defined as the midpoint of the tight world-axis-aligned bounding box of its rendered geometry. For articulated objects, the bounding box encloses the union of rendered link geometries. The reference center is then transformed into the task frame.

Coordinate frame and inputs The task uses a fixed right-handed frame with origin at the initial camera, +x along the initial optical forward direction, +y to the initial left, and +z to the initial up direction. The agent receives RGB images, the inventory, camera intrinsics, camera poses, and target-region radius r = max_i ||p_i||_2. Reference centers, bounding boxes, and asset geometry are kept in the evaluator.

Selection and processing Target selection prioritizes candidates with known semantic categories, then orders them by distance to the initial camera, using identifier order to break ties. This concentrates recognizable targets around the exploration origin. Candidates must have rendered geometry, and each scene must supply the requested target count. Non-selected candidates are hidden in rendered observations. Each catalog entry fixes the source configuration, initial pose, and target count. The authors save the inventory, geometric annotations, and initial view, and verify agreement between camera-pose axes and the projection matrix. Loaded object counts are checked against source configuration counts with scene-builder exclusions accounted for.

Usage SpatialScout evaluates active object-center mapping by VLM agents. Given the object inventory and calibrated camera poses, an agent selects RGB views, associates objects across observations, and maintains a three-dimensional map of object centers. The benchmark measures map accuracy and completeness through object-level evaluation.

Method

The authors design SpatialScout to evaluate active object-center mapping by Vision-Language Model (VLM) agents. The framework operates as an interactive pipeline where an agent is provided with an object inventory and calibrated camera poses to select RGB views, associate objects across observations, and maintain a three-dimensional map of their centers.

As shown in the figure below:

The overall architecture is structured into three primary modules. First, the Scene and Object Inputs module initializes the environment using a fixed scene catalog from platforms such as iTHOR, RoboTHOR, ArchitecTHOR, ReplicaCAD, and ProcTHOR. The agent receives initial RGB views, object identifiers with their corresponding categories, and the camera pose along with its intrinsics.

Second, the Active Tool Interaction module enables the VLM agent to navigate the 3D space. Operating through a shared interaction interface, the agent utilizes a suite of tools for up to 100 steps. It controls camera movement by applying relative translations along the current horizontal forward and right directions, as well as the world vertical axis. The agent acquires new observations through explicit captures, which render 512x512 RGB images, and applies triangulation to estimate spatial relationships between the camera and the objects.

Finally, the 3D Object Mapping module handles the spatial reasoning and output generation. The agent associates observed objects across multiple views and maintains a map of their centers. It submits or revises the positions of identified objects, mapping each object identifier to specific (x, y, z) coordinates defined in the initial camera frame. Each accepted submission replaces the previous estimate for that identifier. To ensure geometric validity, a validator checks the identifiers and finite coordinates, enforcing a public bound constraint expressed as $\| \widehat{\mathbf{p}}_i \|_2 \leq r + 0.5\text{m}$∥p​i​∥2​≤r+0.5m.

Experiment

The benchmark evaluates spatial mapping across 80 indoor scenes with thousands of targets, where seven multimodal agents use a shared camera-movement, inventory, and triangulation interface under fixed interaction budgets. The experiments validate that successful mapping requires multi-view geometric grounding, incremental submission and refinement, and consistent observation, while failures often arise from incomplete coverage or incorrect correspondences that low triangulation residuals alone cannot detect. Overall, Claude Opus 5.5 achieves the strongest localization and completeness but at the highest inference cost, and the open-source models tested show limited accuracy-cost advantages.

The listed spatial intelligence and scene understanding benchmarks primarily evaluate final answer correctness, with output formats ranging from true/false and multiple choice to numerical and open-ended responses. Only a minority incorporate active interaction or multi-round environment feedback, and none include explicit spatial construction or direct 3D coordinate scoring. The comparison highlights a gap in assessing how models actively perceive, explore, and represent 3D scenes. All surveyed benchmarks target answer correctness, while output formats vary across true/false, multiple choice, numerical, and open-ended tasks. Only a minority of benchmarks incorporate active interaction and multi-round environment feedback. No surveyed benchmark includes explicit spatial construction or direct 3D coordinate scoring.

The shared evaluation interface gives all agents a common set of camera, capture, and target-listing tools while deliberately hiding ground-truth target positions and evaluation scores. Agents can submit or revise object centers and inspect the current accepted map, but submissions are validated only against public constraints such as finite coordinates and a distance bound. This design shifts the task toward multi-view geometric grounding and tool-based triangulation, where effective perception and correspondence quality matter more than the number of views captured. All agents use the same tool interface, so no model receives privileged ground-truth positions or evaluation scores. Trajectory analysis indicates that capturing more images is not sufficient; stronger localization is linked to reliable multi-view triangulation and incremental refinement.

Across the 80-scene localization comparison, Claude Opus 5.5 achieves the highest all-target and submitted-object mean IoU, followed by Claude Sonnet 5.5 and GPT-6 Astra. Precision remains limited for every model: strict overlaps above 90% IoU are rare, and the lower all-target scores relative to submitted-object scores show that omitted objects reduce overall mapping performance. Claude Opus 5.5 leads GPT-6 Astra on both all-target and submitted-object mean IoU by a clear margin. Strict high-overlap matches above 90% IoU are uncommon for all models, indicating precise center alignment is still difficult. The gap between submitted-object and all-target IoU highlights how object omissions lower overall mapping scores. GPT-6 Sol and Kimi-K3 trail substantially, with Kimi-K3 showing almost no strict high-overlap matches.

Tool-use distributions show distinct exploration strategies across the four models. Claude Opus 5.5 combines a large share of observation and camera adjustment calls with a low share of submissions, while GPT-6 Astra submits positions more often and uses fewer camera moves. DeepSeek and Qwen place relatively more tool use on image capture or observation but submit positions less frequently, suggesting less efficient mapping behavior. GPT-6 Astra has the highest share of submit_positions calls and the lowest share of move_camera calls, consistent with competitive mapping using fewer tool calls. Claude Opus 5.5, DeepSeek V4.1 Flash, and Qwen3.8-Max devote larger shares to observation, image capture, or camera rotation while submitting positions less often than GPT-6 Astra.

Across the analyzed trajectories, success was associated with repeated multi-view geometric grounding, incremental submission that expanded object coverage, and subsequent refinement of submitted positions. Failures were more varied, with incomplete object coverage being the most common pattern, followed by inaccurate positions despite low triangulation residuals and unresolved geometric inconsistencies. All six successful trajectories shared sufficient multi-view grounding, incremental submission with expanding coverage, and continued refinement. Incomplete object coverage was the most common failure pattern among unsuccessful trajectories. Many failed trajectories retained inaccurate positions despite low triangulation residuals, suggesting correspondence or object-center selection errors rather than weak geometric constraints.

The experiments first survey spatial intelligence benchmarks and find that they mainly assess final answer correctness while rarely involving interactive perception or explicit 3D construction. A shared evaluation interface is then used to test model-driven localization and mapping, showing that reliable multi-view triangulation and incremental refinement matter more than capturing additional images. Model comparisons indicate that Claude Opus 5.5 performs best but precise alignment remains difficult for all systems, and tool-use patterns suggest observation and camera adjustment with selective submission lead to more effective exploration. Trajectory analysis confirms that success depends on repeated geometric grounding and expanding object coverage, while common failures involve incomplete coverage or inaccurate center selection despite low triangulation residuals.