One-sentence Summary

Researchers from Nanyang Technological University, the University of Hong Kong, and Ropedia propose S-Agent 2.0, a spatial reasoning agent that equips a general coding agent with 3D perception tools to actively gather evidence and convert raw readings into reliable conclusions, thereby closing the gap between model and tools for object relations and motion in 3D scenes.

Key Contributions

• S-Agent 2.0 combines a stock general-purpose coding agent with a command-line perception toolkit, and the approach works across coding-agent harnesses including OpenHands, Codex, and Claude Code.
• The toolkit closes the evidence loop by joining tool measurements with the model’s own perception, persisting acquired evidence in scene memory, and returning each measurement as an answer-ready reading already interpreted in spatial language.
• On MMSI-Bench and ReVSI, the perception toolkit adds +13.9 and +20.5 points over the same agent without tools on a Qwen3.8-27B backbone. S-Agent 2.0 surpasses the state-of-the-art agentic spatial reasoning method, and its toolkit gain remains substantial on Gemini-3.8-Flash where the specialized baseline’s gain shrinks far more.

Introduction

Spatial reasoning requires models to determine where objects are in 3D space, how they relate, and how they move, which is essential for agents operating in physical environments. Recent agentic vision-language models improve by actively collecting evidence with 3D perception tools, but a critical gap remains: even when the tools return the right evidence, the model may misread the measurement or fail to carry it into the final answer. Existing specialized systems invest mostly in richer perception libraries and hand-designed control loops, while leaving the memory and interpretation stages of the evidence loop to the model. The authors address this by introducing S-Agent 2.0, which pairs a general-purpose coding agent with a command-line perception toolkit designed to close the evidence loop through combined model and tool perception, persistent scene memory, and readings returned in plain spatial language.

Method

The authors leverage an off-the-shelf general coding agent to build S-Agent 2.0, a spatial reasoning system designed to close the evidence loop between a Vision-Language Model and 3D perception tools. Rather than relying on the model to carry intermediate results through its context window, the architecture equips the coding agent with a command-line interface perception toolkit and a persistent scene memory. This design ensures that raw geometric readings are acquired, stored, and interpreted in plain spatial language before reaching the final answer.

The framework comprises three core components: a general-purpose coding agent $\pi$π acting as the planner and executor, a command-line perception toolkit $\tau$τ, and a persistent per-episode scene memory $S$S. Given a spatial question $q$q and a set of frames $\mathcal{T} = \{I_1, \dots, I_n\}$T={I1​,…,In​}, the agent receives a fixed instruction $u$u detailing the toolkit commands and a recommended workflow. At each reasoning step $t$t, the agent's context includes the instruction, the question, and the history of its chain of thought $z_{<t}$z<t​, actions $a_{<t}$a<t​, and tool responses $o_{<t}$o<t​. The policy generates the next thought and action:

The action space is split into general coding actions $\mathcal{A}_{\text{gen}}$Agen​ (e.g., running Linux commands, reading files) and perception actions $\mathcal{A}_{\text{perc}}$Aperc​ (e.g., reconstruction, segmentation).

To handle multi-step reasoning without overwhelming the context window, the environment maintains the scene memory $S = (\mathcal{I}, P, \{(\nu_i, g_i)\}_i)$S=(I,P,{(νi​,gi​)}i​), where $g_i = (\mathbf{p}_i, E_i, T_i)$gi​=(pi​,Ei​,Ti​). This memory stores the raw frames $\mathcal{I}$I, a fused metric point cloud $P$P in a gravity-aligned world frame, and a registry of named instances. Each instance pairs a name $\nu_i$νi​ with its geometry $g_i$gi​, consisting of a 3D position $\mathbf{p}_i \in \mathbb{R}^3$pi​∈R3, an extent $E_i$Ei​, and a 6-DoF pose $T_i \in \text{SE}(3)$Ti​∈SE(3). Because the memory records position, extent, and pose, measuring objects and their relations becomes a closed-form lookup derived fresh from the stored geometry at each call.

The system divides perceptual labor based on competence. Deciding what to measure is treated as semantic work handled by the VLM, while producing precise masks and measurements is delegated to dedicated perception models. This division is implemented through two write routes into the scene memory. Tool perception $\mathcal{T}_{\text{tool}}$Ttool​ handles measurement commands like recon to fuse frames and segment to ground objects using the model's text prompts. Model perception $\mathcal{T}_{\text{model}}$Tmodel​ allows the agent to inject its own high-level understanding via the add_instance command, grounding imagined observers or recognized objects directly into the geometric registry. The model can also use the drop command to remove incorrectly registered instances, such as reflections, ensuring the memory remains accurate before readouts are taken.

Finally, to prevent the model from misinterpreting raw numerical outputs, the toolkit enforces a language surface where every reply interprets its own numbers. Each command distills its full output $m$m against the scene memory $S$S into a compact structured report:

This report includes typed numeric fields with units and a plain-language conclusion that names the entities, declares the reference frame, and translates the measurement into ordinary spatial vocabulary. This design allows the model to act on a reading without re-deriving the geometry, while the complete measurements remain accessible on disk for deeper inspection if needed.

Experiment

The evaluation covers five multi-view and video spatial-reasoning benchmarks, comparing S-Agent 2.0 against proprietary VLMs, open-weight spatial models, and agentic baselines using a 27B open backbone or Gemini Flash and frozen perception tools. With no training, S-Agent 2.0 outperforms all agentic baselines on every benchmark and, with the stronger backbone, achieves the highest averages on the main MMSI-Bench and ReVSI results, although GPT-6 retains leads on some MMSI-Bench question types. Ablations show that the toolkit and interface components all contribute, with the model perception interface being most important and the interpreted readout least, while the method also works under Codex and Claude Code harnesses. Qualitative analysis indicates that S-Agent 2.0 closes the evidence-to-answer loop that SpatialClaw leaves open by encoding premises into scene memory and cross-checking perception outputs, with remaining errors mainly involving grounding such as simulating required viewpoints or measuring a quantity in the wrong sense.

On MMSI-Bench, S-Agent 2.0 with the 27B backbone and no training outperforms all agentic baselines and all proprietary models except GPT-6. With Gemini-3.8-Flash it reaches the highest overall average, while GPT-6 remains ahead on region-to-region relations, camera motion, and multi-step reasoning. Among the listed proprietary models, GPT-6 has the strongest overall and category-level accuracy. S-Agent 2.0 with the 27B backbone and no training is ahead of every agentic baseline and every proprietary model other than GPT-6. With Gemini-3.8-Flash, S-Agent 2.0 achieves the highest average accuracy, while GPT-6 keeps the lead on region-to-region relations, camera motion, and multi-step reasoning. Among listed proprietary models, GPT-6 ranks first overall, followed by GPT-5.6; Grok 4 and Gemini 2.5 Pro fall slightly below GPT-5.4. GPT-6 shows particularly strong gains on camera motion and multi-step reasoning relative to the other listed models.

On ReVSI, S-Agent 2.0 outperforms every agentic baseline on both backbones, and with the Gemini-3.8-Flash backbone it records the highest overall average among the compared systems. Its largest advantages over the specialized baseline appear on cross-object geometry, relative direction, and route planning. The one area where it trails is absolute distance, which is also where GPT-6 shows its largest relative shortfall. S-Agent 2.0 leads all agentic baselines on ReVSI for both backbones and achieves the top average with Gemini-3.8-Flash. The strongest gains over the specialized baseline occur in cross-object geometry, relative direction, and route planning. Absolute distance is the single deficit for S-Agent 2.0 and is also the area where GPT-6 falls furthest behind.

S-Agent 2.0 with the Qwen3.8-27B backbone leads all listed agentic spatial reasoning baselines across the five evaluated benchmarks. Compared with SpatialClaw, it improves performance on every benchmark, with the largest gains on ReVSI and MindCube and smaller gains on MMSI, SPAR, and MMSI-Vid. This makes S-Agent 2.0 the strongest method in the given backbone comparison. S-Agent 2.0 achieves the highest accuracy on every benchmark in the comparison. The largest improvements over SpatialClaw occur on ReVSI and MindCube, while MMSI, SPAR, and MMSI-Vid show more modest gains.

S-Agent 2.0 remains effective across OpenHands, Codex, and Claude Code, with no single harness dominating all benchmarks. Interface ablations show the model perception interface is the most critical component, scene memory is second, and the interpreted readout contributes a smaller but still meaningful gain. The pattern supports the view that both interpretation and retained scene memory are needed to close the loop. Across the three coding-agent harnesses, S-Agent 2.0 maintains broadly comparable performance, with only moderate benchmark-specific differences. Removing the model perception interface causes the largest accuracy drop, while removing the interpreted readout causes the smallest drop. Removing scene memory causes an intermediate accuracy drop, consistent with readings carried through the context being lost before the answer is written. Tool use gives a large gain on both benchmarks, and S-Agent 2.0 scores above SpatialClaw on every backbone and benchmark.

Tool use raises ReVSI and MMSI scores for both SpatialClaw and the general coding agent across the two backbones tested. The general coding agent outperforms SpatialClaw in every backbone and benchmark setting, and the gap is larger when tools are enabled. SpatialClaw's benefit from tools shrinks more as the backbone becomes stronger, while the general coding agent's tool-enabled scores keep rising. Tool use improves both harnesses on both backbones, but the general coding agent gains more than SpatialClaw in every comparison. The largest scores occur for the general coding agent with tools on the stronger Gemini-3.8-Flash backbone. SpatialClaw's tool advantage is smaller on the stronger backbone, whereas the general coding agent continues to improve with backbone scale.

The experiments evaluate S-Agent 2.0 on spatial reasoning benchmarks such as MMSI-Bench, ReVSI, SPAR, MMSI-Vid, and MindCube, across multiple backbones and coding-agent harnesses. Without training, S-Agent 2.0 surpasses agentic baselines and all proprietary models except GPT-6 on MMSI-Bench, and with Gemini-3.8-Flash it achieves the highest overall average on both MMSI-Bench and ReVSI, with remaining gaps only in areas like camera motion, multi-step reasoning, and absolute distance. Ablations indicate the model perception interface is most important, scene memory is next, and interpreted readout adds a smaller but meaningful gain, while tool use provides consistent improvements, especially for the general coding agent over SpatialClaw. Overall, the results validate S-Agent 2.0 as a strong general spatial reasoning agent across benchmarks, backbones, and harnesses.