One-sentence Summary

To enable unified evaluation of spatial cognition and reveal the complementary strengths of pixel-space expression and text-based reasoning, researchers from Zhejiang University introduce ProVisE, a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models, along with SpatialGen-Bench, a diagnostic benchmark of 470 samples across 14 spatial subtasks.

Key Contributions

• The paper introduces ProVisE, a benchmark-agnostic framework that converts protocol-guided visual answers from image-generation models into structured predictions compatible with original task metrics, and SpatialGen-Bench, a diagnostic benchmark of 470 samples across 14 spatial subtasks and four capability levels.
• An agentic builder is proposed that automatically constructs and validates task-specific generation–parser protocols for new benchmarks, enabling ProVisE to extend beyond manually configured tasks without per-model adaptation.
• Extensive experiments compare image-generation models and text-output VLMs, showing that image-generation models compete where answers can be directly externalized in pixel space, while VLMs retain an advantage in compositional spatial reasoning, and the agentic builder successfully adapts protocols to six external benchmarks.

Introduction

Spatial intelligence is foundational for artificial agents to interact safely with the physical world, yet evaluating it in vision-language models (VLMs) has long been tied to symbolic answer formats like coordinates, labels, or text. This text-only interface is unnatural for spatial tasks and completely excludes image-generation models that can directly mark or synthesize visual answers. Existing benchmarks either measure text-to-image prompt fidelity or score symbolic VLM predictions, leaving no way to compare image generators and text-output VLMs under shared spatial task semantics. The authors introduce ProVisE, a framework that constrains generated outputs into predefined visual formats and parses them into structured predictions compatible with original metrics, and SpatialGen-Bench, a diagnostic benchmark with 14 spatial subtasks. Together they enable systematic, metric-aligned comparison, revealing that pixel-space generation excels when answers can be externalized directly, while text-based models remain stronger at compositional spatial reasoning.

Dataset

SpatialGen-Bench is a diagnostic evaluation benchmark designed to test spatial cognition across heterogeneous answer interfaces. It repurposes and filters samples from multiple public spatial evaluation benchmarks, covering perception, reasoning, grounding, navigation, and path planning tasks. The dataset contains 470 manually curated examples organized into four progressive capability levels with 14 subtasks, all using a unified schema while preserving task-specific scoring metadata.

• Sources and scale

  • Drawn from several public spatial evaluation datasets; no individual source names are listed, but the selected tasks span object-level perception, viewpoint reasoning, spatial grounding, state modelling, affordance, navigation, and trajectory planning.
  • After filtering and deduplication, 470 samples remain across all levels.

• Capability hierarchy and subset sizes

  • Spatial perception (145 samples): counting, relative depth estimation, orientation judgment, object size comparison.
  • Spatial understanding (140 samples): relationship verification, perspective judgment, spatial mental modelling, spatial grounding.
  • Spatial reasoning (90 samples): multi-hop spatial reasoning, spatial state prediction, geometric feasibility.
  • Spatial interaction (95 samples): affordance grounding, navigation, trajectory planning.

• Sample filtering and construction

  • Each candidate sample is manually inspected and retained only if the input media, original annotations, task instruction, and answer can be reliably recovered and scored under the source-task semantics.
  • Near-duplicates, visually unclear instances, inconsistent annotations, and samples with ambiguous or multiple valid answers are removed.
  • After filtering, every sample is assigned to a subtask and capability level based on its spatial objective, not its source benchmark; the assignments are cross-checked against source annotations and task formulation.
  • The final dataset uses standardized shared fields while storing task-specific metadata needed for scoring (e.g., answer formats and evaluation functions).

• Data integrity

  • All released items pass deterministic asset and schema checks plus a manual inclusion review; field checking and review coverage are documented in the appendix.

• How the paper uses the data

  • SpatialGen-Bench is used exclusively for evaluation, not for training. It provides the heterogeneous answer contracts needed to test ProVisE’s ability to serve spatial tasks across different output modes.
  • Evaluation metrics vary by answer type: exact-match accuracy (after normalization) for discrete-answer subtasks, point-in-mask accuracy for spatial grounding, mean mask precision for affordance, partial-credit endpoint scores for prediction, and [email protected] for trajectory planning.

No training split or mixture ratios apply; the benchmark is a fixed evaluation set. The only processing involves answer normalization for discrete tasks and the mapping of raw answers into the appropriate metric function according to the preserved task-specific metadata.

Experiment

The evaluation uses the ProVisE pipeline to compare text-output VLMs and image-generation models on a heterogeneous spatial reasoning benchmark, SpatialGen-Bench. Visual answering excels when the answer can remain a directly externalizable pixel-level state, while text answering is stronger for tasks that demand transformations beyond visible evidence, and the two interfaces provide complementary successes. Most failures arise from incorrect spatial predictions rather than breakdowns in visual communication, indicating that showing preserves but does not replace the required reasoning. Agentic protocol construction transfers across benchmarks, though substituting task-specific parsers with universal VLMs alters scores and rankings, confirming that controlled, protocolized evaluation is critical for reliable comparison.

Existing spatial evaluation benchmarks largely neglect hierarchical capability diagnosis, multiple answer formats, and shared evaluation across text-output and image-generation models. Only a minority fully organize tasks into progressive reasoning levels, and almost none evaluate both modalities under common metrics. This fragmentation limits the ability to isolate whether a model's spatial reasoning or its output modality causes failures. Few benchmarks fully support hierarchical task structures that progress from basic spatial perception to action-oriented reasoning. Multiple evaluated answer forms are rarely provided, with most benchmarks relying on a single response style. Shared evaluation of text-output VLMs and image-generation models under identical task semantics and metrics is almost entirely absent, represented by only one partial implementation. In generative settings, valid but spatially incorrect images are the dominant failure mode, indicating that correct visual output does not imply correct spatial reasoning.

Current models remain far behind human spatial intelligence, with the best text-answering model reaching 61.04 overall and the best visual-answering model 54.49, compared to the human score of 87.79. The largest capability deficit is in Understanding, and the widest task-level gaps occur on Perspective, Relationship, and Mental Modeling, highlighting weaknesses in viewpoint transformation and relational reasoning. Spatially specialized models show isolated strengths but uneven performance across tasks, confirming that current specialization does not yet deliver general spatial competence. Human overall score (87.79) is substantially higher than the best machine scores in text (61.04) and visual (54.49) answering. Understanding shows the largest capability-level gap of 38.57 points between human performance and the best machine score. Task-level deficits are widest on Perspective (51.43 points), Relationship (31.43), and Mental Modeling (28.57), indicating that transforming spatial representations across viewpoints is a principal bottleneck. RoboBrain2.5-8B achieves 88.57 on Size but drops sharply on Orientation, Perspective, Navigation, and Trajectory, reflecting uneven spatial specialization. Among non-correct outcomes, 88.03% are valid predictions that are spatially incorrect, demonstrating that successful image generation does not guarantee accurate spatial reasoning. Non-answer failures cluster on Size and Feasibility (pre-rendering scene preservation issues) and on Prediction and Grounding (answer recovery and parsing difficulties), while Depth incurs no parser failures.

Visual and text answering modes offer complementary strengths, each solving items the other misses. The visual interface recovers a significant share of text-incorrect items, with visual rescue rates of 37.0% for the GPT pair and 30.6% for SenseNova, while text-only successes remain frequent, demonstrating that neither modality dominates spatial reasoning tasks. GPT Image 2 rescued 37.0% of items that GPT-5.4 answered incorrectly, and SenseNova's visual mode rescued 30.6% of its text-mode failures. Text-only wins were most common for SenseNova (30.2% of items), indicating tasks where pixel output alone cannot replace symbolic transformation.

Text-based GPT-5.4 achieves the highest average (60.67%) across the six benchmarks, but visual GPT Image 2 dominates SAT and RoboAfford. Qwen3-VL-8B leads on EmbSpatial, OmniSpatial, and Q-Spatial, though no model excels uniformly. Failure analysis shows that most errors are valid but incorrect spatial predictions, and visual output is most reliable when the spatial state can be directly inspected without hidden transformations or precise coordinate recovery. GPT-5.4 attains the best overall average (60.67%) but is outperformed by visual GPT Image 2 on SAT (74% vs 66%) and RoboAfford (64% vs 48%). Across tasks, 88% of non‑correct outcomes yield valid outputs that fail on spatial correctness; visual output excels when answers map to inspectable image states, but becomes fragile when tasks require hidden geometric transformations, counterfactual placements, or precise coordinate encoding.

Replacing a task-specific parser with a single VLM universal parser changes both scores and rankings of image generation models. Rank correlations with the original parser range from 0.31 to 0.60, and mean absolute score changes exceed five points, while valid-parse rates remain high. This indicates that a universal parser adds a model-dependent interpretation layer, not a uniform calibration offset. The top-ranked model depends on the parser: GPT Image 2 leads under ProVisE, JoyAI-Image leads under Qwen2.5-VL-72B, and Seedream 4.5 leads under Llama 4 Scout. Parser substitution causes sizable and non-uniform score changes, with mean absolute differences of 5 to 6 points, so the effect is not a simple constant shift across all models.

The evaluation framework introduces hierarchical spatial reasoning tasks across text-answering and image-generation models, supporting multiple answer formats and shared metrics to isolate failures in reasoning versus output modality. Results show that all models fall far behind human spatial intelligence, with the largest deficits in viewpoint transformation and relational reasoning, while visual and text modes offer complementary strengths: visual output can rescue many text failures, but the majority of generative errors are valid images that are spatially incorrect, especially when tasks demand hidden transformations or precise coordinate recovery. Replacing task-specific parsers with a universal VLM parser shifts scores and rankings significantly, revealing that the parsing layer adds a model-dependent interpretation rather than a uniform correction.