One-sentence Summary

Researchers from ByteDance Seed, Zhejiang University, and the National University of Singapore introduce GST-Bench, a video question-answering benchmark built from $6,790$6,790 minutes of synthetic video and human-verified questions that evaluates global spatial awareness by requiring models to perform accurate spatial inference from novel viewpoints and map long-horizon egocentric observations onto global top-down images, revealing a severe gap, with the strongest zero-shot model attaining only $42.68$42.68 versus the human score of $79.08$79.08, which they investigate with GST-Bench-Local, finding that despite strong local spatial understanding under the same task formulation, models fail to consolidate long-horizon observations into a globally consistent scene representation, and they further provide GST-Train as a training resource to spur future research.

Key Contributions

• GST-Bench is a video VQA benchmark built from 6,790 minutes of synthetic video that evaluates global spatial intelligence by requiring models to answer spatial questions from novel viewpoints unseen in the input and to map egocentric observations onto top-down images.
• Evaluation of 22 state-of-the-art VLMs reveals a large performance gap: the strongest zero-shot model attains only 42.68 accuracy, far below the human score of 79.08.
• The GST-Bench-Local variant shows that models, despite strong local spatial understanding under the same task formulation, fail to consolidate long-horizon observations into a globally consistent scene representation; the accompanying GST-Train dataset is provided to support future research on this challenge.

Introduction

Vision-language models (VLMs) must build globally consistent spatial representations for embodied tasks such as a household robot retrieving an item, where the agent must integrate long egocentric video streams into a coherent understanding of the whole scene. Prior spatial benchmarks fall short: single-image or few-view evaluations probe only local properties, while existing video-based benchmarks often include questions answerable from a single frame, rely on coarse categorical metrics like left/right instead of precise numerical angles, and do not test the alignment between egocentric observations and an explicit global layout such as a top-down map. The authors address these gaps with GST-Bench, a video question-answering benchmark that enforces cross-frame global reasoning by ensuring target objects are invisible from the query viewpoint, demanding precise numerical answers (distances and angles), and introducing top-down images to assess global-local correspondence. They further contribute GST-Train, a large-scale training dataset that, through fine-tuning, lifts performance past even the strongest zero-shot proprietary models, demonstrating that the spatial reasoning gap can be systematically narrowed.

Dataset

The authors introduce GST-Bench, a benchmark for evaluating whether vision-language models (VLMs) can form globally consistent spatial representations from long egocentric videos. It is complemented by a separate training set, GST-Train, which is built from the same pipeline but uses separate indoor simulation scenes to prevent leakage. Below is a summary of dataset composition, processing, and usage.

1. Dataset sources and overall composition

• All data are generated from diverse indoor 3D simulation assets: BEHAVIOR-1K, HyperSim, ArtVIP, and additional custom scenes. This ensures wide variation in room layouts, object arrangements, and navigable structures.
• GST-Bench contains 2,762 human-verified question-answer pairs, organized into 12 task types across three core competencies: self localization, object localization, and scene structure understanding.
• GST-Train provides training examples built from held-out scenes (details of its size and mixture ratios are not specified in this excerpt).

2. Input modalities and task design The benchmark uses five visual input types to probe spatial reasoning:

• Exploration video – a long egocentric traversal of a whole scene; the primary source for spatial memory.
• Object-annotated video – the exploration video with a red bounding box overlaid on every frame where a target object appears; used for visual object-localization tasks.
• Short trajectory video – a clip from an independent exploration path; used only in the trajectory selection task.
• Top-down images – rendered at three abstraction levels:
  • Easy: photo-realistic bird’s-eye view (ceiling removed).
  • Medium: occupancy-style map (object footprints, no textures).
  • Hard: bare floor plan (only walls).
• Current view – a novel egocentric image from a viewpoint that is not on the exploration trajectory; for object-localization tasks the target object is guaranteed absent from this view, forcing cross-frame reasoning.

These inputs are combined into the following task groups:

• Self localization: predict the camera position or orientation on a medium top-down image given the exploration video and a novel current view.
• Object localization: infer a target object’s location, specified either by a category name (semantic modality) or by the object-annotated video (visual modality). Tasks include egocentric direction, egocentric distance, and global position on a medium top-down map.
• Scene structure understanding: match an exploration video to the correct top-down image among four candidates (at three difficulty levels) or match a short trajectory video to one of four candidate trajectories overlaid on a top-down map.

3. Data generation pipeline The pipeline enforces global reasoning by construction and combines automatic generation with human verification.

• Scene preparation: scenes are reserved for GST-Bench and GST-Train separately to avoid leakage.
• Exploration videos: viewpoints are sampled across the navigable area, connected into efficient traversals, and rendered as RGB videos. Multiple trajectories with varying start points are created per scene.
• Object-annotated videos: target objects are identified via instance/semantic segmentation, and bounding boxes are overlaid on all frames where the instance is visible. This allows instance-level visual target specification.
• Short trajectory videos: separately collected trajectories are segmented into short clips.
• Top-down images: three levels are rendered per scene (easy, medium, hard).
• Current views: off-trajectory viewpoints are sampled with independent orientations. A view is kept only if it has enough visual overlap with the exploration video to make localization possible; for object-localization tasks, the target object must be absent.
• QA generation and automatic filtering: answers are computed directly from simulation geometry (poses, projections, scene/trajectory identities). Filters remove samples with invalid projections, ambiguous visibility, insufficient view overlap, or degenerate choices.
• Human verification: annotators manually verify answerability: they check that the target object is identifiable in the supplied video and that the off-trajectory current view can still be localized given the exploration video. Samples failing these checks are discarded, resulting in the final 2,762-question benchmark.

4. How the data is used in the model

• The paper evaluates VLMs on GST-Bench in a zero-shot manner: models are given the task-specific inputs (videos, top-down maps, textual prompts) and must produce answers without additional fine-tuning on the benchmark itself.
• GST-Train is used as a training set (its construction follows the same pipeline, with scenes disjoint from the benchmark). The excerpt does not provide mixture ratios or training split details, but the hold-out scene principle prevents contaminating evaluation.

Method

The authors leverage an automatic data generation pipeline to construct a benchmark that enforces global spatial reasoning. As shown in the figure below:

This pipeline leverages the scalability and controllability of simulation environments, utilizing diverse indoor assets such as BEHAVIOR-1K, HyperSim, and ArtVIP. The evaluation scenes are held out from the training set to prevent scene-level leakage. The design enforces global reasoning by construction, requiring the target object to be visible in the exploration video but invisible from the current view, which is sampled outside the video trajectory.

The data collection phase generates multiple modalities to support diverse task types. Exploration videos are created by sampling spatially distributed viewpoints over navigable areas and rendering egocentric RGB videos along efficient traversal trajectories. Multiple trajectories with perturbed viewpoints and different starting locations are generated to avoid reliance on a single canonical route. For object-localization tasks, object-annotated videos are produced by overlaying red bounding boxes on visible target instances across frames, specifying targets by appearance rather than category names. Short trajectory videos are also generated as continuous clips to evaluate egocentric motion alignment with global maps. To provide explicit global scene representations, top-down images are rendered at three abstraction levels: photo-realistic bird's eye view, occupancy-style maps preserving object and wall footprints, and floor-plan images retaining only wall structures. Current views are sampled from off-trajectory viewpoints with independent camera orientations. For object-localization, the target object is strictly absent from the current view, forcing models to infer locations from prior video observations. Ground truth information, including camera poses, object poses, and visibility, is extracted directly from the scene geometry.

Following data collection, a rigorous quality control process is applied. Automatic filtering removes samples with invalid projections, ambiguous object visibility, insufficient query-view overlap, degenerate choices, or violations of the target-invisible constraint. Video filtering eliminates instances with mislabeling, incomplete appearance, or heavy occlusion, while current view filtering discards views with low information content.

Finally, the QA generation phase combines the filtered inputs with task-specific templates to produce question-answer pairs for twelve task types. Answers are computed geometrically, such as egocentric directions and distances from camera-object poses. The resulting QA data is split, and for the final benchmark, human verification is conducted. Annotators confirm that the target object is recognizable in the exploration video and that the off-trajectory current view contains sufficient overlap for localization, ensuring the benchmark is both challenging and reliably answerable.

Experiment

The evaluation benchmarks 22 vision-language models (proprietary, open-source, and embodied) on GST-Bench using zero-shot greedy decoding across 12 spatial reasoning subtasks, revealing a large performance gap: even the best proprietary model trails human performance by over 36 points, open-source models perform near random, and embodied models offer no advantage. Controlled experiments disentangling perception from cross-frame integration show that proprietary models can perceive locally but fail at integrating global spatial information, while open-source models struggle with both stages. Fine-tuning Qwen3-VL-8B on targeted GST-Train data significantly boosts its score (25.89 to 53.52), surpassing all zero-shot models yet still far from human performance, confirming that explicit supervision improves but does not fully solve long-horizon spatial reasoning.

Current vision-language models find GST-Bench highly challenging, with the top proprietary model scoring more than 36 points below human evaluators. Open-source models essentially fail beyond coarse scene selection, performing near random on most subtasks. Fine-tuning with targeted spatial supervision dramatically lifts performance above all zero-shot models, but a large gap to human spatial understanding remains. The best proprietary model scores more than 36 points below humans, with the most pronounced gaps in orientation and global position estimation. Open-source models owe their apparent lead entirely to easy and medium top-down selection subtasks; on the remaining ten tasks they stay within a few points of random guessing. Fine-tuning a model on spatial reasoning data raises its average from 25.89 to 53.52, surpassing all zero-shot proprietary models but still falling well short of human performance.

Proprietary models improve markedly when the need for cross-frame reasoning is removed, revealing a bottleneck in global spatial integration rather than local perception. The open-source Qwen3-VL-2B does not benefit from local information and performs worse than in the global setting, indicating a fundamental weakness in basic spatial understanding. GPT-5 gains +29.23 points in the local-video setting and +34.52 in the local-image setting over the global baseline, showing strong single-view perception but weak cross-frame integration. Qwen3-VL-2B degrades by -4.90 points in local-video and -4.55 in local-image compared to the global setting, failing even at simpler local spatial perception tasks.

Evaluating vision-language models on the GST-Bench spatial reasoning benchmark reveals that all models perform far below human level, with the best proprietary model lagging over 36 points behind. Removing cross-frame reasoning substantially improves proprietary models, indicating a global integration bottleneck rather than poor local perception, while an open-source model fails even at local spatial tasks. Fine-tuning with spatial supervision lifts performance above zero-shot models but still leaves a large gap to human spatial understanding.