One-sentence Summary

Researchers from Zhejiang University and SAP propose Spatial-Interactor, a framework that trains vision-language models to model physical-world state transitions through interaction via a three-level curriculum covering passive world-state transitions, active self-state transitions, and long-horizon interaction trajectories; using the LSI-108K dataset, it combines supervised fine-tuning for local transition modeling with on-policy privileged self-distillation to improve long-horizon integration, producing consistent gains across multiple VLMs and spatial benchmarks.

Key Contributions

• The paper introduces Spatial-Interactor, an interaction-centric framework that trains vision-language models to model physical-world state transitions by using interaction trajectories as direct supervision for local transition modeling and long-horizon integration.
• The paper constructs the Learning from Spatial Interaction dataset (LSI-108K), a three-level curriculum of 108K verifiable QA pairs synthesized from simulated interactions and real-world trajectories, covering passive world-state transitions, active self-state transitions, and long-horizon interaction trajectories.
• A two-stage training recipe combines supervised fine-tuning on L1 and L2 tasks for local state-transition modeling with on-policy distillation using privileged segment-level transition descriptions for L3 long-horizon integration. Experiments across multiple vision-language models and spatial reasoning benchmarks show consistent improvements, including overall gains of 16.4 to 25.0 points and cross-benchmark gains of 4.3 to 10.6 points.

Introduction

Vision-language models have advanced substantially on digital-world tasks such as image captioning and visual understanding, but they still struggle in dynamic three-dimensional environments where spatial reasoning requires tracking how observations change across viewpoints and time. The authors show that existing models often rely on order-invariant visual cues, fail to model local state transitions, and cannot integrate consecutive changes over long trajectories, partly because conventional spatial training focuses on static question answering about object attributes and relations. To address this, they introduce Spatial-Interactor, a post-training framework that uses interaction trajectories of the form observation, action, next observation as direct supervision. It includes LSI-108K, a three-level curriculum of 108K verifiable QA pairs covering passive world changes, active ego-motion changes, and long-horizon trajectories, combined with supervised fine-tuning for local transitions and on-policy distillation for long-horizon integration.

Dataset

The authors introduce LSI-108K as a spatial interaction reasoning dataset built from a progressive curriculum.

Dataset composition and sources

• Simulated agents explore reachable scenes and execute camera motions or object operations.
• Real trajectories provide camera poses, robot end-effector states, and object tracks.
• Each record couples ordered observations with an executed or geometrically measured state change.

Subsets and curriculum levels

• L1, passive world-state transitions: covers external-world changes under a stable viewpoint, including object displacement, attribute and articulation changes, occlusion, visibility, relative configuration, single-step and multi-step operations, order, and outcomes.
• L2, active self-state transitions: holds the environment stable while camera translation, rotation, and elevation change the observation; includes ego-motion inference, magnitude comparison, composition, temporal ordering, cross-view correspondence, anchor-based localization, parallax, visibility, and post-motion relation prediction.
• L3, long-horizon transition integration: uses complete camera trajectories; global tasks recover path length, endpoint displacement, and trajectory shape; keynode tasks identify turning intervals and revisited locations and support reverse-path reasoning.

Processing and metadata construction

• State, pose, and trajectory metadata determine the targets.
• Deterministic templates render targets as QA pairs.
• Filtering removes negligible transitions, discontinuous or redundant trajectories, and ambiguous targets.
• A frozen visual-language verifier rejects visually unanswerable instances without generating or revising their ground truth.

How the paper uses the data

• The curriculum enables multi-scale reuse of each interaction record.
• Local windows and ordered action sequences support L1 and L2.
• Complete camera trajectories support L3.
• Multiple complementary questions can be generated from the same verified record, expanding task diversity and reasoning horizon without separate collection pipelines.
• The automatic process produces LSI-108K, with composition and representative tasks shown in Figures 4 and 5.
• The provided excerpt does not specify exact training splits, mixture ratios, or a cropping strategy.

Method

The authors propose a framework for dynamic spatial reasoning grounded in state-transition learning from spatial interaction. In a dynamic environment, visual observations $O_t$Ot​ depend on the world state $W_t$Wt​ and the observer pose $E_t$Et​. To model this, the authors represent interaction trajectories as ordered sequences of local interaction records $\mathcal{T} = (\delta_1, \dots, \delta_{T-1})$T=(δ1​,…,δT−1​), where each record $\delta_t = (O_t, a_t, O_{t+1})$δt​=(Ot​,at​,Ot+1​) aligns consecutive observations with the physical change $a_t$at​ (such as object operations, environmental events, or camera motion). This structure allows the model to learn local transition inference from individual records and trajectory-level reasoning from composed sequences.

To systematically train the model, the authors design a Progressive Spatial Interaction Curriculum that organizes supervision by the state being updated and the temporal horizon. The curriculum is instantiated from complementary simulated and real interaction records, converting executed interactions and geometric trajectories into verifiable transition targets and QA pairs. Refer to the framework diagram for the complete data construction pipeline.

This low-cost automatic process involves removing negligible transitions and using a frozen visual-language verifier to reject visually unanswerable instances, ultimately producing the LSI-108K dataset.

The curriculum is divided into three levels:

• L1 (Passive world-state transitions): Explains external-world changes under a stable viewpoint, covering scene-state transitions like object displacement and occlusion.
• L2 (Active self-state transitions): Holds the environment stable and attributes visual changes to camera motion, combining ego-motion understanding with cross-view spatial inference.
• L3 (Long-horizon transition integration): Extends supervision to complete camera trajectories, requiring the model to preserve the order of intermediate updates and accumulate distributed evidence. As shown in the figure below:

The composition of the dataset and representative QA examples across these three levels are illustrated here.

For local state-transition modeling (L1 and L2), the authors employ standard supervised fine-tuning. Given visual input $V$V, question $q$q, and target response $y^\star$y⋆, the model predicts $y^\star$y⋆ conditioned on observations before and after a transition. This joint supervision links visual differences to their physical causes and resulting configurations, initializing the representations needed for long-horizon learning.

To address the challenge of ordering and accumulating transitions over long trajectories (L3), the authors introduce On-Policy Distillation (OPD). Recognizing that final-answer supervision cannot identify which intermediate update failed, OPD augments verifiable GRPO with a privileged state-transition trace available only during training. For each long video, a frozen visual-language annotator generates descriptions of the environment and changes across uniformly sampled intervals, forming a privileged trace $\boldsymbol{z} = (z_1, z_2, z_3, z_4)$z=(z1​,z2​,z3​,z4​).

During on-policy same-prefix distillation, the behavior policy samples a group of responses. For every response, both the student and a stop-gradient snapshot of the policy (the teacher) evaluate the same student-generated prefix $y_{<t}$y<t​. The student conditions on the standard input $x = (V, q)$x=(V,q), while the teacher additionally conditions on $\boldsymbol{z}$z. This isolates how privileged transition evidence alters the next reasoning step. The process loss $\mathcal{L}_{\mathrm{process}}$Lprocess​ is defined as the teacher-to-student forward KL divergence on the teacher's top-k non-special-token support, applied only to reasoning positions.

The complete objective combines outcome and process optimization:

$$\mathcal{L}_{\mathrm{OPD}} = \mathcal{L}_{\mathrm{answer}} + \beta \mathcal{L}_{\mathrm{KL-ref}} + \lambda_t \mathcal{L}_{\mathrm{process}}$$LOPD​=Lanswer​+βLKL−ref​+λt​Lprocess​

where $\mathcal{L}_{\mathrm{answer}}$Lanswer​ is the clipped group-relative policy loss based on deterministic answer rewards (exact matching or relative-error scores), $\beta$β controls reference-policy regularization from a separate frozen SFT policy, and $\lambda_t$λt​ controls the privileged process signal. The process weight $\lambda_t$λt​ is gradually decayed, allowing early updates to receive explicit transition guidance while later optimization relies on the policy's own reasoning and verifiable outcomes.

Experiment

Experiments evaluate Spatial-Interactor on four Qwen2.5-VL and Qwen3-VL backbones across video and multi-view benchmarks, with additional held-out transfer and closed-loop interaction tasks. The training protocol first applies supervised fine-tuning on local transition data and public spatial QA, then adds privileged process supervision to reinforce long-horizon reasoning, while controlled ablations isolate each component. Results show consistent improvements over base models and matched baselines, with local-transition supervision improving cross-benchmark generalization and process supervision supporting trajectory-level evidence integration. Analyses of training dynamics and temporal order further indicate that the models learn to rely on ordered visual changes and reasoning traces, and qualitative examples highlight stronger cross-view correspondence and long-interval motion interpretation.

Spatial-Interactor improves over its base models across all evaluated spatial reasoning benchmarks, with overall gains ranging from 16.4 to 25.0 points. The benefits appear across Qwen generations and model scales, covering local spatial relations, route planning, camera displacement, and multi-view questions. Among reported systems, its variants lead on VSI-Bench, MindCube, and VSTI-Bench, while SpatialLadder remains strongest on SPBench-MV. Learning from interaction trajectories yields consistent overall improvements across different model capacities and starting performance levels. Relative distance accuracy improves substantially, route planning shows more modest gains, and camera displacement improves across a wider range. MindCube and SPBench-MV also benefit, indicating gains on multi-view correspondence and reasoning beyond single-view settings. The 8B Spatial-Interactor model records the highest overall score among compared systems.

The combined training stages improve cross-benchmark accuracy across the evaluated backbones and benchmarks, with the full SFT+OPD checkpoint achieving the best overall result for each backbone. Gains appear on both multi-view correspondence tasks and interaction-oriented tasks, with SFT accounting for most of the improvement and OPD adding smaller but consistent further gains. SFT accounts for most of the cross-benchmark improvement, with particularly strong gains on SAT-Real for the Qwen2.5-VL-7B backbone. OPD adds further consistent accuracy over SFT across the reported backbones, suggesting transfer beyond the long-horizon tasks used during post-training.

Training on external spatial QA data yields the largest single-stage overall improvement and establishes a much stronger spatial reasoning baseline. Adding interaction-derived transition supervision further improves all benchmark averages, and the final OPD stage achieves the best overall and metric-level performance. The largest gains appear on multi-view, displacement, and relative-distance benchmarks, indicating broad benefits for spatial reasoning. External-data SFT produces the largest overall gain among the training stages, lifting the model well above the base result. Adding interaction-derived L1 and L2 data improves every benchmark average, with consistent gains across video and multi-view tasks. The final Full SFT + OPD configuration achieves the highest overall score and the largest absolute improvements on multi-view and displacement metrics.

Spatial-Interactor improves closed-loop success on WalkerBench relative to the baseline, with clearer gains in angle, height, and navigation tasks. It also reaches these results using substantially fewer executed actions per task. Distance and visibility tasks remain unsolved for both models. Spatial-Interactor shows relative gains in angle, height, navigation, and overall success while using far fewer actions per task. Distance and visibility tasks show no success for either the baseline or Spatial-Interactor.

Spatial-Interactor outperforms the base model on all eight closed-loop ESI-Bench categories. Overall success increases from about 33% to about 41%, and solved episodes require fewer interaction rounds on average. The improvements are broad, with relatively large gains in agent observation, stability, and occlusion. Spatial-Interactor improves success across every reported category, including view, distance, and action order. Solved episodes require fewer interaction rounds on average, falling from roughly 12.3 to 10.6 rounds. The largest gains appear in agent observation, stability, and occlusion, indicating stronger closed-loop reasoning under changing observations.

The experiments evaluate Spatial-Interactor on open-ended spatial reasoning benchmarks, multi-view and interaction-oriented tasks, and closed-loop embodied benchmarks such as WalkerBench and ESI-Bench. Across these settings, the model improves over its base backbones, with the full SFT plus OPD training pipeline providing the largest and most consistent gains, especially in multi-view, camera displacement, relative distance, and agent observation. In closed-loop evaluations, Spatial-Interactor achieves higher success rates while requiring fewer executed actions or interaction rounds, though distance and visibility tasks remain unsolved. Overall, the results show that interaction-derived supervision and staged post-training generalize across model scales and task formats.