One-sentence Summary

Researchers from MirroS, Tsinghua University, and Peking University uncover S-Space, an intrinsic linear representation subspace in multimodal language models where object-token activations encode continuous horizontal, vertical, and distance coordinates, and they demonstrate through activation gradients that this subspace serves as an internal spatial workspace and diagnostic tool, revealing that the models can form capable spatial representations but struggle with viewpoint transformations.

Key Contributions

• The paper identifies and characterizes S-Space, a three-dimensional linear subspace of intermediate multimodal language model representations that acts as an internal spatial workspace, explaining how spatial information is encoded, updated, and causally used.
• A gradient-based readout method is introduced to recover fixed left-right, above-below, and close-far axes, enabling accurate continuous coordinate readouts that generalize across datasets, tasks, and prompts and remain readable under non-spatial questions.
• S-Space is established as a diagnostic tool for spatial reasoning failures: causal interventions alter spatial judgments while largely preserving non-spatial responses, and perception-computation decomposition reveals that models can form capable spatial representations but struggle with the spatial transformations required for reliable viewpoint reasoning.

Introduction

Multimodal large language models are increasingly used for spatial intelligence tasks such as reconstruction, planning, navigation, and manipulation, but robust spatial reasoning remains a major weakness. End-to-end accuracy alone offers little insight into where these failures arise, and prior work on spatial encodings and representational biases has not provided a unified explanation of how spatial information is internally encoded, transformed, and used. The authors uncover S-Space, a three-dimensional linear subspace of intermediate representations that functions as an internal spatial workspace, with fixed axes for left-right, above-below, and close-far directions. They show that this subspace supports generalizable coordinate readouts, causally shapes spatial judgments, and can serve as a diagnostic tool by separating spatial perception from downstream reasoning failures.

Method

S-Space: Formulation, Identification, and Readout

The authors introduce S-Space, a three-dimensional linear subspace of intermediate representations in multimodal large language models (MLLMs) in which object locations are linearly encoded and can be directly read out. The approach has three stages: formulating the subspace, identifying its axes, and reading out spatial coordinates through projection.

S-Space Formulation

Large language models are widely hypothesized to represent concepts linearly in their internal representations. The authors extend this idea to spatial reasoning by hypothesizing that an object’s spatial coordinates, defined in a viewer-centered or egocentric reference frame, are linearly encoded in the hidden representation of its object-name token.

For a given model and layer $\ell$ℓ, S-Space is defined as a three-dimensional linear subspace of hidden representations

$$h_{\ell,t} \in \mathbb{R}^{d_{\mathrm{model}}}.$$hℓ,t​∈Rdmodel​.

S-Space is specific to the model and layer, but shared across all token positions $t$t within that layer. The authors search for S-Space in the model’s middle layers because early layers remain relatively anchored to input representations, while later layers become increasingly aligned with next-token prediction for verbal response generation. For simplicity, the layer index is omitted in subsequent notation.

Identifying S-Space

The authors identify three S-Space directions corresponding to the horizontal, vertical, and distance axes by differentiating the model’s responses to relative spatial questions with respect to object-token activations.

Each relative spatial question example consists of an image and a prompt comparing a query object with a reference object along one axis $g \in \{x, y, z\}$g∈{x,y,z}, for example, “Is the apple to the left or right of the cup?” Let $q$q and $r$r denote the respective token positions of the query and reference object names. The answer contrast for each axis is defined as the corresponding logit difference:

$$s_{p,x} = \mathrm{logit}(\mathrm{right}) - \mathrm{logit}(\mathrm{left}),$$sp,x​=logit(right)−logit(left), $$s_{p,y} = \mathrm{logit}(\mathrm{above}) - \mathrm{logit}(\mathrm{below}),$$sp,y​=logit(above)−logit(below), $$s_{p,z} = \mathrm{logit}(\mathrm{close}) - \mathrm{logit}(\mathrm{far}).$$sp,z​=logit(close)−logit(far).

The gradient $\nabla_{h_t} s_{p,g}$∇ht​​sp,g​ indicates the direction along which perturbing the object representation would most increase the corresponding answer contrast. The authors then form a role-antisymmetric gradient difference and average it across prompt templates, original and swapped object orders, and examples:

$$a_{p,g} = \frac{\nabla_{h_q} s_{p,g} - \nabla_{h_r} s_{p,g}}{2}, \quad v_g = \mathrm{unit}\left(\mathbb{E}_p[a_{p,g}]\right).$$ap,g​=2∇hq​​sp,g​−∇hr​​sp,g​​,vg​=unit(Ep​[ap,g​]).

The resulting unit vectors define the three S-Space directions:

$$\mathcal{S} = \mathrm{span}\{v_x, v_y, v_z\}.$$S=span{vx​,vy​,vz​}.

The role-antisymmetric difference is motivated by the fact that relative spatial judgments treat the query and reference oppositely. A change that makes the query more to the right should favor the answer “right,” while the same change applied to the reference should favor “left.” Subtracting the reference gradient from the query gradient therefore reinforces oppositely acting spatial signals. Averaging across prompt templates, object orders, and examples further suppresses effects from object identity, prompt phrasing, token roles, and scene context that are not consistently associated with the queried spatial relation.

Reading Out S-Space

Once the three axes are identified, reading out S-Space requires only a forward pass. The readout is not restricted to the spatial question-answering prompts used for identification; the same fixed directions can be applied to object-token representations elicited by other prompts and tasks.

The authors stack the three direction vectors into a matrix $V$V and project each object token $t$t to obtain its coordinate in S-Space:

$$V = [v_x, v_y, v_z] \in \mathbb{R}^{d_{\mathrm{model}} \times 3}, \qquad c_t = V^{\top} h_t \in \mathbb{R}^{3}.$$V=[vx​,vy​,vz​]∈Rdmodel​×3,ct​=V⊤ht​∈R3.

For relative spatial judgments, the query and reference coordinates are compared along the queried axis:

$$m_g = v_g^{\top}(h_q - h_r) = c_{q,g} - c_{r,g}.$$mg​=vg⊤​(hq​−hr​)=cq,g​−cr,g​.

The sign of the margin $m_g$mg​ directly selects the predicted endpoint. A positive value indicates right, above, or close, while a negative value indicates left, below, or far.

Experiment

Using relative spatial questions constructed from COCO and readouts evaluated on CV-Bench, EmbSpatial, and Spatial-Tunnel, the experiments establish S-Space as an accurate, persistent internal spatial workspace in multimodal language models; editing its coordinates causally alters spatial judgments while largely preserving non-spatial answers. Case studies suggest that S-Space emerges from spatial language prediction and can reflect relations implied by visual evidence, language, and abstract concepts. Further diagnosis with SpinBench and MMSI-Bench shows that spatial reasoning failures often stem from imperfect viewpoint transformations and coupling between cardinal and image-relative directions rather than from missing perceptual information.

Direct answer generation on SpinBench perspective-taking remains limited across all evaluated models, and providing the correct source-view relation gives only selective improvement. Chain-of-thought reasoning improves accuracy but still leaves substantial errors, especially for smaller models. Reading spatial coordinates from S-Space followed by an explicit rotation yields consistently large gains and near-perfect accuracy when the source-view relation is provided. Across models and premise settings, the S-Space plus rotation pipeline outperforms direct answering and chain-of-thought reasoning by large margins. With the correct source-view relation supplied, the S-Space pipeline reaches near-perfect accuracy, while direct and reasoning modes still leave notable errors. Larger models benefit more from chain-of-thought reasoning, but even the strongest reasoning result falls short of the explicit S-Space transformation.

The evaluation on SpinBench perspective-taking compares direct answer generation, chain-of-thought reasoning, and an S-Space pipeline that reads spatial coordinates and applies an explicit rotation. Direct answering remains limited across all models and improves only selectively when the correct source-view relation is provided, while chain-of-thought reasoning reduces errors but still leaves substantial gaps, especially for smaller models. The S-Space plus rotation approach consistently yields large gains and near-perfect accuracy when the source-view relation is available, outperforming the other modes by wide margins across models and premise settings. Even the strongest chain-of-thought results from larger models fall short of the explicit S-Space transformation.