One-sentence Summary

Researchers at Rice University and NVIDIA propose Sensorimotor Alignment $( \mathsf { S } ^ { 2 } )$(S2), a training-free offline metric that measures how well a model’s sensory representation aligns with the geometry of physical motion, correlates positively with downstream robot task success, and enables screening of vision-language model (VLM) backbones before costly fine-tuning into vision-language-action models (VLAs), thereby extending the Platonic Representation Hypothesis (PRH) to robotics.

Key Contributions

• Introduces Sensorimotor Alignment (S^2), a training-free offline metric of alignment between sensory representations and end-effector trajectories, extending the Platonic Representation Hypothesis to robotics.
• Across VLAs, raw VLMs, WAMs in simulation, and real-robot policies, S^2 correlates positively with downstream task success, serving as a low-cost offline proxy for prioritizing models and identifying promising VLM backbones before costly fine-tuning or closed-loop evaluation.
• S^2 correlates more strongly with task success than VQA scores and video-quality metrics, and provides representation-level explanations for VLA and WAM phenomena including modality dominance, the performance drop from freezing the vision encoder, the small task-success cost of skipping iterative denoising, and the importance of mid-depth blocks for semantic reasoning.

Introduction

Robot foundation models increasingly build on large pretrained vision-language models, video models, and world action models, but selecting among them still relies on slow, noisy, and benchmark-dependent closed-loop rollouts. Existing offline proxies such as visual question answering or video-quality metrics measure passive perception or generation quality rather than whether a sensory representation is organized for action, and prior representation studies in robotics have mostly examined vision encoders without accounting for executed trajectories. The authors address this gap by extending the Platonic Representation Hypothesis to embodied models and proposing Sensorimotor Alignment (S^2), a training-free offline metric that measures whether episodes nearby in sensory feature space are also nearby in motor trajectory space. Across simulated and real-robot settings, S^2 correlates more strongly with downstream task success than VQA or video-quality metrics and also serves as a representation-level diagnostic for VLA and WAM behavior.

Method

The authors propose a metric called Sensorimotor Alignment ($S^2$S2) to evaluate embodied models, motivated by the Platonic Representation Hypothesis. They hypothesize that more capable models exhibit closer alignment between their sensory representations (image and text features) and motor representations (end-effector trajectories). To make this alignment measurable, they construct both representations and compare their neighborhoods using a reference dataset of paired sensory inputs and physical motions.

For each sampled episode $i$i, the authors form a sensory input $\mathbf{x}_i$xi​ by pairing a randomly sampled image frame with the task instruction. They encode this input using the vision and text encoders of a robot foundation model to obtain a sensory embedding:

where $f$f applies the respective encoders and average-pools the output token features into a single $d$d-dimensional vector.

Simultaneously, they extract the end-effector positions, orientations, and gripper states over the next $h$h steps from the sampled frame to form the motor representation:

where the variables represent position coordinates, roll, pitch, and yaw angles, and the gripper coordinate. Positions are encoded as vectors in $\mathbb{R}^3$R3 and orientation angles as rotations in SO(3) to ensure distances reflect the geometry of translations and rotations.

To measure the alignment, the authors employ a mutual nearest-neighbor metric that compares the $k$k-nearest-neighbor sets in both sensory and motor spaces. They utilize cosine distance in the sensory space and Dynamic Time Warping (DTW) in the motor space to accommodate trajectories at varying speeds. The distances between episodes $i$i and $j$j are defined as:

Using these distance metrics, they form the sensory $k$k-nearest-neighbor set $\mathcal{N}_{\text{sensory}}(i)$Nsensory​(i) and the motor $k$k-nearest-neighbor set $\mathcal{N}_{\text{motor}}(i)$Nmotor​(i) for each episode. The alignment score is calculated by taking the intersection of these two sets, dividing its size by $k$k, and averaging over all $N$N episodes:

The authors visualize this neighborhood comparison process to illustrate how shared neighbors reflect alignment. As shown in the figure below:

A greater overlap indicates that episodes with similar sensory representations also follow similar motor trajectories. In practice, the reported $S^2$S2 metric utilizes a centered, normalized variant of this nearest-neighbor approach with $k = 10$k=10 neighbors.

Experiment

The experiments evaluate sensorimotor alignment, a metric comparing sensory and motor episode neighborhoods, as an offline predictor of downstream robot manipulation success. Across vision-language-action models, vision-language model backbones, world action models, and real-robot policies, higher alignment consistently corresponds to higher task success, and it tracks success more reliably than VQA or video-quality metrics while remaining stable across datasets, horizons, and evaluation choices. A mismatched-embodiment control weakens the correlation, supporting the metric as embodiment-sensitive. Representation-level analyses further show that fine-tuned policies rely mainly on visual rather than language features for motion, that freezing vision encoders limits alignment gains, and that early denoising features and mid-depth video backbone blocks already carry the strongest sensorimotor alignment.