One-sentence Summary

Researchers from The Hong Kong University of Science and Technology (Guangzhou), OLA Dimensions, Beijing Academy of Artificial Intelligence, and Tsinghua University propose SLA, a Spatial Language-Action Model that scales implicit spatial alignment to continual pre-training from a pretrained π0.5 policy, using feature-dependent gates and probabilistic state dropout to achieve state-of-the-art performance on RoboDojo-Real, RoboTwin 2.0 under the Clean2Random setting, RoboCasa-Panda, and LIBERO-Plus, along with a 24.0 percentage point improvement in average real-world success rate over π0.5.

Key Contributions

• SLA is a Spatial Language-Action Model that initializes from a pretrained π0.5 policy and jointly optimizes action prediction and alignment between intermediate visual embeddings and spatial representations from a frozen 3D perception model during continual pre-training and downstream post-training.
• A unified heterogeneous robot dataset supports spatial learning across diverse embodiments, tasks, and scenes, with feature-dependent gates that adaptively aggregate visual embeddings from multiple VLA layers for spatial supervision. The teacher exploits partially available depth and camera calibration alongside RGB, and probabilistic state dropout encourages reliance on visual cues and reduces overfitting to demonstration trajectories.
• All auxiliary components are used only during training, allowing direct reuse of pretrained VLA weights and preserving the original inference architecture. Evaluations report state-of-the-art performance on RoboDojo-Real, RoboTwin 2.0 under the Clean2Random setting, RoboCasa-Panda, and LIBERO-Plus, with the real-world average success rate over π0.5 improving by 24.0 percentage points across eight deployment conditions.

Introduction

Robotic manipulation requires precise spatial reasoning and physical control in 3D environments, not just recognizing objects and following instructions. Vision-language-action models built from pretrained 2D vision-language models are promising general-purpose policies, but their 2D-centric pretraining provides limited 3D spatial understanding, and prior attempts to add depth or point-cloud inputs during post-training face noisy sensors, heterogeneous robot hardware, and missing depth in many datasets. Even implicit spatial alignment methods such as Spatial Forcing have remained post-training only, leaving spatial competence as a task-specific add-on rather than a transferable foundation capability. The authors introduce SLA, a spatial-language-action model that develops generalized 3D spatial understanding by aligning intermediate visual embeddings of an existing VLA with spatial representations from a frozen 3D perception foundation model during both continual pre-training and downstream post-training. This supervision preserves the original VLA architecture and action head, allowing the model to reuse pretrained π0.5 knowledge while learning richer geometric representations for precise manipulation.

Dataset

Dataset overview

The authors build a large-scale corpus for continual pre-training of vision-language-action models. It combines six source datasets, totaling approximately 1.7 million episodes and 14,330 hours of demonstrations. All sources pass through a unified data curation and preprocessing pipeline so robot state and action annotations remain consistent across embodiments, tasks, and environments.

Sources and composition

• RoboCOIN: real-world bimanual manipulation demonstrations from heterogeneous robots, adding embodiment and observation diversity.
• AgiBot World Beta: large real-world manipulation collection; the authors retain two-finger-gripper demonstrations without mobile-base motion.
• DROID: Franka robot demonstrations in varied real-world settings; included for scarce single-arm real-robot experience, with some variation in quality and annotation precision.
• InternData-A1: synthetic scripted-policy trajectories; the authors use Split ALOHA, Lift2, and Franka subsets for broader simulated scene, asset, and texture coverage.
• MolmoAct: real-world Franka demonstrations focused on specific tasks and scenes, adding further single-arm real-robot experience.
• RoboCasa365: simulated kitchen household tasks; the authors use pre-training atomic, composite, and MimicGen subsets after removing trajectories with base motion, and they do not use post-train subsets.
• Downstream benchmark data are excluded from the continual pre-training corpus to prevent evaluation leakage.

Processing and annotation schema

• Instead of raw joint coordinates, the authors use end-effector pose representations.
• Observations are absolute EEF states.
• Actions are chunk-wise delta EEF actions.
• All poses are mapped into a consistent robot-centered EEF coordinate frame. The origin is the arm base for single-arm robots and the midpoint between the two arm bases for bimanual robots. Axes are +x forward, +y left, +z upward.
• Orientation uses rotation vectors. Relative rotations are computed through SO(3) composition:
  • Δp_t = p_t - p_0
  • ΔR_t = R_t R_0^T
  • Δr_t = Log(ΔR_t)^∨
• All actions in a chunk use the same current pose as the reference, not the preceding action.
• Rotation-vector channels are normalized channelwise.

Downsampling and filtering

• The authors reduce redundant trajectories by subsampling complete episodes within tasks, keeping the original temporal sampling rate of retained trajectories.
• InternData-A1 retention rates: 3% for selected highly repetitive tasks and 10% for other tasks.
• RoboCasa365 pre-training retention rates: 0.5% for MimicGen episodes and 15% for composite episodes.
• These source-dependent and task-dependent rates bring episode counts closer in scale.
• The authors keep only two-finger-gripper manipulation, excluding dexterous-hand data and trajectories with robot-base motion.
• Action statistics are used to remove anomalous arm-motion speeds.
• When available annotations identify idle frames, frames where both robot and scene remain stationary are removed.

How the data is used

The processed corpus is used for continual pre-training of the VLA model. The provided section does not specify a fixed final mixture ratio across all six datasets. Instead, the authors balance the sources through filtering and source-dependent, task-dependent downsampling, while excluding downstream benchmark data to keep evaluation valid.

Method

The authors propose SLA, a framework designed to develop a scalable 3D Vision-Language-Action foundation model through continual pre-training. Starting from a pretrained 2D VLA backbone, the method leverages external spatial representation supervision to shape the model's visual representations while preserving its original backbone and action-generation architecture.

To provide rich spatial information, the authors input a set of multi-view images into a frozen external 3D perception foundation model. This model outputs pixel-level 3D representations, which are augmented with positional embeddings to preserve critical spatial position order. The final-layer representations are bilinearly resampled to the VLA token resolution to obtain a spatial target at each visual token location:

where $\mathcal{R}[\cdot]$R[⋅] denotes the bilinear resampling operation.

To align the VLA per-pixel visual tokens with these external 3D signals, the authors process each visual token with layer normalization followed by a two-layer MLP. Since the VLA contains multiple causal attention layers, the method adaptively aggregates the projected features across all layers into a unified representation $\mathbf{z}$z:

where $L$L is the number of VLA attention layers and $\odot$⊙ denotes element-wise multiplication. A cosine similarity score is then employed to maximize the alignment between the VLA visual tokens and the external spatial representation signals:

The final training objective combines the standard action generation loss with this 3D alignment loss:

where $\alpha$α controls the spatial alignment strength. During inference, the auxiliary components are discarded, ensuring no additional computational overhead.

The external 3D perception model is built to accommodate uneven geometric annotations across the training corpus. It utilizes a GeoAdapter to encode available camera intrinsics, extrinsics, and depth maps, employing placeholder tokens when annotations are missing. This stochastic multimodal fusion ensures the backbone features contain sufficiently rich spatial information.

To handle the heterogeneous pre-training corpus, the authors implement an adaptive layer selection strategy. Instead of fixing a single alignment layer, they learn a feature-dependent gate for every visual token at each VLA transformer layer:

where $W _ { g }$Wg​ and $\mathbf { b } _ { g }$bg​ are the learnable weight matrix and bias vector, respectively, and $\sigma$σ is the element-wise sigmoid. These gates weight the projected features and sum them across layers, allowing different visual cues to contribute to a unified spatial representation without adding policy computation at deployment. Additionally, a state dropout mechanism is applied during training to randomly omit state-token segments, preventing the policy from relying on proprioceptive shortcuts and forcing it to encode scene geometry.

To ensure consistent control representations across different robot morphologies, the authors express actions through end-effector poses rather than joint coordinates. All poses are mapped to a unified robot-centered coordinate frame. The choice of rotation format is critical for consistent action targets. Directly subtracting Euler angles as delta actions can convert small physical rotations across an angular wrap boundary into large numerical displacements. As shown in the figure below, such wrap-induced outliers severely compress the range of typical action values after normalization, hindering training.

To resolve this, the authors represent orientation with rotation vectors and compute relative rotations through SO(3) composition. For a given current absolute end-effector pose $(\mathbf{p}_0, R_0)$(p0​,R0​) and a target pose $(\mathbf{p}_t, R_t)$(pt​,Rt​), the delta position, relative rotation matrix, and rotation vector are computed as:

where $\mathrm{Log}(\cdot)^\vee$Log(⋅)∨ maps a relative rotation matrix to its rotation-vector representation. This approach ensures that the rotation vector maintains a unified physical meaning through its direction and magnitude. The data pipeline also incorporates downsampling to reduce redundancy from highly repetitive scripted simulation data, alongside filtering to exclude anomalous arm-motion speeds and idle frames.

Experiment

The experiments evaluate SLA by continually pre-training from a pretrained π0.5 policy and post-training on three simulation benchmarks, one real-world benchmark, and multiple real-world deployments that cover visual and language perturbations, scene variation, bimanual coordination, and cross-embodiment manipulation. Across these settings, SLA consistently improves over π0.5, Spatial Forcing, and other state-of-the-art methods, with particularly strong gains in Clean2Random generalization, multi-embodiment adaptation, and challenging real-world conditions such as moving targets and unseen objects. Ablations further validate that applying spatial supervision during both continual pre-training and post-training, using adaptive layer fusion, moderate state dropout, rotation vectors, and camera calibration for the teacher all contribute to the best performance.

Before data selection and processing, the six source datasets collectively contain approximately 1.7 million episodes and 14,330 hours of demonstrations. InternData-A1 and RoboCasa365-Pretrain are the largest contributors by episode volume, while RoboCOIN has the highest task count and MolmoAct is the smallest source overall. The sources span heterogeneous real-world platforms, including bimanual and single-arm manipulation settings. InternData-A1 and RoboCasa365-Pretrain are the largest sources by episode count, far ahead of the other datasets before processing. RoboCOIN contributes the greatest task diversity among the sources, while MolmoAct has the fewest episodes, tasks, and hours. The combined sources provide roughly 14,330 hours of demonstrations, with large-scale manipulation datasets accounting for most of the volume. DROID and MolmoAct represent much smaller demonstration pools compared with the largest datasets.

SLA reports the strongest weighted LIBERO-Plus success rate among the compared policies, outperforming the reported baselines and other listed models. Its gains are broad rather than isolated, appearing across all seven perturbation dimensions evaluated in the benchmark. Improvements are especially visible under challenging visual conditions such as observation noise and background changes. SLA leads the compared methods with a weighted success rate of 87.4%, above the main baseline by 3.0 percentage points and above Spatial Forcing by 2.3 percentage points. SLA improves over the baseline in every perturbation dimension and records the highest lighting perturbation score among compared methods, with clear gains in observation noise and background changes.

SLA reports a 79.2% average success rate on the RoboTwin 2.0 benchmark. Its advantage is strongest in the Clean2Random setting, where it outperforms π0.5 by a wide margin and improves over the strongest average competitor, while trailing that competitor on Clean2Clean. This indicates the main benefit is robustness to domain randomization rather than performance under training-similar conditions. The Clean2Random setting is described as a more demanding test of robustness to scene variation than Clean2Clean. SLA shows its largest relative gain over π0.5 in Clean2Random, with smaller gains in Clean2Clean and average success. Compared with the strongest average competitor, SLA improves Clean2Random success but trails in Clean2Clean, suggesting targeted improvement in generalization.

SLA reports the highest mean success rate on the RoboCasa-Panda benchmark among the compared methods, narrowly ahead of Cosmos Policy and Spatial Forcing under the same limited post-training demonstration budget. Some baselines use larger demonstration budgets but still achieve lower average success. SLA also shows strong task-level results, exceeding 80% on 10 of 24 tasks. SLA has the highest average success rate among compared methods, with a narrow lead over Cosmos Policy and Spatial Forcing at the same demonstration budget. SLA exceeds 80% success on 10 of 24 tasks, including perfect success on TurnOfMicrowave.

SLA achieves the highest overall score and success rate on the RoboDojo real-world benchmark, outperforming prior methods including π0 and the strongest competitor InternW0-∆. Its advantage is especially clear on Piper, where it ranks first in both metrics, while it ranks second on ARX X5. These results support the effectiveness of joint multi-embodiment adaptation with the proposed training approach. SLA leads all compared methods in average score and success rate, surpassing π0 by 19.4 score points and 21.1 percentage points. It outperforms the strongest competing method, InternW0-∆, by 3.4 score points and 2.2 percentage points overall. SLA ranks first on Piper in both score and success rate, and ranks second on ARX X5 behind InternW0-∆.

The evaluation first analyzes six heterogeneous demonstration sources totaling about 1.7 million episodes and 14,330 hours, confirming that large manipulation datasets dominate volume while one source contributes high task diversity. SLA is then tested on LIBERO-Plus, RoboTwin 2.0, RoboCasa-Panda, and RoboDojo to validate perturbation robustness, generalization under domain randomization, limited-budget task learning, and real-world multi-embodiment adaptation. Across these benchmarks SLA consistently achieves the best or near-best results, with especially clear gains under noisy visual conditions, in the Clean2Random setting, and on real-world Piper performance.