One-sentence Summary

Researchers from S-Lab, Nanyang Technological University, SenseTime Research, and Ropedia introduce NEO-VLA, a structurally encoder-free vision-language-action model in which patch-convolutional image tokens evolve jointly with language, enabling a flow-matching action expert to draw on instruction-conditioned visual representations at multiple depths and thereby achieve stronger, more robust manipulation control than encoder-based policies.

Key Contributions

• NEO-VLA is introduced as a structurally encoder-free vision-language-action framework: images are tokenized by lightweight patch convolutional layers and evolve jointly with language through the backbone, with a flow-matching action expert reading instruction-conditioned visual representations at multiple depths.
• In controlled comparisons on LIBERO, LIBERO-plus, and RoboTwin 2.0, native architectures outperform encoder-based counterparts under a matched action expert and training corpus; full fine-tuning reduces but does not erase the gap, and scale alone cannot recover it.
• Layer-wise probing shows action-critical visual signals often peak before the final visual representation, and distribution-shift evaluations show NEO-VLA achieves stronger, more robust control across diverse manipulation settings and observation changes.

Introduction

Robotic manipulation policies must combine semantic understanding with fine-grained spatial reasoning, but most vision-language-action models attach action heads to features from separately pretrained vision encoders, placing the policy downstream of a fixed visual endpoint and hiding intermediate visual computation. Prior work suggests this visual front end can constrain control because visual components are sensitive to pruning and action fine-tuning can degrade pretrained visual representations. The authors introduce NEO-VLA, a structurally native VLA that maps raw pixels directly into a shared vision-language backbone and uses a flow-matching action expert to read cached features from different backbone layers, making visual readout depth an explicit policy choice. They show that early-layer and combined early-late readouts are especially useful for control and that NEO-VLA remains robust across manipulation benchmarks and distribution shifts.

Dataset

The authors build the pretrain data from nine sources: AgiBot-World, Bimanual YAM, RoboCOIN, Open X-Embodiment, Galaxea Open-World, RH20T, RoboMIND, InternData-A1, and DROID. Together, these datasets cover diverse embodiments, tasks, and visual conditions.

Dataset filtering and processing:

• Source checks exclude unreliable measurements and corrupt episodes.
• Joint readings outside ±2π, or more than π from a five-row local median, are marked invalid.
• Joint and hand channels that stay constant in at least 90% of a dataset’s recorded episodes are masked.
• Unresolved joint discontinuities larger than π are masked for the corresponding embodiment.
• Action outliers are checked against expanded per-embodiment bounds. Out-of-band or non-finite values mask only that field group at that step, leaving other targets available.
• Episodes are dropped when data is missing or unreadable, record shapes are inconsistent, values are invalid, or the episode has too few steps.
• Camera calibration is checked where required signals are available. Recorded 3D arm positions are projected using intrinsics and full camera pose. Image motion at projected points is scored against overall image motion and compared with wrong-pose controls. Untrusted poses are marked unknown while their episodes remain.
• Intrinsics checks cover focal length, principal point, field of view, and consistency with the stored image size.

Role in model training:

• The filtered datasets form the pretrain data presented in Table 1.
• The excerpt refers to Table 1 for the per-source composition and training mixture details.
• The processing pipeline preserves usable targets by masking invalid joint, hand, action, or camera-pose fields where possible, while dropping fully unusable episodes.

Method

The authors leverage a structurally native visual interface that projects raw images directly into the backbone token space through a lightweight two-stage convolutional embedder with an overall stride of 32. This produces one visual token for each $32 \times 32$32×32 image region. Visual and language tokens are jointly processed throughout the decoder backbone, while continuous actions are generated by a flow-matching expert, avoiding explicit action tokenization. To handle spatial and temporal dimensions, the model employs Native Rotary Position Embedding to factorize positions into temporal, height, and width coordinates with decoupled rotary frequencies. This enables a unified treatment of images, videos, and interleaved multimodal sequences. The backbone further adopts modality-aware multimodal attention, where text tokens follow causal attention and visual tokens interact bidirectionally within each visual block.

As shown in the figure below, the model converts robot observations into a unified multimodal prefix composed of previous and current multi-view images, structured language, and robot states.

Each temporal block contains up to three camera views, represented by a camera-identity token, projected camera geometry, and patch-level visual tokens. Camera intrinsics and rotations are encoded by lightweight geometry MLPs, while a single patch embedder is shared across all cameras, views, and time steps. Images preserve their native aspect ratios and are dynamically patchified under a fixed budget of up to 88 visual tokens per view. The visual blocks are followed by a structured prompt specifying the control frequency, robot embodiment, gripper configuration, action-frame reference, and task instruction. Robot proprioception is represented by six additional tokens corresponding to the previous and current states of the arms and shared body. State MLPs jointly encode the measured values and their validity masks. All prefix tokens are processed jointly by the native backbone using three-dimensional rotary positions. Rather than collapsing this computation into a single final visual representation, the model exposes the layer-wise keys and values of the backbone directly to a lightweight action expert. The action expert reads from a configurable subset of backbone layers, allowing it to combine this multimodal prefix with noisy action tokens and flow time to model action chunks.

The action expert uses a separate module to predict continuous action chunks. Each expert layer jointly attends to prefix keys and values cached at its mapped backbone layer and to the action tokens own keys and values. Each target contains an arm chunk and a shared body chunk. Given a target action, the authors sample Gaussian noise and a time variable to construct a noisy action. The expert predicts the velocity field with the target as the difference between the clean and noisy actions. The training objective balances four action groups: end-effector motion and gripper, arm joints, hand joints, and body actions. The loss is minimized as a weighted sum of mean squared velocity errors over valid entries in each group:

During inference, the model solves the learned flow from time 0 to 1 using 10 Euler steps while keeping the prefix KV cache fixed, and the resulting action chunk is de-normalized for execution.

To ensure high-quality training data, the authors implement a rigorous data quality filtering workflow.

The pipeline applies source checks to exclude unreliable measurements and corrupt episodes. Joint readings outside a specific range or deviating significantly from a local median are marked invalid. Channels that remain constant throughout most episodes or contain unresolved discontinuities are masked. Action outliers are checked against expanded per-embodiment bounds, masking only the affected field group at that step. Episodes with missing data, inconsistent shapes, or too few steps are dropped. Camera calibration is also verified by projecting recorded 3D arm positions and scoring image motion at the projected points relative to motion over the image. Untrusted poses are marked unknown while their episodes are retained if other data is valid.

The training recipe consists of three progressive stages. In the first stage, the pretrained vision-language backbone is frozen while the action expert and conditioning modules are trained for 113,400 optimizer steps. In the second stage, the full model is unfrozen and trained on the robot pretrain corpus for 75,000 steps, co-training with multi-modal understanding data to retain vision-language capabilities. Packed understanding examples are added on every other optimizer step with an auxiliary next-token prediction loss. In the third stage, the full-parameter finetune continues with in-house action data for another 75,000 steps. Both stages use AdamW with specific beta parameters and weight decay, linear learning rate warmup followed by cosine decay, bfloat16 mixed precision, and gradient norm clipping.

Experiment

The experiments evaluate a native-input VLA backbone against encoder-based baselines under matched pretraining and action-expert settings, finding that encoder-free pretraining produces control-ready representations that transfer well even before backbone adaptation. Layer analysis shows action-relevant information is spread across depths, with early layers supporting frozen transfer and visual, language, and geometry signals peaking at different stages. Co-training with general understanding data preserves multimodal abilities and prevents policy collapse under scene randomization. Simulation benchmarks and real-world tasks further show strong in-distribution performance, robustness to perceptual and semantic shifts, and practical deployment characteristics under limited fine-tuning.

The corpus aggregates nine public dataset families spanning single-arm and bimanual embodiments, with more than 2.2 million episodes and nearly 10,000 hours of robot data. Stored rows are concentrated most heavily in AgiBot-World, while Open X-Embodiment contributes the largest episode count but a relatively small stored row share because it records fewer views per episode. Sampling weights shift somewhat away from the largest stored row share, giving increased relative weight to bimanual and RoboCOIN data. AgiBot-World is the largest single contributor by stored rows and remains the most sampled source during training. Open X-Embodiment has the most episodes but a relatively low stored row share and sampled share, reflecting its lower per-episode view count. Bimanual YAM and RoboCOIN are sampled more heavily than their stored shares indicate, while AgiBot-World is sampled below its stored share.

Across LIBERO suites, the native-input backbone shows stronger frozen-backbone success than Qwen3-VL and Gemma-3 when only the action expert is updated. Its frozen performance on long-horizon tasks exceeds the full fine-tuning results of both encoder-based backbones, and full fine-tuning further raises its average above all compared models. The results indicate that the native backbone provides control-ready representations while retaining room for improvement through full-model adaptation. With only the action expert trained, the native-input backbone attains the highest frozen average and the largest long-horizon score among the compared backbones. Its frozen average is comparable to the fully fine-tuned Qwen3-VL and Gemma-3 baselines. Full fine-tuning improves every backbone, but the native-input model maintains the best average and long-horizon success rates. Updating only the vision encoder and action expert with other layers frozen improves Qwen3-VL over frozen training yet still trails full fine-tuning.

Direct encoder access in Qwen3-VL reduces frozen-backbone performance compared with the original interface, indicating that intermediate vision-tower features are less immediately usable for policy learning. Full fine-tuning largely closes this gap, bringing both interfaces to similar average success rates. The encoder-access setup adds no parameters. Under frozen-backbone training, encoder access performs substantially worse than the original interface. Full fine-tuning nearly equalizes the original interface and encoder access. Encoder access introduces no additional parameters.

This ablation compares how the action expert reads backbone layers on LIBERO. Frozen configurations that retain or read early layers, such as first-28 and first-14, achieve higher average success than late-only maps such as end-28 and last-14. The same trend appears with fine-tuning, though some early-readout maps also shorten the backbone, so readout position is entangled with depth and compute. Frozen first-28, which reads layers 0-27, achieves the highest average success among the compared maps, while skip-28 is intermediate and end-28, reading layers 12-39, is lowest despite retaining the same 40-layer backbone. In the disjoint comparison, first-14 outperforms last-14 on every LIBERO suite, though first-14 also reduces backbone depth, making readout position and backbone depth vary together.

On the LIBERO-plus benchmark, robustness varies widely across perturbation types and models. Among listed open models, π0.5 has the strongest overall result, while earlier open baselines trail by large margins. The cited context reports that NEO-VLA leads among models with public code and weights, with camera and language perturbations as particular strengths. Listed open models range from very low overall robustness for π0 to the strongest open-model total for π0.5. No listed model dominates every perturbation axis, and performance across camera, robot, language, light, background, noise, and layout shifts differs substantially by model. NEO-VLA is reported to achieve the highest overall robustness among models with public code and weights, with especially strong camera and language performance.

The experiments characterize a large multi-embodiment robot dataset, showing that AgiBot-World dominates stored and sampled rows while Open X-Embodiment contributes many episodes but fewer rows per episode, and bimanual YAM and RoboCOIN are oversampled relative to storage. On LIBERO benchmarks, a native-input backbone gives the strongest frozen and fine-tuned control performance, whereas direct encoder access hurts frozen Qwen3-VL policies but recovers after full fine-tuning. Layer readout ablations indicate that reading earlier vision layers improves success over late-only readouts, although depth and compute are partly entangled. Finally, LIBERO-plus robustness varies widely across perturbations and models, with π0.5 leading among listed open models and NEO-VLA reported as strongest among public-code models, especially for camera and language shifts.