One-sentence Summary

Fudan University and NeoteAI researchers introduce N0-VTLA, the first vision–tactile–language–action model pretrained at scale on tactile data, using a training recipe of visuo-tactile pre-training on NeoData, tactile pathway integration, and advantage-conditioned offline RL (ALTER) to achieve $63.8\%$63.8% mean success on 20 simulation tasks and up to $95\%$95% on long-horizon real-robot contact-rich manipulation.

Key Contributions

• N0-VTLA is a vision-tactile-language-action foundation model that treats tactile perception as a prediction target, conditioning a flow-matching action expert on a latent contact-prediction token instead of raw tactile tokens. After pre-training on the large-scale NeoData dataset, the latent token retrieves the matching future tactile target with 92.3% top-1 accuracy.

• A three-stage training recipe integrates tactile perception: visuo-tactile pre-training on NeoData, staged tactile-pathway alignment that freezes the perception stack and masks the vision-language pathway to align latents with the action expert, and joint fine-tuning. N0-VTLA outperforms strong baselines, winning all nine real-robot tasks and reaching 63.8% mean success on a twenty-task simulation suite against 44.0% for the best baseline.

• ALTER is an advantage-conditioned offline reinforcement learning method that converts relative progress estimates and trajectory-event comparisons (such as tactile-detected object drops and logged human corrections) into binary advantage labels for policy improvement on a fixed deployment corpus. N0-VTLA policies trained with ALTER achieve 75-95% success on three long-horizon real-robot tasks.

Introduction

Vision–language–action (VLA) models generalize manipulation across tasks and embodiments, but the absence of tactile sensing leaves contact-rich manipulation unreliable. Prior attempts to add touch either concatenate tactile tokens into the vision–language prefix, where the sparse, mostly silent signal adds little to information-dense views, or inject the current tactile reading into the action pathway, which only reflects contact already produced and fails to anticipate the contact the policy must next enact. The authors present N₀-VTLA, which instead treats touch as a prediction target: a small predictor reads the vision–language context and current tactile tokens to estimate the net tactile change over the coming action chunk, and the resulting latent conditions the action expert directly, so the policy acts on the contact its own actions will cause.

Dataset

The authors use NeoData, a large-scale curated visuo-tactile corpus that combines real-world recordings from single-arm and dual-arm robot manipulators, a handheld UMI-style gripper, and simulated episodes from the UniVTAC simulator. The entire dataset feeds into the pre-training of N₀-VTLA, with a single policy interface that handles both real and simulated data.

• Composition and sources

  • Real-world data: single-arm and dual-arm manipulators, plus a handheld collection gripper.
  • Simulated data: episodes generated by the UniVTAC visuo-tactile simulator (NeoSim suite).
  • Every gripper finger that participates in a manipulation task carries a custom visuo-tactile sensor, so contact is captured as a stream of tactile images rather than a low-dimensional force signal.
  • Exact per-platform stream counts, frame rates, and corpus size are documented in the companion data report (Appendix A).

• Unified action and state schema

  • All embodiments are mapped into a fixed 32-dimensional container (inherited from π₀.₅).
  • The first 20 dimensions are split into two 10‑dimensional slots, one per arm; the remaining 12 dimensions are unused and always zero.
  • Each slot holds: 3D end-effector position, 6D rot6d rotation, 1D gripper channel.
  • Dual-arm episodes fill both slots; single-arm episodes fill only the first slot, leaving the second zero.
  • Actions are stored as absolute end-effector poses. At training time, each chunk is rewritten relative to its own first frame, so the model predicts relative motion. Normalization statistics are computed on this chunk-relative representation for each robot and action‑schema pairing.

• Tactile processing

  • Each gripper finger provides one tactile stream, synchronised with the RGB and proprioceptive channels.
  • Every episode begins with a short zero-contact baseline (gripper open, static for at least 0.5 s). That baseline frame serves as the zero-contact reference, and all subsequent tactile frames are interpreted relative to it.

• Quality filtering

  • Before training, every converted data repository is verified for completeness, schema adherence, and consistency of statistics and media.
  • Repositories that fail the checks are either repaired or excluded. The set of invariants is catalogued in Appendix C.

• Usage in the model

  • The entire verified corpus (real + simulated) is used for pre-training N₀-VTLA.
  • During training, absolute action sequences are converted into chunk-relative motion targets, and the model is trained to predict these relative chunks.
  • At deployment, the predicted relative chunk is transformed back to absolute poses, and inverse kinematics resolves them into joint commands for the robot.

Method

The authors propose $N_0$N0​-VTLA, a policy for contact-rich manipulation that integrates a pretrained vision-language-action backbone with a novel latent tactile pathway. The base architecture pairs a PaliGemma vision-language backbone with a flow-matching action expert. Camera views, instructions, and robot state form the model prefix, while the expert denoises an action chunk over a horizon of $H=50$H=50 steps.

To incorporate touch without destabilizing the pretrained policy, the authors introduce a latent tactile pathway. Instead of feeding raw tactile frames into the vision-language prefix, a frozen self-supervised visual encoder processes the difference between the current tactile frame and an episode-start baseline. This produces tactile tokens. A lightweight predictor then distills these tokens, conditioned on the vision-language context, into latent tactile tokens $z$z that estimate the net contact change expected over the upcoming action chunk.

The integration of this tactile pathway follows a three-stage training recipe to ensure stable grounding.

Refer to the framework diagram for the first stage of training:

The tactile predictor is trained against a future-tactile target. The predictor output $z$z is trained to match a target $z^*$z∗, which is obtained by applying the same tactile encoder to the tactile change over the next $H$H steps. The supervision combines a symmetric InfoNCE contrastive loss and an auxiliary $L_1$L1​ reconstruction loss. This grounds the latent tokens in actual contact dynamics rather than just visual appearance.

In the second stage, the action expert learns to consume these latent tokens.

The latent tokens $z$z are projected to the action-expert width and prepended to the action suffix. Crucially, the vision-language context is masked before the action expert during this phase. This forces the action prediction to rely solely on the latent tactile tokens $z$z, aligning the expert with the new interface before the full policy is unleashed.

The final stage involves end-to-end training of the full policy.

The mask on the vision-language pathway is removed, allowing the expert to see scene and instruction alongside $z$z. Everything except the frozen tactile encoder backbone is trained jointly under the action objective. This refines the pathway, allowing the perception stack to adapt to the expert's needs while retaining the tactile grounding established in the first stage.

The base pre-training of this multi-platform model is conducted at cluster scale. The training process exhibits high stability, characterized by smooth loss descent and flat gradient norms throughout the training progress.

Beyond supervised training, the authors introduce ALTER for offline policy improvement using deployment data.

ALTER constructs a pairwise progress model trained on both dense progress pairs from clean demonstrations and sparse preference pairs from tactile-detected object-drop events and human-in-the-loop corrections. Tactile contact changes, along with kinematic and visual cues, ground stage annotations. The frozen progress model then estimates global task phase and local execution change for stored trajectories. Within each predicted stage, samples are ranked by local change to assign binary advantage labels, which are appended to the task prompt during policy training. At deployment, the policy operates with the positive advantage condition.

Experiment

The evaluation spans nine real-robot tasks and twenty simulation tasks, comparing a vision-language-action model with a tactile pathway against strong baselines under identical protocols. The experiments validate that the anticipatory tactile representation, pretrained via contrastive learning, enables contact-aware retry behaviors during insertion, fine gripper-force modulation for delicate manipulation, and action divergence precisely at contact-critical moments. These benefits persist under offline RL, where the tactile-pretrained backbone consistently outperforms its vision-only counterpart, demonstrating that touch transforms manipulation from a visual commitment into a closed-loop, contact-adaptive process.

Integrating tactile feedback dramatically improves success on contact-rich manipulation tasks. The proposed N0-VTLA model achieves an average success rate of 83.1% on the UniVTAC suite, far exceeding all baselines, including the strongest external model at 67.1%, and nearly saturates performance on several insertion tasks. Tactile sensing enables closed-loop recovery from blocked insertions and fine gripper force control, proving decisive when contact determines the outcome. Tactile-enabled N0-VTLA reaches 83.1% average success on UniVTAC, a 16-point lead over the best external baseline (InternVLA-A1 at 67.1%) and more than 2.5 times the vision-only ACT (30.9%). Insertion tasks benefit most: N0-VTLA scores 95% on Insert Hole and 99% on Insert Tube, while vision-only methods frequently fail after blocked attempts. ACT+UniVTAC (48.0%) improves over vision-only ACT (30.9%), but VITaL (40.5%) and several external models fall behind, showing that not all tactile integration strategies are equally effective. On Grasp Classify, tactile methods VITaL and ACT+UniVTAC achieve near-perfect success (100% and 99%), while the best vision-only model reaches only 50%. External baselines vary widely: InternVLA-A1 dominates on several tasks (e.g., 90% Lift Can, 95% Insert Tube) but struggles on Insert HDMI (12%), whereas Xiaomi-Robotics-0 shows strength on Insert Hole (96%) and Insert Tube (98%) but falls on Lift Can (13%).

On the NeoSim benchmark, N0-VTLA reaches an overall success rate of 50.8%, ahead of the strongest baseline π0.5 at 45.8%, while all other external baselines fall between 8.6% and 23.4%. Performance is strongest on the four single-arm tasks, where N0-VTLA averages 73.8% versus 68.8% for π0.5, and the method retains a lead on the much harder eight dual-arm tasks despite a steep drop in success for every method. The tactile pathway enables closed-loop recovery from blocked insertion attempts and fine grip-force regulation, turning contact-rich tasks from single planned motions into contact-aware processes. N0-VTLA achieves the highest single-arm mean (73.8%) and the best overall NeoSim mean (50.8%), exceeding π0.5 (68.8% and 45.8%) and leaving other baselines in the 8.6–23.4% range. On the eight dual-arm tasks, success rates collapse: N0-VTLA drops to 39.4% while InternVLA-A1 falls to 1.0%, and π0.5 to 34.3%, showing the regime is far more challenging for all methods. N0-VTLA turns insertion into a closed-loop recovery process: it lifts and realigns after contact, reaching 85% on Socket Plugging (vs. 60% for π0.5) and 25% on Board Insertion where π0.5 scores 0%. Tactile-based fine control of gripper force allows N0-VTLA to stabilize objects without excessive grip, achieving 30% on a real-robot Bottle Standing task while baselines remain at 0%. Specialist baselines are severely degraded on NeoSim, with overall suite means between 8.6% and 23.4%, highlighting that the tactile pathway provides a decisive advantage when contact determines success.

The authors evaluate the N0-VTLA model on the UniVTAC and NeoSim benchmarks, which cover contact-rich manipulation tasks such as insertion and grasping. Tactile sensing enables closed-loop recovery from blocked insertions and fine grip-force regulation, turning these tasks from vision-only open-loop attempts into contact-aware processes. The model achieves large improvements over vision-only and external baselines, especially on insertion tasks, though dual-arm scenarios remain challenging. The results show that effective tactile integration is decisive for contact-dominated manipulation, and not all tactile strategies deliver equal benefits.