One-sentence Summary

Researchers at NeoteAI and Fudan University's Institute of Trustworthy Embodied AI introduce N0-TWAM, a tactile-native world-action model that combines large-scale visuo-tactile joint training, the NeoForce unified force-based tactile representation, and an asymmetric Mixture-of-Transformers architecture to jointly predict future vision, contact, and actions for contact-rich manipulation, demonstrating strong performance on both real and simulated benchmarks.

Key Contributions

• The paper introduces N-TWAM, the first tactile-native world-action model trained at scale, which jointly predicts future vision and contact using the NeoForce force-based tactile representation and is pre-trained on 450 contact-rich tasks across six embodiments.
• An asymmetric Mixture-of-Transformers architecture pairs a full-capacity pre-trained visual expert with lightweight tactile and action experts in a causal cascade, positioning predicted contact as an intermediate target for action generation while keeping streaming inference computationally inexpensive.
• On simulated and real-robot benchmarks, N-TWAM achieves state-of-the-art performance (84.5 on UniVTAC, 49.4 on NeoSim, 46.3% average real-robot success), and ablations show that both predicted and observed tactile pathways contribute and that data scaling improves prediction accuracy.

Introduction

In contact-rich manipulation tasks such as tightening a screw or handling stacked cups, the key control signals come not from the overall visual scene but from subtle fingertip forces and contacts that cameras cannot resolve. Policies for these settings need both tactile sensing and the ability to anticipate immediate interaction futures, but prior work supplies only one of these ingredients. Vision-language-action models compress observations directly to low-dimensional actions without modeling what happens next, while video world-action models predict only visual futures and ignore tactile evolution. Recent tactile-aware methods either use touch as a passive input channel without prediction, bolt on a separate frozen tactile forecaster, or gate tactile tokens to protect a visual stream, always keeping touch out of the joint predicted future.

The authors close this gap with N₀-TWAM, the first tactile world-action model trained at scale. They reorganize a video-diffusion transformer backbone into a Mixture-of-Transformers with three slim per-modality experts (video, action, tactile) that share one self-attention, enabling the model to co-generate future vision and future contact in a causal cascade, then condition action on both the predicted and observed tactile streams. This asymmetric design maintains the pretrained visual expert's full capacity while keeping training and inference cheap. Pre-trained on tens of thousands of hours of real-robot data with synchronized touch, N₀-TWAM sets new state-of-the-art results across simulated and real contact-rich benchmarks, and ablation studies confirm that both the predicted foresight target and the observed conditioning pathway contribute significantly.

Dataset

Here is a concise dataset description based on the provided paragraphs, covering composition, processing, and usage.

• Data source and composition The authors train on NeoData, a large multi-robot manipulation corpus of over 30,000 hours spanning six embodiments and 450 contact-rich tasks. Each episode includes synchronized per-finger tactile sensors alongside multi-view RGB. Real-robot post-training episodes use InTac S1 visuotactile sensors. The massive tactile coverage distinguishes NeoData from vision-only pre-training corpuses.

• Latent encoding (offline preprocessing) All inputs are converted to latent tokens once and cached to disk so that encoders never run during training.

  • Vision and touch: A single frozen causal video VAE encodes all camera views and tactile streams. Tactile streams are treated as small videos, sharing the same latent space as vision.
  • Language: A frozen umT5 text encoder embeds the language instruction. The resulting latents are what the model actually consumes.

• Chunking into training windows Each episode is segmented into fixed windows of 33 latent frames.

  • Raw frames captured at 30 fps are subsampled to 10 fps, giving 129 frames, which the VAE compresses 4× temporally into 33 latent frames.
  • Consecutive windows advance with a stride of 24 latent frames, producing a ~27% overlap.
  • Tail windows are right-aligned to the episode end.
  • A one-time validation pass flags windows that are complete across all required modalities; only complete windows are used for training.

• Unified action space To train one policy across heterogeneous robots, every embodiment’s action is canonicalized into a 20-dimensional end-effector space (10 dimensions per arm).

  • Position and 6D-rotation targets are formed as chunk-anchored deltas, while the gripper value is kept absolute.
  • Actions are normalized per robot rather than globally, with quantile clipping applied to the heavy-tailed gripper channel.

• Contact-event stage segmentation For the long-horizon pipeline, the authors automatically mark stage boundaries within a demonstration.

  • Primary signal: tactile stream, because sharp changes in tactile readings reliably indicate transitions (first contact, loss of contact, slip, insertion seating).
  • Fallback: when tactile cues are weak, gripper aperture is used to detect grasps and releases.
  • Remaining boundaries are corrected with a final human check. This yields stage labels that are keyed on meaningful contact events and are used both for sub-task labeling and inference-time scheduling.

• How the data is used in training The model is trained directly on the cached 33-latent-frame windows with the corresponding action targets. Only windows that are complete across all required modalities enter the training mixture. The unified action space, chunking scheme, and contact-driven stage segmentation are all baked into the offline data preparation.

Method

$\mathcal{N}_0$N0​-TWAM is a world-action model that learns a generative model to roll out the future video and tactile signals a robot will experience, reading actions from that predicted future. Conditioned on a language instruction and observation history, the model predicts coupled streams of future video, tactile feedback, and actions autoregressively over chunks. The joint distribution is factorized such that the model first predicts the future video and tactile signals, and then denoises the action conditioned on these anticipated futures.

The authors leverage a Mixture-of-Transformers (MoT) backbone to handle the multimodal inputs efficiently. Instead of a single monolithic transformer, the architecture splits the backbone into three modality-specific experts: video, tactile, and action. These experts maintain private weights for normalization, modulation, and feed-forward networks but interact through a single shared self-attention mechanism at each layer. This design allows vision and touch to remain fully attentive to one another while isolating capacity in the weights, preventing the sparse tactile signals from degrading the visual dynamics. To manage computational cost, the video expert runs at full width, while the action and tactile experts are slimmer and trained from scratch, relying on the shared attention to access the visual prior.

The model is trained using a conditional flow-matching objective at latent-frame granularity. For each modality and temporal group, noise is added to the target, and the network regresses the velocity of the path from noise to the clean target. The loss combines SNR weighting with validity masks to handle single-arm embodiments.

To enforce the predict-then-act factorization within a single forward pass, the authors employ a diffusion-forcing cascade governed by a causal mask. This mask orders tokens by latent-frame position and noise state. Noisy tokens attend to clean history and co-generate with other noisy tokens at the same position, such as video and tactile co-generating. Action tokens attend to the clean copies of the just-predicted video and tactile frames. This structure ensures the policy acts on an anticipated future rather than ground truth history during the current chunk generation.

A key innovation is the dual role of touch: it serves as both a foresight target to be predicted and a reactive observation to condition the action.

For the predicted pathway, touch is treated as a generative target alongside video. Tactile streams are VAE-encoded into the same latent space as video, allowing direct comparability in the shared attention. The model predicts future touch in residual form relative to the initial frame, capturing the informative changes at contact onset and release. This predicted tactile latent is denoised jointly with the video, ensuring consistency between the anticipated scene and contact.

For the observed pathway, the model reads current tactile input in force space rather than latent space. A frozen estimator converts raw tactile images into dense three-axis surface force maps. These maps are processed by a NeoForce encoder to produce representation tokens, which are cross-attended into the action stream just before the action head. This pathway is initialized as a no-op and fine-tuned during post-training, allowing the policy to react to immediate contact feedback without disrupting the pretrained generative capabilities.

At inference, the model runs autoregressively, utilizing a rolling key/value cache and asynchronous pipeline for real-time performance. The observed tactile pathway operates at sensor rate, refreshing between chunk-level predictions, while the predicted pathway operates at chunk rate.

For long-horizon tasks, the authors introduce tactile punctuation to structure execution. Since mid-task observations can be stage-ambiguous, the model uses contact events like grasp onset and release to segment demonstrations into sub-task clips. During inference, a lightweight scheduler advances the sub-task prompt queue based on tactile feedback. The predicted future tactile signal triggers an early advance, which is confirmed by the observed tactile stream, ensuring the robot transitions between sub-tasks precisely when contact events occur.

Experiment

The evaluation spans public and in-house simulation benchmarks and real-robot contact-rich tasks, comparing a tactile world-action model against vision-language-action and vision-only baselines under closed-loop asynchronous control. Incorporating predicted tactile foresight and observed force feedback yields large gains on tasks where contact state cannot be resolved visually, and the policy degrades gracefully under visual perturbations by falling back on touch. Ablations confirm that large-scale pretraining and both tactile pathways are critical, while analysis shows the model accurately predicts future contact and a pretrained force encoder transfers effectively to realistic simulated tactile, further boosting performance.

The asymmetric Mixture-of-Tokens backbone keeps a full-width video expert while using narrower action and tactile experts, roughly halving the total parameters relative to an all-full-width design. The video expert is warm-started from a pretrained model, and shared self-attention runs at a hidden size of 3072 across all layers. Total parameters drop by about half with the asymmetric design (7.2B vs. an estimated 15B). Action and tactile experts use a hidden size of 1024, while the video expert stays at 3072. Shared self-attention operates at 3072 dimensions with 24 heads of 128 across all 30 layers. The video expert is warm-started from a pretrained video model; action and tactile experts are trained from scratch.

The N-TWAM world-action model with native tactile achieves the highest average success on the UniVTAC benchmark (84.5%), outperforming the best vision-language-action policy (InternVLA-A1, 67.1%) by 17 percentage points. Vision-only world-action baselines fall well behind even the weakest VLA policy, demonstrating that predicting future scenes alone is insufficient for tasks defined by contact. Incorporating tactile prediction into the world-action model closes this gap and enables strong performance across insertion, grasping, and shelf-placement tasks. N-TWAM reaches 84.5% average success, a 17-point lead over the best VLA policy (InternVLA-A1, 67.1%). Vision-only world-action models (e.g., GigaWorld-Policy 16.5%, FastWAM 48.0%, LingBot-VA 31.4%) trail all VLA policies, showing that scene prediction without contact modeling fails on these tasks. Among VLA policies, no single method dominates all tasks: Xiaomi-Robotics leads on Insert HDMI, Insert Hole, Insert Tube, and Pull-out Key; InternVLA-A1 on Grasp Classify and Lift Can; and StarVLA-α on Put Bottle in Shelf.

On the four single-arm NeoSim tasks, performance depends strongly on the task and model type: the world-action model N0-TWAM achieves perfect success on the precise insertion task Insert USB, while vision-language-action policies excel on the dexterous Pour Ball task and lead narrowly on Grasp Chip. All methods fail to reliably solve Unplug & Plug Charger, where even the best success rate is only 23%. N0-TWAM reaches 100% on Insert USB, far above the best VLA policy (π0.5 at 74%). On Grasp Chip, StarVLA-α (93%) barely edges out N0-TWAM (92%), with both well ahead of other methods. For Pour Ball, π0.5 dominates at 92% while N0-TWAM reaches only 63%, reversing the trend from the insertion task. Unplug & Plug Charger proves extremely challenging: most methods score 0%, and the leader π0.5 achieves just 23%.

Generalization is evaluated along three axes: unseen objects, unseen positions, and visual perturbations. The tactile model N0-TWAM is not the strongest on every axis but achieves the highest average success rate by remaining robust when vision becomes unreliable, holding its performance under lighting and background shifts. A large vision-language model leads on unseen objects but degrades sharply under visual perturbation, showing that direct tactile feedback provides a complementary, vision-independent fallback. Under visual perturbations, N0-TWAM maintains 45% success while vision-based methods drop to 25–30%, because tactile sensing directly reports contact and offers a vision-independent signal. On unseen objects, the vision-language model π0.5 achieves the highest success (80%), benefiting from broad semantic priors, while N0-TWAM scores 65%. All methods tie or perform closely on unseen positions (45% for π0.5 and N0-TWAM), making visual perturbation the decisive axis for robustness.

On four alignment-sensitive manipulation tasks, switching from a delta end-effector parameterization to an absolute pose parameterization substantially improves success rates by eliminating accumulated drift. The delta formulation, while translation-invariant, relies on a moving anchor, causing small per-chunk errors that degrade precision in tasks requiring a fixed world-frame target. Absolute end-effector parameterization raises average success across tasks to 82.5% versus 50.0% for delta. The biggest gains appear on Lift Can and Grasp Chip, where absolute poses more than triple and more than double the success rate, respectively. Put Bottle in Shelf reaches high success with both parameterizations (86–87%), indicating that the drift penalty is mild when alignment constraints are looser.

The evaluation spans contact-rich manipulation benchmarks and single-arm dexterous tasks, comparing a tactile-predicting world-action model against vision-language-action policies and vision-only world models. Findings show that incorporating tactile prediction is essential for contact-defined tasks, enabling a substantial overall lead over the best VLA policy, though model strengths vary by task: world-action models excel at precise insertion while VLA policies dominate on dexterous pouring. Generalization experiments reveal that tactile sensing provides a robust, vision-independent fallback under lighting and background shifts, while large vision-language models benefit from broad semantic priors for unseen objects. An absolute end-effector parameterization eliminates accumulated drift from chained delta predictions, more than doubling success rates on alignment-sensitive manipulation.