One-sentence Summary

NVIDIA, Peking University, and UC Berkeley propose PhysWord, a reasoning-before-acting foundation model that uses Physical Words, atomic action verbs as shared semantic anchors, to represent continuous physical dynamics, align contextual hidden states with latent targets derived from future observations, and organize visual and optional tactile action variants across time, achieving state-of-the-art success rates of $90.3\%$90.3% on visuotactile UniVTAC and $99.6\%$99.6% on vision-based LIBERO-Long while outperforming competitors on real-world bimanual gripper and dexterous-hand tasks.

Key Contributions

• PhysWord is a reasoning-before-acting foundation model that uses atomic action verbs as Physical Words to anchor continuous physical dynamics by predicting upcoming action words and aligning their contextual hidden states with latent targets from future observations.

• The PhysWord space is temporally structured with visual and tactile variants of each action word, and autoregressive prediction across future timesteps captures transitions between execution stages for action generation.

• PhysWord reaches 90.3% success on visuotactile UniVTAC and 99.6% on vision-based LIBERO-Long, with real-world experiments using bimanual grippers and dexterous hands demonstrating contact-rich, precise, long-horizon manipulation and generalization to unseen scenarios.

Introduction

The authors study vision-language-action (VLA) robotics, where general-purpose robot control requires not only interpreting visual and language instructions but also anticipating how physical states change during manipulation. Prior reasoning-before-acting methods help plan actions, but language-based reasoning is often too coarse to describe fine-grained dynamics, while latent or world-model approaches capture physical evolution without organizing it around reusable action meanings. The authors introduce PhysWord, which uses atomic action verbs such as pick, rotate, and push as semantic anchors for latent future dynamics. This creates a structured physical context for action generation, supports visual and tactile modeling, and improves long-horizon, contact-rich robotic manipulation.

Method

The authors propose PhysWord, a framework designed for structured latent reasoning in language-conditioned robotic manipulation. The model leverages a mixture-of-transformers architecture comprising two primary branches: a temporal perception branch initialized from Cosmos-Predict2.5-2B and a reasoning-and-action branch initialized from Qwen3-VL-2B.

As shown in the figure below:

The temporal perception branch encodes visual history and anticipates scene evolution, while the reasoning-and-action branch generates Physical Words followed by robot actions. For multimodal inputs, the temporal branch processes a sequence of head-view observations to predict future frames. The reasoning-and-action branch handles remaining camera views and language instructions. When tactile sensing is available, a T3 encoder extracts features from 3D visuotactile observations. These features are aggregated based on the end-effector configuration by concatenating sensor features for grippers or pooling fingertip features for dexterous hands to form tactile tokens that match the language model embedding dimension. Both visual encoders remain frozen during training. The two branches retain independent QKV projections and feed-forward networks but concatenate their query, key, and value features along the token dimension for shared attention. At inference, the temporal branch performs a single denoising step to cache predictive features for subsequent token generation.

To bridge high-level semantics and continuous physical dynamics, the authors introduce Physical Words as reasoning primitives. Each Physical Word couples an atomic action label with a contextual hidden state representing future visual or tactile dynamics. The framework defines 16 atomic action labels, each instantiated with separate visual and tactile variants to avoid conflicting supervision across distinct latent spaces. These variants share a common semantic initialization from pretrained embeddings but maintain independent trainable entries. The hidden state of each Physical Word is jointly supervised using cross-entropy loss for the action label and cosine distance loss to align with future sensory latent targets. Future visual targets are derived by encoding and pooling future head-view images, while tactile targets are constructed using the same T3 encoder applied to future observations.

To capture the continuous evolution of action semantics and contact conditions, the authors organize Physical Words into a temporally structured space. Instead of dense prediction, Physical Words are predicted at intervals over the action horizon. These representations are organized into blocks ordered by future timestep, with left and right end-effector predictions grouped together. The model generates Physical Words autoregressively over these future timestep blocks. A blockwise-causal attention mask is applied, allowing full attention within each end-effector timestep block and causal attention across its own blocks, while strictly blocking cross-effector attention.

The training process consists of two stages: large-scale pretraining on human and robot demonstrations, followed by task-specific fine-tuning. In both stages, the authors jointly optimize four objectives: the flow-matching loss for video prediction, the cross-entropy loss for Physical Word labels, the cosine alignment loss for future latent prediction, and the flow-matching loss for action generation. The latent alignment loss is defined as:

where $\varepsilon$ε ensures numerical stability. The total objective combines all four losses:

This joint optimization ensures that action semantics are tightly coupled with future physical dynamics within a shared contextual representation.

Experiment

the paper evaluate PhysWord on two simulation benchmarks, UniVTAC and LIBERO-Long, to compare it with vision-based, tactile, VLA, and world-model baselines, then use ablations to examine the contributions of Physical Word supervision and temporal organization, and finally test real-world manipulation on robotic platforms. The experiments show that PhysWord is effective for both contact-rich and long-horizon tasks, with predicted tactile latents closely matching measured tactile information and atomic-word supervision producing more consistent, semantically grounded action representations. Ablations indicate that joint semantic and latent supervision, direct hidden-state alignment, and an appropriately dense prediction interval improve performance. Real-world results further demonstrate stronger robustness across unseen configurations and long-horizon tasks, suggesting that Physical Words can reuse learned action semantics as context evolves.

PhysWord obtains the strongest mean success rates on the visuotactile and long-horizon comparisons, improving over both tactile-enabled and vision-based baselines. Its largest benefits appear on tasks requiring sustained motion coordination and sensitive contact handling. Predicted tactile latents nearly match measured tactile inputs, and visual-only Physical Words also support long-horizon task success. PhysWord leads tactile-enabled and vision-based baselines on the visuotactile benchmark. Predicted tactile latents retain most of the full model's performance when replacing measured tactile input. PhysWord also improves long-horizon manipulation over VLA and world model baselines.

PhysWord achieves the highest success rates on all four real-world manipulation tasks, outperforming both a VLA baseline and a tactile foundation policy. It is especially strong on contact-rich dexterous tasks and long-horizon gripper-based tasks, showing better average success with lower run-to-run variability. The method also remains more robust than both baselines when tested on unseen backgrounds, objects, and positions. PhysWord outperforms both baselines across all four real-world tasks, with the clearest advantage on Pick Paper. The method shows broad robustness in unseen configurations, maintaining a large success-rate advantage over both baselines.

Experiments evaluate PhysWord on visuotactile, long-horizon, and real-world manipulation tasks against vision-only, VLA, tactile-enabled, and tactile foundation baselines. PhysWord achieves the strongest overall success rates, especially on tasks requiring sustained motion coordination, sensitive contact handling, contact-rich dexterous manipulation, and long-horizon gripper control. Predicted tactile latents preserve most of the full model’s performance, and visual-only Physical Words remain effective, indicating strong robustness and generalization to unseen backgrounds, objects, and positions.