One-sentence Summary

Researchers from NVIDIA, Brown University, Columbia University, and Harvard University introduce Hydra-0, a generalist world model conditioned on action flow that represents robot actions as pixel motion to unify modeling and control across embodiments and tasks, achieving $90.4\%$90.4% lower robot-motion error and $60.2\%$60.2% lower object-motion error than the action-conditioned baseline, a Pearson correlation of $?? = 0.96$??=0.96 on the RoboLab benchmark, and an emergent inverse mode that predicts compatible robot motion from desired object flow without requiring task-specific expert demonstrations.

Key Contributions

• Introduces Hydra-0, a generalist world model that represents robot actions as pixel-level action flow, enabling a shared visual interface for action-conditioned video prediction across embodiments, tasks, environments, and video-generation backbones.
• Reduces robot-motion error by 90.4% and object-motion error by 60.2% relative to an action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation.
• Achieves a Pearson correlation of r = 0.96 between replayed and reference success rates on the RoboLab benchmark, and demonstrates an inverse mode where a world action model predicts compatible robot motion from desired object flow in a human demonstration, with a trained action head converting latent features into executable actions without task-specific expert demonstrations.

Introduction

Robots operating in the physical world benefit from world models that predict how scenes or objects evolve under their actions. However, existing world models and action-conditioned video models are often tied to a specific robot embodiment, limiting their ability to generalize across different robots, tasks, and environments. A key challenge is that native robot commands, such as joint-space or end-effector commands, encode robot-specific structure and kinematics, forcing the model to learn an embodiment-dependent mapping from commands to visual dynamics. While motion- and trajectory-conditioned video models have shown that image-plane motion can serve as a shared visual condition, a gap remains in grounding this visual representation in executable robot commands across embodiments.

To address this, the authors introduce action flow, a shared image-plane motion representation that supports both forward dynamics prediction and inverse motion prediction. Action flow represents sparse trajectories of visible robot or object points, allowing the same format to describe embodiment motion and task intent. In the forward direction, a candidate motor command is executed through the robot controller and physics simulation, and the resulting visible robot-surface trajectories are projected into the image plane using robot geometry and camera calibration. This provides a directly visual condition tied to an executable command. In the inverse direction, the same interface is used for robot control: given desired object flow as task intent, the model infers compatible robot motion, and a target-embodiment readout decodes latent motion features into executable actions. The authors instantiate this model on different video-generation backbones, demonstrating that action flow is a portable conditioning interface. Their best configuration achieves 90.40% lower robot-motion error and 60.16% lower object-motion error than an action-conditioned Cosmos 2.5 baseline. They also show zero-shot and data-efficient transfer to robot interaction with deformable objects, open-loop policy evaluation with a Pearson correlation of 0.96 across RoboLab policies, and real-robot control by converting object flow from a human demonstration into executable actions.

Dataset

The authors build a multi-embodiment training corpus from seven sources covering both robot and human interactions. Table 1 summarizes the corpus, including size before and after filtering, plus the license for each source. DROID provides large-scale, scene-diverse single-arm Franka manipulation and is the primary robot source. ABC-130k and MolmoAct2 provide bimanual teleoperation. EgoDex provides egocentric human-hand manipulation captured with Apple Vision Pro, while Deform360 provides handheld-gripper demonstrations of deformable-object manipulation using UMI grippers. XVLA-Soft-Fold and H1-Fold-Clothes contribute smaller bimanual and humanoid cloth-folding sets. The Interactive World Simulator (IWS) tasks are used only for data-efficiency evaluations and are excluded from the training corpus.

• Preprocessing and annotation: Since the paper focuses on deformable-object interaction, the authors keep mainly deformable-object interaction data (cloth, cable, rope, bag, and paper). The main corpus sources are converted to a shared per-episode, per-camera layout: frames are resampled to 480p at 16 fps and cut into non-overlapping 81-frame windows, with dense point tracks, a language caption, precomputed VAE latents, and source-dependent embodiment and object annotations where available. Dense tracks come from AllTracker on a 128 x 128 query grid per window. Depending on the source, embodiment masks come from calibrated robot rendering, projected hand geometry, SAM 3 refinement, or text prompting. Object masks use either source-provided refined masks or text-prompted SAM 3 and are unavailable for DROID and EgoDex. Captions are generated per episode using dataset-provided captions or a vision-language model.

• Filtering: Beyond the task-level deformable subsetting, the authors apply source-specific combinations of three window-level filters, with thresholds tuned from motion-score histograms. A static-window filter drops windows whose 90th-percentile visible track path length is below 50 pixels at 480p. A frozen-gripper filter drops windows whose embodiment tracks fall below the same 50-pixel floor. A contentless-caption filter drops DROID episodes whose language annotation describes no actionable content. For wrist-view coverage, they add a separate DROID wrist-camera stream of 17-frame windows restricted to deformable-object episodes identified from language annotations; it is used as a standalone viewpoint and is not combined with the external-camera views.

• Action flow construction: Rather than conditioning the model directly on an embodiment-dependent command, the authors map the command to action flow, a set of camera-plane trajectories describing the commanded motion of the visible robot embodiment. For N tracked points over a prediction horizon of H future steps, the image-plane location of point n at time t is written as x(n,t) = (u(n,t), v(n,t)) and its visibility as m(n,t) in {0, 1}, so the shared trajectory condition is F = {tau_n} for n = 1 to N, with tau_n = {(x(n,t), m(n,t))} for t = 0 to H. Because F is expressed in the observation plane, the same interface can represent robot arms, grippers, and even human hands without exposing their native action spaces to the video model. The authors consider two complementary construction scenarios: geometry-aware and video-only.

• Geometry-aware construction: When robot geometry and camera calibration are available, the authors sample points on the robot surface visible in the initial observation and obtain a sequence of robot states and link transforms. At deployment, Isaac Lab produces this sequence by executing the candidate command through the robot controller and physics simulation. The authors propagate the sampled surface points using these link transforms and project the resulting motion into the camera plane. For tracked point n, the image position at time t is computed using the robot configuration, link transform, camera extrinsics and intrinsics, and perspective projection. Visibility is set to 1 only when the projected point has positive camera depth, lies within the image bounds, and agrees with the rendered depth buffer within a small tolerance; a 3 x 3 neighborhood avoids false rejection at silhouette boundaries. The projected visible trajectories form the deployment action flow. Moderate calibration error perturbs the two-dimensional motion condition without altering the target video, whereas large projection errors can degrade spatial correspondence.

• Video-only construction: Many large interaction datasets provide videos but omit robot description files or camera calibration. For these data, the authors first recover dense image-plane trajectories and visibility labels with a flow tracker, then segment the tracks using grounded masks for the visible embodiment and manipulated objects. This produces the same trajectory representation for training without requiring privileged metadata. The tracked future is used only to construct training conditions; at deployment, robot action flow is computed causally from the corresponding physics simulator command rollout.

• Training-time flow sampling: During training, the authors sample multiple flow conditions rather than relying on a single track category. This enables the model to use different motion cues across applications. At each training step, they sample one of four conditioning strategies from the grounded track set: Embodiment, Object, All, or None. Instead of generating a separate learned token, the sampled mode only determines which trajectories populate the shared motion tensor. Embodiment selects tracks on the acting body and is the primary action-conditioning strategy; it includes visible robot links and grippers or human hands. Object selects observed future tracks on the manipulated object during training; when an object trajectory is explicitly supplied at inference, the same mode can instead provide desired-motion conditioning that elicits compatible robot motion through the world model. All samples across grounded and unassigned tracks so that training retains approximate motion correspondence when semantic grounding is incomplete. None removes the trajectory condition and acts as conditioning dropout, so the model predicts based only on text and image conditioning. If the tracks required by a strategy are unavailable, the authors sample from the remaining valid strategies.

Method

The authors propose a kinematically grounded world model that operates in a partially observable setting. The framework encodes an initial RGB observation $\mathbf{o}_0$o0​ into a spatial latent state $\mathbf{s}_0 = e_\phi(\mathbf{o}_0)$s0​=eϕ​(o0​) and predicts future latent states $\hat{\mathbf{s}}_{1:H} = g_\theta(\mathbf{s}_0, \mathcal{F})$s^1:H​=gθ​(s0​,F) conditioned on an action flow $\mathcal{F}$F. A decoder then generates the corresponding RGB sequence $\hat{\mathbf{o}}_{1:H} = d_\psi(\hat{\mathbf{s}}_{1:H})$o^1:H​=dψ​(s^1:H​).

Rather than conditioning the model directly on embodiment-dependent commands, the authors map commands to action flow, which consists of camera-plane trajectories describing the commanded motion of the visible robot embodiment. This shared interface allows the model to handle robot arms, grippers, and human hands without exposing native action spaces to the video model. For offline training, action flow is constructed from interaction videos using dense tracking and grounded masks for the embodiment and manipulated objects. During training, the model samples from four conditioning strategies: Embodiment, Object, All, or None, to enable the use of different motion cues across applications.

For online deployment, when robot geometry and camera calibration are available, the system executes candidate commands through a physics simulation to obtain robot link transforms. Visible robot-surface points are propagated using these transforms and projected into the camera plane to form the deployment action flow.

The action-flow-conditioned video prediction module retains the spatial structure of the initial encoded state. The authors align each image-space trajectory with the temporally compressed video latent by pooling positions and visibility over the image frames. They bilinearly sample a source feature $\mathbf{h}_n = \mathbf{s}_0(\tilde{\mathbf{x}}_{n,0})$hn​=s0​(x~n,0​) and propagate it along the temporal dimension using Gaussian weights. The motion feature at latent time $k$k and normalized latent-grid location $\tilde{\mathbf{p}}$p~​ is computed as:

where the weights are defined by $\tilde{w}_{n,k}(\tilde{\mathbf{p}}) = \tilde{m}_{n,0} \tilde{m}_{n,k} \exp(-\beta \|\tilde{\mathbf{p}} - \tilde{\mathbf{x}}_{n,k}\|_2^2)$w~n,k​(p~​)=m~n,0​m~n,k​exp(−β∥p~​−x~n,k​∥22​). A presence gate $g_k(\tilde{\mathbf{p}})$gk​(p~​) is also computed to indicate where the visual condition contains trajectory-propagated appearance. The complete motion condition $C_{\text{motion}} = (M, g)$Cmotion​=(M,g) is concatenated with the noisy video latent before being processed by the DiT blocks.

To optimize the model, the authors use a standard flow-matching objective:

For efficient robotics deployment, the framework combines autoregressive conversion for causal long-horizon generation with improved distribution-matching distillation for few-step sampling. The latent sequence is partitioned into temporal chunks, and generated chunks replace the clean history and are reused through a KV cache.

The system also supports an inverse control mode by constructing a world action model. By conditioning on desired object flow without providing gripper flow, the model encodes compatible robot motion in the latent dynamics. Lightweight action and state heads are trained to read clean DiT token features and learn the embodiment-specific inverse mapping to executable commands, optimized with a combined loss function including flow-matching, action, state, and velocity smoothness terms.

Experiment

Experiments evaluate action flow as a conditioning interface for video world models across Cosmos 2.5 and Wan2.2 backbones, showing it outperforms native relative 6D actions on fidelity and motion metrics for deformable-object interaction. Multi-embodiment mid-training with action flow improves zero-shot transfer and reduces task-specific data needs, with most gains achieved by 20% of training data. Cached causal autoregressive sampling with distillation yields a 16x generation-only speedup, while open-loop policy replay in RoboLab preserves success rates with high correlation to reference outcomes. A flow-conditioned world action model qualitatively converts desired object motion into executable robot commands for a real-world pipe-bending task.

The corpus combines seven robot and human interaction sources, with DROID as the largest single-arm robot source and ABC-130k as the largest bimanual source. Filtering removes static windows, frozen-gripper windows, and contentless captions, reducing episode and window counts for several sources while preserving most data for others. The sources cover diverse embodiments including single-arm robots, bimanual arms, human hands, and handheld grippers, with a focus on deformable-object interactions. DROID contributes the most episodes but loses over half its windows after filtering, while ABC-130k retains nearly all of its windows. EgoDex provides human-hand data with no window loss after filtering, and Deform360 retains about 69% of its windows. The corpus includes deformable-object-focused sources such as Deform360, XVLA-Soft-Fold, and H1-Fold-Clothes, alongside general manipulation data.

Actionable flow conditioning consistently improves video generation fidelity and motion accuracy across multiple embodiments compared to native action representations. The Wan2.2 A14B 4-step variant achieves the best overall performance, particularly in PSNR, SSIM, FID, and FVD, while also reducing gripper and object motion errors. Zero-shot baselines generally underperform, and VLM scores are mixed due to their focus on qualitative aspects rather than precise trajectory following. Replacing native relative 6D actions with actionable flow in the Cosmos 2.5 backbone improves PSNR, SSIM, gripper EPE, FID, and FVD on all datasets, and object EPE where reported. The Wan2.2 A14B 4-step model shows the best point estimates across most metrics, with notably higher PSNR and SSIM and lower FVD compared to other variants. Zero-shot baselines (ATI and Wan-Move) exhibit higher object and gripper EPE and lower fidelity scores, indicating a disadvantage without multi-embodiment mid-training. VLM scores are inconsistent with flow-based metrics because they evaluate physical plausibility and temporal consistency rather than exact trajectory adherence.

The table compares inference speed across three model stages: a bidirectional teacher, an autoregressive teacher, and a few-step student. The autoregressive teacher is faster than the bidirectional baseline, and the few-step student achieves a substantial further speedup, reaching about 62 frames per second. These measurements are generation-only, excluding guidance and VAE decoding, and are based on a short horizon with a single H100 GPU. Cached causal autoregressive conversion is about 1.68 times faster than full-sequence bidirectional sampling despite using more network evaluations. The few-step student reduces network evaluations to 15 and achieves a further approximate 9.5 times speedup over the autoregressive teacher. The few-step student reaches about 62 frames per second, a 16 times generation-only speedup relative to the bidirectional teacher.

The evaluation spans a diverse multi-embodiment corpus, including single-arm, bimanual, human-hand, and deformable-object sources, with filtering that trims redundant windows while retaining most data. Across all datasets, replacing native action representations with actionable flow conditioning consistently improves video fidelity and motion accuracy, with the Wan2.2 A14B 4-step variant achieving the best overall performance and zero-shot baselines lagging due to lack of multi-embodiment training. Inference speed tests show that a few-step student model, built from cached causal autoregressive conversion, delivers a 16x generation-only speedup over the bidirectional teacher, reaching about 62 frames per second.