One-sentence Summary

Joy Future Academy researchers present ZimaBlue, a scalable framework for learning generalizable World Action Models from large-scale egocentric videos via a three-stage training curriculum and an asynchronous Slow-Fast architecture, achieving $30\,\text{Hz}$30Hz real-time control and raising zero-shot robotic manipulation success from $36.1\%$36.1% to $77.8\%$77.8% by leveraging over $120,000$120,000 hours of embodied video.

Key Contributions

• ZimaBlue follows a three-stage training curriculum: causal embodied video pre-training on large-scale egocentric video, video-action mid-training with a unified action representation to align heterogeneous robot trajectories, and target-robot specialization.
• To enable real-time control, ZimaBlue adopts a Slow-Fast dual-system architecture where a high-capacity Slow world model provides generalizable spatiotemporal representations and a lightweight Fast branch delivers action predictions at 30 Hz on an NVIDIA RTX 4090.
• Zero-shot real-robot evaluations show that scaling pre-training from target-robot data alone to over 120,000 hours of embodied video improves success rate from 36.1% to 77.8%, and the method achieves strong performance across multiple benchmarks with particularly pronounced gains on unseen tasks.

Introduction

The authors tackle the challenge of scaling generalist robot policies that can follow open-ended language instructions and adapt to new scenes, dynamics, and embodiments. While vision-language-action models inherit strong semantic priors, their generalization remains constrained by limited action-labeled robot data, and recent world action models that leverage video for physical understanding often rely on generic video generators ill-suited for sequential, action-relevant control. The authors present ZimaBlue, a three-stage training framework that first pre-trains a causal world model on over 120,000 hours of egocentric video to learn physical dynamics, then aligns it with multi-embodiment robot actions via a unified representation, and finally fine-tunes on target-robot data. They further introduce a Slow-Fast architecture that decouples high-level world reasoning from fast reactive control, enabling 30 Hz closed-loop deployment while demonstrating that scaling video pre-training significantly boosts zero-shot manipulation generalization.

Dataset

The authors construct a two-stage training dataset that first builds a video world model and then incorporates action supervision.

Stage I: Video Pre-training

• Phase 1 – Broad mixture: Combines egocentric human videos (EPIC-KITCHENS, Egocentric-100K, EgoDex, HOT3D-Aria), simulated manipulation trajectories (DreamDojo, GenRobot), and real robot demonstrations (RoboCOIN, DROID, AgiBot, Galaxea, RoboMIND2, InternData-A1). Human sources supply diverse hand-object interaction patterns, simulated data gives clean motion sequences, and robot data introduces deployment-like viewpoints and object dynamics.
• Phase 2 – Curated embodied subset: Filters out noisy web-scale sources and retains only manipulation-centric videos: DreamDojo, RoboCOIN, DROID, AgiBot, Galaxea, RoboMIND2, InternData-A1, plus a large proprietary human egocentric dataset. This step removes low-resolution clips and those lacking detailed task instructions, shifting the distribution toward high-quality robotic viewpoints and instruction compliance.
• Processing: Raw RGB streams are kept, spatio-temporal scales are normalized, and long recordings are split into trainable episodes. Clips undergo random cropping, resizing, and photometric jitter before VAE encoding. Each sample contains a fixed-length visual trajectory (8K+1 frames across K temporal chunks; K=4 yields 33 frames), language instructions when available, and null state/action placeholders. A dummy action stream spans a 24-step horizon but carries no semantic supervision.

Stage II: Video-Action Mid-training

• Embodiment families: Four representative families are selected – DROID, AgiBot, Galaxea, and RoboMIND2-Franka – covering single- and dual-arm manipulation. All data is projected into a standardized interface while preserving embodiment-specific dynamics.
• Sample structure: Each training sample is anchored at a key timestep and bundles four synchronized modalities: multi-view video context, proprioceptive state, a future action chunk (24 steps), and language instructions. The visual context and action chunk cover the exact same temporal horizon, with video frames sampled at evenly spaced offsets. Language supervision comes from original annotations (task instructions, camera view descriptions). Temporal sampling windows are strictly confined within language-consistent segments to avoid mixing disjointed subgoals.
• Visual layout: Each embodiment is mapped onto a shared three-row vertical canvas based on its sensor configuration:
  • DROID: two exterior cameras + wrist camera.
  • AgiBot: head camera + left/right hand cameras.
  • Galaxea: head camera + left/right wrist cameras.
  • RoboMIND2-Franka: top camera + left/right wrist cameras. A shared pipeline applies crop, resize, photometric augmentation, and canvas packing to standardize input geometry.
• Proprioceptive and action unification: Native states and actions are mapped into a unified 100-dimensional space. Sub-vectors are populated per embodiment (e.g., DROID uses a single-arm subset; AgiBot and Galaxea fill bimanual arm, hand/gripper, torso, and mobile-base slots; RoboMIND2-Franka uses dual-Franka end-effector, gripper, and joint slots). Binary validity masks accompany each state and action tensor to handle inactive dimensions.

Method

The authors address the challenge of incompatible control interfaces across diverse robot datasets by mapping native interfaces into a 100-dimensional semantic state-action space. This ensures every coordinate retains consistent physical meaning across different embodiments. The state and action share the same slot layout, where the state describes the current robot configuration and the action represents a future control chunk. Each end-effector pose occupies 9 dimensions, utilizing 3D translation and a continuous 6D rotation representation. Additional slots cover grippers, arm joints, torso, mobile base, and dexterous hands, with each embodiment activating only the slots defined by its native interface.

To improve alignment across robots and scenes, the authors represent end-effector actions relative to the proprioceptive state at the first frame of each action chunk. Denoting the anchor pose as $(p_0, R_0) \in \mathbb{R}^3 \times \mathrm{SO}(3)$(p0​,R0​)∈R3×SO(3) and the target pose as $(p_h, R_h) \in \mathbb{R}^3 \times \mathrm{SO}(3)$(ph​,Rh​)∈R3×SO(3) at horizon $h$h, the relative pose is transformed as:

Here, $\Delta p_h$Δph​ and $\Delta R_h$ΔRh​ denote the relative translation and rotation, with the latter parameterized in a 6D rotation format. Joint targets follow a similar relative definition. Undefined slots are zero-filled and accompanied by validity masks to exclude inactive coordinate supervision during training.

To achieve low-latency control, the authors design a Slow-Fast dual-system architecture that separates high-capacity world modeling from high-frequency action generation.

As shown in the figure below:

The Slow DiT maintains a strong video-centric world model. It processes RGB observations, proprioceptive state $s_t$st​, and text instructions $\ell$ℓ, encoding them into visual latent tokens, state tokens, and language embeddings. During training, it also ingests noisy future video and action tokens to model causal visual dynamics and predict future video latents. Auxiliary action supervision is attached to the Slow branch to enforce semantic alignment between predicted video dynamics and action sequences.

The Fast DiT serves as the dedicated action generation module for real-time execution. Instead of running the heavy world model at every control step, the Fast DiT reuses the layer-wise video key-value (K/V) cache from the first 12 layers of the Slow branch as conditioning guidance. It leverages updated observation latents, current state tokens, language conditions, and noisy action tokens to predict the final action sequence. The coupling occurs within the self-attention layer, where action queries in the Fast DiT cross-attend to the cached Slow video tokens, effectively transferring visual dynamics without generating full future video sequences.

The model is trained using a three-level data pyramid that progressively adapts a pre-trained text-to-video diffusion transformer to embodied intelligence.

As illustrated in the figure below:

The curriculum consists of three stages. Stage I, video pre-training, uses large-scale heterogeneous videos to adapt the visual generative prior to causal embodied dynamics without action annotations. The authors employ a unified three-view visual interface to handle heterogeneous camera availability, padding absent views and excluding them from the flow-matching loss via validity masks. The video loss is computed over valid visual tokens using a block-causal teacher-forcing attention scheme.

Stage II, video-action mid-training, introduces robot trajectories with synchronized observations, proprioceptive states, language instructions, and actions. The Slow world model jointly predicts future video latents and action chunks using a unified flow-matching objective:

This joint prediction grounds the pre-trained visual dynamics with cross-embodiment action semantics. The video and action noise levels are coupled within a denoising step to align visual rollout uncertainty with motor command uncertainty.

Stage III, post-training, specializes the model to a target embodiment. The authors first specialize the Slow branch to the target domain, preserving its unified state and action interface or attaching lightweight embodiment-specific encoders. Then, they freeze the Slow branch and train the Fast branch using an action-only flow-matching objective. To simulate asynchronous closed-loop execution, the Fast branch is trained with Real-Time Chunking (RTC), where it receives a short prefix from the previous action sequence and predicts the remaining suffix conditioned on the latest observation and Slow guidance.

To accelerate closed-loop control, the authors employ asynchronous Slow-Fast inference and diffusion step distillation.

As depicted in the figure below:

The Slow and Fast branches operate at different frequencies. The Slow stream acts as a low-frequency world-model predictor, performing long-horizon rollouts and exporting layer-wise video K/V caches. The Fast stream runs at a high frequency, continuously refining actions using the latest observations and the available Slow guidance without waiting for the next Slow rollout. Temporal consistency is maintained via the RTC inference mechanism, where each Fast request uses a prefix from the previous action sequence to anchor transitions.

Following post-training, Distribution Matching Distillation (DMD) is applied sequentially to both branches. The Slow branch is distilled from eight to two DiT evaluations per rollout, preserving its joint video-action prediction. Subsequently, the Fast branch is distilled from eight to two DiT evaluations per request, conditioned on the layer-wise video K/V guidance from the distilled Slow branch. This two-stage distillation significantly reduces inference latency while preserving the original model architecture and asynchronous closed-loop execution protocol.

Experiment

The evaluation spans real-robot zero-shot tasks (Standard and Perturbed suites) and simulation benchmarks (LIBERO-Plus, RoboTwin 2.0, RoboCasa365) to isolate the effects of egocentric video pre-training, multi-embodiment action data, and the dual-system architecture. Multi-embodiment trajectories primarily strengthen contact-rich manipulation skills, while large-scale egocentric video pre-training dramatically improves visual generalization and robustness to environmental perturbations. The Slow-Fast dual-system design further enhances closed-loop execution on tasks requiring frequent visual feedback, and benchmark results confirm that video pre-training is crucial for generalizing to unseen composite tasks and maintaining performance under visual domain shifts.

A unified 100-dimensional state–action space maps diverse robot control interfaces into a consistent semantic layout. Each embodiment activates only the slots that correspond to its physical capabilities, with unused coordinates zero-filled and masked. Actions are expressed relative to the first frame of each action chunk to improve alignment across different robots and scenes. A single-arm robot like DROID activates only 17 of the 100 dimensions, covering the left end-effector, left gripper, and left arm joints, while bimanual systems can additionally engage right-side slots and optional torso, base, and hand slots. End-effector actions are transformed into a chunk-relative frame: the target pose is expressed relative to the proprioceptive state at the chunk's start, which serves as a stable reference anchor across embodiments.

The perturbed evaluation suite introduces two visual perturbation types to test policy robustness beyond standard lab conditions. Dynamic illumination applies glare and flashing lights with surrounding distractor objects, while scene appearance uses a tablecloth background and unseen distractors. Each perturbation type is paired with two manipulation tasks. Dynamic illumination perturbation includes glare, flashing lights, and surrounding distractor objects. Scene appearance perturbation introduces a tablecloth background and unseen distractor objects.

Progressive pre-training data scaling yields monotonic gains in zero-shot real-robot success. The full ZimaBlue model, with multi-embodiment and 120K hours of egocentric video, reaches 87.9% on the Standard suite and 57.5% on the Perturbed suite, outperforming external baselines by a large margin. Egocentric video is particularly effective for visual generalization, delivering a 22.5-point improvement on the Perturbed suite when scaled from 60K to 120K hours. Adding 6K hours of multi-embodiment action data lifts overall success from 36.1% to 46.1%, primarily by strengthening executable control on contact-rich tasks. Scaling egocentric video from 60K to 120K hours boosts Perturbed suite success by 22.5 points, with all four perturbed tasks improving under visual shifts.

The Dual-System architecture substantially improves success on tasks requiring spatial grounding, articulated-object manipulation, and multi-object transport, while slightly reducing performance on a few simple stacking tasks. The DMD-distilled variant preserves most of these gains with only a minor drop, offering a faster inference alternative. This reveals a trade-off where the fast reactive branch helps in contact-rich and dynamic scenarios but can interfere with precise, slow motions. Dual-System turns a low 6/10 on Basket spatial grounding into a perfect 10/10, and raises Microwave articulated-object success from 6/10 to 9/10. On simple stacking tasks like Bowls and Blocks, the Slow baseline outperforms Dual-System, suggesting the fast branch can disrupt precise stacking motions. The DMD-distilled model, using only 2 inference steps, closely tracks the full Dual-System, achieving 9/10 on Basket and Air Fryer and 27/30 on Toys.

In zero-shot transfer on LIBERO-Plus, ZimaBlue achieves the highest overall success rate, outperforming InternVLA-A1.5. It demonstrates strong robustness to variations in robot initial states and lighting, while camera viewpoint changes remain a key weakness. Supervised fine-tuning raises overall performance to 92.0%, driven primarily by a dramatic improvement in camera viewpoint generalization. ZimaBlue leads zero-shot transfer with an 86.7% average success rate, surpassing InternVLA-A1.5 by 0.9 percentage points. Zero-shot performance is strongest on lighting and robot state perturbations, but camera viewpoint accuracy is a bottleneck at 58.1%. With supervised fine-tuning, overall success rises to 92.0%, and camera viewpoint robustness jumps to 95.4%. The 5.3-point overall gain from fine-tuning is largely due to the camera viewpoint improvement, highlighting that zero-shot viewpoint generalization is limited by pre-training data diversity.

The evaluation uses a unified 100-dimensional action space and a perturbed suite with dynamic illumination and scene appearance changes to test policy robustness. Pre-training data scaling reveals that egocentric video greatly improves visual generalization, while multi-embodiment action data strengthens execution on contact-rich tasks. A Dual-System architecture enhances spatial grounding and articulated-object manipulation but can interfere with precise stacking, with a distilled variant retaining most gains. Zero-shot transfer on LIBERO-Plus shows strong robustness to lighting and robot state variations but limited camera viewpoint generalization, which is largely recovered through supervised fine-tuning.