One-sentence Summary

The DreamX Team presents DreamX-Phi 1.0, an action-conditioned video world model for robotic manipulation that predicts future observations from observed frames, language instructions, and prescribed end-effector poses and gripper states, using per-arm $\mathrm{SE}(3)$SE(3) geometric attention encoding, a lightweight depth branch, SAM3 masks with a frozen V-JEPA teacher, and distribution-matching distillation, achieving first place on Track 1 and second place on Track 2 of the WorldArena 2.0 Challenge.

Key Contributions

• DreamX-Phi 1.0 is an action-conditioned video world model for robotic manipulation that predicts future observations from an observed frame, a language instruction, and a prescribed bimanual action sequence of end-effector poses and gripper states.
• The model injects per-arm SE(3) transformations into attention via PRoPE-style geometric encoding to preserve each arm’s commanded path, and adds a lightweight depth branch, SAM3 masks, and a frozen V-JEPA teacher to supervise scene geometry and object consistency.
• The multi-step generator is distilled into a few-step student via distribution-matching distillation for efficient deployment; on the August 12, 2026 WorldArena 2.0 snapshot, the submission ranked first among 31 Track 1 entries with an EWMScore-P of 60.65 and tied for second on Track 2 with a 67.19% success rate on Adjust Bottle.

Introduction

World models support planning by letting robots imagine future observations under candidate actions, but faithful action conditioning remains difficult. Modern video generators provide strong appearance and motion priors, yet prior action interfaces often encode commands as compact tokens or feature modulations that lack explicit SE(3) rigid-body geometry and do not indicate where the commanded motion should appear in the image. Consequently, a rollout can look realistic while still showing wrong arm motion, missed objects, or confused grasping and release. The authors introduce DreamX-Phi 1.0, a geometry-aware action-conditioned video world model that preserves SE(3) end-effector trajectories via PRoPE-based attention, grounds them with image-space motion cues, and adds depth and object-centric supervision to improve scene geometry and manipulated-object consistency.

Dataset

The authors build the dataset from three complementary sources:

• Action-free egocentric video
• Real-robot demonstrations
• Simulated robot trajectories

Overall scale is summarized in Table 1 of the paper. Specific scale details mentioned in the provided text include:

• Filtered AgiBot imitation-learning split: 178.7 hours
• RoboTwin component in the action-conditioned pool: 25,000 bimanual clips, spanning clean and randomized variants
• WorldArena 2.0 Track 1 evaluation set: 1,000 episodes
• WorldArena 1.0 Track 1 evaluation set: Clean-50 protocol with 50 manipulation tasks and 10 held-out episodes per task

Dataset processing

The authors apply several filtering and normalization steps:

• Remove robot trajectories dominated by mobile-base motion, dexterous-hand operation, or stationary segments
• Deliberately retain failed task executions because they expose failure modes and non-ideal interaction dynamics
• Normalize observations, instructions, robot states, and actions into a common LeRobot v2.1 representation
• Preserve the original camera structure: single-view recordings remain single-view examples
• Spatially concatenate synchronized multi-view streams into unified multiview videos
• For RoboTwin videos in the action-conditioned pool, apply DreamX-Refiner to super-resolve the clips into high-resolution visual data

No explicit cropping strategy is described in the provided sections.

How the data is used

The dataset is split into two training pools:

Action-agnostic pretraining

• Uses every retained video, regardless of whether the source provides action annotations
• Includes egocentric video, real-robot operation, and simulation
• Supports both single-view and multi-view downstream settings through view-adaptive organization

Action-conditioned fine-tuning

• Restricted to videos with synchronized action annotations
• Each video is paired with its corresponding robot trajectory
• Visual and control streams remain temporally consistent
• Includes the 25,000 RoboTwin bimanual clips after DreamX-Refiner super-resolution

For evaluation, the authors use WorldArena datasets built from RoboTwin 2.0 trajectories:

• WorldArena 2.0 Track 1 provides an initial RGB observation, a language instruction, and a robot action trajectory; the model predicts the subsequent rollout conditioned on either signal
• WorldArena 1.0 Track 1 uses the Clean-50 protocol
• WorldArena 2.0 Track 2 uses the submitted world model as the rollout environment for optimizing a policy with an organizer-provided initialization and a fixed reward model, evaluated on held-out Adjust Bottle episodes in RoboTwin 2.0

Method

The authors propose a framework for action-conditioned video prediction that models the conditional distribution $p_{\theta}(\mathbf{x}_{1:T} \mid \mathbf{x}_0, \mathbf{a}_{1:T}, \mathbf{c})$pθ​(x1:T​∣x0​,a1:T​,c) of future RGB frames given an observed frame $\mathbf{x}_0$x0​, a language instruction $\mathbf{c}$c, and a prescribed bimanual action trajectory $\mathbf{a}_{1:T}$a1:T​. The architecture is built upon a Wan2.2-TI2V-5B video diffusion transformer, where the latent of the first frame provides visual context and future-frame latents are learned under a flow-matching objective. The overall system is organized into three main components: action-conditioned video prediction, training supervision, and few-step post-training.

In the action-conditioned video prediction module, the model integrates structured action conditioning with the generative path. Instead of treating actions as low-dimensional tokens, the authors employ a Pose-Rotary Position Embedding (PRoPE) mechanism. This approach inserts known relative SE(3) transforms directly into the self-attention mechanism. The bimanual action trajectory is processed through an arm-grouped PRoPE stream, which applies geometric transformations to the query, key, and value projections, alongside a separate gripper bias injection. Complementing this 3D geometric conditioning, a robot-only optical flow cue provides image-plane motion signals. These two streams interact via joint attention within the video diffusion transformer to predict the future RGB video.

To ensure physical and geometric consistency during training, the authors introduce a multi-faceted supervision strategy. First, to prevent the static background from dominating the loss, SAM3-derived object masks are used to compute a mask-weighted RGB loss, focusing the flow-matching objective on the manipulated object. Second, an auxiliary depth supervision branch is added to enforce 3D consistency. Depth maps generated by Depth Anything 3 are encoded into latent targets, and a lightweight depth branch predicts these targets using a latent-space mean-squared error loss. Third, to regularize the spatiotemporal evolution of the manipulated object, a frozen V-JEPA teacher provides object relational supervision. By aligning the Gram matrices of student and teacher features over masked object tokens, the model is encouraged to maintain coherent object identity and state across contact intervals.

Finally, to enable efficient inference, the authors apply a few-step post-training distillation process. Following the DMD2 approach, the multi-step teacher model is distilled into a few-step student model. This is achieved by combining a distribution matching objective, which minimizes the KL divergence between the student and real data marginals, with a noised non-saturating GAN objective. The student model is optimized using both the distribution matching loss and an adversarial loss, allowing it to generate high-quality future videos in significantly fewer denoising steps.

Experiment

DreamX-Phi 1.0 is evaluated on WorldArena 2.0 along two axes: Track 1 measures fidelity of language- or action-conditioned visual rollouts, while Track 2 tests the model as a learned rollout environment for training a policy on held-out Adjust Bottle episodes. The model ranks first on the Track 1 leaderboard with an EWMScore-P of 60.65, achieves a 67.19% success rate and ties for second on Track 2, and qualitatively maintains coherent arms, grippers, and manipulated objects under randomized backgrounds, textures, lighting, and distractor layouts. An offline WorldArena 1.0 evaluation further reaches 76.88, above the leading official entry, though evaluation remains limited to WorldArena and RoboTwin and covers only video prediction rather than closed-loop control.

The curated corpus draws from egocentric video, real-robot demonstrations, and simulated robot trajectories to support both action-agnostic and action-conditioned learning. Egocentric video and simulated robot data contribute the largest shares by duration, while real-robot sources provide hundreds to nearly two thousand hours of physically executed manipulation. RoboTwin is reported in action-annotated clips rather than duration and adds simulated bimanual coverage. Egocentric video is the largest single source by duration, while simulated robot data forms the largest action-annotated pool. Among real-robot sources, AgiBot World is the largest, and the other real-robot collections are smaller by comparison. RoboTwin contributes simulated bimanual clips rather than hours, with action annotations used for action-conditioned fine-tuning.

At the August 12, 2026 snapshot, DreamX-Phi-1.0-FDM-0730 ranks first on the 31-entry Track 1 leaderboard with an EWMScore-P of 60.65, ahead of Alpha-World and FlowWAM-FiveAges. The top three systems are closely matched on most reported component metrics, with DreamX-Phi showing the strongest background consistency and motion smoothness while Alpha-World and FlowWAM-FiveAges lead on visual quality and photometric consistency. Selected open-source reference models generally trail the leading group, showing notable weaknesses in subject consistency, photometric consistency, or JEPA similarity. DreamX-Phi-1.0-FDM-0730 leads Track 1 with the highest overall EWMScore-P among the reported entries. Alpha-World and FlowWAM-FiveAges complete the top three and are close to DreamX-Phi on most component metrics, with better image or aesthetic quality and photometric consistency. DreamX-Phi shows the best background consistency and motion smoothness among the top three, while differences in dynamic degree and flow score remain small. GigaWorld-0, Vidar, and WoW trail the top three, with especially large gaps in JEPA similarity, subject consistency, or photometric consistency.

At the August 12, 2026 snapshot, WOVR-PLUS has the highest Adjust Bottle success rate among the displayed systems, ahead of DreamX-Phi-1.0-FDM-0730 and Lute, which are tied for second. The remaining open-source reference models all score lower, with OpenSora recording the lowest success rate in this set. WOVR-PLUS leads with 68.75%, 1.56 percentage points above the shared second-place score of DreamX-Phi-1.0-FDM-0730 and Lute. CtrlWorld, IRASim, RoboScape, and OpenSora trail the top group by roughly 6 to 8.6 percentage points, with OpenSora lowest at 60.16%.

An offline DreamX-Phi-1.0-FDM-0730 evaluation is compared against the top official WorldArena 1.0 leaderboard entries and selected open-source reference systems. The DreamX system records the highest overall score among the models shown, reaching 76.88 and exceeding the leading official snapshot entry by 3.24 points. In the reported component metrics, it shows competitive or leading results on several visual and motion quality measures relative to the official top three. DreamX-Phi-1.0-FDM-0730 reaches an overall score of 76.88, the highest among the displayed systems and 3.24 points above the leading official snapshot entry. The official top three systems are tightly clustered, while the offline DreamX result is higher than all of them. Among the reported component metrics, DreamX records the highest visual quality, motion quality, and aesthetic quality among the compared systems.

The experiments evaluate a curated training corpus and benchmark performance across Track 1 world generation, the Adjust Bottle manipulation task, and an offline WorldArena comparison. The DreamX-Phi system leads Track 1 with strengths in background consistency and motion smoothness, while close competitors score higher on visual quality and photometric consistency. On Adjust Bottle, WOVR-PLUS records the highest success rate, ahead of DreamX-Phi and Lute, with the remaining reference models trailing. In the offline WorldArena evaluation, DreamX-Phi achieves the highest overall score and leads on visual, motion, and aesthetic quality among the compared systems.