One-sentence Summary

Researchers from GigaAI and Tsinghua University present WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts, and a systematic study of 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions, enriched by large-scale community submissions from the CVPR 2026 GigaBrain Challenge, curated synthetic trajectories, and over 12,000 hours of training videos, revealing that reliable policy evaluation hinges on long-horizon, action-faithful rollout consistency rather than short-term visual realism, that pretraining gains require balancing general world knowledge with robot-specific controllability, and that architectural choices including action encoding, memory design, and evaluator-focused post-training strongly determine alignment with real-world robot behavior, leading to a practical design roadmap realized in GigaWorld-1, a world model optimized for scalable policy assessment, with full code, models, datasets, and toolkits publicly released.

Key Contributions

• WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts, enables controlled comparisons of world models as surrogate policy evaluators across diverse manipulation tasks, model families, action representations, rollout horizons, and evaluation metrics.
• A study of 7 video world models, 4 action encoding schemes, and over 324,000 simulated rollouts reveals three core insights: evaluator quality is dominated by long-horizon action-faithful consistency rather than short-term visual realism; pretraining benefits come from balancing general world knowledge with robot-specific controllability; and architectural choices like action encoding and memory design strongly determine alignment with real-robot behavior.
• A practical design roadmap derived from these findings is realized in GigaWorld-1, a world model optimized for policy evaluation, and the full release of code, models, datasets, and toolkits supports scalable evaluation research for embodied foundation models.

Introduction

Evaluating robot policies remains a major bottleneck because real-world rollouts demand continuous human supervision, occupy hardware for lengthy cycles, and are difficult to scale with consistent reset conditions. While learned world models offer a promising alternative by simulating visual dynamics, prior work has only demonstrated that evaluation is possible without clarifying which design choices produce reliable, action-conditioned policy evaluators. The authors address this gap by building WMBench, a benchmark spanning eight manipulation tasks with paired real-world and world-model rollouts, and by conducting a large-scale study of over 324,000 rollouts across seven world models and four action representations. Their key insight is that reliable evaluators must maintain action-faithfulness over long horizons, retain pretrained world knowledge under robot-domain adaptation, and support stable iterative rollout, culminating in GigaWorld-1, which improves evaluator-alignment metrics by 14.9% over strong baselines.

Dataset

The authors construct two complementary data resources: WMBench, a benchmark for evaluating whether world models can replace real-world execution for policy evaluation, and a large-scale training corpus for the GigaWorld-1 world model. This section outlines their composition, filtering, processing, and usage.

• WMBench benchmark

• Composition: 2,989 paired trajectories across eight manipulation tasks, drawn from teleoperated real-world data and policy rollouts generated by GigaBrain checkpoints. The ratio of teleoperated to rollout data is roughly 1:1, and rollouts include both successes and failures.

• Train/test split: Episode-disjoint split with outcome balance to avoid memorization of visually similar success/failure cases. After filtering, the training set contains 82,470 seconds of video and the test set 7,200 seconds.

• Data cleaning: Removed corrupted or truncated videos, clips with camera desynchronization, missing robot states, misaligned control timestamps, ambiguous outcome labels, and near-duplicate teleoperation episodes.

• Large-scale rollout dataset: An additional pool of 324,000 world model rollout segments submitted by over 100 teams in the CVPR 2026 GigaBrain Challenge. Segments were chained into long-horizon interaction episodes (20–30 segments each) and manually annotated with a four-level World Model as Evaluator Score (WMES) by three independent annotators, with spot checks by a senior annotator. Scores range from accurate outcome with high visual fidelity (3) to incorrect outcome with severe generation collapse (0).

• Usage: WMBench serves as the official benchmark for closed-loop policy evaluation and provides the annotated rollout data for analyzing evaluator reliability.

• GigaWorld-1 training corpus

• Scale and sources: Approximately 12,980 hours of video from four complementary sources:

• Internet and physics videos (1,298 hours) for generic physical dynamics.

• Open-source robot datasets (5,377 hours) covering diverse embodiments (e.g., AgiBot, RoboMind, Galaxea).

• Egocentric human-hand data (2,411 hours) emphasizing hand-object interaction.

• Giga-collected humanoid and dual-arm demonstrations (3,894 hours) with calibrated robot trajectories.

• Embodiment coverage: The corpus spans humanoid robots, dual-arm manipulators, single-arm manipulators, and dexterous hands.

• Data curation pipeline

• Video quality filter: Applies a clip-level acceptance test. It verifies metadata consistency, frame decodability, timestamp monotonicity, and resolution validity, then computes per-frame image-quality scores (sharpness, exposure, noise, contrast, and compression artifacts). An aesthetic-semantic scorer rejects videos with irrelevant overlays, extreme occlusion, poor framing, or non-manipulation content. Temporal integrity is checked via histogram and embedding differences between adjacent frames to detect scene jumps, stitching errors, frozen frames, or dropped frames.

• Motion and trajectory filter: Estimates dense optical flow to compute motion magnitude. Clips with average motion below a threshold are removed (except for labeled waiting, holding, or contact-stabilization segments). High-frequency motion artifacts are penalized using the temporal jerk of the flow signal, discarding clips with abrupt oscillations or unstable hand-object contacts. For robot data, a vision-language verifier checks whether the observed motion is consistent with the action stream, filtering synchronization errors and calibration drift.

• Distribution filter: After filtering, remaining clips are hierarchically balanced by source type, embodiment, camera view, task family, and motion intensity, then annotated with task descriptions, success labels, action synchronization metadata, and quality tags.

• Annotation with Giga DataCrafter

• Semantic masks: SAM2 generates frame-level semantic masks for manipulable objects, robot arms, hands, and task-relevant regions. Masks are propagated temporally and assigned category tags, providing object-centric supervision.

• Monocular depth: Depth Anything 3 produces dense depth maps per frame. For multi-view robot data, depth predictions are normalized per camera and checked against calibration metadata.

• Language captions: A fast-slow Vision-Language Model captioner operates offline. A fast stream produces high-frequency, short-term subtask descriptions (e.g., reaching, grasping, contact), while a slow stream generates low-frequency, long-term descriptions of scene layout, object attributes, and task context. Captions are cached and reused as lightweight conditioning signals during training.

• Usage in the model: The curated and annotated training corpus is used to train the GigaWorld-1 world model. WMBench is employed to measure the model’s ability to act as a policy evaluator, with the WMES-annotated rollout dataset providing a foundation for studying evaluator reliability.

Method

The authors design GigaWorld-1 as an evaluator-oriented world model built upon a pretrained video diffusion backbone. To adapt the model efficiently to robot domains while preserving spatiotemporal priors, the architecture keeps the VAE and text encoder frozen, concentrating robot-domain learning in trainable LoRA adapters and lightweight control pathways within an autoregressive diffusion transformer.

As shown in the figure below:

The overall architecture extends the pretrained backbone into an autoregressive world generator. Historical frames are encoded through memory patchification, while future noisy latents are processed via standard patchification. Structured controls, including actions, depth, semantic maps, and captions, are injected as temporally aligned conditions. The model maintains state across iterative rollouts to ensure long-horizon policy outcomes remain stable.

To support controllable robot world generation, the authors introduce a unified control representation. For static head camera views, an end-effector pose map encodes future manipulation intent. For dynamic wrist camera views, a ray map stores ray origins and directions to distinguish camera motion from scene dynamics. These are concatenated and encoded into a unified latent control representation that is continuously provided throughout autoregressive generation.

Autoregressive generation requires preserving both short-term motion continuity and long-term scene consistency. The authors implement a hierarchical history memory consisting of a first-frame anchor and multi-scale temporal memories. The anchor preserves the initial scene configuration, while long-, mid-, and short-term memories capture scene-level, task-level, and motion-level information. In the self-attention layer, the historical context serves as a persistent guidance source for future prediction, while cross-attention is applied only to the current noisy window to inject semantic information from the task description.

For long-video generation requiring smooth semantic transitions across different temporal stages, the authors avoid abrupt prompt switching by interpolating text embeddings using Spherical Linear Interpolation.

As illustrated in the figure below:

Given two text embeddings, the angle between them is computed, and intermediate text conditions are sampled along the spherical path. These interpolated embeddings are progressively injected into successive generation windows, allowing the world model to smoothly evolve from one semantic state to another while preserving the angular structure of the embedding space.

The training process follows a progressive curriculum that separates robot-domain pretraining, autoregressive rollout learning, scene-level adaptation, and few-step distillation.

As summarized in the figure below:

In Stage 1, the authors initialize from a pretrained video backbone and continue training on a curated multi-source robot corpus to learn a bidirectional robot video prior. The model is optimized with a flow-matching objective to transfer general spatiotemporal knowledge into the embodied domain. In Stage 2, the bidirectional foundation model is converted into an autoregressive world model using Relative RoPE, Hierarchical History Injection, the First-Frame Anchor, and Unified Control Injection. The model learns to denoise future observations conditioned on past context and robot actions. Stage 3 involves optional Low-Rank Adaptation for deployment in specific workspaces, adapting the model to scene-specific appearance and lighting while preserving general dynamics. Finally, in Stage 4, the autoregressive teacher is distilled into a few-step student to accelerate inference. An optional ODE distillation provides a warm start, followed by mandatory DMD2 distillation, which combines teacher distribution matching, score consistency, and adversarial supervision to produce realistic rollouts with substantially fewer denoising steps.

Experiment

The paper introduces WMBench, a benchmark for evaluating world models as policy evaluators via closed-loop rollout and a hierarchical metric system. Through systematic experiments, it finds that reliable evaluator quality hinges on visual and geometric fidelity, long-horizon assessment, transferable physical priors, and a balanced data mixture of robot and broad physical videos. Model design choices such as spatially aligned action control and persistent memory are critical for accurate trajectory following and long-term stability. The resultant GigaWorld-1 model demonstrates improved evaluator reliability, robust OOD generalization, and closer alignment with real-world policy outcomes.

A VLM-based annotation pipeline demonstrates near-perfect agreement with human expert scores on over 5,000 videos. Predictions are almost always within one score level of the ground truth, large errors are extremely rare, and rank correlation is strong, indicating the VLM can reliably replicate human outcome assessments at scale. Adjacent accuracy reaches 0.9916, meaning VLM predictions differ from human scores by at most one level in 99% of videos. Large errors, where predictions differ by two score levels, occur in only 0.84% of cases.

Adding broad physical videos (PhysData) to the GigaData robot-only baseline yields the best overall evaluator quality, improving the average metric by 0.049 while substantially boosting photo consistency and subject fidelity, and largely preserving trajectory accuracy. In contrast, adding AgiBot robot data also enhances subject and photo consistency but degrades JEPA similarity and trajectory metrics, resulting in a smaller average gain of 0.029. GigaData + PhysData achieves the highest average improvement (+0.049) over the GigaData baseline, driven by a large increase in Photo Consistency (+0.307) and a modest gain in Subject fidelity (+0.020), with only a minor drop in Trajectory (-0.007). Adding AgiBot data improves Subject fidelity (+0.140) and Photo Consistency (+0.303) but sharply reduces JEPA Similarity (-0.243) and Trajectory accuracy (-0.108), limiting the overall gain to +0.029.

The way action control is injected into a world model heavily influences its reliability for policy evaluation. Spatially aligned control interfaces, especially channel-concatenated control maps, achieve the best trajectory accuracy and motion coherence, while cross-attention control yields negligible improvement over no control and degrades motion quality. Channel-concatenated control maps lead across all metrics, delivering the highest trajectory accuracy, dynamic degree, motion smoothness, flow score, subject consistency, and photometric consistency. Cross-attention control provides only a marginal gain in trajectory accuracy (0.1620 vs. 0.1576 for no control) and worsens dynamic degree, motion smoothness, and flow score. ControlNet-style conditioning raises trajectory accuracy to 0.2566, a clear improvement over cross-attention, but still falls far short of channel concatenation.

Long-horizon rollout quality is evaluated over 40 seconds with PSNR, FID, and FVD. GigaWorld-1 consistently achieves the best scores across all 8-second intervals, while the SVD baseline shows a sharp quality collapse after the first interval. Cosmos2.5 and LTX-Video maintain more stable but lower-quality performance than GigaWorld-1. GigaWorld-1 attains the highest PSNR and lowest FID and FVD across every 8-second interval from 0 to 40 seconds. SVD's PSNR drops from 14.05 at 0-8s to below 7 after 16s, and its FID and FVD increase sharply, confirming rapid visual and temporal degradation.

GigaWorld-1 instantiates a design map that balances generic world knowledge with robot-specific controllability by combining a diverse, quality-filtered training corpus with explicit pixel-aligned action representations and memory-augmented rollout. Structured supervision and relative temporal encoding further stabilize geometry and reduce drift, enabling the model to serve as a policy evaluator calibrated to real-world task success rather than visual realism alone. The training data blends physical videos, open-source robot data, egocentric footage, and Giga-collected data, then applies quality, motion, and distribution filtering to remove noisy and static samples. An explicit action interface represents end-effector pose and ray maps in pixel-aligned form, and a memory module with a first-frame anchor and hierarchical history stabilizes long-horizon rollouts.

A VLM-based annotation pipeline is validated on over 5,000 videos, achieving near-perfect agreement with human expert scores and confirming reliable outcome assessment. Data mixing experiments show that incorporating broad physical videos improves evaluator quality, while action control interface studies demonstrate that pixel-aligned control maps outperform alternatives like cross-attention. Long-horizon rollout comparisons over 40 seconds reveal that GigaWorld-1 maintains stable generation quality, in contrast to baselines that suffer sharp degradation. These evaluations collectively confirm that GigaWorld-1's explicit action representations, memory-augmented rollout, and diverse training data yield a calibrated world model suitable for policy evaluation.