One-sentence Summary

Researchers from NVIDIA, University of Washington, KAIST, and UCLA introduce DREAMGEN, a $4\text{-stage}$4-stage pipeline that trains generalizable robot policies via synthetic neural trajectories from image-to-video world models, recovering pseudo-actions with latent action or inverse-dynamics models to enable a humanoid to perform $22$22 new behaviors from single-task teleoperation data, alongside DreamGen Bench, a benchmark whose video generation quality strongly correlates with downstream policy success.

Key Contributions

• Introduces DREAMGEN, a four-stage pipeline that adapts image-to-video generative models to a target robot embodiment, generating photorealistic synthetic videos of familiar or novel tasks in diverse environments, then recovers executable pseudo-action sequences using either a latent action model or an inverse-dynamics model.
• Demonstrates strong behavior and environment generalization, showing that a humanoid robot can perform 22 new behaviors in both seen and unseen environments while requiring teleoperation data from only a single pick-and-place task in one environment.
• Presents DreamGen Bench, a video generation benchmark that establishes a strong correlation between benchmark performance and downstream policy success, offering a low-cost, diagnostic way to evaluate video world models for robotics without requiring a physical robot in the loop.

Introduction

Robot foundation models trained on large-scale human teleoperation data show strong potential for dexterous real-world manipulation, yet the paradigm is bottlenecked by the high cost of manually collecting demonstrations for every new task and environment. Synthetic data generation in simulation offers an alternative, but it typically demands significant manual engineering and suffers from the sim-to-real gap when deploying visuomotor policies on physical robots. Prior generative approaches that augment existing demonstrations with diffusion or video-to-video models also produce limited motion diversity, and most video world model research has focused on real-time planning rather than scalable data generation.

The authors introduce DREAMGEN, a synthetic data pipeline that repurposes state-of-the-art video world models as data generators rather than planners. The pipeline follows a four-step recipe: fine-tune a video world model on a target robot to capture its specific dynamics, prompt the model with initial frames and language instructions to generate large volumes of robot videos, extract pseudo-actions via a latent action model or inverse dynamics model, and finally train downstream visuomotor policies on the resulting video-action pairs, which the authors call neural trajectories. This approach requires minimal manual labor and works across different embodiments, environments, and tasks.

The authors validate DREAMGEN on the RoboCasa simulation benchmark with synthetic data scaled up to 333 times the original demonstrations, achieving log-linear policy improvements. In the real world, they show consistent gains across 9 tasks on Fourier GR1, Franka Emika, and SO-100 robots using only 10 to 13 real trajectories per task. More notably, DREAMGEN unlocks true generalization: it enables a GR1 humanoid trained only on pick-and-place to perform 22 novel behaviors (pouring, tool manipulation, and opening articulated objects) and to operate in 10 unseen environments, where the baseline achieves near 0% success but DREAMGEN reaches 43.2% on new behaviors in seen environments and 28.5% in unseen ones. The authors also release DreamGen Bench, a video generation benchmark that evaluates how well video world models adapt to novel robot embodiments, offering a low-cost diagnostic that correlates with downstream policy performance.

Dataset

The authors introduce DreamGen Bench, a benchmark for evaluating video generative models as world models for robotics. It measures how well existing video generation models can adapt to a specific robot embodiment, internalize rigid body physics, and generalize to new objects, behaviors, and environments.

Dataset composition and sources

• The benchmark covers two training and evaluation setups: simulation on the Franka Emika robot arm and real-world data on the Fourier GR1 humanoid.
• Video data comes from robot environments including RoboCasa (a multiview simulation environment) and other diverse robot environments.
• Four video world models are benchmarked: Hunyuan, CogVideoX, WAN 2.1, and Cosmos.

Key metrics and evaluation details

• Instruction following (IF): generated videos are fed to Qwen-VL-2.5 with specific prompts, producing a binary score (0 or 1) for consistency between the video and task instructions. Human evaluations are also provided, showing an average Pearson correlation above 90% with the model-based scores.
• Physics adherence (PA): videos are scored by VideoCon-Physics, a VLM trained for physics adherence, giving a score from 0 to 1. Because VideoCon-Physics was not trained on multiview RoboCasa data or diverse robot environments, the authors also use Qwen-VL-2.5 to score each video, then average the two scores.

How the data is used

• The four models are evaluated in zero-shot mode (without embodiment adaptation) and in adapted setups, with results and dataset statistics reported in Table 2.
• To test whether DreamGen Bench can proxy downstream robot policy performance, the authors train RoboCasa policies only on neural trajectories generated by each video world model, using 7K trajectories per model. The DreamGen Bench score is the average of IF and PA scores, and the results show a positive correlation with RoboCasa performance.
• For the GR1 humanoid, the authors replay inverse dynamics model (IDM) actions in simulation using the robot's digital twin to empirically assess the quality of the IDM actions.

Method

The authors propose DREAMGEN, a comprehensive pipeline designed to generate synthetic training data for visuomotor robot policies by leveraging video world models. The overall framework, as illustrated in the provided overview, consists of four sequential stages: fine-tuning a video world model, generating synthetic video rollouts, extracting pseudo-actions to form neural trajectories, and training the final policy.

The process begins by adapting a pre-trained video world model to the target robot embodiment. This is achieved by fine-tuning the model on human-teleoperated robot trajectories. To mitigate catastrophic forgetting of prior internet video knowledge, the authors employ Low-Rank Adaptation (LoRA). During this phase, the model learns the specific physical constraints and movement capabilities of the robot. For datasets containing multiple viewpoints, such as RoboCasa and DROID, the frames are concatenated into a grid format before fine-tuning.

Once the model is adapted, it is used to generate synthetic data. The model is prompted with an initial frame and a language instruction to produce video rollouts depicting the intended behavior. This allows for the generation of large volumes of data, capturing both familiar behaviors from the fine-tuning phase and novel behaviors in unseen settings.

Since these generated videos lack action annotations, the third step involves extracting pseudo-actions to create "neural trajectories." The authors utilize two distinct architectures for this purpose, as detailed in the architecture diagram below.

The first approach employs an Inverse Dynamics Model (IDM). The IDM architecture utilizes a diffusion transformer with a SigLIP-2 vision encoder and is trained with a flow-matching objective. It is conditioned on two image frames (current and future) to predict action chunks between them. The model focuses solely on capturing robot dynamics without explicit language or proprioception inputs. To generate labels, a sliding window approach is applied: the model predicts $H$H actions for a window, then slides by one frame to predict the next sequence.

The second approach utilizes a Latent Action Model (LAPA). This model features a transformer encoder-decoder architecture trained with a VQ-VAE objective. It captures visual delta information between frames. To extract latent actions, the model is conditioned on the current frame and a future frame (one second ahead) from the generated trajectory. The output is a pre-quantized continuous embedding representing the latent action. A key advantage of this method is that it does not require ground-truth actions for the target robot during training.

Finally, the visuomotor policy is trained on these neural trajectories. The policy is conditioned on image observations and language instructions, with state information zeroed out since neural trajectories lack explicit state data. The authors demonstrate this training process using three different policy architectures: Diffusion Policy, $\pi_0$π0​, and GR00T N1. The training can be performed solely on neural trajectories or co-trained with real-world trajectories at a 1:1 sampling ratio. For specific embodiments like GR00T N1, separate action encoders and decoders are used to handle the distinct trajectory types.

Experiment

DREAMGEN was validated across three applications: data augmentation, behavior generalization, and environment generalization. Co-training with neural trajectories consistently improved downstream policy performance across simulation and real-world settings, with gains scaling log-linearly with trajectory count. Policies trained solely on generated videos learned entirely new behaviors and transferred to novel environments without physical data collection, far exceeding the baseline. A new benchmark, DreamGen Bench, quantifies instruction and physics following of video world models and shows positive correlation with downstream robot policy performance.

The DreamGen Bench evaluates video generation models on instruction following and physics alignment across simulation and real robot setups. Fine-tuned variants consistently improve over zero-shot baselines, and model-based evaluations align closely with human judgments. The benchmark also correlates positively with downstream robot policy performance. Fine-tuning on embodiment-specific data boosts both instruction following and physics alignment scores relative to zero-shot inference. Human and model-based evaluations are highly consistent, with an average correlation exceeding 90%. Models that score higher on DreamGen Bench tend to yield better downstream robot policy performance.

The DreamGen Bench evaluates video generation models on instruction following and physics alignment across simulation and real robot setups. Fine-tuned variants consistently outperform zero-shot baselines, and model-based evaluations align closely with human judgments, with an average correlation exceeding 90%. Higher benchmark scores also correlate positively with downstream robot policy performance, indicating that fine-tuning on embodiment-specific data improves both metrics.