One-sentence Summary

Researchers from KAIST and NVIDIA introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner by extracting compact latent actions from its rich spatiotemporal features, compressing each latent video frame $300\times$300× from 1,200 to 4 tokens, and leveraging the model’s generative prior to capture action-relevant dynamics disentangled from visual appearance, yielding state-of-the-art performance on LARY-Bench among latent action models and zero-shot transfer across embodiments, environments, and action parameterizations.

Key Contributions

• DreamX repurposes a pretrained video diffusion model as a universal latent action learner, extracting compact latent actions from the model’s spatiotemporal features and compressing each latent video frame by 300x, from 1,200 to 4 tokens.
• Using the video model’s generative prior for future-frame reconstruction, the learned latent actions capture action-relevant dynamics disentangled from visual appearance and provide a shared action interface across human and robot embodiments, environments, and action parameterizations.
• On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models, supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments without paired source-target training data, and enables policy learning in a compact latent action space that reduces computational cost.

Introduction

Generalist robot policies need diverse manipulation data, but robot demonstrations are expensive and action labels do not transfer cleanly across different embodiments. Human egocentric videos are a scalable alternative, yet the same behavior can map to different low-level actions for different bodies or control spaces, so a shared action representation is necessary. Prior latent action models learn action codes from visual transitions, but reconstruction alone allows them to encode appearance or viewpoint information and produces representations tied to their training embodiments. The authors introduce DreamX, a framework that reads latent actions directly from a pretrained video diffusion model using learnable query tokens. DreamX condenses frame-to-frame change into four latent tokens and supervises them through future-frame reconstruction, yielding state-of-the-art latent action performance on LARYBench and enabling human-to-robot transfer without paired training data.

Method

The authors build DreamX upon Cosmos-Predict2.5, a pretrained video diffusion transformer. A causal video autoencoder compresses a video clip into $N$N latent frames $\mathbf{V} = (V_0, \ldots, V_{N-1})$V=(V0​,…,VN−1​). The model learns future-video generation through flow matching by minimizing the objective:

where $v_{\theta}$vθ​ predicts the flow velocity. This generative prior captures spatial context and temporal relationships.

DreamX adapts this pretrained backbone to encode observed dynamics into latent actions and supervise them via future-frame reconstruction.

Refer to the framework diagram.

In the encoder, the authors append learnable query tokens $\mathbf{q}$q to the video token sequence. A group of $K$K query tokens $\mathbf{q}_g$qg​ is assigned to each latent frame $V_g$Vg​. Processed jointly through the backbone spatiotemporal attention, their output features form the latent-action group $Q_g$Qg​. The extraction is formulated as:

where $\ell$ℓ is the language instruction. The attention mechanism is frame-causal for query-to-video interactions, allowing $Q_g$Qg​ to attend to $V_{0:g}$V0:g​, while query-to-query attention remains unrestricted.

For the decoder, the same pretrained backbone reconstructs future frames conditioned on $Q_{0:N-1}$Q0:N−1​ and the initial frame $V_0$V0​, without text conditioning. This forces the latent actions to encode motion rather than visual cues. When action labels are available, the decoder also predicts action chunks $\hat{A}$A^ from the proprioceptive state $p_0$p0​:

The training objective combines video and action generation:

The learned latent actions capture motion transferable across embodiments. Given a source video, the frozen encoder extracts latent actions $Q^{\text{src}}$Qsrc.

As shown in the figure below:

For video transfer, $Q^{\text{src}}$Qsrc is combined with a target initial frame $V_0^{\text{tgt}}$V0tgt​ to generate a target video. For action transfer, it is combined with a target robot state $p_0^{\text{tgt}}$p0tgt​ to decode executable low-level actions:

For downstream policy learning, the authors compress the latent actions into a low-dimensional code $Z$Z using a variational bottleneck, reducing the dimension from $D$D to $d \ll D$d≪D while preserving the token layout. The bottleneck is trained with feature reconstruction and KL regularization. The DreamX policy $\pi(\cdot)$π(⋅) then generates these compact codes from the current frame $V_0$V0​ and instruction $\ell$ℓ. A separate DiT action head predicts low-level actions conditioned on the latent-action features, observation, instruction, and state $p_0$p0​:

The policy is trained with:

Experiment

The experiments evaluate DreamX, a latent action model built on Cosmos-Predict2.5 and pretrained on mixed robot and human videos, across representation quality, cross-scene and cross-embodiment transfer, downstream policy learning, and ablations. DreamX outperforms baselines on motion regression and classification, transfers human motion to robots zero-shot, and enables open-loop action transfer that reproduces demonstrated motion. In a real-world drawer-closing policy test it achieves 96.7% average subgoal success, while ablations show auxiliary action labels add little and attention analysis confirms the latent actions focus on hands and grippers to specify motion.

On LARYBench, DreamX provides the strongest action representation quality, outperforming latent action models that encode either frame pairs or video clips. It achieves better normalized action regression error and higher action classification accuracy across all reported benchmarks. Against the strongest baseline, DiLA, DreamX cuts regression error by more than half and improves classification accuracy by roughly seven percentage points. DreamX achieves the best reported results on LARYBench for both action regression and action classification. Compared with DiLA, DreamX lowers regression error by more than half and improves classification accuracy by roughly seven percentage points. Among the compared frame-pair latent action methods, LAPA-SigLIP2 is the strongest, with lower average regression error and higher average classification accuracy than the other listed frame-pair methods.

In zero-shot human-to-robot video transfer, DreamX achieves better generation quality than AdaWorld and VideoWorld 2 across all reported metrics. It records the lowest FVD and LPIPS values while attaining the highest PSNR and SSIM, indicating stronger motion transfer and visual preservation. This advantage occurs without paired human-robot training data. DreamX attains the lowest FVD and LPIPS, suggesting improved video fidelity and perceptual similarity compared with the baselines. DreamX achieves the highest PSNR and SSIM, reflecting better reconstruction quality and structural consistency. AdaWorld shows visual degradation in the out-of-distribution setting, while VideoWorld 2 occasionally introduces a human hand despite its appearance-disentangling design.

In the real-world drawer-closing evaluation, DreamX reaches higher success rates than Mimic-Video across all tested drawer configurations, with the strongest gains on tasks where Mimic-Video performed poorly. DreamX also achieves substantially faster inference, reducing per-call latency from tens of seconds to roughly one and a half seconds by predicting compact latent actions rather than future video. DreamX outperforms Mimic-Video in every evaluated drawer configuration, with particularly large gains on the lower-drawer and both-drawer tasks. Inference latency is about nineteen times lower for DreamX, dropping from tens of seconds to approximately one and a half seconds per policy call. The higher success and lower latency are attributed to DreamX generating compact latent actions alongside low-level actions instead of generating future video.

Ablating encoder supervision shows that adding action labels changes regression and classification performance only slightly. Regression MSE improves marginally with action supervision, while classification accuracy dips by a small amount. The comparable results suggest DreamX learns motion and action semantics from video alone without requiring action labels. Adding action supervision gives a small regression MSE improvement but also a slight drop in classification accuracy. RGB-only performance is close to RGB+action performance, indicating that video alone is sufficient for learning action semantics.

The experiments evaluate DreamX on action representation quality, zero-shot human-to-robot transfer, real-world manipulation, and encoder supervision. DreamX outperforms latent action baselines on LARYBench, achieves better motion transfer and visual preservation than AdaWorld and VideoWorld 2 without paired human-robot data, and attains higher real-world drawer-closing success rates with substantially lower inference latency. Ablations show that adding action labels changes performance only slightly, indicating that video alone is sufficient for learning action semantics.