One-sentence Summary

Researchers from Tuojing Intelligence, University of Chinese Academy of Sciences, Institute of Automation, Chinese Academy of Sciences, Tsinghua University, and others present GaussianDream++, a compact extension of the GaussianDream VLA policy that inserts World State and Prediction Tokens into the backbone, uses a training-only World Representation Head with static-dynamic factorization to supervise 3D structure and future dynamics, and achieves $98.6\%$98.6% on LIBERO, $87.8\%$87.8% on LIBERO-Plus, and a real-robot success improvement from $29.2\%$29.2% to $52.5\%$52.5% over $\pi_{0.5}$π0.5​ while removing all auxiliary components at inference.

Key Contributions

• Introduces GaussianDream++, a policy-native extension that replaces the dense VGGT/TGE-based prefix of GaussianDream with 20 World State Tokens and World Prediction Tokens embedded directly in the PaliGemma VLA backbone, enabling compact current-geometry and future-dynamics supervision without a dedicated runtime pathway.
• Employs a training-only World Representation Head that decodes these tokens into a Current World and coupled Future Prediction over shared Gaussian primitives, using static–dynamic factorization to preserve persistent structure while focusing residual motion on interaction-relevant regions, and removes the head, renderer, auxiliary objectives, and VGGT/TGE pathway at inference.
• Achieves 98.6% on LIBERO and 87.8% on LIBERO-Plus with clear gains under Camera and Layout shifts, and improves pooled real-robot success from 29.2% to 52.5% over reproduced π₀.₅ while maintaining efficient closed-loop control.

Introduction

General-purpose robotic manipulation requires agents to interpret language goals and convert visual observations into precise actions, a capability that recent Vision-Language-Action (VLA) policies have advanced by transferring semantic priors from large-scale vision-language pretraining to robot control. However, these policies face three key challenges: scene geometry and contact-relevant spatial relationships are often encoded implicitly in 2D features, robot trajectories contain dense physical evidence that behavior cloning leaves largely unused, and current-state understanding and future evolution are typically learned through separate or entangled objectives rather than being jointly structured. Prior approaches either add geometric grounding (e.g., depth, point clouds, 3D representations) to improve current scene anchoring without supervising physical state evolution, or use predictive policies and world models that model future states in RGB or latent spaces, which do not guarantee metrically consistent transitions and can increase deployment cost.

The authors introduce GaussianDream++, a compact, policy-native upgrade to GaussianDream that replaces a dense external Gaussian prefix with 20 World State Tokens and World Prediction Tokens embedded directly in the VLA backbone. World State Tokens reconstruct the current physical scene, while World Prediction Tokens model short-horizon evolution over shared Gaussian primitives, with static-dynamic factorization preserving persistent structure and concentrating residual motion on interaction-induced changes. During training, a lightweight World Representation Head decodes these tokens into renderable current and future Gaussian states, providing dense supervision through RGB, depth, visibility, and motion objectives, but at deployment all auxiliary heads are removed, leaving only the compact tokens for action generation. Experiments show GaussianDream++ achieves 98.6% success on LIBERO and 87.8% on LIBERO-Plus, improving over GaussianDream by 0.8 points overall and by 2.8 and 1.6 points under camera and layout shifts, respectively, while real-robot success improves from 29.2% to 52.5% over a reproduced pi_0.5 baseline.

Method

The authors propose GaussianDream++, a framework that integrates Gaussian supervision directly into the native vision-language-action backbone rather than relying on a dedicated temporal geometry pathway. The system processes multi-view observations and language instructions through PaliGemma alongside two distinct groups of learnable tokens. The overall architecture is shown in the figure below:

These tokens consist of World State Tokens, which encode the current physical scene, and World Prediction Tokens, which capture short-horizon evolution. Their contextualized hidden states form a unified multimodal prefix that directly conditions the flow-matching Action Expert.

Let $o_t$ot​ denote the current multi-view observation, $\ell$ℓ the language instruction, and $r_t$rt​ the robot state. The policy predicts an action chunk $A_t$At​. The authors augment the multimodal prefix with World State Tokens $Z^S \in \mathbb{R}^{N_s \times d}$ZS∈RNs​×d and World Prediction Tokens $Z^P \in \mathbb{R}^{N_p \times d}$ZP∈RNp​×d. In the primary configuration, the model uses $N_s = 16$Ns​=16 state tokens and $N_p = 4$Np​=4 prediction tokens. PaliGemma jointly contextualizes these world tokens with the original visual and language tokens to produce hidden states. The Action Expert directly conditions on this complete augmented prefix to generate actions without requiring an additional world-to-action projector.

During the training phase, a lightweight World Representation Head decodes the same hidden states to generate explicit Gaussian supervision. This auxiliary decoding path converts robot trajectories into dense supervision for geometry, appearance, visibility, and motion. Crucially, future observations are utilized exclusively as supervision targets and never enter the policy forward pass, ensuring that neither rendered images nor explicit Gaussian primitives are consumed by the Action Expert.

To constrain the state representation, the authors enforce Current World reconstruction. The contextualized hidden states of the World State Tokens retain a coarse spatial organization and are expanded into a dense Gaussian feature field. The method decodes geometry and appearance through separate branches. The geometry branch predicts metric depth and non-appearance Gaussian attributes, while the appearance branch operates on stop-gradient geometry features to prevent photometric shortcuts from altering the metric scene structure. Gaussian centers are obtained by unprojecting the predicted metric depth using calibrated camera parameters. The resulting Current World is rendered via differentiable Gaussian splatting and supervised using a combination of photometric, structural similarity, depth, and coverage losses.

For short-horizon anticipation, the model preserves the reconstructed Current World as a shared template and predicts only horizon-dependent geometric changes. For a given prediction horizon, the corresponding World Prediction representation is combined with the Current World features and a learnable horizon embedding to predict a motion coefficient and geometric displacement. Scale, rotation, opacity, and appearance are inherited from the Current World. This shared-primitive construction preserves current-to-future correspondence and allocates predictive capacity strictly to geometric changes. Each Future World is rendered using the camera parameters associated with its specific prediction horizon to prevent viewpoint changes from being absorbed into predicted physical motion. The future supervision combines rendering, metric depth, three-dimensional motion, and static-consistency terms.

The authors jointly optimize action generation and Gaussian world supervision. The complete objective combines the original flow-matching action loss with the Current World and Future Prediction losses:

$$\mathcal{L} = \mathcal{L}_{\mathrm{act}} + \lambda_{\mathrm{cur}} \mathcal{L}_{\mathrm{cur}} + \lambda_{\mathrm{fut}} \mathcal{L}_{\mathrm{fut}}$$L=Lact​+λcur​Lcur​+λfut​Lfut​

This ensures the world-token-augmented prefix remains physically structured and prevents it from collapsing into unconstrained action-only features. The training and deployment paths are deliberately asymmetric. During training, the World Representation Head, Gaussian renderer, and all auxiliary objectives transform trajectories into dense geometric and temporal supervision. At deployment, these components are entirely discarded. The deployed policy retains only the PaliGemma backbone, the World State and Prediction Tokens, and the Action Expert. Consequently, inference requires no online Gaussian decoding, rendering, future rollout, or dedicated prefix-construction pathways, making the additional policy computation minimal.

Experiment

GaussianDream++ is evaluated on LIBERO and LIBERO-Plus for manipulation performance, zero-shot robustness under spatial, visual, linguistic, and embodiment shifts, real-world transfer, and design contributions. It achieves 98.6% average success on LIBERO and 87.8% Overall on LIBERO-Plus, with the largest gains under Camera and Layout perturbations, and improves pooled real-world success from 29.2% to 52.5% over the reproduced baseline. Ablations confirm that structured Gaussian world supervision, not extra token capacity, drives the gains, with metric depth, 3D flow, and static-dynamic factorization each contributing to robustness. The policy adds only 44 ms of inference latency over the baseline by removing the runtime geometry pathway, and qualitative visualizations show that compact world tokens retain decodable scene geometry and coarse future evolution.

GaussianDream++ achieves strong performance on both standard LIBERO and the zero-shot robustness benchmark LIBERO-Plus, with the largest gains under camera and layout perturbations. The improvements are attributed to structured Gaussian world supervision and role-separated world tokens, which help preserve scene geometry and layout under distribution shifts. GaussianDream++ outperforms the reproduced baseline on LIBERO-Plus Overall, with notable gains under Camera and Layout shifts. The model also improves robustness to Noise perturbations, suggesting that dense Gaussian supervision helps when local image evidence is degraded. Performance under Robot shift remains below the reproduced baseline, indicating that embodiment-level generalization is not improved by the proposed approach.

In real-world manipulation trials, GaussianDream++ consistently outperforms the reproduced baseline across both tasks and all condition groups, with the largest gains under standard conditions and the smallest under camera shifts. The method's overall success rate is roughly 20 percentage points higher than the baseline when pooling all tasks and conditions. GaussianDream++ achieves higher success rates than the baseline in every task and condition, including standard, layout, and camera variations. The performance gap between GaussianDream++ and the baseline is most pronounced in the standard condition and smallest under camera shifts. Across all tasks and conditions, GaussianDream++ improves overall success by about 20 percentage points over the reproduced baseline.

GaussianDream++ avoids explicit Gaussian decoding or rendering at runtime by forming a compact world representation directly inside the VLA backbone, resulting in lower latency than GaussianDream's reference. The measured latency for GaussianDream++ is 330 ms, compared to 286 ms for the reproduced baseline and 531 ms for the reported GaussianDream reference, though the latter was measured under different conditions. GaussianDream++ removes the dedicated runtime geometry pathway, unlike GaussianDream which still uses VGGT and TGE to build a 1024-token prefix. The latency of GaussianDream++ is lower than the reported GaussianDream reference, but direct speedup claims are limited by differing measurement environments. The overhead of GaussianDream++ comes from processing 20 additional world tokens, not from online Gaussian construction or rendering.

Ablation studies show that structured Gaussian world supervision, not just added token capacity, drives performance gains in GaussianDream++. Progressively adding current world reconstruction, future prediction, coupled decoding, and static consistency each improves robustness, with the full model achieving the best results. Removing any single supervision objective, especially metric depth, reduces performance, confirming the complementary roles of appearance, geometry, coverage, and motion constraints. Adding world tokens without Gaussian supervision improves LIBERO-Plus Overall only slightly, remaining below the full model, indicating capacity alone is insufficient. Constraining world state tokens with current world reconstruction improves robustness, especially under camera and layout shifts. Coupling current and future predictions over shared Gaussian primitives further boosts performance, and static consistency adds the final gain. Removing metric depth causes the largest drop, particularly affecting camera and layout robustness. Removing metric 3D flow weakens noise and layout robustness, showing rendering alone is insufficient. RGB and alpha supervision are complementary, with RGB affecting lighting and background and alpha affecting visibility-sensitive shifts.

GaussianDream++ outperforms the reproduced baseline on both standard LIBERO and the zero-shot robustness benchmark LIBERO-Plus, with the largest gains under camera and layout perturbations, while performance under robot shift remains below baseline. In real-world trials, it consistently improves success rates across all tasks and conditions, achieving roughly 20 percentage points higher overall success than the baseline. The method also reduces runtime latency by avoiding explicit Gaussian decoding, though direct speedup comparisons are limited by differing measurement environments. Ablation studies confirm that structured Gaussian world supervision, not just added token capacity, drives robustness gains, with metric depth being the most critical supervision objective.