One-sentence Summary

CUHK-SZ, SLAI, NUS, et al. introduce SolarWM, an open foundation for interactive video world models that employs a reconfigurable multi-source data engine and backbone-native adaptation to train 5B–33B models on unified frame-aligned data, enabling real-time interaction over minutes-to-hours rollouts after only 5-second training sequences through a three-stage recipe combining bidirectional adaptation, teacher-forced autoregressive initialization, and distribution matching distillation.

Key Contributions

• A reconfigurable multi-source data engine normalizes 1.43 million clips from 10 heterogeneous datasets into a unified frame-aligned contract, separating source processing from mixture construction to enable reproducible and auditable training.
• A backbone-native adaptation framework and a unified three-stage training recipe consisting of bidirectional camera-conditioned adaptation, teacher-forced autoregressive initialization, and distribution matching distillation instantiate four camera-controllable world models (5B–33B parameters) across Wan2.2, LTX-2.5, and MiniMax-H3 while preserving each generator's native representations.
• The complete stack (processed data, pipeline, training recipes, model weights, and inference framework) is released openly, and the resulting causal models support real-time interaction and continuous hour-scale rollouts after training on only 5-second sequences, providing a reproducible and extensible foundation for interactive world-model research.

Introduction

Interactive video world models enable the generation of future visual observations conditioned on camera motion or actions, transforming passive video generators into explorable environments for simulation and content creation. However, existing systems struggle to scale from short clips to long-horizon rollouts because they must maintain visual quality and temporal coherence across heterogeneous data sources and diverse model architectures, while most open implementations remain tied to narrow datasets and backbone-specific training. The authors introduce SolarWM, a fully open and unified foundation that addresses these challenges with a reconfigurable multi-source data infrastructure, a backbone-native adaptation framework that preserves each generator’s native representations, and a simple three-stage training recipe. This yields a family of camera-controllable models spanning 5B to 33B parameters that support open-ended rollouts from minutes to hours after training on only 5-second sequences, providing a reproducible and extensible basis for future world-model research.

Dataset

The authors construct the SolarWM dataset as a flexible open data engine rather than a fixed list of clips. The corpus contains 1.43 million canonical clips from 10 source datasets, organized into 14 independently addressable dataset owners. Every clip is fully processed and annotated, and filtering decisions are applied later through configurable recipes, so rejected clips remain available with full metadata.

Dataset composition and sources

• The 10 source datasets are ABOT-World, DL3DV, MiraData, RealCam, SpatialVID, Sekai-Game, Sekai-Walking, MIND, MultiCamVideo, and OmniWorld.
• DL3DV is split into two temporal views (DL3DV-10s and DL3DV-60s), and three additional Clean Plate owners are derived from MiraData, Sekai-Walking, and SpatialVID, yielding 14 total owners.
• The Clean Plate subsets are created with the LTX-2.3 Clean Plate pipeline to remove people and vehicles while preserving scene layout and camera motion. This produces 543k clean clips (298k SpatialVID-Clean, 135k MiraData-Clean, 109k Sekai-Walking-Clean), each treated as an independent owner with recomputed annotations.

Unified schema and processing

• Every sample follows a unified contract: video, metric camera-to-world poses, per-frame intrinsics, dense captions, metadata, quality metrics, and provenance.
• Camera annotation uses a robust pipeline (Pi3X, MoGe-2, VIPE SLAM) to produce metric-scale 6-DoF trajectories and per-frame intrinsics. When ground-truth or COLMAP poses exist, they are preserved and aligned to metric scale via Sim(3) fitting.
• All 1.43M clips are captioned with the Kimi-K2.6 pipeline. The caption describes only the persistent environment and stable scene content, explicitly excluding dynamic entities, camera motion, and speculative wording to avoid leaking camera-control information.
• A vector of quality and motion metrics is computed for every clip: camera integrity checks, visual quality (DOVER scores, saturation), motion (VMAF Motion, dense correspondence magnitude), temporal consistency (scene cuts, transitions), and semantic suitability (quality flags, entity density, scene type).

Filtering and quality tiers

• Source-aware filtering policies assign each clip to one of three physical labels: xhigh (satisfies both kept and stricter promotion rules), high (satisfies kept rule only), or rejected (with recorded failure reasons).
• Policies are asymmetric across owners. For example, Sekai-Game preserves high-motion game clips without generic camera-geometry gates, while clean owners require post-transformation geometry checks.
• Of the 1.43M clips, 876k are kept: 471k in the high tier and 404k in xhigh. The remaining 549k clips are retained in the rejected partition with full annotations.

How the data is used

• The authors separate the physical corpus from logical recipes and model views. Recipes define split membership, tier policy, source weights, and repeat factors without duplicating video data.
• A short-horizon recipe (81 frames) spans ten owners and contains 600,320 physical training rows. Source balancing applies a repeat factor of 6 to ABOT, MiraData, and Sekai-Game, producing 870,210 virtual occurrences per epoch. A heldout test view contains 1,000 rows.
• Temporal views for short- and long-horizon generation are constructed from the same physical clips, enabling flexible recipe design without re-running expensive preprocessing.

Key statistics

• Total corpus: 1.43M canonical clips, 29k shards, approximately 25.85 TB.
• Kept clips: 876k (471k high, 404k xhigh); rejected: 549k.
• Clip length distribution: the majority (858k clips) fall in the 153–956-frame range; all 78k clips shorter than 81 frames are rejected; 82k kept clips have at least 957 frames.

Method

The authors propose SolarWM, an interactive world-model framework that unifies multiple video-generation backbones under a shared data recipe, processing contract, and camera-geometry interface. To enable precise camera control across different architectures, the framework leverages fused projective rotary positional embeddings (fused-PRoPE). The backbone first applies its native video RoPE to represent temporal and spatial positions. Camera poses and intrinsics then determine projective rotations that are directly applied to the query, key, and value tensors within the existing self-attention path. This design injects camera motion through the attention computation itself, avoiding the need for separate control branches or additional attention passes.

To ensure high-quality supervision for camera-controlled generation, the authors implement a rigorous data processing pipeline. Dynamic entities such as people and vehicles often introduce ambiguous motion that is neither explained by the camera nor consistently controllable. To address this, the authors employ the LTX-2.3 Clean Plate IC-LoRA pipeline to remove these dynamic elements while preserving the static scene layout and source camera trajectory.

As shown in the figure below:

For each accepted source interval, the transformation utilizes eight denoising steps with a strength of 1.0 and a specific Clean Plate prompt. Inputs are processed at a resolution of 1248x704, and outputs are saved at 1280x720 and 16 fps. Because the removal process can introduce texture artifacts or temporal discontinuities, captions, visual metrics, semantic metrics, and camera diagnostics are recomputed for every output window to guarantee data quality.

The world model is trained through a structured three-stage pipeline designed to transition from a pretrained video generator to a causal, few-step autoregressive model.

The first stage is Bidirectional Adaptation. Given a clean video latent $\mathbf{z}_0$z0​ and conditions $\mathbf{c}$c consisting of text, image, and camera inputs, the model predicts the native flow target $\mathbf{u}_t$ut​ from a noisy latent $\mathbf{z}_t$zt​. The objective minimizes the flow matching loss:

using bidirectional attention over the complete training window. This adapts the pretrained visual prior to camera-conditioned world data while retaining unrestricted temporal context.

The second stage is Teacher-Forced AnyFlow Initialization, which introduces causality while keeping the optimization target fully supervised. The latent sequence is divided into ordered blocks. When predicting the current block, the model attends only to the current noisy state and clean ground-truth history, hiding future blocks. For a sampled pair of noise levels $(t, r)$(t,r), the training objective is:

where $\ell_{\mathrm{AF}}$ℓAF​ supervises the flow map between arbitrary noise levels. Initialized from the bidirectional model, this stage acts as a short causal adaptation that directly yields a few-step autoregressive checkpoint, bypassing the need for separate Causal ODE or Causal Consistency Distillation stages.

The final stage is DMD-based Causal Training, which addresses the exposure gap between teacher-forced training and inference on model-generated trajectories. A causal student generates trajectories using a detached rollout-and-replay procedure to allow KV Cache gradients. A frozen bidirectional teacher estimates the target distribution, while a trainable fake-distribution model tracks the evolving rollout distribution of the student. The distribution-matching direction for the student is derived from their difference. The DMD update takes the form:

where $J_{G_{\psi}}$JGψ​​ maps the signal back to the generator parameters and $w(t)$w(t) controls its scale across noise levels. This separation preserves a stable quality reference while allowing the training signal to adapt to the changing student distribution.

Experiment

The evaluation first tests bidirectional pretrained models on out-of-domain initial images from external generators, showing that the unified data and camera-conditioning contract generalizes across unseen visual domains while preserving each backbone's visual prior. Distilled causal generation is then assessed in-domain and out-of-domain, demonstrating that the model can follow prescribed camera trajectories, maintain scene consistency, and plausibly infer newly revealed content from a single image. In long video generation, the model sustains coherent minute-scale and hour-scale rollouts, with hour-long sequences remaining visually coherent despite training only on 5-second clips, highlighting strong generalization and sustained camera control.

Most interactive video world models release only inference code and partial weights, while training code, data, and full reproducibility remain scarce. Among the surveyed systems, only DIAMOND provides an executable training recipe, and none offer a complete source-to-training pipeline with exact data mixtures. SolarWM is introduced as a fully open training framework with a unified recipe across multiple video backbones, aiming to address these gaps. DIAMOND is the sole system in the table that releases training code and an exact recipe, but it does not provide data or a full source-to-training pipeline. All other listed systems restrict public artifacts to inference code and either full or reduced weights, omitting training, data, and reproducibility components. SolarWM promises a fully open stack with training stages, processed data, exact recipes, and support for multiple video backbones, contrasting with the partial releases of prior systems.

The Kimi-K2.6 captioning pipeline generates a dense English description of persistent scene elements such as architecture, terrain, and lighting, while explicitly excluding dynamic entities, actions, and camera motion to prevent control-signal leakage. The model outputs a strict JSON object with exactly six fields, using only the dense caption as the text condition; the remaining fields serve as structured annotations for downstream use. Deterministic generation is enforced with temperature 0 and disabled thinking. Captions describe only persistent environment and stable scene content, explicitly excluding people, animals, vehicles, actions, and camera motion. The response is a fixed JSON schema with six fields, generated deterministically at temperature 0 and with thinking disabled. Only the dense_caption field becomes the text condition; entity density and visual quality scores are stored as structured annotations.

Each dataset owner uses a frozen, asymmetric policy that assigns samples to xhigh, high, or rejected tiers. The policies differ in which visual and geometric gates are applied: game trajectories skip camera-geometry and saturation checks, while clean sources enforce post-transformation geometry. The xhigh tier imposes stricter thresholds on metrics such as quality score, entity density, and motion limits. Sekai-Game accepts quality scores as low as 3, whereas all other sources require a minimum quality of 4. MiraData uniquely mandates a non-empty caption, a frame count of at least 81, and a fixed resolution of 1280×720.

The released corpus contains 1.4M clips from seven dataset owners, with 876k retained across high and xhigh quality tiers. Owner-level contributions differ sharply: MultiCamVideo and ABOT are predominantly kept, while the two MiraData variants together supply the vast majority of rejected clips. Temporal analysis reveals that all clips shorter than 81 frames are rejected, and most retained clips fall in the 153–956 frame interval. MultiCamVideo provides the largest share of high-tier clips, while ABOT consists almost entirely of xhigh clips. MiraData and MiraData-Clean each have over 85% of their clips assigned to the rejected partition. The high and xhigh tiers contribute roughly equally to the 876k kept clips. All 78k clips shorter than 81 frames are rejected, whereas the long tail of clips with at least 957 frames contains 82k kept clips. The 153–956 frame range holds the majority of all clips and over half of the kept clips.

The SolarWM model family preserves each backbone's native interfaces while standardizing data and camera contracts. Image conditioning, audio handling, and VAE compression ratios vary across the four routes, reflecting the original design choices of the underlying video-generation models. The 5B model uses a text-to-video latent path with no separate image input, the 14B model employs an explicit mask-and-latent image conditioning, and the larger models adapt or disable native audio components. Image conditioning differs markedly: the 5B model has no separate image tensor and treats the first latent as a clean anchor, while the 14B model uses a 4-channel mask plus 16-channel image latent as an explicit input. Audio handling is either absent (Wan-based models), removed (LTX drops its 3.69B audio stream and cross-attention), or retained but disabled (Minimax keeps audio target rows with loss disabled), focusing all routes on visual generation.

SolarWM is introduced as a fully open training framework that releases training code, processed data, and exact recipes across multiple video backbones, directly addressing the reproducibility gaps left by prior systems that typically provide only inference artifacts. A deterministic captioning pipeline generates dense environment descriptions while explicitly excluding dynamic entities to prevent control-signal leakage, producing structured JSON annotations for downstream use. Dataset filtering applies owner-specific, asymmetric policies to assign clips into quality tiers, yielding a curated corpus of 876k clips where contributions vary sharply by source, with the vast majority of rejected clips originating from MiraData variants. The model family standardizes data and camera contracts while preserving each backbone's native interfaces, focusing all routes on visual generation through varied image conditioning strategies and the removal or disabling of audio components.