One-sentence Summary

Researchers from ARC Lab, Tencent IEG, and Peking University introduce WorldCrafter, a video world model whose camera-queryable implicit 3D-aware memory uses a pose-conditioned readout module to compress multiview evidence into target view-specific tokens, enabling streaming scene exploration from a single image or text prompt with improved long-horizon consistency and camera-control accuracy.

Key Contributions

• WorldCrafter is a video world model that learns a camera-queryable implicit 3D-aware memory. A jointly trained memory encoder and pose-conditioned readout module compress multiview historical observations into fixed, target-view-specific tokens before denoising, without explicit depth-based correspondences.
• The model combines this memory with recent temporal context and few-step distillation to enable streaming, real-time scene exploration from a single input image or text prompt.
• Experiments on static and dynamic scenes show a 47.6% improvement in revisit consistency over the strongest baseline and improved camera-control accuracy, while preserving visual quality during minute-scale exploration; controlled ablations validate the memory design.

Introduction

Video world models enable interactive exploration by generating new views as users move the camera, but maintaining consistency over long trajectories requires memory beyond the recent context. Prior methods either incur high computational cost with full-history attention, trade view coverage for efficiency with retrieval, depend on accurate geometry and struggle with dynamic scenes using spatial memories, or favor geometric prediction over the appearance fidelity needed to reproduce previously observed content. The authors introduce WorldCrafter, a video world model with an implicit 3D-aware memory that encodes historical latent frames into a compact representation initialized from pretrained 3D encoders and jointly optimized with the video diffusion transformer. Pose-guided readout extracts view-relevant tokens under a fixed budget, and few-step distillation enables real-time streaming while preserving camera control and minute-scale revisit consistency.

Dataset

The authors describe the following dataset construction and usage.

Sources and composition

• OSP dataset: Large-scale video data covering diverse indoor and outdoor scenes and object motion. The paper uses 760,000 OSP videos for initial training and a filtered subset of 40,000 videos for camera-control and memory-related training.
• DL3DV: Used as an additional real video source, mainly for camera-conditioned training and memory encoder adaptation.
• MIND: Synthetic videos used to improve dynamic subject modeling in the final training stage.

Processing and filtering

• The authors apply Depth Anything 3 to obtain metric-scale camera pose annotations across all data sources.
• They use Qwen2.5-VL to generate video captions.
• Using the generated captions and estimated camera trajectories, they filter a subset of the OSP dataset to keep videos where the camera follows moving subjects. This filtered OSP subset is intended to help the model learn coordinated camera and subject motion.

How the data is used in training

• Stage 1: 760,000 videos from the OSP dataset are used for 5,000 iterations with a global batch size of 32.
• Stage 2: 40,000 videos from the filtered OSP subset and 6,000 videos from DL3DV are used with a global batch size of 128.
• Stage 3: DL3DV and the filtered OSP subset are used to warm up the memory encoder for 5,000 iterations with a global batch size of 16.
• Stage 4: DL3DV and the filtered OSP subset are used for 8,000 iterations with a global batch size of 32, followed by 1,000 additional iterations with synthetic videos from MIND.

Method

The authors propose WorldCrafter, a camera-controllable autoregressive video world model equipped with an implicit 3D-aware memory mechanism. The foundation of the system relies on latent video diffusion, where a video variational autoencoder maps clean video frames into a spatiotemporal latent space. A Diffusion Transformer (DiT) based denoiser operates on these latents, trained via conditional flow matching to predict the velocity field. To extend generation beyond a fixed clip, the model employs a chunk-wise autoregressive approach. At each rollout step, the current latent is evolved through a conditional flow that incorporates both a compact memory representation and a target camera trajectory.

As shown in the figure below:

The pipeline overview illustrates this autoregressive process. The video DiT generates the initial chunk conditioned solely on camera poses since no history is available. As generation progresses, the system accumulates history latent frames. To maintain long-horizon consistency without exceeding computational budgets, the authors implement a memory encoder that maps selected history latents and their corresponding camera parameters into an implicit 3D-aware representation. Instead of retrieving a fixed-length context directly, the memory encoder aggregates geometry and appearance information across the input history. To bound the encoding cost, the system restricts the input to a fixed number of history latent frames. During inference, it retains the latest latent frame and greedily selects complementary frames to maximize the field of view coverage along the upcoming camera trajectory.

Following the memory writing phase, a pose-guided memory readout module queries the implicit representation using a fixed-size set of query poses sampled from the target camera trajectory. This mechanism extracts a fixed number of memory tokens that are highly relevant to the upcoming generation. At each denoising step, the DiT processes the concatenated sequence comprising the memory tokens, recent history frames, and the noisy current chunk. This design adds a dedicated memory stream while preserving chunk-level causality.

For camera conditioning, the target trajectory specifies the camera-to-world pose and intrinsics for each frame. The authors encode relative camera geometry as a positional transformation within self-attention. This is implemented using a parallel camera-attention branch that adopts independent query, key, and value projections. The output is added to the original self-attention through a zero-initialized projection, and this camera branch is applied exclusively to the noisy part of the concatenated sequence.

The base model training is conducted in four distinct stages. Initially, the video DiT is fine-tuned to adapt to a modified inference window that incorporates memory tokens. The second stage introduces camera control by training the camera conditioning branch while keeping the DiT backbone frozen. In the third stage, the memory encoder is adapted to process VAE latents by replacing shallow image-processing layers with a latent patch embedding layer. The final stage jointly trains the memory readout module with the memory encoder, video DiT, and camera conditioning branch to co-adapt the learned memory representation and the DiT token space.

To achieve real-time interaction, the authors apply a pyramid distillation scheme. They adopt a coarse-to-fine pyramid denoising approach with distribution matching distillation to reduce sampling steps, utilizing multiple spatial resolutions. To balance visual fidelity and subject-following ability, a hybrid distilled model is constructed. A low-noise model is distilled using datasets focused on natural appearance, while a high-noise model is distilled with a mixture of datasets to retain dynamic subject modeling capabilities. During inference, the high-noise model performs the preceding denoising steps, and the low-noise model executes the final step, enabling high-speed generation.

Experiment

The evaluation uses a 145-image benchmark covering dynamic and static scenes with five metric camera trajectories per image, comparing WorldCrafter and WorldCrafter-fast against eight recent camera-controllable video world models. Memory and closed-loop revisit experiments show that the proposed models best preserve previously observed appearance and scene structure over long horizons. Camera-control and visual-quality evaluations further indicate accurate trajectory following, strong scene consistency, and temporal coherence, with WorldCrafter achieving the highest overall visual-quality score. Ablations validate the learned implicit 3D-aware memory, joint optimization of the memory encoder and video generator, pose-guided readout, and max-coverage history retrieval, while also showing substantially lower memory-processing cost than depth-based spatial warping.

WorldCrafter and WorldCrafter-fast lead long-horizon revisit consistency, with WorldCrafter-fast achieving the best overall results and WorldCrafter also placing in the top two across all four metrics. Relative to Lyra 2.0, WorldCrafter reduces LPIPS from 0.487 to 0.255 and raises PSNR from 14.050 to 18.016 dB. These gains indicate more faithful recovery of appearance and structure when revisiting previously seen locations. WorldCrafter-fast ranks first and WorldCrafter ranks second on MEt3R, LPIPS, PSNR, and SSIM among the evaluated methods. WorldCrafter cuts LPIPS by roughly half relative to Lyra 2.0 and improves PSNR from 14.050 to 18.016 dB. Lyra 2.0 is the strongest baseline on these consistency metrics, but the proposed models outperform it across all four metrics.

WorldCrafter achieves the lowest camera-control errors across rotation, translation, and pose-matrix discrepancy metrics on the evaluated generated videos. WorldCrafter-fast ranks third on every metric, indicating a speed-accuracy tradeoff relative to the full model. The results show improved camera trajectory alignment over all compared approaches. WorldCrafter outperforms every compared method on all three camera-control metrics. WorldCrafter-fast is the third-best model on rotation error, translation error, and pose-matrix discrepancy. The full model improves over geometry-based baselines such as Lyra 2.0 and Alaya-EVOKE in camera trajectory accuracy.

WorldCrafter achieves the highest aggregate VBench visual quality score among the compared methods, while WorldCrafter-fast records the best temporal flickering score. The proposed models lead in five of the eight reported visual quality dimensions, with gains concentrated in scene consistency and temporal coherence. Among the listed baselines, Echo-WM and Alaya-EVOKE are the strongest overall, while Matrix-Game 3.5 and HY-WorldPlay show lower overall consistency. WorldCrafter leads the aggregate VBench score, and WorldCrafter-fast leads temporal flickering. The proposed models rank first in five of eight visual quality dimensions, reflecting stronger scene consistency and temporal coherence. Echo-WM and Alaya-EVOKE are the strongest listed baselines overall, with Echo-WM leading subject consistency, background consistency, and overall consistency.

The full memory design achieves the best memory reconstruction and camera control across all reported metrics. Replacing learned memory tokens with retrieved context latent frames leads to the largest degradation, while frozen encoders, pose-free readout, and similarity-based retrieval each reduce performance. Joint optimization, pose-guided readout, and max-coverage history selection all contribute to more consistent revisits and better camera estimation. Replacing the learned memory tokens with retrieved history latent frames gives the weakest memory and camera-control results among all variants. Jointly optimizing the memory encoder with the video generator improves memory quality and camera estimates compared with freezing the encoder. Pose-guided memory readout outperforms a pose-free readout, showing that target-pose queries help both memory and camera control. Max-coverage history retrieval improves revisit consistency and camera control over similarity-based retrieval without increasing the number of history inputs. The full model achieves the best memory fidelity, PSNR, SSIM, and camera error metrics among all tested memory design variants.

The experiments evaluate WorldCrafter on long-horizon revisit consistency, camera trajectory control, visual quality, and memory design ablations. WorldCrafter and its fast variant lead in revisit consistency and VBench visual quality, with stronger scene consistency and temporal coherence, while the full model also achieves the lowest camera-control errors. Ablation results show that learned memory tokens, joint encoder optimization, pose-guided readout, and max-coverage history selection all improve memory fidelity and camera estimation. Overall, the proposed memory and control mechanisms enhance generation quality, spatial-temporal consistency, and trajectory accuracy.