One-sentence Summary

Alaya Lab introduces AlayaWorld, an open-source, 15B-parameter interactive video world model that autoregressively generates long-horizon 24-fps video at 540p and 720p under camera trajectories and switchable text prompts, using a bounded visual context that combines a persistent sink frame, compressed temporal history, geometry-aligned spatial memory, and recent-frame conditioning, trained with corrupted histories and prediction residuals from its own roll-outs to reduce drift, and distilled via discrete autoregressive distillation (distribution-matching distillation, self-forcing++, and consistency distillation) to 4-step inference, achieving state-of-the-art on iWorld-Bench.

Key Contributions

• AlayaWorld is a 15B video diffusion transformer that generates 24-fps, 540p/720p interactive video by autoregressively producing short latent chunks under camera trajectories and switchable text prompts, and maintains spatiotemporal consistency with a bounded visual context combining a global scene anchor, compressed temporal history, geometry-aligned spatial memory, and recent-frame conditioning; on iWorld-Bench, it achieves the best overall performance across generation quality, trajectory following, and memory ability.
• Long-term drift is reduced by training the model with corrupted histories and prediction residuals from its own roll-outs, teaching recovery from imperfect context and improving stability over extended sequences.
• A discrete autoregressive distillation method combines distribution-matching distillation, self-forcing++, and consistency distillation to reduce inference from approximately 30 sampling steps to four steps per chunk, preserving the full control and memory stack while enabling low-latency interaction.

Introduction

Video world models promise to generate interactive virtual environments directly from user inputs, sidestepping the expensive, tightly coupled production pipelines of modern game development. However, adapting a video generator into a genuine world model means simultaneously satisfying four competing demands: accurate response to camera trajectories and evolving intent (interaction), preservation of spatial structure and visual identity across viewpoint changes (consistency), stability over long roll-outs without drift, and low-latency efficiency. These properties cannot be tackled independently, as broader interaction complicates consistency and longer generations amplify residual errors. The authors present AlayaWorld, a unified autoregressive framework built on a 15B video diffusion transformer. It generates short latent chunks under continuous camera control and switchable text prompts while maintaining a bounded visual context that combines a global scene anchor, compressed history, geometry-aligned spatial memory, and recent-frame conditioning. Drift-aware training and a discrete autoregressive distillation procedure further improve long-horizon robustness and reduce inference latency, enabling stable, controllable, and memory-aware navigation across diverse scenes.

Dataset

The authors construct a single training corpus of 222,147 video clips from seven diverse sources, deliberately mixing real-world captures and synthetic renderings. All sources are normalized into a unified record consisting of the video, per-frame camera intrinsics and poses (recovered if absent), and a hierarchical caption aligned to the clip timeline.

Real‑world sources

• Sekai‑Real – first‑person urban walking trajectories.
• SpatialVid – short clips with dense indoor camera motion.
• RealEstate10K – real‑estate walkthroughs with broad indoor coverage.
• DL3DV – long, contiguous multi‑view walkthroughs of real environments, kept as a standalone source.
• MUGEN – internally curated from YouTube videos, featuring varied camera trajectories and accurate annotations. For real‑world clips without camera metadata, per‑frame intrinsics and poses are recovered using ViPE.

Synthetic sources

• GameVerse – internal large‑scale gameplay corpus containing 124,116 clips of approximately 66 seconds each.
• GenEvent – 6,490 event‑centric clips (average 10.6 s) synthesised by a generative video model, exposing the model to open‑domain, action‑triggered events.

Unified filtering pipeline All clips must pass six mandatory gates, with thresholds calibrated per source on a small profiling slice:

• Technical validation: decode integrity, resolution ≥720p, duration ≥3 s, frame rate 24–65 fps, codec H.264/HEVC/AV1.
• Photometric validation: reject extreme exposure and black borders (maximum border ratio 0.10).
• Shot‑boundary filtering: ensure each retained clip is a single continuous shot.
• Motion analysis: remove static clips, score temporal consistency, assign camera‑motion buckets, and detect pose‑free shake.
• Text/interface suppression: EasyOCR‑based text detection and pixel‑stability UI mask (maximum overlay ratio 0.04).
• Person control: YOLO11 detector bounds foreground‑human count and screen occupancy. For sources with estimated camera trajectories, an additional pose stability gate evaluates jitter, peak acceleration, reconstruction residual, and drift. Final training clips are those that pass every active gate.

Caption annotation Each clip receives a two‑level annotation generated by a vision‑language model (Kimi‑K2.6 as default) using timestamped frames at 1–2 fps.

• Video‑level context: global attributes (weather, time of day, location type, camera perspective, camera motion, video style) drawn from a reduced vocabulary of 26 values.
• Segment‑level tracks: the clip is partitioned into timestamped segments, each with separate semantic tracks for primary subject motion, environmental dynamics, static scene attributes, and camera viewpoint/motion. These are fused into a full_prompt (5–9 present‑tense sentences) and a short_prompt (15–45 words) used for caption dropout and multi‑caption augmentation.

How the data is used The entire filtered corpus of 222,147 clips is used as the training mixture for the video world model. The heterogeneous blend of real and synthetic data, together with explicit camera metadata and hierarchical captions, teaches the model photorealistic scene evolution and controllable camera‑conditioned navigation. Training leverages the short_prompt for caption dropout, and the separated subject/camera tracks let the model distinguish, for example, a walking subject from a dolly‑in camera motion.

Method

The authors introduce AlayaWorld, an interactive video world model built upon the LTX-2.3 architecture. By removing the audio module from the original 22B multimodal checkpoint, they retain a 13B video Diffusion Transformer (DiT) as the core backbone. The model is designed to generate video autoregressively in the VAE latent space, chunk by chunk, enabling controllable camera-conditioned navigation and photorealistic scene evolution.

To anchor the model in diverse visual domains and motion geometries, the authors construct a heterogeneous training corpus combining real-world captures and synthetic renderings.

As shown in the figure below:

This mixture exposes the model to various camera motions, such as forward walking, indoor traversal, and panoramic sweeps, across domains ranging from crowded streets to game-engine renderings.

Autoregressive Formulation and Conditioning The causal video VAE encodes a clip into a latent sequence partitioned into chunks of $K=4$K=4 latent frames. At each step $i$i, the model generates the target chunk $z_i$zi​ conditioned on the past latents $z_{<i}$z<i​, the target camera trajectory $\pi_{\le i}$π≤i​, and an optional text prompt $y_i$yi​. The generation factorizes causally as:

Instead of using separate cross-attention banks, the authors inject conditioning via an in-context prefix. For chunk $i$i, the model prepends four clean conditioning streams to the $K$K noised target frames, forming a single token sequence processed by full self-attention:

The sink $s$s acts as a global identity anchor pinned at temporal position 0. The temporal memory $h_i$hi​ is a compressed embedding of the last $L=6$L=6 latent frames. The nearby condition $n_i$ni​ is the most recent latent frame providing frame-to-frame continuity. The spatial memory $g_i$gi​ provides geometry-aligned visual evidence for the queried viewpoint.

Spatial Memory Construction To maintain long-range spatial consistency, the spatial memory maintains an explicit cache of previously generated frames with their monocular depth and camera pose. To render this cache into the current target view, the authors retrieve up to 10 frames via greedy maximum-coverage selection. Each selected frame is warped into the target view by forward splatting:

The warped image is VAE-encoded to form $g_i$gi​, and a coverage mask is used as a self-attention key bias to ignore unobserved regions.

Training Pipeline The training process proceeds in three distinct stages.

Stage 1: Bidirectional Pre-Training The authors first adapt the general video prior to their domain through a full-parameter fine-tune. The model remains fully bidirectional, trained on a weighted mixture of scene and camera-pose corpora, gameplay recordings, and event clips. An adaptive sigma-shift schedule scales the flow-matching timestep shift with clip length.

Stage 2: Autoregressive Model Training This stage transforms the bidirectional generator into a controllable world model. It begins with history pre-training, where a history-compression module $H_{\phi}$Hϕ​ is trained via a LoRA adapter to compress recent frame history into a lightweight embedding, optimized under a flow-matching objective:

Subsequently, full-stack fine-tuning unfreezes the backbone and trains dedicated modules. Camera control is injected via an AdaLN module using Fourier-embedded relative-pose increments:

A next-forcing head reinforces causal continuity by predicting the next chunk from intermediate features, supervised at a shifted noise level $\tilde{\tau} = \frac{10\tau}{1+9\tau}$τ~=1+9τ10τ​:

To combat error accumulation during long roll-outs, the authors employ an anti-drift training strategy. This includes Helios drift simulation, which degrades context tokens with additive noise, blur, or saturation shifts, and an error bank that replays the model's own reconstruction residuals into the context.

Stage 3: Post-Training Acceleration To enable interactive generation, the authors distill the multi-step teacher into a 4-step student. They utilize Distribution-Matching Distillation (DMD) to match the teacher's output distribution. The student is trained to minimize the score-difference gradient:

Self-forcing++ is incorporated by having the student roll out its own multi-chunk trajectories and scoring them against the teacher, closing the train-inference gap. Finally, a consistency distillation loss enforces trajectory-invariant outputs against an EMA target to suppress flicker at chunk boundaries:

The combined objective allows the distilled model to generate at 4 sampling steps per chunk while retaining full camera control and memory capabilities.

Experiment

AlayaWorld is evaluated on the iWorld-Bench and WorldMark test suites against open-source models, where it demonstrates leading performance in trajectory following and long-term memory, as well as strong visual consistency. Qualitative experiments showcase its ability to perform camera-controllable navigation, dynamic prompt-driven actions, and consistent world generation, with limited degradation over long autoregressive rollouts. Overall, the spatial and temporal memory mechanisms prove effective for stable, coherent interactive generation.

The training corpus contains 222 147 clips from seven sources, blending real-world captures (91 541 clips) for photorealism and geometry with synthetic renderings (130 606 clips) for scalable camera and action controllability. Internally curated sources dominate: GameVerse contributes 124 116 synthetic game clips and MUGEN adds 21 436 real first-person clips, together providing over 65% of the data. This mixture enables the model to achieve strong trajectory following and memory retention while maintaining visual quality over long autoregressive rollouts. Real-world clips from five sources, including first-person walking and indoor navigation, supply photorealism and scene layout priors. Synthetic clips from GameVerse (124k) and GenEvent (6.5k) outnumber real clips and provide scalable access to controlled camera motions and action-driven dynamics. Internally curated MUGEN and GameVerse together account for 145k clips, ensuring consistent quality and camera-motion coverage across the mixture. AlayaWorld trained on this mixture yields the best trajectory smoothness and accuracy, alongside leading memory ability, demonstrating effective integration of real and synthetic data for stable long-horizon generation.

Each video clip is annotated with segment-level tracks that disentangle subject motion, environment dynamics, static scene attributes, and camera description. The free-text descriptive tracks are fused into a full prompt for the text encoder and a short prompt for caption dropout, while a separate 16-value camera_path enum provides a discrete camera-trajectory control signal independent of subject motion. Subject motion and camera motion are annotated in separate tracks, enabling the model to distinguish cases like a walking agent under a dolly-in camera or a static subject under a panning viewpoint. The camera_path track uses a fixed set of 16 discrete values as a direct control signal, decoupled from the descriptive text tracks. All descriptive tracks are free-text and are merged into a 5–9 sentence full_prompt and a 15–45 word short_prompt for training.

AlayaWorld achieves the best overall performance on most metrics under a reduced sampling budget. It delivers substantially higher brightness consistency, color-temperature constraint, and sharpness retention than all competitors, while being competitive in image quality. AlayaWorld also records the highest motion smoothness and leads on both memory ability metrics, reflecting strong visual stability, trajectory control, and long-term memory. Brightness consistency and color-temperature constraint are far higher than those of any other model, with AlayaWorld nearly reaching the maximum score. Sharpness retention improves markedly over the next best method, indicating effective mitigation of blur and sharpness degradation during autoregressive generation. AlayaWorld does not achieve the highest image quality score but remains competitive with existing video and interactive world models. Motion smoothness is the best among all evaluated methods, showing that generated observations follow action trajectories with high temporal coherence. AlayaWorld consistently leads on both memory ability metrics, demonstrating strong preservation of visual appearance and spatial structure when revisiting previous regions.

AlayaWorld is trained on a mixture of real