One-sentence Summary

Alaya Lab presents AlayaWorld, a full-stack open-source framework that autoregressively generates long-horizon, interactive video worlds from gameplay and real-world videos, and unifies data preparation, model training, inference acceleration, and deployment to deliver real-time, playable environments with diverse actions, advancing generative world models for gaming and embodied intelligence.

Key Contributions

• The paper presents AlayaWorld, a full-stack open-source framework for interactive generative worlds that supports real-time, open-ended user interaction, including free navigation and actions such as combat, spell casting, and monster summoning.
• The framework integrates an autoregressive diffusion transformer with a prompt-switching mechanism, an AdaLN-style camera-control module, a 3D cache, a history-compression module, an error bank, and few-step distillation.
• The release includes reproducible pipelines, reference implementations, evaluation tools, and comprehensive documentation, providing a practical foundation for future research and real-time applications of generative world models.

Introduction

Interactive virtual worlds are pivotal for 3D games and embodied AI research, where agents need coherent, persistent observations in response to actions. Conventional production pipelines require costly manual specification of assets and rules, making worlds predefined and hard to extend. Video world models that generate future frames conditioned on user input promise a more scalable, data-driven alternative, but they still struggle with unlimited control, spatial-temporal consistency, long-horizon stability, and real-time runtime. The authors review these challenges and introduce AlayaWorld, an autoregressive DiT that combines a prompt-switching mechanism, an AdaLN-style camera-control module, a 3D cache, history compression, an error bank, and few-step distillation to deliver a playable, open-source generative world system.

Method

AlayaWorld is a full-stack framework for interactive generative worlds, built by fine-tuning the LTX-2.3 video backbone and integrating a set of purpose-designed modules that address four core challenges: interaction, consistency, stability, and runtime. The system is designed to support real-time exploration and dynamic user actions while maintaining visual coherence over long horizons and across return visits.

The interaction module supports two complementary modes. For navigation, the authors combine explicit rendered evidence with lightweight architectural injection. A 3D cache is maintained from depth-unprojected frames and is continuously rendered under the player’s target camera trajectory. This rendered cache provides the generator with spatially grounded visual evidence for the queried viewpoint, improving trajectory following and cross-view consistency. The camera condition is then injected into the backbone through an AdaLN-style modulation, which introduces minimal parameter and computation overhead while preserving the efficiency needed for responsive control. The second mode, prompt-driven action, enables freely triggered events such as spell-casting or weapon combat. The authors introduce a chunk-granularity prompt switching mechanism: the text condition can be replaced at any chunk boundary, so that the newly generated content does not affect previously generated frames and avoids re-generation of the existing sequence. This allows the world to react to changing user intent without full-sequence regeneration.

Consistency across long horizons and revisits is handled by a dual memory system. The authors recognize that spatial memory (indexed by viewpoint) and temporal memory (indexed by recency) are complementary. Following the principle of explicit geometric anchor, a 3D cache is maintained and reprojected into the queried viewpoint, providing spatially grounded evidence for previously observed regions and ensuring place identity under loop-closing trajectories. However, because the static cache cannot encode recent temporal dynamics, the system additionally compresses the recent frame history into a lightweight embedding, adopting a method akin to Frame Preservation. The explicit spatial cache supplies persistence for revisited locations, while the compressed temporal history preserves short-term motion and transient changes, together covering the failure modes of purely spatial or purely temporal memory.

Stability over long forward rollouts is treated as a training-time robustness problem. The authors follow the Helios philosophy of exposing the model to drifted histories during training, so that the generator learns to correct errors rather than relying on clean conditioning segments. They further introduce an error bank that stores residual artifacts accumulated during rollout and reuses them as structured perturbations. Critically, error-bank samples are injected into both the memory condition and the target segment, matching the conditions of long-horizon inference where the model must generate from imperfect memory while simultaneously correcting errors in the next segment. This joint perturbation teaches the model to stabilize generation under corrupted history and prevents errors from compounding autoregressively.

For runtime, the goal is to achieve real-time generation under interactive constraints. The authors adopt standard DMD-based distillation to reduce the number of denoising steps per chunk. A small temporal chunk size is used to keep per-chunk latency low and to allow frequent condition update points. Prompt switching is supported at chunk boundaries by simply updating the text condition before the next chunk is generated, which avoids heavy KV-recache mechanisms and keeps the interaction loop simple and predictable. This combination of few-step distillation, short chunks, and frequent prompt switching provides the speed, interaction granularity, and controllability needed for responsive, user-driven exploration.

Experiment

AlayaWorld, fine-tuned from LTX-2.3 for autoregressive 720p generation, is compared against baselines under identical conditions. The experiments demonstrate precise camera control, seamless transitions when switching text prompts mid-generation, and strong loop-closure consistency where revisited regions remain geometrically and texturally stable. Unlike prior interactive world models, which exhibit visual degradation and camera inaccuracies, AlayaWorld maintains faithful control adherence and visual quality over long horizons and diverse artistic styles, confirming its reliable interactive world generation.