One-sentence Summary

Alaya Lab presents Programmable World Model, a framework that decouples persistent world-state evolution from visual generation by translating natural-language instructions into executable programs, maintaining explicit state through state-augmented 3D oriented bounding boxes, and leveraging a pretrained video renderer to achieve 94% Count Accuracy and 98% State Accuracy on Combat-StateBench.

Key Contributions

• The paper introduces Programmable World Model, a framework that decouples explicit world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, while a lightweight engine maintains a persistent global world state, including off-screen entities and non-visual attributes.
• The paper proposes state-augmented 3D oriented bounding boxes as an intermediate representation connecting explicit world state to visual generation. These boxes, together with the target camera trajectory, are deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer.
• The paper presents CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, the method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation.

Introduction

Recent video world models aim to build interactive environments by predicting visual observations from past frames and user actions, but turning them into reliable world engines remains challenging. Existing systems offer limited entity-level control, lack a persistent global state that tracks off-screen objects and nonvisual information, and cannot execute user-defined rules that consistently govern world behavior. Prior work either relies on implicit state in pixel-level generation or learns state transitions that can accumulate errors. The authors introduce Programmable World Model, which decouples world-state evolution from visual generation. A coding agent translates instructions into executable programs that define entity states and rules, while a deterministic state compiler converts state-augmented 3D oriented bounding boxes into pixel-aligned spatiotemporal controls for a generative renderer, enabling entity-level control, persistent state management, and user-programmable world rules.

Dataset

The authors construct a training dataset from unlabeled gameplay videos and a controlled benchmark for evaluation, both processed through an automatic data engine.

• Training data sources

  • Raw HUD-free footage from three games: Cyberpunk 2077 (first-person), Forza Horizon 6 and Grand Theft Auto V (third-person with varied camera angles).
  • The exact number of videos and total duration are not disclosed; the dataset is defined by the diversity of perspectives and in-game environments.

• Automatic annotation pipeline (Data Engine)

  • The engine turns raw videos into structured, camera-aligned conditioning maps without manual labels.
  • Camera estimation: ViPE recovers intrinsics, per-frame poses, and metric depth, lifting observations to 3D and establishing a shared world coordinate system.
  • Semantic discovery: A Qwen3-VL agent generates a global caption and identifies discrete, countable object categories for the scene. This vocabulary can be replaced with a predefined set for domain-specific data.
  • Instance segmentation and tracking: SAM3 produces temporally consistent instance masks, 2D boxes, semantic labels, and track IDs. Masks with insufficient visible area are discarded to suppress unreliable tiny or heavily occluded objects.
  • Object trajectory recovery: For each tracked instance, Wild-Det3D estimates per-frame 3D oriented bounding boxes (center, dimensions, orientation, semantic label, track ID). Boxes are transformed into the world frame using camera poses, forming temporally consistent trajectories. Object motion is computed from world-space displacement, rotated into the current camera frame, and quantized into seven states: static, left, right, up, down, forward, backward. A small displacement threshold identifies static objects.
  • Conditioning map generation: Projected 3D boxes are rasterized per frame with z-buffer visibility resolution, producing identity, semantic, and direction maps. From these, the final identity, semantic, and object-motion maps are built to condition the model.

• Training data composition and usage

  • The processed videos yield paired video-control training samples: each frame is associated with identity, semantic, and object-motion maps.
  • The data is used to train the video generation model; no explicit mixture ratios or split details are given.

• Evaluation benchmark: CombatStateBench

  • 50 curated clips designed to test whether generated videos faithfully reflect engine-maintained world states.
  • For each clip, the Data Engine reconstructs the initial 3D layout (entities, 3D boxes, semantics, camera parameters) from the first frame. An AI agent then evolves a short combat scenario, producing a full sequence of box-based world states that include camera and entity motion, interactions with off-screen entities, and persistent death states.
  • Each sequence contains synchronized 3D boxes, entity states, camera parameters, projected entity controls, and instance masks.
  • An automatic verifier checks reprojection, metric depth consistency, box geometry, ground contact, temporal continuity, prescribed motion, and state transitions; only sequences passing all checks are retained.

• Processing details (cropping/filtering)

  • No explicit frame cropping is described; the pipeline operates on full frames.
  • Filtering: instance masks with too small a visible area are removed during tracking; a displacement threshold is used to label static objects in motion quantization.

Method

The authors formulate the task as an interactive generative world loop. Starting from an initial visual observation $I_0$I0​, a player provides an action $a_t$at​ at each step. The system maintains a canonical world state $s_t$st​ that records persistent information needed to execute interactions. The overall loop is defined as

where $F$F is a lightweight engine that transitions the state, $C_{t+1}$Ct+1​ is the target camera, $P$P is a deterministic state compiler that projects the updated state into camera-aligned spatial controls, and $G$G is a generative renderer that synthesizes the visual observation $I_{t+1}$It+1​. The following subsections detail each component.

Agent-Orchestrated Box World. The programmable world is represented as a set of persistent entities embedded in 3D space. Each entity is grounded by a 3D oriented bounding box (OBB) that specifies position, extent, and orientation, and is associated with state variables such as identity, semantic category, and functional attributes. The canonical state at step $t$t is

where $\mathcal{E}_t$Et​ contains entities and their 3D poses, $\mathcal{A}_t$At​ holds semantic and functional attributes, $\mathcal{Q}_t$Qt​ captures inter-entity relations, and $\mathcal{R}_t$Rt​ denotes executable world rules. An off-the-shelf 3D detector recovers visible entities and their OBBs from $I_0$I0​, initializing the geometric component. An agent orchestrator then combines this layout with a natural-language world description to produce an engine-readable program that specifies initial states, attributes, relations, supported actions, and transition rules. The engine executes this program: at each step, given $s_t$st​ and action $a_t$at​, it evaluates validity, applies effects, resolves triggers, and produces the updated canonical state $s_{t+1}$st+1​. This state may contain latent variables (e.g., health, inventory) not directly visible, which nonetheless influence future transitions.

Control Compilation. The state compiler $P$P transforms the updated canonical state into a renderer-facing spatial representation without altering the world state. It projects entity OBBs under the target camera $C_{t+1}$Ct+1​ and augments the projected regions with identity, semantic, and object-motion attributes. Three spatially aligned control maps are constructed:

• Identity map $M_{t+1}^{\text{id}}$Mt+1id​: A fixed bank of $K$K learnable identity embeddings $\mathcal{E}^{\text{id}} = \{q_j^{\text{id}}\}_{j=1}^K$Eid={qjid​}j=1K​ is maintained. Each persistent entity is assigned a unique slot, and the corresponding embedding is rasterized over the visible projection of its OBB. This provides a spatial key for persistent instance correspondence across frames, even under occlusion or re-entry.

• Semantic map $M_{t+1}^{\text{sem}}$Mt+1sem​: For each entity, a pretrained text encoder produces a semantic embedding $q_i^{\text{sem}} = E_{\text{text}}(y_i)$qisem​=Etext​(yi​) from its category label $y_i$yi​. This embedding is rasterized over the visible OBB region, giving category-level guidance complementary to the instance-specific identity map. Entities of the same category share the same semantic representation but remain distinguishable via identity slots.

• Direction map $M_{t+1}^{\text{dir}}$Mt+1dir​: The entity’s world-space velocity is rotated into the target camera coordinate system and quantized into one of seven states (forward, backward, left, right, up, down, static). A bank of learnable embeddings $\mathcal{E}^{\text{dir}} = \{q_\ell^{\text{dir}}\}_{\ell=1}^{7}$Edir={qℓdir​}ℓ=17​ is rasterized over the projected OBB. Because the direction is computed from world-space velocity rather than image-space displacement, it explicitly separates object motion from apparent motion caused by camera movement.

The three maps are concatenated channel-wise to form the structured projected-box control:

Together with the projected OBB geometry and target camera trajectory, these maps provide explicit spatial, identity, semantic, and object-motion guidance to the renderer.

Generative Renderer. The renderer $G$G is built on a pretrained camera-controlled video generation backbone. A trainable Structured Spatial ControlNet is attached to incorporate the compiled controls. For a rendering window of $T$T frames with camera trajectory $\mathcal{C} = \{C_t\}_{t=1}^T$C={Ct​}t=1T​, the state compiler produces the control sequence $M_{1:T}^{\text{ctrl}}$M1:Tctrl​. The control sequence is encoded at the video latent resolution and processed by the ControlNet, which injects layer-wise features $r^{(\ell)}$r(ℓ) into the frozen backbone’s main blocks:

The camera-conditioning pathway controls the global viewpoint evolution, while the spatial control branch constrains where each entity appears, its persistent identity, semantic category, and camera-relative motion. During training, the pretrained backbone (including its camera module) is frozen, and only the spatial-control branch parameters $\phi$ϕ are optimized. The rendering process for a clip is

where $\theta$θ denotes the frozen backbone parameters.

For long-horizon generation, the renderer operates in a chunk-autoregressive manner. Denoising is bidirectional within each chunk, while information propagates causally across chunk boundaries. Each chunk after the first receives a temporal history $\mathcal{H}_{\text{temp}}^{(n)}$Htemp(n)​ and a geometry-aligned spatial memory $\mathcal{H}_{\text{spa}}^{(n)}$Hspa(n)​:

The temporal history is a multi-scale latent representation of previously completed chunks, organized into recent, mid-range, and long-range contexts. It is stochastically degraded during training to tolerate imperfect autoregressive rollouts. The geometry-aligned spatial memory lifts past RGB frames into a world-space memory using estimated depth and camera parameters, then retrieves and reprojects relevant observations into the target views of the new chunk. This provides a persistent global visual memory beyond the finite temporal window, mitigating drift under large camera motion or viewpoint revisitation. The additional parameters $\psi$ψ introduced for cross-chunk conditioning are trained jointly with the spatial-control branch.

Automatic Training Data Pipeline. Training the renderer requires structured annotations of camera geometry, persistent instance identities, semantics, and object motion, which are rarely available for in-the-wild videos. The authors develop an automatic data engine that recovers these from unlabeled videos. First, camera intrinsics, poses, and metric depth maps are estimated per frame using ViPE. An agent powered by Qwen3-VL analyzes each video to discover a set of discrete, countable object categories. SAM3 then performs video instance segmentation and tracking, yielding temporally consistent instance masks, bounding boxes, semantic labels, and track identities. For each tracked instance, Wild-Det3D estimates a per-frame 3D OBB from the 2D box, RGB frame, depth map, and intrinsics. The per-frame boxes are transformed into a shared world coordinate system using the estimated camera poses, forming object trajectories. Object motion is derived from world-space box center displacements, rotated into the camera coordinate system, and quantized into seven direction states. Finally, the 3D OBBs are projected into each frame to generate identity, semantic, and direction maps, which serve as conditioning for the renderer. This pipeline enables scalable creation of training data with the required structured annotations.

Experiment

The evaluation uses gameplay videos from multiple titles processed by a data engine to create paired video-control data, and a controlled benchmark with AI-generated combat scenarios to test whether generated videos faithfully reflect engine-maintained world states. Quantitative comparisons against prompt-switching baselines show that external state management with structured spatial controls yields substantially more reliable preservation of entity counts and death events, while also improving perceptual and temporal video quality. Qualitative results confirm that the approach consistently maintains world state across camera motions and interactions, generalizes to unseen characters, genres, and heterogeneous object categories, and supports complex autoregressive sequences without sacrificing visual realism.

The proposed method matches or slightly exceeds baseline video quality and consistency metrics, with the highest subject consistency, background consistency, and temporal stability. Its clearest advantage is in world-state evaluation, where count accuracy and state accuracy are substantially higher than both compared interactive video world models. Baselines struggle to maintain entity counts and death states, while the proposed method reflects engine-maintained state far more reliably. The proposed method achieves the best subject, background, and temporal consistency among all compared methods. Count accuracy and state accuracy show large relative gains over baselines, with the proposed method reaching near-perfect state tracking on death events.

The evaluation assesses video quality and world-state consistency. The proposed method matches or exceeds baselines on visual consistency metrics, achieving the highest subject, background, and temporal stability. Its most significant advantage is in world-state tracking, where it attains large gains in count and state accuracy over interactive video world models, including near-perfect death state detection, while baselines struggle to maintain entity counts and death states.