One-sentence Summary

Researchers from SkyworkAI, The Chinese University of Hong Kong, UC San Diego et al. introduce Matrix-Game 4.0, the first playable game-world system that separates game execution from visual generation through an executable harness, coupling a lightweight engine, a multimodal coding agent, and a distilled autoregressive video renderer to achieve 480p 60 FPS gameplay at 112 ms key-press-to-display latency on a single NVIDIA RTX 5090.

Key Contributions

• Matrix-Game 4.0 is introduced as the first system built on an executable game harness that separates game execution from visual generation, using a lightweight engine for world state and rules, a multimodal coding agent for natural-language game code and dynamic scene captions, and an autoregressive video renderer that converts low-poly engine proxies into detailed visuals conditioned on dynamic captions and memory references.
• The rendering model is trained on pixel-aligned low-poly and high-fidelity video pairs and distilled into a causal generator that produces four-frame chunks with one or two denoising steps, with targeted inference optimizations for interactive latency.
• On a single RTX 5090, the system achieves 480p 60 FPS gameplay with 112 ms key-press-to-display latency, 83 FPS single-step steady-state throughput, and 18.2 GiB peak GPU memory. Fixed-prompt throughput reaches 67 FPS on RTX 5080 and 58 FPS on RTX 5070 Ti; demonstrations include customizable shooter and driving games, with code, model weights, and low-poly game assets released.

Introduction

Interactive world models such as the Genie series have extended generative modeling from passive video viewing to active environment interaction, but systems that rely primarily on video generation to maintain world state face several limitations: no continuous external correction, accumulating autoregressive errors, frame-based memory that grows with rollout length, and limited gameplay mechanics. Diffusion models also tend to learn statistical correlations rather than causal or physical structure, while playable worlds further require real-time interaction, rich mechanics, and shared multiplayer state. The authors introduce Matrix-Game 4.0, a framework that separates game execution from visual rendering by placing a video generation model inside an external harness. A multimodal coding agent translates player intent into game code, a lightweight game engine maintains persistent world state and rules while rendering low-poly proxy frames, and the video model transforms those structural signals into detailed visuals. The system achieves real-time 480p 60 FPS gameplay on a single consumer GPU with 112 ms key-press-to-display latency, and it is released as an open-source foundation for generative-game research and development.

Dataset

The authors build a cross-domain dataset of aligned low-poly and high-fidelity videos in Unreal Engine 5 for world modeling and video generation.

Dataset composition and sources:

• Four scene types: CitySample megacity, open desert, complex castle, and high-dynamic air combat.
• Recorded at 1280 × 720. Each capture contains three aligned videos: UE5 RGB render, low-poly structure render, and depth render of proxy geometry, plus per-frame metadata.
• Shooter and air-combat clips last about one minute.
• Held-out data: castle, desert, and city hold out three complete capture sessions each; air combat holds out whole capture batches. No held-out session contributes any training clip.
• Training data quantities per scene are listed in Table 1.

Processing and filtering:

• Scenes are abstracted into lightweight geometric representations. Camera trajectories and timestamps are synchronized, so low-poly and high-fidelity frames align pixel by pixel.
• The proxy keeps spatial topology, rigid poses, and motion trajectories.
• Adaptive geometric fitting and view-frustum culling remove material reflections, high-frequency textures, and complex lighting.
• In the proxy, the sky is black and the ground is a gray checkerboard. Buildings and walls become flat-colored boxes and prisms, trees become a cylinder with a sphere, and characters become block figures. Every primitive has a black outline and no texture, reflection, or shadow.
• No separate cropping strategy is described in this section.

Metadata and dynamic captions:

• Per-frame metadata records object instance IDs, visibility states, and causal interaction events.
• Each object ID links to a semantic description. The system reads visible objects and real-time states from the game engine and fills them into a prompt template, producing a caption that matches the current frame.
• Captions update when the view changes, when entering a new area, or when an event triggers.

Use in the model:

• The data defines a conditional mapping from lightweight geometric topology to high-fidelity visuals.
• It provides deterministic supervision for spatiotemporal consistency and interaction stability in long rollouts.
• The low-poly proxy supplies structure: position, layout, and motion. Dynamic captions supply semantics: which objects are in the frame and their current states. Together they constrain generation so frames follow engine-produced structure and event or state changes.
• Mixture ratios are not specified in this section; training clips are reported per scene in Table 1.

Method

Matrix-Game V4 aims to build a truly playable world by wrapping a video model in a harness that maintains world state and provides an executable foundation for gameplay. The architecture integrates three core modules: a game engine, a multimodal coding agent, and a video generation model. The game engine acts as an external manager, maintaining world state, defining rules, and outputting structural renderings. The multimodal coding agent bridges player ideas and game code, translating scenes into visual descriptions and implementing new mechanics. The video model autoregressively generates realistic future frames based on the structural renderings and text descriptions.

To train the rendering model, the authors build a cross-domain dataset of aligned low-poly and high-fidelity videos. The capture pipeline abstracts complex scenes into lightweight geometric representations, where buildings become flat-colored boxes and characters become block figures, preserving spatial topology and motion trajectories while removing high-frequency textures. This low-poly proxy provides structural constraints for the video model.

In an interactive environment, the visual context changes with every player action. To address this, the system employs dynamic captions. Each object in the scene is linked to a semantic description via an instance ID. The system reads the currently visible objects and their real-time states from the game engine and populates a prompt template. This ensures the video model receives up-to-date semantic conditions regarding object positions, states, and events, complementing the structural information from the low-poly proxy.

The rendering model is built upon the Wan2.2 5B Fun Control variant, adapted as a video-to-video model. It conditions each chunk, representing one latent frame or four video frames, on four inputs: the low-poly proxy, the dynamic caption, spatial-memory reference frames, and the first frame of the sequence. The noisy latent and the TAEHV-encoded low-poly proxy are concatenated along the channel dimension to form 148 channels, which are patchified into 405 tokens per latent frame.

The backbone consists of 30 causal DiT blocks. Each block applies causal self-attention over a KV cache containing spatial-memory frames, reference frames, and the latest 20 chunks. It also performs cross-attention to the cached text keys and values from the umT5-xxl encoded dynamic caption. The output head predicts a velocity, and the latent is updated via flow matching for one or two steps before being decoded by the lightweight TAEHV decoder. After a chunk is generated, the model re-runs on the finished latent at $t=0$t=0 without the control input to write clean keys and values into the cache for future chunks.

The training begins with Supervised Fine-Tuning. Starting from pre-trained weights, the model is fine-tuned as a bidirectional video-to-video model using a flow-matching objective on 81-frame windows. The windows are conditioned on the low-poly proxy, dynamic caption, first frame, and up to three context frames serving as spatial memory. To support classifier-free guidance, the first frame, prompt, and proxy control are dropped with probabilities of 0.3, 0.2, and 0.2, respectively.

To enable real-time generation, the authors apply Causal Few-Step Distillation in three stages. Stage 1 converts the bidirectional model into a causal autoregressive teacher using teacher-forced autoregressive diffusion, where each latent frame attends only to clean ground-truth frames before it. Stage 2 performs causal consistency distillation. An online student is trained to match the predictions of an EMA student, which is updated from a frozen causal teacher, enabling generation in four steps. Stage 3 utilizes distribution matching with asymmetric DMD. The student learns from its own autoregressive rollouts, matching the real score from the frozen bidirectional model. This stage incorporates a rolling cache and mixed proxy encodings to align with the deployed inference setting, ultimately reducing the denoising steps to one or two per chunk.

To achieve real-time performance, several optimizations are implemented. The DiT backbone computation is recorded as a CUDA Graph and replayed in a single launch, utilizing 8-bit weights. Fused GPU kernels are written for RoPE, normalization, and FP8 quantization. The KV cache operates as a ring buffer, overwriting the oldest slot in place as the window slides. Finally, the full Wan VAE decoder is replaced by the lightweight TAEHV decoder, and model I/O runs in an asynchronous pipeline, significantly reducing latency and increasing the steady-state generation rate.

Experiment

The experiments evaluate Matrix-Game 4.0 as an interactive video generation system, with real-time inference optimized through CUDA graphs, fused kernels, a key/value cache ring buffer, and a lightweight decoder, substantially improving frame rate and response latency. Held-out rollouts across three distilled game models show that generated frames preserve the low-poly proxy layout while adding stable textures, effects, and character appearance over long sequences. Comparisons with recent interactive systems indicate that Matrix-Game 4.0 achieves the lowest end-to-end latency and runs on a single consumer GPU. The coding agent experiments further validate natural language gameplay customization, with rule changes executed in the engine and rendered faithfully by the video model.

The dataset consists of paired low-poly and high-fidelity videos across four UE5 scene types: castle, desert, air combat, and city. In total it contains over 114,000 training clips and about 1,700 hours, with the desert shooter contributing the largest share and the castle shooter the smallest. Most scenes are captured at 24 FPS, while the city driving scene runs at 17 FPS, and held-out clips are separated by complete capture sessions or batches. The Ancient Egypt desert shooter provides the largest training split, while the medieval castle shooter provides the smallest. All scenes are captured at 24 FPS except the city driving scene, which is captured at 17 FPS. Held-out clips are grouped by complete capture sessions for castle, desert, and city, and by whole capture batches for air combat.

Cumulative inference optimizations raise steady-state generation from a few frames per second to over 65 FPS and cut latency from over one second to about 120 ms. The largest throughput improvements come from adding a RoPE kernel with cached text keys and values, and from one-step distillation. The final fused kernels mainly reduce latency rather than adding much throughput. The RoPE kernel with cached text KV provides the largest speedup relative to the previous configuration at 2.4x. Fused Triton kernels raise FPS by only 1.1x but lower latency from 367 ms to 121 ms.

Recent real-time interactive generation systems differ widely in latency, hardware requirements, and release openness. Matrix-Game 4.0 reports the lowest end-to-end latency by a large margin and is the only compared system that runs on a single consumer GPU. Its code and weights are also released at launch. Matrix-Game 4.0 achieves about one-fifth the end-to-end latency of the next fastest compared system. It is the only system running on a single consumer GPU, while the others rely on data-center GPUs or multi-GPU configurations.

The evaluation uses paired low-poly and high-fidelity videos across four UE5 scene types, with over 114,000 training clips and held-out splits grouped by capture session or batch. Inference experiments show cumulative optimizations raise throughput from a few frames per second to real-time rates and cut latency to interactive levels, with the largest gains from cached text key/value RoPE and one-step distillation, while fused kernels mainly reduce latency. A comparison with recent real-time systems finds Matrix-Game 4.0 achieves the lowest end-to-end latency, runs on a single consumer GPU, and provides code and weights at launch.