One-sentence Summary

Researchers at Alaya Lab and the University of California, Merced introduce AlayaRenderer-Flash, a real-time generative forward world renderer that reformulates the original AlayaRenderer as a few-step autoregressive streaming model with lightweight distilled codecs, achieving 31.54 FPS (up from 0.56 FPS) while preserving scene structure from physics-engine G-buffer and text-prompt interfaces, and enabling fully playable generative worlds at 30 FPS.

Key Contributions

• AlayaRenderer-Flash adopts an autoregressive streaming formulation that propagates rendering history across windows, enabling continuous generation over unbounded G-buffer sequences and removing the fixed-window limitation of the original model.
• Lightweight distilled encoder and decoder networks replace the computationally expensive G-buffer encoder and Wan VAE decoder, reducing the latency of latent encoding and frame reconstruction.
• A progressive 4-step distillation compresses the 50-step denoising schedule into a compact renderer, increasing speed from 0.56 FPS to 31.54 FPS and enabling real-time, prompt-controllable rendering integrated with a physics engine at 30 FPS.

Introduction

The authors build on the engine-plus-generative-renderer paradigm, where a game engine simulates physics and produces G-buffers (depth, normals, albedo, etc.), and a generative model replaces the shading stage to generate appearance from text prompts. This approach preserves scene structure and gameplay logic while enabling prompt-controllable visual restyling. The prior state-of-the-art, AlayaRenderer, delivers high-quality rendering on complex AAA-style scenes but operates as an offline renderer: its 50-step denoising, heavy G-buffer encoder, and VAE decoder make inference too slow, and its bidirectional fixed-window design prevents autoregressive streaming over unbounded G-buffer sequences. The authors introduce AlayaRenderer-Flash, which transforms the model into a real-time streaming renderer through three key improvements: autoregressive generation that propagates rendering history across windows, distillation of the 50-step denoising into a 4-step process, and replacement of the expensive encoder and decoder with lightweight distilled networks. These changes preserve rendering quality and controllability while reducing inference cost enough to enable interactive, prompt-controlled gameplay.

Method

The authors present AlayaRenderer-Flash, a real-time stylized rendering pipeline that transforms synchronized G-buffer streams from a game engine into stylized RGB frames.

Refer to the framework diagram below:

The system integrates autoregressive streaming, hierarchical history compression, four-step diffusion, and lightweight distilled codecs to enable playback-rate rendering.

G-Buffer-Conditioned Latent Rendering The base renderer builds upon Wan 2.1, a latent video diffusion model. The game engine provides five synchronized physical buffers: albedo, depth, metallic, normal, and roughness. Each G-buffer channel is treated as a video stream and encoded by the Wan causal 3D VAE to yield frame-aligned latent representations. These features are concatenated along the channel dimension to form the G-buffer condition $g_k$gk​, which is concatenated with the noisy RGB latent before the first 3D patch embedding:

where $x_k$xk​ denotes the noisy RGB latent of chunk $k$k, and $g_k$gk​ is the corresponding G-buffer condition. This strategy requires only widening the input projection of the first 3D patch embedding, leaving the remainder of the diffusion transformer unchanged.

Autoregressive Streaming To support unbounded live rendering, the authors partition the latent video into fixed-length chunks and operate autoregressively at the chunk level. Previously generated chunks are retained as history, organized into a three-tier compression hierarchy based on temporal distance. Recent history is preserved at full fidelity, while distant history is progressively compressed. The first generated latent frame is retained as a global appearance anchor. During denoising, all retained history is inserted into self-attention alongside the current chunk as clean latent tokens. To preserve prompt controllability over arbitrary rollouts, every self-attention layer is augmented with a dedicated text sink consisting of persistent key-value entries projected from the style-prompt embedding. The encoding and decoding pipeline is implemented causally, carrying temporal feature caches across chunks.

Few-Step Distillation To accelerate inference, the authors target a four-step denoising schedule. They employ a three-stage distillation pipeline consisting of guidance distillation, progressive step reduction, and Mean Flow Distillation (MFD). First, in Guidance Distillation, the student retains the original 50-step schedule while learning to reproduce the teacher's classifier-free guidance combined velocity field with a single network evaluation. Second, Progressive Step Reduction gradually reduces the denoising budget from 32 to 16, 8, and finally 4 steps, providing a stable initialization. Finally, MFD under self-rollout refines the four-step student. The student conditions exclusively on its own previously generated chunks, matching the deployment setting and reducing the discrepancy between training and autoregressive inference. To compensate for the loss of high-frequency details caused by aggressive step reduction, lightweight GAN heads are attached to intermediate transformer features and the final latent prediction.

Distilled Tiny Codecs To further reduce latency dominated by VAE modules, the authors distill both the encoder and decoder into lightweight replacements. The tiny decoder adopts the TAEHV architecture and is distilled from the frozen Wan VAE decoder using pixel reconstruction and perceptual losses. The tiny G-buffer encoder replaces five separate VAE encoding passes with a single shared forward pass to predict the G-buffer condition $g_k$gk​. It is first distilled to match the frozen Wan VAE encoder and then jointly fine-tuned with the entire renderer.

Real-Time Deployment During inference, incoming G-buffer streams are continuously encoded by the distilled tiny encoder, rendered by the four-step diffusion model, and decoded into RGB frames by the stateful tiny decoder. All components operate autoregressively while maintaining persistent temporal states across chunk boundaries, enabling continuous rendering over unbounded input streams.

Experiment

Experiments are conducted on a Black Myth: Wukong dataset with synchronized G-buffer inputs, where progressive design analysis reveals that autoregressive streaming, few-step distillation, and lightweight codecs together enable real-time rendering while preserving content similarity, temporal stability, and cross-window consistency. Comparisons with baselines show that AlayaRenderer-Flash is the only method that simultaneously supports causal streaming, prompt control, and few-step inference, striking the best balance between quality and deployment efficiency. Qualitative results demonstrate stable long-horizon illumination and geometry, seamless prompt switching during continuous streaming, and end-to-end deployment on SuperTuxKart confirms practical interactive gameplay at the target 30 FPS.

Replacing the original 50-step denoising with an autoregressive then 4-step distilled formulation raises throughput from below 1 FPS to over 30 FPS, while lightweight distilled encoder and decoder nearly halve GPU memory. Quality metrics remain stable or improve across content similarity, boundary consistency, temporal smoothness, and prompt alignment, with the final compact model achieving the top scores in all categories. Autoregressive streaming alone improves content similarity and boundary accuracy but degrades temporal stability, and throughput stays far from real-time. Switching to a 4-step distilled diffusion model multiplies frame rate by over four times without extra memory, while restoring temporal coherence and maintaining rendering quality. Adopting lightweight distilled encoder and decoder further boosts throughput to interactive rates and cuts peak GPU memory, with all quality indicators reaching their best values.

The table compares methods for G-buffer conditioned rendering on support for autoregressive streaming, prompt switching, and few-step inference. AlayaRenderer-Flash is the only method that provides all three capabilities while running at real-time rates, whereas prior approaches lack prompt control or temporal modeling and exhibit lower spatiotemporal coherence. RGB↔X uses G-buffers per-frame without temporal modeling, resulting in the weakest spatiotemporal coherence and highest FVD among evaluated methods. FrameDiffuser adds autoregressive temporal modeling but does not support prompt switching and shows substantially weaker temporal stability. AlayaRenderer-Flash uniquely enables autoregressive streaming, prompt switching during continuous rendering, and few-step inference, achieving real-time interactive rates of 31.54 FPS.

AlayaRenderer-Flash achieves the best trade-off between rendering quality and real-time performance, leading in content similarity (0.847), temporal stability (tLPIPS_warp 0.155), and frame rate (31.54 FPS) among streaming methods. RGB↔X lacks temporal modeling and shows the highest FVD (1031.3), while FrameDiffuser reduces FVD but suffers from the worst temporal drift (0.440) and lacks prompt control. A retrained bidirectional baseline obtains higher offline quality but runs at only 1.10 FPS, confirming that AlayaRenderer-Flash uniquely enables prompt-controlled, real-time interactive rendering. AlayaRenderer-Flash delivers the best content similarity (0.847) and temporal stability (0.155 tLPIPS_warp) while running at 31.54 FPS, far surpassing the other streaming methods in both quality and speed. FrameDiffuser improves similarity over RGB↔X (0.844 vs. 0.820) but yields the worst temporal drift (0.440 tLPIPS_warp) and lacks prompt control, while RGB↔X exhibits severe flicker and the highest FVD (1031.3). A bidirectional baseline achieves higher offline scores (S_CLIP-I 0.870, FVD 335.5) but is limited to 1.10 FPS, showing that AlayaRenderer-Flash uniquely combines real-time interactive rendering with prompt switching and streaming support.

The evaluation compares rendering methods by progressively replacing a slow 50-step denoising pipeline with an autoregressive streaming formulation, a 4-step distilled diffusion model, and lightweight distilled encoder/decoder components. This combination raises throughput from below 1 FPS to over 30 FPS, halves GPU memory, and maintains or improves quality across content similarity, boundary accuracy, temporal smoothness, and prompt alignment. Compared to prior approaches, the final model uniquely supports autoregressive streaming, prompt switching during continuous rendering, and few-step inference, achieving the best spatiotemporal coherence and real-time interactive rates.