One-sentence Summary

Researchers at KAIST AI, Ewha Womans University, and KAIST Urban AI Institute propose L3G-World, a real-time infinite world generator built on a Latent 3D Gaussian Memory (L3G) in which a memory updater and a video diffusion transformer alternate autoregressively to render VAE latent features directly without re-encoding, achieving state-of-the-art generation quality, long-horizon self-consistency, and 9 FPS generation.

Key Contributions

• The paper introduces a Latent 3D Gaussian Memory (L3G) that combines feed-forward reconstruction with a compact scene representation, where each Gaussian stores a VAE latent feature so the memory can be rendered directly into the video diffusion transformer's latent space without VAE re-encoding.
• Building on this memory, L3G-World alternates a memory updater and a video diffusion transformer autoregressively, with the updater spawning Gaussians in newly observed regions and refining existing ones so memory scales with scene extent rather than stream length.
• Experiments show L3G-World achieves state-of-the-art generation quality and long-horizon self-consistency, with up to 38.6× fewer 3D primitives, 3.2× faster memory reconstruction, and 359× faster VAE feature rendering than the most efficient prior 3D-memory baseline, enabling real-time world generation at 9 FPS.

Introduction

Interactive video world models generate visual content conditioned on user actions and are promising for real-world simulation, interactive game generation, and embodied simulation. A central challenge is preserving spatial consistency during long-horizon exploration and loop closures while maintaining real-time performance. Prior temporal-context methods let new frames attend to earlier history, but their memory and attention costs grow with stream length and resolution, while explicit 3D memory methods either accumulate dense per-pixel primitives that become redundant and slow to render, or rely on optimization-based reconstruction that is too slow for real-time use. The authors introduce L3G-World, which uses a compact latent 3D Gaussian memory that is continually updated only in uncovered regions and refined in explored regions. The memory stores VAE latent features and is rendered directly into the latent space of a video diffusion transformer, enabling efficient conditioning without VAE re-encoding and supporting real-time, long-horizon world generation.

Method

The authors present L3G-World, a framework for real-time interactive world generation that maintains geometric consistency over a long horizon. To balance the competing demands of storing previously generated content and keeping the memory compact for fast read and write operations, the system adopts 3D Gaussians as the memory representation. The overall architecture realizes this through two modules that alternate autoregressively: a memory updater, L3G-Mem, which reads and updates the latent 3D Gaussian memory with each generated chunk, and a video generator, L3G-Gen, which generates the next chunk conditioned on the memory rendered in its latent space.

As shown in the figure below:

The L3G-Mem module processes a stream of images and their corresponding camera parameters in windows of size $V$V. At each autoregressive step $k$k, the model updates the latent 3D Gaussian memory $\mathcal{M}_{k-1}$Mk−1​ using the current input window $\mathcal{W}_k$Wk​. The update process is formulated as:

To keep the memory compact, the authors design an adaptive Gaussian spawning mechanism. By rendering the existing memory from the current camera views, the system identifies regions with low opacity or high RGB error. These insufficiently represented regions form a spawn mask. The model then encodes the input images with a geometric foundation model to obtain geometry features and applies K-means clustering to the features within the spawn mask to initialize a compact set of spawning query tokens. This ensures that the spawning budget is allocated only to content missing from the memory.

Simultaneously, a Gaussian refinement stream updates existing Gaussians that are visible in the current window. The memory tokens associated with visible Gaussians are retrieved as refinement tokens. Both the spawning and refinement tokens are concatenated and processed jointly by a transformer. In each block, the streams cross-attend to the geometric features, followed by streamwise self-attention and a global self-attention layer that enables information exchange between newly observed and previously reconstructed scene content. Furthermore, a dedicated latent-lifting branch initializes latent queries from the VAE features and cross-attends to them, producing VAE latent tokens spatially aligned with the query tokens. The processed tokens are then decoded into new Gaussian primitives and residual updates for the retrieved Gaussians. The memory is updated by appending the newly spawned Gaussians and applying the predicted residuals to the retrieved ones. The authors train L3G-Mem autoregressively over consecutive windows, supervising the renderings with a combination of RGB, LPIPS, depth, VAE latent MSE, and visibility losses.

The L3G-Gen module is a memory-conditioned autoregressive video diffusion model designed for long-horizon world generation. At each step, the VAE latents stored in the memory are rendered under the target cameras to obtain spatially aligned VAE latents and an alpha mask. These are concatenated with the target video latents along the channel dimension to provide geometric conditioning. The previously generated reference latents are also concatenated with zero-filled channels to match the dimension. The input sequence is formed by token-wise concatenation, and residual camera ray rotary positional embeddings are added to the self-attention in each DiT layer. To handle temporal consistency, the reference latents are assigned a temporally offset rotary positional embedding preceding the current generation chunk.

The memory-conditioned teacher model is trained using rectified flow. The loss is applied only to the target video latents, conditioned on the rendered memory latents, opacity map, retrieved reference latents, target cameras, and text prompts. To reduce the sampling cost for real-time interaction, the authors distill the teacher model into a few-step autoregressive student. The student is conditioned on both the source and destination noise levels and learns flow-map transitions between arbitrary points along the denoising trajectory. By jointly optimizing the flow-map objective and on-policy distribution matching, the student generates samples through its own autoregressive rollout, which are re-noised and compared against the frozen teacher, effectively reducing the train-test mismatch in few-step autoregressive generation.

Finally, the L3G-World framework combines these two modules in an autoregressive loop. Given an input image, L3G-Mem initializes the persistent Gaussian memory. At each generation step, the VAE features stored in the memory are rendered from the target cameras to provide spatially aligned latent conditioning for L3G-Gen. The generated latents are passed directly to L3G-Mem for VAE feature lifting, while the corresponding RGB frames are obtained via the VAE decoder. By repeatedly reading from and writing back to the same persistent 3D memory, the system continuously extends and corrects the generated world while avoiding redundant latent encoding between the two modules.

Experiment

The experiments evaluate L3G-World on DL3DV and RealEstate10K using long loop-closure trajectories, comparing against explicit and implicit memory baselines on self-consistency, generation quality, camera control, geometric consistency, and memory efficiency. Results show that the compact Gaussian memory with selective spawning and refinement uses substantially less memory and accelerates rendering while achieving strong long-horizon self-consistency and generation quality. Qualitative comparisons indicate that persistent, spatially anchored memory avoids the drift, blur, and artifacts seen in temporal-context and other 3D approaches, and ablations confirm the importance of adaptive spawning, residual refinement, global self-attention, RGB supervision, and direct VAE latent rendering.

Temporal-context and dense feed-forward scene representations scale with generated frames and resolution, leading to high memory use or out-of-memory failures. Optimization-based refinement scales with spatial coverage but greatly reduces throughput. The proposed approach combines feed-forward construction with refinement, scaling with spatial coverage while attaining the lowest GPU memory and substantially higher throughput. Temporal-context and dense per-pixel 3D approaches show memory scaling with frame count and resolution, with one method encountering out-of-memory errors. Scene representations with spatial scaling but without feed-forward construction remain far from real-time throughput. The proposed method achieves the lowest peak GPU memory and highest total FPS among all compared approaches, enabling real-time generation.

L3G-World and L3G-World Fast combine strong generation quality and long-horizon self-consistency with substantially lower memory and primitive counts than explicit 3D baselines. Across DL3DV and RealEstate10K, the L3G-World variants rank among the best for perceptual and self-consistency metrics while also achieving faster memory reconstruction and RGB or VAE rendering. These efficiency advantages do not come at the cost of generation quality, as L3G-World leads several generation and 3D consistency metrics on both datasets. Compared with Gaussian- and point-cloud-based memory baselines, L3G-World uses far fewer primitives and much less memory while rendering RGB and VAE features substantially faster. L3G-World variants achieve the best or second-best self-consistency and perceptual quality on both datasets, and L3G-World leads multiple generation and 3D consistency metrics.

The ablation evaluates L3G-Mem components on DL3DV novel view synthesis. The full model achieves the best VAE MSE, VAE cosine similarity, and RGB PSNR while keeping Gaussian count and reconstruction throughput competitive. Removing adaptive spawning, refinement, global self-attention, RGB supervision, or VAE lifting degrades quality or efficiency, with VAE lifting and RGB supervision showing especially large drops. The full model attains the strongest reconstruction quality across most RGB and VAE metrics with a compact Gaussian memory and comparable reconstruction throughput. Without adaptive spawning, the model uses roughly 1.8x more Gaussians yet still degrades RGB and VAE novel view synthesis, showing that selective spawning reduces redundant primitives. Removing the refinement stream lowers reconstruction quality, and further removing global self-attention between spawn and refine streams degrades performance. Encoding RGB renderings with the VAE instead of lifting VAE features substantially slows VAE rendering and reduces rendered latent quality. Ablating RGB supervision reduces PSNR and SSIM, increases LPIPS, and worsens VAE latent metrics.

The ablation compares full L3G-World against variants without memory refinement and without direct VAE latent lifting. Disabling memory refinement consistently degrades self-consistency, generation quality, pose accuracy, and 3D consistency, showing that progressive correction of persistent scene memory is important. Replacing direct latent rendering with RGB-to-VAE encoding improves some fidelity and 3D consistency metrics but lowers generation quality and translation accuracy, so the full model offers a strong balance. Removing memory refinement degrades all reported metrics, with clear losses in self-consistency, pose accuracy, and 3D consistency. Replacing direct VAE latent rendering with RGB rendering followed by VAE encoding improves PSNR, SSIM, and 3D consistency but reduces generation quality and translation accuracy. Direct VAE latent rendering removes encoding overhead and improves generation quality scores while keeping translation error low.

The experiments evaluate a hybrid feed-forward and refinement scene representation on DL3DV and RealEstate10K, comparing it against temporal-context, dense per-pixel 3D, Gaussian, and point-cloud baselines. The efficiency study shows the method scales with spatial coverage rather than frame count or resolution, achieving the lowest peak GPU memory and highest throughput while maintaining real-time generation, and the benchmark results show it ranks among the best for perceptual quality and self-consistency with far fewer primitives. Ablations validate that adaptive spawning, refinement, global self-attention, RGB supervision, and direct VAE latent lifting improve reconstruction quality or efficiency, while memory refinement and direct latent rendering are important for self-consistency, pose accuracy, and generation quality.