One-sentence Summary

The ABot-World Team presents ABot-World-0, an action-conditioned video world model trained on a multi-source data infrastructure spanning AAA games, simulation engines, and internet videos and supported by agent-driven collection via WorldExplorer guided by training feedback and a unified pipeline applying 14 deterministic quality checks, VLM-based assessment, and synchronized action and text annotation, which progressively distills a bidirectional action-conditioned teacher into a causal student through teacher forcing and ODE distillation, introduces LongForcing to align long student self-rollouts with an extended-horizon teacher to mitigate distribution shift and autoregressive drift, employs raw keyboard actions as a unified control interface for scene roaming and third-person character interaction with reference-character memory providing persistent appearance cues, and co-designs a streaming inference stack with lightweight VAE decoding, efficient attention, memory-aware scheduling, and low-bit DiT inference that streams 720P video at up to 16 FPS on a single NVIDIA RTX 5090 desktop GPU with 1.2 s action-to-first-frame latency and approximately 19 GiB peak VRAM, achieving competitive controllability and coherent long-horizon world evolution on WorldRoamBench.

Key Contributions

• A progressive distillation pipeline converts a bidirectional action-conditioned teacher into a causal student via teacher forcing and ODE distillation, and LongForcing aligns the student's long self-rollouts with an extended-horizon teacher to mitigate distribution shift. Evaluations on 60-second rollouts show reduced visual error accumulation relative to a causal-forcing baseline.
• A streaming inference stack co-designed with a lightweight VAE decoder, efficient attention, memory-aware scheduling, and low-bit DiT inference achieves 720P streaming at up to 16 FPS on a single NVIDIA RTX 5090 with 1.2 s action-to-first-frame latency and peak VRAM below 19.3 GiB, enabling real-time desktop deployment.
• Unified interactive control uses raw keyboard actions for scene roaming and third-person character manipulation, supported by reference-character memory for persistent appearance cues. Results on WorldRoamBench and extended rollouts demonstrate competitive controllability and coherent long-horizon world evolution.

Introduction

AI is moving from generating static clips to building persistent, interactive worlds where actions continuously change the state and the visual response must remain coherent over long rollouts. Prior systems such as Genie, GameNGen, and Oasis have shown that video models can be steered by user inputs, but scaling video generation alone fails to address four coupled bottlenecks: acquiring large-scale data with reliable, synchronized action labels; representing both camera navigation and embodied character control in a single interface; preventing accumulated visual drift as the generated output becomes the next input; and deploying the full generation-and-decoding pipeline at interactive speeds on practical hardware. The authors introduce ABot-World-0, a unified action-conditioned world model built on a multi-source data infrastructure that combines AAA game, simulation, and internet video data with agent-driven collection and rigorous quality checks. The model uses raw keyboard inputs for scene roaming and character control, and a progressive training pipeline that distills a bidirectional teacher into a causal student, then applies a technique called LongForcing to supervise long self-rollouts and reduce distribution shift. An optimized streaming stack co-designs lightweight decoding, low-bit inference, and memory-aware scheduling to deliver 720P output at up to 16 FPS with 1.2 s action-to-first-frame latency on a single RTX 5090, addressing interactive world modeling as a full-stack systems challenge.

Dataset

The authors construct the training corpus from three complementary data sources, processed through a unified pipeline and used for both action-conditioned and text-conditioned world model training:

• Overview of sources

  • AAA game data: captures high-fidelity, synchronous action and observation signals with ground-truth control labels.
  • Simulation engine data: provides geometrically precise, photorealistic synthetic scenes with deterministic action labels.
  • Real-world internet data: introduces natural camera dynamics, lighting variation, and domain diversity via pose-estimated pseudo-labels.

• Game data details

  • Collected by the WorldExplorer agent across multiple AAA titles (open-world, driving, equestrian) in both first- and third-person viewpoints.
  • RGB video at 1920×1080, synchronized with per-frame camera poses, control inputs, and environment metadata.
  • Control signals are captured directly from the game’s runtime API, yielding noise-free action labels converted to a canonical representation.

• Simulation data details

  • Uses Unreal Engine and ABot-3DGS reconstructions of proprietary street aerial and street-scanning assets to create large outdoor and indoor environments.
  • Trajectories generated procedurally (geometric patterns with interpolation) or imported from real-world device motion paths to preserve authentic browsing behaviour.
  • Action labels are deterministically derived by projecting translational/rotational displacements onto camera basis vectors and binarizing.

• Internet data details

  • Passively sourced from driving, walking, and aerial videos; clips are segmented by scene coherence.
  • 6‑DoF camera poses are estimated per clip (selected by indoor/outdoor, static/dynamic, and texture richness), then action intent labels are derived via displacement projection and thresholding—producing pseudo-labels that expand diversity despite estimation noise.

• Quality filtering

  • All raw data passes through a three-stage, 14‑check pipeline spanning file integrity, visual validity, geometric consistency, game state correctness, action-label alignment, and metadata quality.
  • VLM-based semantic filtering removes UI overlays, loading screens, popups, and rendering anomalies. Soft signals from later checks are used for training‑time sample weighting rather than hard rejection.

• Annotation and usage

  • Action labels are unified into a canonical format across all sources, enabling joint model training.
  • A large VLM generates structured scene descriptions (with game identifier tokens for game data) as text‑conditioning inputs, while deliberately omitting camera motion information to decouple control from scene description.
  • Each clip receives semantic tags (indoor/outdoor, weather, time of day, vehicle type, viewpoint) for stratified sampling and targeted fine‑tuning.
  • For third‑person clips, person identity thumbnails are cropped from four viewing angles and a canonical frontal portrait is synthesized via image‑based face completion, supporting identity‑conditioned generation.

• Closed‑loop data production

  • WorldExplorer adapts collection ratios based on training feedback: a monitoring module tracks per‑category performance, diagnoses weaknesses, and adjusts task templates—making the dataset an evolving part of the model development cycle rather than a one‑time preprocessing step.

Method

The authors formulate the interactive world modeling task as predicting future visual observations conditioned on past frames, user actions, and multimodal conditions. Formally, given visual history $\mathbf{v}_{0:t-1}$v0:t−1​, future action sequence $\mathbf{a}_{t:t+L-1}$at:t+L−1​, and multimodal conditions $\mathbf{c}$c, the model aims to predict the next video chunk $\mathbf{v}_{t:t+L-1}$vt:t+L−1​ of length $L$L. The overall training pipeline consists of two main phases: adapting a pretrained video generator into a bidirectional teacher, and progressively distilling it into a causal autoregressive student for efficient rollout.

Bidirectional Teacher Training To equip the pretrained video generator with action-conditioned dynamic knowledge, the authors inject raw keyboard inputs and reference images into the backbone. Raw keyboard actions are represented as an 8-dimensional multi-hot vector per frame. To align with the video tokenizer's temporal compression, every 4 consecutive frame-level actions are packed along the channel dimension, yielding a 32-dimensional action token sequence. These packed action tokens are processed by an Action Control Adapter, which uses PixelUnshuffle and convolutional blocks to match the spatial compression ratio of the VAE. The action-conditioned latent tokens are then injected additively into the DiT backbone.

To prevent identity drift during long-horizon third-person rollout, a reference-character memory module is introduced. Canonical reference images of the character are encoded and converted into identity-memory tokens, which are prepended to the video-token sequence. An asymmetric attention pattern allows video tokens to attend to memory tokens while keeping memory tokens isolated, ensuring persistent identity retrieval.

Causal Progressive Student Distillation Since the bidirectional teacher relies on full-sequence generation and iterative denoising, it cannot be deployed for real-time autoregressive rollout. The authors employ a three-stage progressive distillation pipeline to convert the teacher into a causal student.

In the first stage, Teacher Forcing, the causal student is initialized from the bidirectional teacher and trained under a causal attention mask. History frames are provided as clean ground-truth latents, while the target chunk is corrupted for diffusion training. This adapts the model to causal generation while retaining the visual priors learned by the teacher.

In the second stage, ODE Distillation, the authors compress the multi-step autoregressive denoising process into few-step inference. A frozen Stage 1 causal model defines a probability-flow ODE trajectory. The distilled model is trained to directly predict the clean endpoint from intermediate noisy latents under the same causal context, significantly reducing inference latency.

In the third stage, LongForcing, the authors address the accumulation of distributional errors during long-horizon rollout. By exposing the student to extended self-rollout contexts and applying distribution-level corrective supervision from an extended-horizon teacher, LongForcing mitigates progressive degradation and autoregressive drift without requiring explicit frame-level trajectory matching.

Full-Stack Co-Design for Real-Time Rollout To achieve real-time interactive generation on a single consumer-grade GPU, the authors implement a full-stack co-design. A lightweight VAE decoder reduces first-frame latency and peak memory usage. Memory-aware runtime scheduling stages model modules according to their execution order to minimize VRAM pressure. Fast-RoPE re-anchors temporal positional encoding within the local attention window to reduce computational overhead. Furthermore, low-bit mixed-precision inference is applied to the DiT backbone, and efficient attention kernels accelerate transformer operations. Bounded KV caching with rolling eviction prevents unbounded context growth, ensuring scalable long-horizon generation.

Experiment

The end-to-end system analysis demonstrates that real-time, high-resolution world rollout on a single desktop GPU requires full-stack co-optimization of VAE decoding, transformer quantization, attention, and runtime scheduling, rather than isolated kernel acceleration. On WorldRoamBench, ABot-World-0 achieves strong action controllability and visual quality, while qualitative stress tests reveal sustained scene coherence, out-of-domain generalization, and emergent physical plausibility over hour-scale rollouts. A LongForcing ablation shows that extended-horizon teacher supervision reduces visual artifact accumulation, maintaining higher HPSv3 and lower blur and repetition during long autoregressive generation. These results collectively validate that the distilled model learns persistent, stateful interactive dynamics suitable for practical interactive world simulation.

ABot-World-0 runs interactively on a single consumer GPU with streaming inference, delivering up to 16 FPS and 1.2 seconds action-to-first-frame latency at modest VRAM usage. The model generalizes action-conditioned character motion to out-of-domain scene–character combinations and spontaneously produces physical interactions such as collisions, persistent tracks, and environmental blocking without explicit physical rules. With LongForcing, long-horizon rollouts better preserve visual quality over 60 seconds, showing less error accumulation than a shorter-horizon baseline. A single NVIDIA RTX 5090 runs the model at up to 16 FPS with 1.2 s latency and peak VRAM under 19.3 GiB. Action-conditioned motion transfers to novel scenes and characters, while physical effects like collisions, water traces, and snow footprints emerge without symbolic physics annotations.

On a single RTX 5090 at 1280×704 and batch size 1, the Base model and a SageAttention2-only variant run out of memory, showing that a faster attention kernel alone is insufficient. Replacing the default VAE with LightVAE provides the first feasible configuration, and further stack optimizations—DiT quantization, Fast-RoPE, and aggressive low-bit formats—progressively increase throughput from 9.1 FPS to 15.8 FPS, with peak VRAM determined by the complete runtime, not by a single operator. The Base configuration and the variant with only SageAttention2 both fail with out-of-memory errors, indicating that accelerating attention alone cannot enable high-resolution deployment. Introducing LightVAE yields the first feasible configuration, achieving 9.1 FPS and 20.5 GiB peak VRAM, revealing the original VAE as a critical memory bottleneck. Quantizing the DiT backbone to FP8 reduces DiT time from 1191 ms to 845 ms per chunk, raises throughput to 12.4 FPS, and lowers peak memory from 20.5 GiB to 15.9 GiB. Adding Fast-RoPE further cuts DiT time to 786.9 ms and increases FPS to 13.3, but peak VRAM rises to 19.3 GiB, demonstrating that peak memory depends on the full runtime configuration. Using MXFP6 or MXFP4 precision with Fast-RoPE pushes throughput to 14.1 and 15.8 FPS respectively, while peak memory drops to 18.3 GiB and 17.1 GiB, achieving the best frame rates and lowest memory usage.

ABot-World-0, the smallest model at 5B parameters, achieves the highest or second-highest scores in five of the seven reported sub-dimensions, substantially outperforming the larger LingBot-World (14B) and HY-World 1.5 (8.3B). HappyOyster leads overall on strict accuracy, partial accuracy, trajectory score, aesthetic, and memory, while Genie 3 shows particular strength in imaging quality and physical mechanics. ABot-World-0 (5B) ranks first or second on strict accuracy, partial accuracy, trajectory score, aesthetic, and imaging, trailing only HappyOyster in these categories. HappyOyster achieves the top scores in strict accuracy, partial accuracy, trajectory score, aesthetic, and memory. Genie 3 records the best imaging quality and physical mechanics, with memory close behind HappyOyster. LingBot-World (14B) and HY-World 1.5 (8.3B) are surpassed by the much smaller ABot-World-0 across all sub-dimensions.