One-sentence Summary

Researchers from HKUST, PKU, et al. propose EchoWM, an omnimodal world model for enterable media that jointly generates 720p video, environmental sound, music, and speech while mapping camera intent to metric-scale relative 6-DoF trajectories, and through progressive training followed by autoregressive post-training it achieves strong trajectory following and synchronized long-horizon generation across first- and third-person scenes on public world-model benchmarks.

Key Contributions

• EchoWM is an omnimodal world model that jointly generates 720p video, environmental sound, music, and speech while responding to continuous navigation through a shared metric-scale relative 6-DoF trajectory and first- or third-person camera intent.
• A complementary gameplay, simulation, and Internet data engine with dataset-level velocity calibration supports a progressive curriculum: audio-visual pretraining builds a world prior, action fine-tuning learns visual navigation, and low-learning-rate joint fine-tuning with autoregressive post-training consolidates long-horizon audio-visual generation and trajectory control.
• Evaluations report strong trajectory following and high visual quality on public world-model benchmarks, with first- and third-person interaction and synchronized environmental sound and speech over long-horizon generation.

Introduction

Generative video models are becoming powerful audio-visual media engines, but user interaction remains largely prompt-driven and passive. World models that generate future states in response to inputs usually produce silent visual rollouts or rely on subject-specific action spaces, missing the spatial, ambient, and social cues that audio provides. The authors introduce EchoWM, an enterable omnimodal world model that continuously generates synchronized video, environmental sound, and speech from user navigation. They organize interaction around a subject-agnostic camera intent, mapping diverse inputs to a shared relative 6-DoF trajectory, and combine a multi-source data engine with a progressive training curriculum that first learns audio-visual priors, then isolates trajectory control, and finally jointly refines both to achieve responsive, long-horizon generation across first- and third-person scenes.

Dataset

The authors build a multi-source dataset for training an interactive world model. It combines four complementary data sources because no single source provides diverse appearance, natural audio, interactive motion, and reliable camera geometry at once.

Sources and composition

• Internally collected gameplay: Includes first- and third-person play across open-world exploration, urban driving, and equestrian traversal. Programmatic scripts issue controls and produce clean action-observation pairs. A ReShade plugin removes HUD and interface overlays before capture, preserving the original field of view without post-hoc cropping. The recordings include high-resolution RGB video, native stereo game audio, script-issued action logs, and environment metadata.

• Human-played Internet gameplay: Complements internal gameplay with more natural control timing, richer speech, player commentary, in-game dialogue, and context-dependent reactions. Source-native action logs are unavailable. Clips with reliably recovered camera trajectories contribute to trajectory-conditioned training, while the broader pool supports audio-visual and speech modeling.

• Unreal Engine simulation: Provides ground-truth metric camera poses, synchronized action logs, and controlled translation and rotation coverage, but no natural audio. Characters are driven with local-frame navigation commands and engine physics. Each character carries one first-person camera and four third-person cameras; training randomly selects one view per rollout. UE audio is treated as missing audio rather than semantic silence.

• General Internet video: Broadens appearance and acoustic diversity with real-world and cinematic content, speech, environmental sound, and varied camera-subject relationships such as tracking, orbiting, and reframing. It has no native action logs or standardized navigation. A smaller subset with successful metric pose recovery and motion-quality filtering enters trajectory-conditioned training.

Processing paths

• Audio-visual path: Uses shot-boundary detection to isolate continuous spans in Internet and cinematic footage. AV-eligible sources are sliced directly into short model-ready clips because visual, speech, and environmental sound learning does not require long-range geometric reconstruction.

• Geometry path: For non-UE trajectory-conditioned data, the authors preserve continuous windows of about one minute before slicing. This allows neighboring clips to share a common geometric reconstruction and provides longer temporal support for pose optimization.

• Canonicalization: All retained media are re-encoded into canonical video and audio formats for stable decoding and batched training, preserving stereo channels when present.

Metric pose recovery and filtering

• UE provides camera-to-world poses directly in metric units. Other trajectory-eligible sources require pose recovery from video.
• The authors use ViPE with VGGT-Omega for temporally coherent long-sequence geometry and MoGe-2 for per-frame metric-depth constraints. This supports per-frame intrinsic optimization to handle zoom, stabilization, cropping, and imperfectly known intrinsics in Internet videos.
• For efficiency, long windows are temporally subsampled before pose estimation. Recovered poses are then aligned back to original timestamps using spherical linear interpolation for rotations and linear interpolation for translations. Video observations are not synthesized or modified.
• Trajectory quality filtering checks reconstruction reliability, intrinsic-calibration stability, temporal consistency, frame-to-frame jitter, abrupt orientation changes, motion plausibility, displacement, degenerate trajectories, and extreme scale outliers.
• UE trajectories skip reconstruction but follow the same convention conversion and motion-range checks.

Structured metadata and leakage control

• Qwen3-Omni and Gemini-3-Pro process synchronized video and audio into a common schema with fields: scene, style, viewpoint, subject, narrative, speech, and sound.
• Persistent context fields are scene, style, viewpoint, and subject. Temporal audio-visual event fields are narrative, speech, and sound.
• Source-specific field names are normalized to these canonical fields. Speech markers are normalized across None, empty, null, and equivalent values.
• Raw annotation responses and generation metadata are retained for provenance but are not exposed as semantic conditioning fields.
• For trajectory-conditioned Action-SFT, the narrative field is removed to prevent text from leaking subject motion or camera evolution. AV-CPT keeps all fields. Joint-FT keeps static fields plus speech and sound, while still excluding narrative.

Training mixtures and usage

• AV-rich mixture: Combines internally collected gameplay, human-played Internet gameplay, and general Internet video selected for visual and acoustic quality, with speech-rich examples explicitly sampled. Reliable camera trajectories are not required. Used for AV-CPT to build a broad audio-visual prior.

• Control-clean mixture: Combines internally collected gameplay, human-played Internet gameplay, UE simulation, and pose-eligible general Internet video selected for reliable metric trajectories and smooth, interpretable motion. Used for Action-SFT, with the motion-bearing narrative field removed.

• Balanced high-quality mixture: A smaller intersection of reliable trajectory supervision and high-quality audio-visual content, including environmental sound and speech when available. Used for Joint-FT to consolidate trajectory following with the audio-visual prior.

The current model uses camera trajectories as its unified motion condition. Native action logs are preserved as a distinct signal for alignment, data analysis, and future interaction modeling. The paper does not report exact total clip counts or mixture ratios. For motion coverage analysis, the authors sample 28,605 trajectories from the control-clean mixture, where examples include straight, diagonal, return-loop, and orbiting motion.

Method

World Data Engine Training an enterable omnimodal world requires diverse appearance, natural audio, interactive motion, and reliable camera geometry. The authors combine four complementary sources: internally collected game recordings, human-played Internet game recordings, Unreal Engine simulation, and general Internet video. These sources are processed through two complementary paths: an audio-visual path prioritizing appearance and sound, and a geometry path preserving long-range temporal continuity for metric camera recovery. The resulting examples are organized into three stage-aligned mixtures for AV-CPT, Action-SFT, and Joint-FT.

Model Architecture The authors build on a pretrained joint audio-visual diffusion transformer, extending it with metric camera-trajectory conditioning. Given a media context, structured text condition, and user control sequence, the goal is to jointly generate synchronized video and audio while following the requested camera motion. The user controls are converted into a calibrated relative 6-DoF camera trajectory.

The Unified Camera-Intent Interface represents user navigation through desired camera motion over time. For clip $i$i, the relative pose at frame $k$k is expressed as $\Delta T_{i,k} = T_0^{-1} T_t$ΔTi,k​=T0−1​Tt​. To resolve translation-scale differences across heterogeneous data sources, the authors estimate a robust global scale $s_{\text{global}}$sglobal​ from the metric training trajectories:

The calibrated trajectory is injected into the video backbone using Unified Camera Positional Encoding (UCPE). For a video latent token at frame $t$t and spatial cell $s$s, the reference-space ray direction is computed as $\mathbf{d}_{t,s} = \mathbf{R}_t \operatorname{norm}(K_t^{-1} \mathbf{p}_s)$dt,s​=Rt​norm(Kt−1​ps​). A local ray frame is constructed, and the geometry is injected through a lightweight camera-attention branch parallel to the original video self-attention.

Progressive Training Reliable audio-visual supervision and reliable trajectory supervision are concentrated in different parts of the training data. As shown in the figure below:

AV-rich examples provide stronger audio supervision but typically cover a narrower range of camera motion, whereas control-clean examples contain more reliable and diverse trajectories. The training is separated into three stages. Stage 1 (AV-CPT) adapts the complete backbone on the AV-rich mixture without trajectory conditioning. Stage 2 (Action-SFT) freezes the audio-visual backbone and trains only the camera-branch parameters on the control-clean mixture. To prevent text from directly specifying the target motion, the narrative field is removed, leaving the trajectory as the only condition specifying camera evolution. The control-clean mixture exposes the model to varied realized camera evolution rather than a narrow set of discrete command templates.

Stage 3 (Joint-FT) updates both parameter groups on a balanced high-quality subset with a reduced learning rate.

Streaming Audio-Visual Post-Training The authors progressively adapt the pretrained bidirectional model into a streaming autoregressive generator. First, audio-visual teacher forcing converts bidirectional temporal attention into causal autoregressive computation. The causal attention pattern ensures each noisy chunk can attend to all preceding clean audio-visual chunks and the noisy tokens within the current chunk, while clean targets and future chunks are masked out.

To address discrepancies between teacher-forcing training and inference, the authors introduce Self-Gradient Forcing (SGF). Short-horizon SGF trains on self-generated autoregressive histories while recovering gradients through reconstructed causal representations, combined with Distribution Matching Distillation (DMD) to convert the multi-step generator into a few-step sampler. For sustained streaming generation, long-horizon SGF extends this to longer trajectories using a sink-plus-FIFO cache to bound the causal context.

The long-horizon mask restricts access to clean history by the sink-plus-FIFO policy, while layer-wise propagation allows gradients to reach earlier representations through intermediate chunks.

Experiment

The evaluation combines two public benchmarks, WBench Navigation and SANA-WM-Bench, with a user study, qualitative analysis, and ablations to assess interactive world modeling. EchoWM and its distilled causal variant, EchoWM-Flash, achieve leading results on navigation and long-horizon trajectory benchmarks, with strong controllability, visual consistency, and reduced drift over extended rollouts. Subjective results favor EchoWM in overall preference and spatial-temporal consistency, while qualitative experiments show generalization across first-person and third-person viewpoints, unseen domains, coherent 3D reconstruction, synchronized audio-visual generation, and visual memory. Ablations further support the fixed global trajectory scale and shared camera-intent modeling across viewpoints.

Representative generative video and world models consistently report high-resolution video output and first- and third-person participation. Most support continuous trajectory-based navigation, while one game-oriented model reports discrete keyboard input instead. None of the listed models report native audio feedback for environmental sound, background music, or spoken speech. Most compared models support continuous trajectory interaction, while one game-oriented model supports discrete keyboard navigation. All listed models lack reported native audio feedback, including environmental sound, background music, and spoken speech.

Collected gameplay provides native audio, speech, and action logs; Internet gameplay adds human operation and richer speech; Unreal Engine renders contribute precise pose reliability and action clarity without audio; and general Internet video broadens appearance and audio diversity to non-game domains. Their complementary availability of logs, human operation, and out-of-distribution coverage jointly supports audio-visual world modeling and trajectory-conditioned interaction. Action logs are present in collected gameplay and UE renders, providing supervised interaction signals, while Internet gameplay and general video lack them. General Internet video is the only source covering non-game domains, significantly expanding visual and acoustic diversity beyond game-rendered scenes.

EchoWM and its distilled four-step causal variant, EchoWM-Flash, rank first and second on the WBench Navigation benchmark. The undistilled model achieves the highest overall average and consistency, while EchoWM-Flash retains most of that capability and delivers the top interaction score, slightly surpassing the original model. The undistilled EchoWM leads with the highest consistency (89.8), indicating strong multi-turn scene and viewpoint stability. EchoWM-Flash achieves the best interaction score (87.9) and substantially improves setting adherence (88.3) compared to the base model’s 79.4.

Across the front 241 frames, EchoWM achieves the best visual quality on both Simple and Hard splits, recording the highest VBench Overall scores. On Simple trajectories it also has the lowest translation and camera-motion consistency errors while staying close to SANA-WM in rotation accuracy. On Hard trajectories, SANA-WM attains lower pose errors, but EchoWM maintains the strongest visual quality. EchoWM records the highest short-horizon VBench Overall on both Simple and Hard splits. On Simple trajectories, EchoWM achieves the lowest translation and camera-motion consistency errors and rotation accuracy close to SANA-WM. On Hard trajectories, SANA-WM has lower pose errors while EchoWM keeps the strongest visual quality.

Long-horizon evaluation shows that EchoWM preserves strong visual quality and scene consistency on simple trajectories, achieving the lowest translation and camera-motion consistency errors and the highest revisit PSNR among the compared methods. LingBot-World attains the highest overall VBench score and the smallest temporal imaging-quality drop, though its pose errors are larger. Accumulated rotation drift remains EchoWM's main weakness, worsening from the short horizon to the full 961-frame evaluation and further on hard trajectories. EchoWM achieves the best scene revisit consistency and the lowest translation and camera-motion consistency errors on simple long-horizon trajectories. Rotation drift accumulates over the long horizon and worsens on hard trajectories, while LingBot-World has the least temporal imaging-quality degradation among the reported methods.

The experiments first compare existing generative video and world models on interaction and audio support, and compare training data sources, finding that most models enable continuous trajectory navigation but none provide native audio, while game logs, internet videos, and UE renders together supply complementary action, pose, speech, and out-of-domain supervision. EchoWM and its distilled EchoWM-Flash are then evaluated on navigation and short- and long-horizon generation, where EchoWM consistently achieves the best scene consistency and visual quality, and EchoWM-Flash shows the strongest interaction behavior. Long-horizon results confirm EchoWM's strong visual quality and revisit consistency, with LingBot-World showing less temporal quality degradation, while accumulated rotation drift remains EchoWM's main weakness.