One-sentence Summary

The Tencent Hy Visual Generation Team introduces Hy-Nomas, an $8\mathrm{B}$8B diffusion video world model that supports free-form text instructions and continuous camera control across multiple interaction turns by aligning structured video captions with action-aware causal attention, using a vision-language Prompt Enhancer, and applying multi-stage curriculum training, thereby achieving superior human evaluation results and releasing model weights, code, benchmarks, and a lightweight adaptation workflow.

Key Contributions

• Hy-Nomas is an 8B-parameter diffusion video world model that supports free-form text instructions and continuous camera control across multiple interaction turns, generating variable-length segments conditioned on current inputs and cached history.
• Its central design aligns structured video captions with action-aware causal attention, keeping descriptive context accessible across segments while restricting action instructions to their intended temporal windows, and a vision-language Prompt Enhancer converts user requests and multimodal interaction histories into structured conditions for the next segment.
• A multi-stage curriculum progressively introduces causal generation, text interaction, camera control, and training on model-generated histories, followed by diffusion distillation. Hy-Nomas achieves stronger text-driven action control than larger models evaluated in the benchmark and superior human evaluation performance over existing world models, and the release includes model weights, code, benchmarks, and a lightweight adaptation workflow.

Introduction

Interactive video world models are gaining attention because they let users influence future visual observations through commands issued during generation, unlike conventional fixed-prompt video synthesis. Recent systems such as Genie 3, HappyOyster, and LingBot-World v2 have improved navigation and long-horizon generation, but adapting pretrained video generators remains difficult due to causality constraints, error accumulation across segments, persistent scene memory, and the need to coordinate free-form text actions with camera control. The authors introduce Hy-Nomas, an 8B-parameter world model built on HunyuanVideo 1.5 that supports multi-turn free-form text instructions and continuous camera navigation in a single autoregressive diffusion transformer. Their approach combines structured chronological captions, timestamped action annotations, and action-aware causal attention, along with a progressive training recipe and a vision-language prompt enhancer for segment-level generation.

Dataset

Dataset composition and sources

• The authors construct a large-scale, heterogeneous video corpus spanning real and virtual environments.
• Raw videos vary in encoding type, duration, frame rate, resolution, and encoding quality.

Preprocessing and filtering

• Each video is divided into 5-minute segments and then split at detected shot boundaries.
• The shot boundary detector is based on OmniShotCut and fine-tuned on high-quality annotations.
• Clips are transcoded into a unified format and filtered by basic properties such as duration, aspect ratio, and resolution.
• Visual quality filters include motion, clarity, and aesthetic scores.
• Additional filters screen for derivative content and abrupt transitions.

Distribution and deduplication

• Visual embeddings are extracted with Qwen3-VL-Embedding.
• Videos are grouped by visual similarity through hierarchical clustering.
• SemDeDup is applied within each cluster to remove semantically redundant clips.
• A hierarchical taxonomy covers semantic content and visual attributes.
• Content categories include interaction, object manipulation, human and object movement, and camera motion.
• Visual attributes describe camera perspective, subject type, subject and camera motion patterns, and recording methods.
• A fine-tuned VLM labels each video according to this taxonomy.
• Videos are also associated with labels derived from hierarchical clustering.
• Intermediate cluster levels are manually annotated for data selection.
• The authors select videos using taxonomy labels, cluster-level annotations, or both.
• Sampling weights are adjusted across training stages to target weaknesses identified in earlier stages.

Camera trajectory processing

• Camera trajectories are estimated with ViPE for a subset of videos suitable for trajectory estimation.
• Camera calibration and segmentation modules are upgraded with stronger pretrained models while retaining the existing depth estimator.
• Adaptive spatiotemporal downsampling is applied to reduce memory usage and improve throughput.
• SLAM sliding window and optimization schedules are tuned to balance optimization scope and trajectory drift.
• A video-based inverse dynamics model is trained from a pretrained VPT backbone to predict discrete camera motion such as WASD keyboard inputs.
• Training data for the inverse dynamics model includes synthetic camera-motion videos and ViPE-derived motion labels refined using dense optical flow.
• Manually annotated shake and zoom data provide additional supervision, and unavailable labels are masked during training.
• Trajectory filtering occurs before and after ViPE estimation.
• Filtering includes four sequential steps:
  1. Optical-flow motion-score range filtering to reject static scenes and abrupt spikes.
  2. CoTracker3 point tracking on a uniform grid, rejecting clips where inter-frame visible-point displacement exceeds 20% of max(H, W) or median total tracked-point displacement is below 30% of max(H, W).
  3. AnyCalib joint estimation of per-frame focal length and radial distortion, penalizing excessive focal-length variation, abrupt focal-length changes, overly large focal lengths, or strong distortion.
  4. ViPE trajectory audit, rejecting trajectories with repeated large orientation jumps, excessive inter-frame viewing-angle offsets, translation spikes, or insufficient overall camera displacement.

Video captioning and annotation

• Captions use structured JSON with video-level and segment-level information.
• Video-level fields specify visual style, camera perspective, and a shared subject list.
• Each subject has an explicit identifier and a static profile anchored to its earliest clearly identifiable appearance.
• Later actions and state changes reference this identifier to maintain continuity across segments.
• Within each segment, scene descriptions and camera-motion sentences are interleaved chronologically.
• Subject actions and visible outcomes are recorded separately with start and end timestamps.
• The first segment establishes the scene.
• Subsequent segments describe changes relative to the preceding segment without repeating unchanged context.
• Camera-motion sentences are enclosed in explicit <CameraN> and </CameraN> tags.
• For training segments with ViPE-estimated trajectories, camera-motion descriptions and tags are removed and the estimated trajectories are used for explicit camera-motion supervision.
• For segments without trajectory annotations, the tagged sentences are retained as textual supervision.
• Multi-turn annotation proceeds in two stages:
  • First, V-JEPA 2 visual features are extracted and clips are divided into shorter segments based on temporal changes.
  • Second, a VLM annotates the segments sequentially in a multi-turn conversation.
• The first turn establishes the initial scene and subject profiles.
• Subsequent turns describe scene changes and current actions while refining subject identities and video-level labels.
• Previous segments and annotations remain in the conversation context.
• For later segments, the VLM uses visual evidence only from the current and preceding segments to reduce hallucination.
• A final text-only turn reviews accumulated annotations, corrects earlier drafts, resolves inconsistencies, and produces the final English caption and structurally aligned Chinese translation.
• This final turn checks JSON structure, required fields, subject references, tags, segment and action timestamps, and consistency between language versions.
• Caption models vary by training stage:
  • Gemma 4 and Qwen3.5-122B-A10B are used for efficient large-scale captioning in data-intensive stages.
  • Kimi K3 is used for detailed segment-wise annotation in the annealing stage.

Use in training

• The processed clips, taxonomy labels, cluster labels, captions, and camera trajectories form training samples.
• The structured captions support segment-wise autoregressive rollout training.
• Camera control is supervised through either keyboard inputs mapped to trajectories or natural-language camera motion instructions.
• Sampling weights are adjusted across training stages using taxonomy and cluster-based data selection.
• For prompt enhancer training, the authors build a multi-turn multimodal conversation dataset from dense captions and real user prompts.
• User prompts are synthesized from existing dense captions and augmented from real user prompts.
• These prompts are run through the prompt enhancer and video generation pipeline, recording the caption response and generated video at each turn.
• This dataset includes scene initialization and follow-up interactions across subjects, environments, viewpoints, and control modes.
• It is used to supervised fine-tune a lightweight prompt enhancer model based on Qwen3.8-27B.

Method

The authors construct a large-scale, heterogeneous video corpus spanning real and virtual environments. The end-to-end pipeline converts raw videos into training samples through standardized preprocessing, filtering, semantic deduplication, camera-trajectory annotation, and structured captioning. The sampling distribution is adjusted across training stages to address weaknesses identified in each stage.

To preserve subject identity, scene context, and action chronology, the authors design structured video captions with camera motion explicitly tagged. The captions use structured JSON with video-level and segment-level information. A VLM annotates the segments sequentially in a multi-turn conversation. The first turn establishes the initial scene and subject profiles, while subsequent turns describe scene changes and current actions. A final text-only turn reviews the accumulated annotations, corrects drafts, and produces aligned English and Chinese captions.

The model, Hy-Nomas, builds upon HunyuanVideo 1.5. It introduces segment-wise causal attention for autoregressive world generation and camera cross-attention for explicit camera control. The model uses a DCAE-V video autoencoder and a frozen Qwen2.5-VL-7B text encoder, followed by a timestep-aware token refiner. Starting from Gaussian noise, the model denoises video latents one segment at a time, conditioned on the current text prompt, an optional camera trajectory, and cached keys and values from preceding segments. A dedicated cross-attention module in each MMDiT block injects camera conditioning.

Unlike conventional video generation, video-based world models simulate evolving world states in temporal order. The authors replace fully bidirectional attention with segment-wise causal attention aligned with the temporal structure of the input captions. Attention is bidirectional within each segment and causal across segments. To address within-segment action misalignment and cross-segment action repetition, they introduce action-aware causal attention. This extends segment-wise causal attention with a finer-grained masking scheme to distinguish actions within and across segments. Each segment text tokens are divided into a descriptive head and action tails. Query tokens within an action window are restricted to attending only to the associated text tokens and the descriptive head tokens from the same segment, while descriptive heads remain accessible to current and subsequent segments.

To adapt the model to the evolving history it consumes during autoregressive inference, the authors employ Segment Forcing. This combines sequential segment-wise denoising, one-step context estimation, and streaming backward. Given a video divided into multiple segments, the autoregressive DiT processes the segments sequentially. At each segment, the model predicts its flow-matching velocity conditioned on the KV cache constructed from preceding segments. The flow-matching MSE objective is applied only to the current segment prediction, and the loss is immediately backpropagated. To construct the context for the next segment, a one-step denoising is performed from the same noisy latent using the predicted velocity:

The resulting low-noise latent is appended to the KV cache. Streaming backward retains only the latest segment computation graph in memory, accumulating gradients across segments for a single optimizer step.

Experiment

The evaluation introduces Hy-Nomas-Bench, a 280-case benchmark spanning multi-turn text interaction, camera navigation, interleaved text and camera control, and spatial memory, and compares Hy-Nomas with Genie 3, LingBot-World v2, and Happy Oyster under native interfaces and pairwise human judgment. Results show that Hy-Nomas consistently leads in prompt following, physical plausibility, and interleaved control, while its camera-control advantage is smaller and spatial memory still trails Genie 3 despite being competitive with Happy Oyster. Additional experiments validate that LoRA adaptation improves cinematic or robotics-specific behavior but can slightly reduce camera-control accuracy, and that the PE model improves instruction and format adherence while maintaining control performance.

The architecture builds on HunyuanVideo 1.5 and uses a DCAE-V video autoencoder with 32x spatial and 4x temporal compression, producing a 32-channel latent representation at 1x1x1 patch size. Text conditioning comes from a frozen Qwen2.5-VL-7B encoder, while the denoising backbone has 54 dual-stream MMDiT blocks with hidden width 2048 and MLP width 8192. Segment-wise causal attention supports autoregressive generation over video segments, and camera cross-attention injects explicit camera control. The video autoencoder applies substantially higher spatial compression than temporal compression, yielding compact per-position latent tokens. The transformer backbone uses a deep dual-stream MMDiT design with wide MLP expansion and camera conditioning through dedicated cross-attention.

Hy-Nomas training is organized as a staged curriculum that first adapts a text-to-video initialization to a new latent space, then uses continual pre-training to add segment-causal generation, action-aware temporal structure, and camera-conditioned control. Later stages apply high-quality annealing, segment forcing, and diffusion distillation to refine quality, align training with autoregressive inference, and enable few-step generation. The curriculum jointly advances causal, interaction, and context capabilities rather than introducing all skills at once. Segment-causal continual pre-training establishes segment-wise autoregressive generation before action-aware and camera-conditioned training are introduced. Action-aware training uses temporally localized action captions to improve action timing and cross-segment isolation while expanding maximum context length. Camera-conditioned training mixes videos with and without camera trajectories to add continuous camera control without losing text-driven generation. Segment forcing trains with cached context from the model's own predictions, aligning training more closely with segment-wise autoregressive inference. Diffusion distillation enables few-step generation and further reduces error accumulation from the training-inference mismatch.

The evaluation settings reflect the heterogeneous interaction interfaces of the compared models. Hy-Nomas and LingBot-World v2 are evaluated across multi-turn text, camera navigation, interleaved text and camera, and spatial memory, while other models have more limited applicability. Hy-Nomas and LingBot-World v2 support all four benchmark settings, including interleaved text and camera control. Genie 3 is limited to camera navigation and spatial memory because its interface does not support text-prompt based interactions. Happy Oyster supports text interaction and navigation separately but is not evaluated on interleaved text and camera control. LingBot-World v2 is adapted from its open-source model for free-form text and interleaved evaluation.

The experiments examine Hy-Nomas across its architecture, staged training curriculum, and benchmark evaluation settings. The model builds on HunyuanVideo 1.5 with a DCAE-V autoencoder, a frozen Qwen2.5-VL-7B text encoder, and a 54-block dual-stream MMDiT backbone that supports segment-wise causal generation and camera cross-attention. Its training curriculum progressively adds segment-causal generation, action-aware temporal structure, camera-conditioned control, segment forcing, and diffusion distillation to align training with autoregressive inference and enable few-step generation. Evaluation covers multi-turn text, camera navigation, interleaved text and camera control, and spatial memory, where Hy-Nomas and LingBot-World v2 support all settings while other models have more limited applicability.