One-sentence Summary

Researchers from Alaya Lab, Shanghai Innovation Institute, Wuhan University, and Tsinghua University propose GameVerse, a minute-scale gameplay video dataset comprising 69.7K clips spanning 1,655 hours and 349.6 million frames, including 1,358 hours of continuous two-minute sequences, with temporally stable camera trajectories and event-aligned captions that separate static scene context from dynamic foreground events, surpassing existing public datasets in scale, diversity, and annotation quality and substantially extending the temporal horizons of interactive world models.

Key Contributions

• GameVerse is a minute-scale gameplay video dataset containing 69.7K clips, 1,655 hours, and 349.6 million frames, including 1,358 hours of continuous two-minute gameplay sequences.
• The dataset is built through a unified processing and annotation pipeline that yields quality-filtered, frame-level camera trajectories and four-level hierarchical captions; trajectories are filtered against jumps, jitter, and drift, while captions separate static scene context, dynamic events, and agent actions aligned to event boundaries.
• Training on GameVerse substantially extends the temporal horizon of interactive world models and improves temporal coherence, geometric consistency, and visual quality in 60-second generation, with same-backbone comparisons validating the benefit of GameVerse fine-tuning.

Introduction

Interactive world models aim to learn dynamic environments that can be rolled out from action and camera inputs, which matters for embodied decision-making, simulation, and game-style content generation. Recent gameplay systems such as GameGen-X and Matrix-Game show strong short-horizon control, but rollouts degrade after roughly one minute because their training data consist mostly of sub-ten-second clips with whole-clip captions that mix static scene context and dynamic events. Existing datasets also fragment the needed supervision, separating minute-scale footage, validated camera trajectories, dense interaction coverage, and event-aligned semantic annotations. The authors address this gap with GameVerse, a minute-scale gameplay dataset providing 69.7K clips and 1,655 hours of footage, including 1,358 hours of continuous two-minute sequences, along with validated frame-level camera trajectories and four-level event-aligned captions that disentangle scene context from agent behavior.

Dataset

Dataset: GameVerse

The authors construct GameVerse from long-form YouTube gameplay videos and convert them into minute-scale data units. Each unit contains an aligned video segment, a validated camera trajectory, and a multi-granular caption set.

Sources and raw composition

• Source: curated per-title YouTube gameplay videos from open-world AAA and indie games, selected for first-person and third-person viewpoints and diverse player behaviors.
• Raw scale: 11,002 source videos, totalling 4,692 hours, 195.5 days, and about 986.6M frames, kept at up to 1080p and 30 to 60 FPS.
• Game coverage: at least 110 distinct game titles. The distribution is long-tailed, with top titles including God of War, Red Dead Redemption 2, Ghost of Tsushima, Elden Ring, and The Witcher 3, each contributing over 1,800 clips.
• Perspective mix: 24% first-person, 73% third-person follow camera, and 3% specialized viewpoints such as drone, vehicle-mounted, gimbal-follow, or stationary.
• Behavior coverage: exploring, combat, walking, running, driving, interacting, climbing, and idle.
• Environment coverage: urban, rural, forest, highway, temple, and industrial settings with varied architecture, weather, time of day, seasons, and crowd density.

Processing and filtering

• Temporal segmentation: TransNetV2 detects hard cuts, while CLIP-based cosine semantic distance detects gradual transitions such as camera pans or UI changes. The pipeline extracts scene-consistent segments up to 120 seconds, and these can be concatenated into longer sequences.
• Segmentation output: 272,128 clips totalling 4,661 hours, with 120-second clips contributing about 2,773 hours.
• Multi-stage quality evaluation: filtering removes low-resolution or corrupted video; OCR-based detection removes subtitles, HUD elements, and overlays; stability checks identify unstable motion and temporal artifacts; deduplication and diversity selection reduce redundancy; temporal metrics such as COVER and VBench refine temporal quality.
• Quality scoring: each clip receives a weighted sum of per-stage scores, with stage-specific constraints preventing severe defects from being masked by high scores elsewhere.
• Camera pose processing: ViPE estimates frame-wise intrinsic and extrinsic camera parameters. Estimated trajectories are validated and filtered for sudden jumps, high-frequency jitter, and long-term drift.
• Captioning: the vision-language model Kimi-K2.5 generates structured captions with four components: global static, local static, dynamic events, and agent actions.
• The described processing focuses on temporal segmentation and filtering. No separate spatial cropping strategy is specified in the provided sections.

Final dataset statistics

• Final high-quality set: 69.7K gameplay clips totalling 1,655 hours, including 1,358 hours of two-minute continuous video.
• Long-horizon distribution: over 40.6K clips reach the 120-second upper bound. Clips of 60 seconds or longer form a substantial minute-scale portion.
• Camera motion distribution: forward motion accounts for 34.7%; complex translation plus rotation accounts for 22.0%; static, lateral, orbit, rotational, translational, and mixed motions are also represented.
• Caption statistics: local static captions have a median length of 895 characters; dynamic event captions have a median length of 775 characters. Global static and agent action captions are shorter, with agent actions drawn from a constrained vocabulary.
• Caption separation: the four caption levels use largely disjoint vocabularies covering scene attributes, spatial layouts, environment transitions, and explicit motion.

How the paper uses the data

• Each structured data unit combines aligned video, a validated camera trajectory, and a four-level caption set.
• The multi-granular captions provide hierarchical semantic supervision and separate static context from dynamic events and agent actions, supporting conditional generation.
• The camera trajectories support camera-controllable video synthesis, geometry-aware generation, and interactive world modeling under viewpoint change.
• The provided sections do not state explicit train, validation, or test split sizes or mixture ratios.

Method

The authors design a unified data construction pipeline to transform long-form gameplay videos into structured, minute-scale data units. Each unit comprises an aligned video segment, a validated camera trajectory, and a multi-granular caption set. As shown in the figure below, the pipeline processes raw video pools through several distinct stages to overcome limitations in temporal structure, camera pose stability, and semantic annotation.

To address fragmented temporal structures, the authors construct continuous segments by removing scene transitions. They employ TransNetV2 to detect abrupt hard cuts based on frame-level transition probabilities. To capture gradual transitions like camera panning or UI state changes, they compute a CLIP-based cosine semantic distance between sampled frames, treating large semantic jumps as smooth-transition boundaries. This combination extracts scene-consistent segments of up to 120 seconds.

Following segmentation, raw videos may still contain distracting visual signals such as subtitles, watermarks, UI overlays, and compression artifacts. To ensure visual fidelity and temporal coherence, the authors implement a multi-stage video quality evaluation pipeline. Refer to the framework diagram for a detailed breakdown of this filtering workflow.

The pipeline progressively executes basic filtering for low-resolution or corrupted videos, followed by content and quality assessment using OCR-based detection to remove subtitles and HUD elements. Subsequent stages include stability and UI detection to identify unstable motion patterns, deduplication and diversity selection to eliminate highly similar samples, and temporal refinement utilizing metrics like COVER and VBench consistency scores. The overall clip quality is computed as a weighted sum of per-stage scores, with stage-specific constraints preventing severe defects from being masked by high scores in other dimensions.

For geometric modeling, the authors apply ViPE to estimate frame-wise camera poses, obtaining both intrinsic and extrinsic parameters. Because pose estimation in dynamic scenes is prone to noise and tracking failures, they validate every estimated pose against three common error modes: sudden jumps, high-frequency jitter, and long-term drift. A unified trajectory-quality function combines local relative-pose increments and a global trajectory smoothness term, applying threshold-based filtering to remove unreliable samples.

To capture fine-grained gameplay semantics without conflating static context with dynamic events, the authors introduce a multi-granular structured captioning framework leveraging the vision-language model Kimi-K2.5. This framework separates static context from dynamic processes and organizes each clip’s semantics into four complementary components. Global static captures clip-level environment and scene context such as terrain, weather, and lighting. Local static describes relatively stable scene configurations within temporal segments, including functional areas and landmarks. Dynamic events track changes in the surrounding world like crowd movement and environmental transitions. Finally, agent actions detail the main subject’s behavior using a unified action vocabulary that decouples action semantics from viewpoint, providing explicit semantic supervision at multiple granularities.

Experiment

The evaluation covers 60-second image-to-video generation on 103 inputs using LingBot, HY-World 1.5, HY-World-FT, and LTX-FT, with VBench measuring video quality and WorldScore measuring controllability and world generation quality. Results show that fine-tuning on GameVerse consistently improves temporal coherence, visual quality, geometric consistency, and controllability under the same backbone, and it also preserves reference frames longer and performs better across diverse gameplay activities. Additional validation confirms that the multi-stage filtering pipeline and structured annotation process yield stable camera trajectories and high-quality captions with low hallucination and temporal mismatch.

Existing world modeling datasets each cover parts of the desired supervision stack, but none simultaneously provides minute-scale continuous footage, long-horizon coherence, released camera trajectories, multi-granular semantics, and dense interaction. The proposed dataset is the only one that satisfies all five criteria, while prior efforts remain fragmented across single axes. This fragmentation helps explain why models trained on a single source inherit specific blind spots such as short rollouts or missing pose and semantic supervision. Most datasets show single-axis optimization: Sekai provides exploration footage but lacks dense interactions, validated camera trajectories, and structured semantics. Matrix-Game and HY-World support long-horizon and interaction but only partially provide long-duration footage and do not release reliable camera poses as supervision. Only the proposed dataset combines long-duration, long-horizon, released pose, multi-granular annotations, and interaction in one collection.

On the full VBench evaluation set, LTX-FT leads in temporal consistency metrics, including subject consistency, background consistency, and motion smoothness, while LingBot leads in perceptual quality and text-video alignment. Fine-tuning the same HY-World 1.5 backbone on GameVerse improves all six VBench metrics, with particularly clear gains in subject and background consistency. LTX-FT achieves the best temporal consistency, leading in subject consistency, background consistency, and motion smoothness. GameVerse fine-tuning consistently improves all six VBench metrics over the HY-World 1.5 baseline, especially subject consistency, background consistency, and overall consistency.

LTX-FT achieves the strongest overall static world generation quality, with top scores in 3D consistency, photometric consistency, and style consistency. LingBot leads in camera control, content alignment, and subjective quality, while HY-World-FT leads in object control. Fine-tuning the HY-World backbone on GameVerse improves every WorldScore component, most notably 3D consistency. LTX-FT records the highest static world score and the best geometric, photometric, and style consistency among the compared models. GameVerse fine-tuning on the same HY-World backbone improves all WorldScore components, with a particularly large gain in 3D consistency.

Long-horizon consistency is measured through reference-frame similarity over time and subject consistency across gameplay activities. LTX-FT demonstrates the strongest overall stability, with the highest late-time similarity and the latest observed drop. GameVerse fine-tuning also yields consistent gains over the HY-World base model in both temporal stability and activity-wise subject consistency. LTX-FT reaches the highest reference-frame similarity at 30 seconds and sustains stability longest before the first drop. Under the same HY-World backbone, GameVerse fine-tuning improves 30-second similarity and delays the first drop below the 0.6 threshold. Across driving, riding, running, and walking, HY-World-FT consistently scores higher subject consistency than HY-World 1.5, while LTX-FT leads all activities.

Human evaluation on 400 clips indicates that structured captions are highly accurate for scene/event and agent-action content, with temporal alignment and camera-motion accuracy also rated positively. Overall alignment is strong and annotator agreement is consistent. Clip-level hallucination and temporal mismatch remain low. Scene/event accuracy and agent-action accuracy are the highest-rated dimensions, while camera-motion accuracy is the lowest among the caption quality scores. Only a small share of clips show hallucination or temporal mismatch, and overall annotator agreement is consistent.

The work evaluates a new dataset that uniquely combines long-duration footage, long-horizon coherence, released camera trajectories, multi-granular semantics, and dense interaction. On VBench and WorldScore, LTX-FT achieves the strongest temporal and static consistency, while LingBot leads in perceptual quality, camera control, and alignment; fine-tuning HY-World on GameVerse consistently improves all metrics over its baseline, especially subject, background, and 3D consistency. Long-horizon tests show LTX-FT maintains reference-frame similarity longest, and GameVerse fine-tuning delays stability drops. Human evaluation confirms structured captions are highly accurate for scene/event and agent-action content with low hallucination and temporal mismatch.