One-sentence Summary

Researchers from ARC Lab, Tencent, et al. introduce GameHorizon, a unified gameplay data and evaluation suite comprising an automated multi-horizon instruction annotator, a 5,000-hour AAA dataset spanning 21 games, and reproducible offline and stepwise online benchmarks, and they evaluate 47 models across more than one million invocations to reveal task-difficulty hierarchies and capability differences.

Key Contributions

• GameHorizon-Annotator is a scalable, automated annotation pipeline that generates a dense instruction pyramid spanning short-horizon operations, medium-horizon goals, and long-horizon strategies.
• GameHorizon-Data is a large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions, containing 5,000 hours of recordings from 21 games collected by 100 human expert players.
• GameHorizon-Bench combines reproducible offline evaluation with stepwise online testing; the offline track includes thousands of standardized questions across three primary tasks and a series of diagnostic variants, while the online track checks whether offline scores reflect actual gameplay capability and localizes failures to specific steps. Evaluation of 47 models through more than one million model invocations reveals a hierarchy of task difficulty and model capability differences.

Introduction

The authors address AI gameplay in modern AAA games, where a single stream of primitive actions must support short-term operations, medium-term goals, and long-term strategies. Prior work is limited in two ways: dedicated game agents favor high-frequency action control but lack planning ability, while general-purpose vision-language models can plan but lack standardized action evaluation; existing datasets also have narrow game coverage, sparse instruction annotations, and low-confidence online rollouts. The authors introduce GameHorizon, a unified data and evaluation suite that combines an automatic multi-horizon instruction annotator, a large-scale dataset of aligned video, action, and instruction triplets across 21 AAA games, and reproducible offline plus stepwise online benchmarks for comparing diverse model families.

Dataset

Dataset composition and sources

• The authors present GameHorizon Suite, which consists of GameHorizon-Annotator, GameHorizon-Data, and GameHorizon-Bench.
• Raw gameplay is collected from 100 experienced human players using a dedicated recording-and-upload system.
• The recordings contain 2K video and timestamped keyboard-mouse actions, avoiding pseudo-labels from inverse dynamics models or gamepad segmentation.
• GameHorizon-Data is described as the first large-scale AAA gameplay dataset aligning videos, player actions, and multi-horizon instructions.

Scale and subset details

• The corpus covers 5,000 hours of human gameplay from 21 game titles across open-world, action role-playing, competitive shooter, sandbox survival, and creature-collecting adventure genres.
• Valorant is the largest title with 617.5 hours, or 12.35%, while Honor of Kings: World is the smallest with 35.9 hours, or 0.72%.
• Low-quality or uninformative recordings are filtered using automated rules and VLM assessments. Examples include prolonged action-free cutscenes and switches away from the game window.
• After filtering, 4,341 hours, or 86.8%, are retained for instruction annotation. Synchronized keyboard-mouse actions remain available for all 5,000 hours.
• The retained data contains 4,571 videos recorded at 60 fps and 411.03 million keyboard-mouse action events.
• Action annotations average 22.84 events per second, combining mouse inputs sampled at 20 Hz with an average of 2.84 keyboard events per second.
• Instruction counts by horizon:
  • Short-horizon operations, L1: 5,947,588 instructions, average span of 2.63 seconds.
  • Medium-horizon goals, L2: 189,158 instructions, average span of 82.6 seconds.
  • Long-horizon strategies, L3: 47,290 instructions, average span of 330.4 seconds.
• Every frame is aligned with instructions at all three horizons, creating a dense multi-horizon instruction pyramid.

Processing pipeline

• The annotation pipeline works bottom-up because VLMs struggle to resolve fine-grained visual and action details over very long videos.
• Action-aware video segmentation maps raw keyboard-mouse events to game-specific action semantics, groups consecutive actions within sustained events, uses discrete actions such as jumping or attacking as initial boundaries, and uses a VLM to disambiguate input semantics and refine short clips.
• Bottom-up temporal merging fuses adjacent L1 clips into L2 clips and adjacent L2 clips into L3 clips.
• A dynamic programming step enforces duration ranges: 1-5 seconds for L1, 1-2 minutes for L2, and 5-8 minutes for L3.
• Multi-horizon instruction annotation uses level-specific VLM prompts:
  • L1 uses sampled frames and aligned actions to generate action-grounded operations with details such as coordinates, object descriptions, and spatial relations.
  • L2 uses frames, actions, and constituent L1 instructions to generate medium-horizon goals.
  • L3 uses frames and constituent L2 instructions without action traces to generate high-level strategies.
• Prompts include game title, clip duration, frame rate, sampled frame indices, and game-specific keybinds for L1 and L2.
• No spatial cropping strategy is described in the provided text; processing focuses on temporal segmentation, merging, and instruction annotation.

How the data is used

• GameHorizon-Data is used to construct GameHorizon-Bench for reproducible offline and stepwise online evaluation.
• The offline track builds standardized multiple-choice questions from aligned frames, instructions, and player actions.
• Offline tasks include:
  • Single-horizon action selection based on frames and L1 instructions.
  • Multi-horizon decomposition from L2 goals to ordered L1 operations.
  • Cross-horizon consistency across L3, L2, L1, and action sequences.
• Variant tasks compare settings such as frames only, frames with short instructions, frames with multi-horizon instructions, top-down decomposition versus bottom-up abstraction, and instruction-only versus instruction-and-action options.
• The online track contains 10 causal tasks and 10 thematic tasks with 62 short-horizon subtasks. It uses stepwise resets so failures can be localized to specific steps.
• The online track is implemented in Minecraft because underlying game state is inaccessible in most AAA titles.
• The provided section does not specify a training split or dataset mixture ratios; the described use focuses on evaluation and benchmark construction.

Method

The authors present the GameHorizon Suite, a comprehensive framework designed to evaluate and train agents in complex game environments. As shown in the framework diagram, the suite comprises three integrated components: GameHorizon-Annotator, GameHorizon-Data, and GameHorizon-Bench.

The workflow begins with high-fidelity data acquisition. To overcome the limitations of pseudo-labels derived from Inverse Dynamics Models or gamepad segmentation, the authors recruit 100 experienced human players. A dedicated recording system synchronously captures gameplay videos at 2K resolution alongside timestamped keyboard and mouse actions. This authentic data serves as the foundation for the subsequent annotation and evaluation pipelines.

The core of the methodology is the GameHorizon-Annotator, which constructs a three-level pyramid of textual instructions: short-horizon operations ($L_1$L1​, 1–5 seconds), medium-horizon goals ($L_2$L2​, 1–2 minutes), and long-horizon strategies ($L_3$L3​, 5–8 minutes).

The annotation pipeline operates bottom-up through three distinct stages: action-aware segmentation, temporal merging, and instruction generation.

First, the authors employ action-aware video segmentation to partition long gameplay sessions into manageable clips. Unlike visual-only segmentation tools that may over-segment during rapid camera movements, this method utilizes keyboard-mouse traces to identify key action transitions. Raw inputs are mapped to game-specific semantics (e.g., mapping the 'Shift' key to sprinting). Discrete actions define initial boundaries, and a Vision-Language Model (VLM) is employed to disambiguate context-dependent inputs, such as distinguishing between an attack and item selection via a left click.

Next, the system performs bottom-up temporal merging. The VLM assesses whether adjacent $L_1$L1​ clips form a continuous progression toward a medium-horizon goal, and similarly merges $L_2$L2​ clips into $L_3$L3​ strategies. A dynamic programming algorithm enforces specific duration constraints for each level to ensure the clips remain within the reasoning capacity of current models while avoiding the fragmentation of primitive actions.

Finally, multi-horizon instruction annotation is performed using specific prompts for each level. For $L_1$L1​ clips, the VLM receives video frames and aligned actions to generate detailed, action-grounded instructions. For $L_2$L2​ clips, the input includes frames, actions, and constituent $L_1$L1​ instructions to synthesize medium-horizon goals. For $L_3$L3​ clips, the VLM uses frames and $L_2$L2​ instructions (omitting low-level actions) to generate high-level strategies. This hierarchical approach ensures that instructions are grounded in actual player controls while abstracting appropriately for long-term planning.

Applying this annotator yields GameHorizon-Data, a large-scale dataset aligning videos, actions, and multi-horizon instructions across 21 AAA game titles. Leveraging this data, the authors introduce GameHorizon-Bench, which features two evaluation tracks. The offline track utilizes thousands of multiple-choice questions to test single-horizon action, multi-horizon decomposition, and cross-horizon consistency. The online track evaluates long-horizon gameplay through verifiable subtasks in causal (order-dependent) and thematic (order-flexible) scenarios. Crucially, the online protocol resets the environment to the last success state upon failure, allowing for stepwise error localization and reproducible evaluation.

Experiment

The study evaluates 47 models across five categories using a 5,000-question offline benchmark, then samples 12 models for a stepwise online track with short-horizon subtasks and long-horizon tasks. Offline primary experiments validate the benchmark's discriminability and difficulty gradient, showing strong performance from proprietary VLMs and coding agents, weaker transfer from unified multimodal models, GUI agents, and dedicated game agents, and reasoning gains only for models with reliable thinking. Variant experiments confirm that current-action perception is easier than future-action planning, textual and multi-horizon instructions improve goal understanding, and top-down decomposition rather than action decoding is the key bottleneck. The online results align with offline rankings and reveal that long-horizon gameplay remains difficult, while stepwise resets support fine-grained failure diagnosis.

GameHorizon-Data contains 5,000 hours of human gameplay across 21 game titles, with 4,341 valid hours retained after filtering, for an 86.8% valid rate. The corpus jointly provides direct human actions and multi-horizon instructions, with the largest duration shares from Valorant, Minecraft, and Grand Theft Auto V. Instruction spans increase sharply across horizons, from a few seconds for short-horizon instructions to about five minutes for long-horizon instructions. Valorant is the largest contributor by recording duration, while Honor of Kings: World is the smallest among the 21 titles. Quality filtering retains most data, and long-horizon instructions span roughly five to five and a half minutes on average across the reported games.

The compared gameplay datasets are largely single-game benchmarks, with most entries focused on Minecraft. Large-scale data, direct human actions, and instruction annotations appear in separate datasets, but none of the listed datasets jointly provides large-scale AAA coverage, direct human actions, and dense multi-horizon instructions. GameHorizon-Data is presented as the first corpus to combine these properties across multiple AAA titles. Existing datasets are predominantly single-game and Minecraft-focused; MineRL, MineDojo, VPT, STEVE-1, and PLAICraft are all confined to Minecraft. Large-scale coverage, direct human actions, and instruction annotations appear only partially across existing datasets, whereas GameHorizon-Data is described as the first to jointly provide large-scale AAA gameplay, direct human actions, and dense multi-horizon instructions.

Existing gameplay benchmarks are limited in both model diversity and evaluation scope. Single-game benchmarks concentrate on Minecraft and support only specific model families, while the multi-game benchmark covers non-AAA titles and only a narrow set of model paradigms. GameHorizon-Bench contrasts by combining broad model diversity with direct human action annotations, multi-horizon tasks, and both reproducible offline and stepwise online evaluation. Single-game benchmarks are confined to Minecraft and evaluate only narrow model families such as game agents or VLMs. No listed existing benchmark combines reproducible offline evaluation with stepwise online evaluation, and none includes the full set of broad model diversity, AAA-focused settings, direct human actions, and multi-horizon annotations.

Across evaluated models, mean accuracy is well above the random baseline, and the benchmark separates models into clear capability tiers. Proprietary models dominate the top tier, while recent open-weight models are competitive in middle ranks. Unified multimodal models and GUI agents tend to fall into lower tiers, and task difficulty is highest for fine-grained action prediction and lowest for cross-horizon understanding. Proprietary models lead the benchmark and occupy most top-tier positions, while some recent open-weight models rank competitively below that tier. Unified multimodal models and GUI agents generally place in lower tiers; within the tasks, fine-grained action prediction is the hardest and cross-horizon understanding is the easiest.

All three game agents are evaluated zero-shot on the single-horizon action task and show weak absolute performance. JARVIS-VLA ranks highest among them, followed by Open-P2P and NitroGen, but the gap among the agents is small. Their results cluster near the random baseline, indicating limited zero-shot action prediction capability. All three game agents are tested zero-shot on the single-horizon action task. JARVIS-VLA achieves the highest T1 accuracy, followed by Open-P2P and NitroGen. NitroGen and Open-P2P fall below the 25% random baseline, while JARVIS-VLA only slightly exceeds it.

The experiments introduce GameHorizon-Data, a 5,000 hour human gameplay corpus across 21 AAA titles with 4,341 valid hours retained after filtering, and validate it as the first dataset to jointly combine large scale AAA coverage, direct human actions, and dense multi-horizon instructions. GameHorizon-Bench is compared against existing benchmarks to show broader model diversity, multi-horizon task coverage, and both offline and stepwise online evaluation. Results indicate proprietary models occupy the top tier while open weight models are competitive in middle ranks, unified multimodal and GUI agents place lower, fine grained action prediction is the hardest task, and game agents evaluated zero shot perform near the random baseline.