One-sentence Summary

Researchers from Shanghai Innovation Institute and Fudan University introduce GTAGym, a GTA V evaluation and reinforcement learning environment whose GTAVBench comprises 103 online main-story tasks across nine families plus 47 image-based shooting tasks with automatic engine-state outcome checks and parallel testing, while LoRA reinforcement learning of Qwen3.8-27B on generated navigation tasks improves in-domain performance, transfers to out-of-domain categories, and raises native GTA V task success from 15% to 23%.

Key Contributions

• The paper introduces GTAGym, a GTA V environment and evaluation framework containing GTAVBench: 103 online main-story mission tasks across nine families and 47 image-based shooting and sniping tasks, served through a web interface with parallel screenshot and keyboard-mouse control and automatic engine-state outcome checks.
• The paper contributes a scalable template-based generator that samples parameters such as movement mode, route length, turn count, and actor speed within user-specified bounds to create batches of 3D and image-based shooting environments with automatic reward and termination signals for reinforcement learning.
• Evaluation of six frontier LLMs on GTAVBench and on GTAVBench-AGI, where instructions appear only in the first screenshot and history is compressed automatically, reports a best overall accuracy of 70.9%; LoRA RL of Qwen3.8-27B on generated on-foot navigation tasks improves in-domain completion and transfers to out-of-domain categories, with native GTA V task performance rising from 15% to 23%.

Introduction

Building agents that can perceive, reason, and act in real-world environments as flexibly as humans is a central AGI goal, and sim-to-real transfer is a common path toward that goal. AAA open-world games provide a demanding testbed because navigation, vehicle control, and dynamic interaction require spatial understanding, temporal reasoning, and extended action execution. Prior work advances separate directions: verifiable tasks in sandbox or purpose-built games, agents in commercial 3D worlds, and benchmarks based on visual trajectories or offline questions. However, these approaches do not fully combine real AAA mission objectives, automatic outcome verification, and generation of new executable training tasks. The authors introduce GTAGym, an environment and framework that decomposes GTA V main-story missions into reusable tasks, provides GTAVBench with 103 online tasks across navigation, vehicle control, and dynamic interaction plus 47 image-based shooting tasks, and adds a GTAVBench-AGI branch that presents goals only in the first screenshot with automatic context compression. The framework also exposes a shared screenshot-and-action interface with engine-state evaluators and a template-based generator for batched RL environments with automatic reward and termination signals.

Dataset

Dataset description: GTAVBench

The authors construct GTAVBench as a 150-task benchmark derived from GTA V gameplay. It shares a task suite with the GTAVBench-AGI branch and is evaluated through GTAGym, which provides a 3D environment for task initialization, interaction, and outcome verification.

• Overall composition and sources

  • 75 GTA V main-story missions are manually decomposed into subtasks based on collected walkthrough videos.
  • The resulting annotations and source game images support 150 benchmark tasks across eleven categories.
  • The eleven categories are grouped into navigation, interaction, and gun use.
  • The benchmark has two tracks: an online 3D track and an image-based track.

• Online 3D track

  • Contains 103 tasks across nine dimensions.
  • Reconstructed in GTAGym using Babylon.js.
  • Covers urban, indoor, water, and aviation scenarios.
  • Uses native engine states for automatic verification of online task outcomes.

• Image-based track

  • Contains 47 shooting and sniping tasks.
  • Built from source gameplay imagery for visual target localization and aiming.
  • Shooting tasks select gameplay frames and associate targets with annotated hit regions.
  • Sniping tasks use images captured directly in GTA V to create unscoped and scoped views with magnification-dependent rendering.

• Task construction and processing

  • Each subtask specifies a goal, required interaction order, completion constraints, and entity behaviors.
  • For 3D navigation tasks, routes are manually traced on source minimaps and adapted to available city geometry with comparable turn counts.
  • Climbing surfaces and landing areas are added where required by the task.
  • Dynamic interaction tasks use predefined NPC behaviors and mechanics such as collision-induced slowdown.
  • Each subtask is instantiated as a scenario template with an initial state, instruction, task budget, and executable success and failure predicates.
  • Completion predicates check movement mode, height or depth, target distance, and interaction events.
  • Following and pursuit tasks use predefined target routes; evasion tasks use pursuer controllers that update detection and capture states.
  • The provided text does not describe a separate cropping-specific pipeline beyond frame selection and target region annotation for image-based tasks.

• Usage in model training and evaluation

  • Task templates are used for procedural RL environment generation.
  • Each template fixes the scene, available assets, interaction logic, evaluator, and episode budget.
  • Users control variation through numeric intervals or finite candidate sets, such as route length ranges and turn count candidates.
  • The generator samples feasible configurations, rejects incompatible ones, and saves resolved configurations and seeds for reuse.
  • Environments share a screenshot-action interface and use task outcomes as sparse rewards.
  • Success and failure end an episode; budget exhaustion is recorded as truncation.
  • The evaluator uses engine-state predicates for 3D tasks and target-hit checks for image-based tasks.
  • The provided text does not specify fixed train, validation, or test split ratios or mixture ratios.

Method

The authors introduce GTAGym, a comprehensive evaluation and training environment designed to assess agents across eleven dimensions of GTA-style gameplay, including navigation, vehicle control, dynamic interaction, and visual aiming. The framework is divided into two primary tracks: an online 3D track and an image-based track, collectively comprising 150 tasks.

The online 3D track encompasses 103 tasks across nine categories, evaluating navigation, vehicle control, and dynamic interaction within urban, indoor, water, and aviation scenarios. The image-based track adds 47 shooting and sniping tasks that utilize source gameplay imagery for visual target localization and aiming.

Task construction begins with the manual decomposition of main-story missions into distinct subtasks based on walkthrough videos. For each subtask, the authors specify the goal, required interaction order, completion constraints, and entity behaviors. Online scenario reconstruction is implemented using Babylon.js, where task controllers expose task-specific states. For navigation tasks, paths are manually traced on source minimaps to find routes with comparable turn counts in the available city geometry. Dynamic-interaction tasks utilize predefined NPC behaviors and mechanics. Image-based tasks are constructed by selecting gameplay frames and associating targets with annotated hit regions, or by capturing distinct unscoped and scoped views for sniping.

Each subtask description is instantiated as a scenario template containing an initial state, an instruction, a task budget, and executable success and failure predicates. The evaluation framework employs a shared interaction interface for both benchmark branches, following a closed-loop screenshot-action protocol with keyboard-mouse control. The agent specifies a hold duration $d \in (0, d_{\max}]$d∈(0,dmax​] for each key, with $d_{\max} = 1$dmax​=1 second. Keys execute concurrently, and the turn duration is determined by the longest scheduled hold.

To support reinforcement learning, the authors implement procedural environment generation. The system organizes environment generation around task templates, where users control variation through numeric intervals or finite candidate sets:

where $\theta$θ contains sampled parameter values and $\xi$ξ is the generation seed. A rule-based generator samples configurations subject to user-defined bounds and template feasibility constraints, rejecting incompatible configurations. This process generates batches of RL environments where instances share the screenshot-action interface and use task outcomes as sparse rewards, with success and failure ending an episode and budget exhaustion recorded as truncation.

Experiment

Experiments evaluate six models on GTAVBench and GTAVBench-AGI, closed-loop screenshot-action benchmarks with keyboard and mouse control over navigation, interaction, and gun-use tasks, where the AGI branch presents the goal only in the first screenshot and compresses context. GPT models lead overall but remain below human performance, with dynamic control, calibration, and regulation as the main bottlenecks and better error recovery in GPT models; reinforcement learning with LoRA on generated on-foot routes improves held-out on-foot tasks and some out-of-distribution navigation but not road driving, aerial navigation, or interception. Harness context can aid memory yet also cause goal drift, and evaluation settings are important, since shorter action durations improve performance via more frequent feedback while removing inference-time pausing severely degrades dynamic tasks.

The comparison shows that prior game benchmarks specialize in subsets of evaluation capabilities: many support 3D interaction and state-based scoring, but only GameVerse targets AAA open-world tasks, and it lacks state-based scoring, web execution, and RL instance generation. GTAGym is presented as the environment that combines all five listed features, covering AAA open-world tasks, state-based scoring, web execution, and RL instance generation. GameVerse is the only prior benchmark with AAA open-world task support, but it does not report state-based scoring, web execution, or RL instance generation. State-based scoring and RL generation co-occur in Procgen and MineDojo, while state-based scoring and web execution co-occur in GameWorld; no prior benchmark combines all five features.

GPT-class models show a substantial accuracy advantage over the other evaluated models, but the best configuration still remains well below human performance. Navigation tasks, especially on-foot, indoor, and vertical navigation, are strong for these models, while aerial navigation, evasion, and dynamic control remain difficult. The AGI protocol raises overall accuracy across all six models, with the most consistent gains in navigation and mixed changes in gun-use tasks. GPT models outperform DeepSeek and GLM by a wide margin under both protocols. The best setup reaches 70.9% overall accuracy, 18.9 points below mean human accuracy. GTAVBench-AGI improves overall accuracy for all six models by 0.9 to 10.0 points, with the most consistent gains in navigation. GPT models reliably handle on-foot, indoor, and vertical navigation but struggle with aerial navigation and evasion. Dynamic control remains a bottleneck, and interception performance separates Astra from the other models.

The first experiment compares GTAGym with prior game benchmarks, showing that GTAGym is the only environment combining AAA open-world tasks, state-based scoring, web execution, and RL instance generation, while alternatives such as GameVerse cover only part of these capabilities. The second experiment evaluates six models on GTAVBench under standard and AGI protocols, finding that GPT-class models lead but still fall below human performance, with strengths in on-foot, indoor, and vertical navigation and persistent weaknesses in aerial navigation, evasion, and dynamic control. The AGI protocol improves overall accuracy across all models, mainly in navigation, while interception remains a challenging task.