One-sentence Summary

Researchers from the National University of Singapore and Nanyang Technological University survey foundation-model-era AI for games by organizing the literature into six roles spanning from playing to testing, and they argue that the central challenge lies in reusing or transferring capabilities across these roles without re-validating them within the game-specific structures, interfaces, and player contexts where they are ultimately deployed.

Key Contributions

• Introduces a taxonomy organizing foundation model research in games into six roles by immediate use of AI output, from AI that Plays and Acts to AI that Tests and Evaluates Games.
• Maps cross-role connections such as interaction traces training world models and design specifications driving executable implementations, while identifying that control schemes, engine interfaces, and player contexts remain setting-specific and require validation in target environments.
• Synthesizes evidence showing evaluation is most standardized for bounded game playing and selected learned environments, whereas persistent state in learned worlds, repeated software revision, sustained runtime adaptation, and representative automated testing are less established.

Introduction

AI for games has expanded beyond just playing. Recent models like GPT-6 Astra are now used in unfamiliar interactive environments (as in ARC-AGI-3) and in development tools like Playbot to create playable prototypes, placing the same pretrained model in different roles across the game lifecycle. Prior work in gameplaying, procedural generation, and player modeling developed largely in separate research communities, with limited connections between tasks such as designing, implementing, and evaluating games. The authors' main contribution is a survey that organizes AI for games into six roles, connects them through the outputs that pass between them, and analyzes how foundation models, with their language, multimodal, and tool interfaces, reshape these roles. They specifically examine three recurring findings: broad pretraining expands interfaces but does not remove game-specific structure, gameplay increasingly provides data and feedback beyond a final score, and progress is task-dependent with stronger benchmarks for bounded play than for sustained creation, adaptation, or human experience.

Dataset

Dataset Description: Player Modeling and Interactive Generation

The authors draw on a wide range of datasets to support player modeling, action interfaces, and long-horizon generation. These datasets vary in source, scale, and annotation, and are used differently depending on whether the goal is behavioral inference, affect prediction, or training interactive video generators.

1. Behavioral and Player Modeling Data

• Chess move datasets: The Maia line of models uses 12 million online games per rating range to model human-like moves at different skill levels. Maia-2 conditions a shared model on player skill, while Maia-3 uses board-square tokens. For individual adaptation, Maia4All learns player prototypes and adapts an embedding from as few as 800 positions (about 20 games) per player, with held-out move accuracy rising from 51.4% (Maia-2) to 53.2% in that low-data setting.
• Battle-royale behavioral features: Engineered from more than 75,000 matches, these features predict player rank better than three mainstream rating systems.
• Interactive fiction framework: The "Learning to Play Like Humans" framework guides language models through interactive fiction, using structured map building and feedback-driven experience analysis, aligning behavior with narrative intent rather than task score.

2. Affect and Experience Datasets

• AGAIN dataset: Includes more than 1,100 in-game videos from nine games, with gameplay data annotated for arousal by 124 participants in a continuous, first-person fashion, totaling more than 37 hours of annotated gameplay.
• Gameplay video for affect prediction: A survival shooter dataset uses 50 videos for leave-one-video-out validation, classifying high versus low arousal at over 78% average and 98% best accuracy. A commercial title, Tom Clancy's The Division 2, provides nearly 20 hours of annotated play from 25 players, fusing footage with gamepad actions to predict long-term engagement (up to 72% average, 88% best accuracy). Angry Birds is used for label-free affect approximation from Let's Play videos.
• Foundation-model affect studies: A study of three vision-language models under six prompting strategies on nine first-person shooters finds zero-shot engagement predictions generally weak, often failing to beat per-game majority-class baselines.

3. Subjective Judgment and Playtesting Data

• MeepleLM: Trained on 1,727 corrected rulebooks and 150,000 board-game reviews to simulate different player groups' experiences and critiques.
• Judgment prediction study: Collins et al. use 121 novel strategy games, comparing model evaluations with judgments of expected payoff and fun from over 450 participants. Agreement with game-theoretic estimates and with people varies non-monotonically, indicating that stronger optimal-play reasoning alone does not yield a more faithful model of human judgments.
• Beyond Playtesting: Adapts language models to a massively multiplayer game using supervised fine-tuning and reinforcement learning on large-scale real player behavior, with offline simulations reproducing player reasoning and reactions to interventions.

4. Interactive Video Generation Data

• Action-conditioned datasets:
  • SCOPE: Trains on 69,000 clips from seven first-person shooters with aligned 10-DoF controls, separating spatially local actions (firing) from global camera and movement signals.
  • Game2World Engine: Introduces a removal pipeline for heads-up-display elements, plus 96,000 paired synthetic videos and 1,079 in-the-wild clips from 303 games. Training on UI-free gameplay improves overall VideoReward by 6.83% relative to UI-overlaid footage.
  • H3-World: Uses 7,872 training clips and 128 held-out clips with structured character and camera instructions, testing language-conditioned control with action interventions that keep initial observation and generation conditions fixed.
• Latent-action datasets: Genie infers discrete actions between frames from ordinary video without ground-truth controls. ShadowDancer learns action latents from paired renderings that preserve motion while varying appearance.
• Entity and language control: Incantation reports 89% action-control accuracy versus 43% for an action-index baseline on five held-out entity-action pairs, and 90% versus 0% on four out-of-vocabulary probes. ReactiveGWM tests strategy-module transfer between two fighting games.
• Long-horizon and serving studies: Yume-1.5 combines compressed history and linear attention with distilled streaming generation, adding textual event control. AlayaRenderer-Flash uses a physics engine and a four-step streaming model, reporting 31.54 fps on one H200 and 30 fps for live SuperTuxKart after fine-tuning.

5. Cropping, Filtering, and Preprocessing

• Heads-up-display removal: Game2World Engine applies a dedicated removal pipeline to strip overlays from gameplay video, separating scene dynamics from game-specific display conventions. The authors argue that health, ammunition, and timers may be essential state observations and suggest retaining them as a separate channel if needed.
• Visual drift mitigation: CausVid distills a bidirectional diffusion transformer into a causal few-step generator, supervising the causal student with the teacher to reduce error accumulation. Self Forcing trains on the model's own rollouts with key-value caching, applying a video-level loss to the whole sequence. FramePack adds drift-prevention methods including early-established endpoints and adjusted sampling orders.

6. Data Use in Model Training and Evaluation

• Training splits: Maia uses separate models per rating range, with held-out positions to prevent memorization, held-out players to test the population model, and later games by the same person to test individual representation usefulness. The three splits answer different questions even when scored by move accuracy.
• Mixture ratios and adaptation: Beyond Asking infers profiles from observed behavior, tests recovery of controlled synthetic traits, and includes an exploratory 12-participant pilot. A match-three deployment uses language-model reasoning to assign coarse player types and evaluates the resulting personalized levels.
• Validation strategy: The authors emphasize separating observations, inferred constructs, and intervention outcomes. For models inferred from interaction, calibration and temporal stability are measured before deployment, while randomized content comparisons assess downstream benefit. This prevents an improvement in level completion from being treated as proof that every intermediate profile is accurate.
• Long-rollout evaluation: MultiGen produces synchronized generated viewpoints backed by an editable external map, but its multiplayer evaluation uses simulated Doom deathmatches on a single map. MASS separates a learned global-state transition model from per-camera rendering, evaluating both state accuracy and agreement between views in multiplayer Snake.

Method

The authors present a comprehensive framework for integrating AI across the game development and interaction lifecycle, categorized into four primary roles: playing and acting, modeling players and games, building and maintaining games, and generating and adapting content at runtime.

For agents that play and act, the architecture transitions from specialist policies to generalist systems through algorithm reuse, parameter sharing, and interface constraints. As shown in the figure below, the system employs control hierarchies where language planners handle high-level reasoning, skill libraries provide reusable behaviors, and controller training manages low-level motor execution. Test-time adaptation is achieved through retrieval-based methods, dynamics identification, and episodic memory retention to handle unfamiliar game mechanics and layouts.

When modeling players and games, the framework separates planning, training, and simulation from world state representation and player behavior inference. Learned dynamics support planning models and training environments, while long-horizon interactions require robust action representation and generation serving. World state representation combines aleatoric and epistemic uncertainty estimation with memory and spatial consistency, alongside explicit structured state to track numerical variables and game mechanics. Player behavior inference leverages behavioral representation learning, human action prediction, and affect modeling to estimate engagement, churn, and retention.

In the domain of building and maintaining games, the authors describe a pipeline where tool-using development agents coordinate multi-agent efforts to integrate code, scenes, and engine components. These agents utilize repository access, editor state, and runtime feedback to execute and debug projects. The maintenance phase focuses on regression avoidance and update-aware testing to ensure that new features do not break existing functionality, facilitating developer handoff and long-term project evolution.

Finally, for runtime generation and adaptation, the system manages generative content such as dialogues, props, and tasks while maintaining long-term character memory and state-presentation agreement. Dynamic rules and mechanics are implemented through generated code and constrained rule languages, validated by engine-side checks. Personalization is driven by affect and model-driven adaptation, difficulty adjustment, and adaptive characters to tailor the interactive experience to individual players.

Experiment

Experiments span multiple roles, including multi-agent coordination and social interaction, player behavior modeling, game generation and repair, runtime personalization, and automated testing. Key findings show that execution feedback improves repair and that project scale sharply reduces runtime success, while structured intermediate representations can improve mechanic adherence at the cost of runtime performance. Reactive adaptation studies report mixed player outcomes, with effects depending on integration choices and response timing, and testers that combine exploration with diverse oracles uncover more defects, though diversity is not equivalent to representativeness of human play. Evaluation across roles consistently distinguishes partial progress from task completion, separates model capability from harness configuration, and cautions that judge agreement or convergence does not establish independent verification.

The analysis across roles focuses on three recurring questions: boundary setting between game/workflow and AI, transfer and reuse of capabilities and artifacts, and the evidence supporting claims at the point of use. Evaluation must consider not just initial success but also robustness to changes in rules, requirements, and player contexts, including multiplayer fairness and system updates. Boundary questions ask what is supplied by the game or workflow versus what is assigned to AI, and evidence must be grounded at the point of use. Transfer and reuse depends on which capabilities and artifacts generalize, while some elements remain setting-specific. Automated testing faces limitations related to state coverage, oracle accuracy, and sampling representativeness, which can be addressed by combining complementary exploration policies and independent oracles. Using model judges for evaluation risks self-preference and rewarding shortcuts, so independent verification is required beyond repeating or changing the judge. End-to-end evaluations should report compute and latency, and test rollback or fallback after failed updates, as well as compatibility with shared multiplayer state.

The table organizes AI game systems into six roles, each defined by its outputs, applications, and a principal empirical claim. The roles span playing, modeling, designing, building, generating runtime content, and testing, with a focus on how pretrained and multimodal models extend these tasks. The Play and Act role produces actions, plans, and messages for players, teammates, or NPCs, with the claim centered on action quality. Model Players and Games outputs states, transitions, and player forecasts, applied to planning and simulation, with claims about predictive quality and policy transfer. Design generates levels, rules, and story structures for PCG and co-creation, with claims about validity and control. Build and Maintain produces code, scenes, and project edits for engine agents and debugging, with the claim of working software. Generate and Adapt at Runtime outputs dialogue, quests, and live content for characters and adaptive narrative, with claims about consistent content and player response. Test and Evaluate provides traces, verdicts, and diagnoses for playtesting and verification, with claims about state coverage and verdict accuracy.

The table organizes game-agent training approaches by what is reused across tasks versus what is game-specific, with examples from each category. It distinguishes per-game training, formal-specification reuse, parameter sharing, and test-time adaptation, highlighting the different generalization claims each setup supports. Per-game training reuses algorithms and settings but requires game-specific policy training and dynamics learning, as seen with DQN and AlphaZero. Formal-specification approaches reuse solvers and interfaces while keeping game rules and forward models unique, such as in General Game Playing. Parameter sharing reuses policy weights and tests on held-out games, exemplified by Gato and Multi-Game Decision Transformers. Test-time adaptation reuses base models and retrieval mechanisms with game-specific demonstrations and exploration, as in REGENT and Code World Models.

This table compares seven design methods by the type of generated object, the feedback or checks used, and the game structure supplied to the generator. The methods range from generating tile sequences or 3D scenes to rule programs, dialogue, and plans, with oversight mechanisms including compilers, simulated play, and iterative repair. These distinctions show how different approaches embed validity constraints and designer control in the generation process. MarioGPT and DreamCraft generate tile-based content but differ in feedback: MarioGPT uses A* checks and prompt features, while DreamCraft relies on text alignment and adjacency losses. GAVEL and ScriptDoctor both generate game logic, but GAVEL uses compile and simulated play, whereas ScriptDoctor adds iterative repair with a BFS solver. KNUDGE focuses on branching dialogue with lore and quest consistency checks, in contrast to structure-generating methods. DreamGarden combines compiled visual checks with user edits, highlighting human-in-the-loop verification.

This table provides a guide for interpreting evaluation results across six AI roles in games, clarifying what each role's targets measure and where interpretations should stop. It emphasizes that success in one context or metric does not imply validity in another, such as short-term fidelity not guaranteeing persistence or coverage not equating to correctness. For Play and Act, evaluation outcomes are protocol-specific, and transfer across rules, controls, or timing is untested. In Model Players, short-horizon fidelity does not establish persistence, and synthetic traits do not confirm real-player validity. Design evaluations do not establish authoring efficiency or the intended player experience. Build and Maintain results do not ensure long-term maintenance or smooth developer handoff. Runtime adaptation evidence is single-session, and adaptation effects need suitable control conditions. For Test and Evaluate, coverage does not equal correctness, and sampled behavior does not represent the full population.

The evaluation framework addresses recurring questions about AI-game boundaries, capability transfer, and evidence grounding, emphasizing robustness to rule changes, multiplayer dynamics, and system updates. The six AI roles (playing, modeling, designing, building, generating runtime content, and testing) each carry distinct claims, but results are protocol-specific and do not generalize across contexts without additional validation. Training approaches vary from per-game training to parameter sharing and test-time adaptation, with different claims about reuse. Design methods embed validity constraints via compilers, simulated play, or iterative repair, yet success in one metric does not confirm validity in another, such as coverage not equating to correctness or short-term fidelity not ensuring persistence.