One-sentence Summary

The authors propose NCP-Bench, a benchmark of 100 narrative environments derived from movie synopses and equipped with structured specifications, to systematically evaluate long-horizon narrative commitment preservation in LLM-driven interactive storytelling, revealing that even the strongest model (GPT-5.2) achieves only a 42% survival rate after 20 turns and fact conflict rates across models range from 40% to 68%.

Key Contributions

• Narrative Commitment Preservation (NCP) is formalized as a task for maintaining logical commitments in open-ended interactive narrative, providing a domain-agnostic abstraction that requires instantiating commitments, state, and violation checks per application.
• NCP-Bench is introduced, a benchmark of 100 narrative environments built from movie synopses, each containing a structured narrative specification (trajectory, commitments, initial facts) and an automated auditing protocol that evaluates commitment preservation throughout agent interactions.
• Experiments across six state-of-the-art LLMs show that the best-performing model (GPT-5.2) achieves only 42% survival after 20 turns, with fact conflict rates ranging from 40% to 68%, and only isolated runs satisfy all achievement commitments within 100 turns, demonstrating that linguistic fluency does not guarantee long-horizon logical consistency.

Introduction

The authors tackle interactive narrative in AI for Games, where LLM-based narrator agents must maintain coherent story worlds and adhere to plot constraints despite free-form user inputs. While LLMs can generate fluent text, they often fail to enforce logical consistency, allowing user actions that contradict established facts or skip mandatory plot milestones. To address this, the authors formalize Narrative Commitment Preservation (NCP) as a long-horizon constraint satisfaction problem and introduce NCP-Bench, a benchmark of 100 narrative specifications with explicit fact ledgers and commitments, enabling automatic auditing of state consistency. Their experiments show that even state-of-the-art models like GPT-5.2 struggle to survive beyond 20 turns, with fact conflicts dominating failures, revealing that fluency does not guarantee reliable logical commitment.

Dataset

The authors construct NCP-Bench, a benchmark for evaluating narrative commitment preservation in interactive storytelling environments. The dataset is built from the CMU Movie Summary Corpus and consists of 100 curated environments, each grounded in a movie synopsis and a specific in-story character whose perspective limits the available information.

Dataset composition and sources

• Source: CMU Movie Summary Corpus, using the subset of classic movie characters (not necessarily protagonists).
• Initial candidate pool of (movie, character) pairs is processed through cleaning and deduplication.
• After manual cleaning and filtering, 100 movies are selected to maximize genre diversity and narrative quality.
• Each finalized environment includes a structured narrative specification: an initial fact ledger, a set of narrative commitments, and an ordered reference trajectory that tracks major plot steps.

Key details for each subset

• The dataset contains a single set of 100 environments, with no predefined training/validation/test splits.
• Every environment is tied to one player character, and all annotations are constrained to that character’s point of view (no omniscient information).
• Two genre tags (primary and secondary) are manually assigned per movie; the final set covers 18 genres (e.g., Action, Sci-Fi, Comedy, Crime, Western).
• The narrative specification format uses three component types:
  • Initial fact ledger: world state and player knowledge at the start, with strict future-sight isolation.
  • Commitment set: rules that must be respected (Invariants, Orderings) or goals that must be satisfied (Achievements); violations or unmet Achievements cause failure.
  • Reference trajectory: ordered plot steps, each with a description, trigger event, and key delta.
• Quality verification: after construction, three experts independently reviewed all 100 specifications, flagging only seven for localized ambiguities (no recurring errors).

How the paper uses the data

• NCP-Bench is used as an evaluation benchmark for the commitment preservation task.
• The task is modelled as a free-form, open-ended interaction between a player (model) and a narrator agent.
• The benchmark provides the structured specification per environment; the model must interact while preserving the given commitments, with success determined by whether all Achievement commitments are satisfied without any violations.
• No training mixture ratios or dataset splits are applied, as the dataset serves purely for evaluation.

Processing and cropping details

• Cleaning: human experts remove duplicate/nearduplicate entries, resolve character-movie mismatches, and filter out synopses too incomplete to support interaction.
• Diversity-oriented selection: from the cleaned candidates, 100 movies are manually chosen to cover orthogonal narrative styles and genres, prioritizing detailed, coherent synopses.
• Genre annotation: two tags are manually assigned per movie to guide selection and enable later analysis.
• No cropping or downsampling is applied; the full 100 environments are used in evaluation.
• The narrative specification is generated per environment and reviewed by experts; specifications with major issues are regenerated, while those with minor issues are corrected.

Method

The authors study interactive narrative as a turn-based interaction where a narrator agent acts as a Game Master, responding to a player's free-form actions while maintaining a coherent story world and adhering to authorial plot constraints. At each turn $t$t, the interaction is characterized by the dialogue history $H_t$Ht​ and an evolving story world. The player produces a free-form utterance or action $u_t$ut​, and the agent outputs a narrative continuation $y_t$yt​. The agent response serves two coupled functions: logical reaction, which acknowledges the action and updates the narrative state, and narrative steering, which guides the interaction toward plot-relevant events while preserving player agency.

To evaluate this open-ended task reliably, the authors model it as a commitment preservation task and construct NCP-Bench. They start from the CMU Movie Summary Corpus, collecting candidate movie and character pairs. After cleaning and deduplication by human experts to remove noise and resolve mismatches, they manually select 100 movies to maximize coverage of orthogonal narrative styles and genres.

The final benchmark is constructed from these 100 curated synopsis and player character pairs, ensuring broad genre coverage across 18 distinct categories. For each synopsis, the associated character is designated as the player role, and all annotations are constrained to this role's point of view to avoid omniscient information leakage.

The core of the method is Narrative Commitment Preservation (NCP), which maintains an explicit representation of the story state, narrative constraints, and trajectory progress:

where $F_t$Ft​ is a fact ledger recording the currently established world state, $C$C is a set of commitments, $\mathcal{R}$R is an ordered reference trajectory, and $i_t$it​ records progress along that trajectory. Each trajectory node represents a salient plot step evaluated using its trigger event and key delta.

The evaluation framework separates the narrator agent being evaluated from a fixed, externally defined auditing protocol. The narrator agent produces narrative text, while a set of prompt-fixed evaluation components determines consistency, updates the explicit state, and tracks progress.

The framework evaluates each turn through four distinct steps. First, the Conflict Check examines the narrator agent response for fact conflicts with the active ledger, commitment conflicts, and player-input conflicts. Any initially detected conflict undergoes a secondary confirmation step to mitigate false positives from LLM auditors. Second, the Fact Update extracts incremental updates, comprising added and negated facts, from the response, committing them to the fact ledger only if the Conflict Check passes. Third, the Trajectory Node Update determines whether the current node's trigger event and key delta have occurred, advancing the index if both are explicitly completed. Finally, the Commitment Check tracks whether each commitment's satisfaction condition has been met, marking it as PENDING or SATISFIED. If any violation is confirmed, the interaction terminates as a failure; otherwise, it succeeds when all achievement commitments are satisfied without contradiction.

Experiment

The evaluation framework pairs a narrator agent with a fixed, off-the-shelf auditing protocol that checks for fact conflicts, commitment violations, and player-input acknowledgment while tracking trajectory progress. Across multiple state-of-the-art LLMs evaluated as narrators, survival rates decline steeply with interaction length, and fact hallucinations emerge as the primary failure mode, with even the best model achieving only 42% survival after 20 turns. Qualitative analysis reveals that models often prematurely reveal plot-critical information, rewrite established story facts, or ignore player actions. Ablation studies confirm that these findings are robust to auditor choice, and neither memory augmentation nor cooperative player inputs resolve the core difficulty of preserving narrative commitments over long horizons.

All models struggle to preserve commitments over long interactions, with near-zero survival rates well before the 100-turn limit and low overall satisfied commitment percentages. GPT-5.2 achieves the longest average interactions but makes slower plot progress, while DeepSeek-V3.2 advances more aggressively, reaching later trajectory nodes and achieving the highest satisfied commitment percentage despite fewer turns. Fact conflicts are the primary failure mode, with GPT-5.2 exhibiting the lowest conflict rates overall. GPT-5.2 sustains the longest interactions (32.92 average turns) but shows lower trajectory progress and commitment satisfaction than DeepSeek-V3.2, which advances further per turn. Fact conflicts dominate as the failure mode, ranging from 40% of interactions for GPT-5.2 to 68% for Qwen3-235B-A22B.

Fact conflicts dominate across all evaluated models, comprising at least 40% of all conflicts, while commitment and player-input conflicts vary more widely. Human verification on GPT-4o-mini outputs confirmed that evaluator errors are rare and largely limited to nuanced fact-transition boundary cases, with no disputes found in commitment or player-input conflict categories. Fact conflicts are the most frequent failure mode for every model, with GPT-5.2 showing the lowest share (40%) and Qwen3-235B-A22B the highest (68%). Human review of GPT-4o-mini found only 4% of fact-conflict cases were disputed, all in state-change or epistemic-update boundaries, and no errors in commitment or player-input conflicts.

Three different auditor models produce broadly consistent conflict profiles when evaluating the same narrator agent, with fact, commitment, and player-input conflict rates falling within narrow ranges. While GPT-5.2 is stricter, recording no successful runs and lower trajectory progress, all auditors yield highly correlated outcomes, confirming that evaluation conclusions are not tied to a single backbone. Human verification shows rare errors, limited to boundary-case fact conflicts. Fact conflict rates are similar across auditors, ranging from 60% to 67%. Commitment and player-input conflict rates remain comparable, varying by no more than 15 percentage points. GPT-5.4-mini and Gemini-2.5-Flash each report 5 successful runs, while GPT-5.2 records none. Trajectory progress is highest for GPT-5.4-mini (16.64%) and lower for GPT-5.2 (7.67%), but satisfied-commitment rates stay near 10–11% for all. Pairwise Pearson correlations remain above 0.96, indicating strong agreement between auditors. Human review of 100 runs found only 4 fact-conflict false positives, all in nuanced state-change cases, and no errors in commitment or player-input conflicts.

Integrating hierarchical memory reduces commitment conflicts dramatically but substantially increases player-input conflicts, and the overall success rate declines. The memory-augmented agent achieves longer interactions yet fails to satisfy all achievement commitments in any run, illustrating a trade-off between long-term plot tracking and local input fidelity. HiAgent cuts commitment conflicts from 26% to 4%, but player-input conflicts more than double from 13% to 38%. Runs satisfying all achievement commitments drop from 2 with plain GPT-4o-mini to 0 with HiAgent, and conflict-free runs decrease from 5 to 3. Fact conflict rates remain nearly identical (60% vs 59%), while average turns increase from 22.16 to 30.05.

Switching from adversarial to natural player inputs more than doubles average interaction length and sharply reduces commitment conflicts, but the number of runs that satisfy all achievement commitments drops to zero. Even cooperative play does not eliminate conflicts: the majority of interactions still end in conflict, demonstrating that adversarial inputs amplify difficulty but are not the root cause; the core challenge of long-horizon commitment preservation persists. Natural inputs more than double the average number of turns (from 22.16 to 46.08) and raise the number of runs reaching the 100-turn limit from 3 to 19. Commitment conflicts drop from 26% to 9%, yet runs satisfying all achievement commitments fall from 2 to 0. Player-input conflicts increase from 13% to 18%, likely because cooperative players generate more inputs that the narrator must acknowledge. Despite the shift to cooperative play, most interactions still end in conflict, confirming that adversarial difficulty is not the only barrier.

All models fail to preserve plot commitments over long interactions, with fact conflicts emerging as the primary failure mode across all tested systems. GPT-5.2 sustains the longest interactions but makes slower progress, while DeepSeek-V3.2 advances more aggressively and achieves higher commitment satisfaction. Hierarchical memory sharply reduces commitment conflicts but amplifies player-input issues, and switching to cooperative play more than doubles interaction length yet still cannot produce runs that satisfy all commitments, demonstrating that the core challenge of long-horizon consistency is intrinsic, not merely adversarial.