One-sentence Summary

A team from Alaya Lab, Shanghai Jiao Tong University, et al. introduces AgenticSTS, a bounded-memory testbed where LLM agents receive fresh prompts via typed retrieval rather than accumulating transcripts, allowing isolated ablation of memory layers, and their ablations in the game Slay the Spire 2 show that triggered strategic skills improve win rate from $\frac{3}{10}$103​ to $\frac{6}{10}$106​, demonstrating a reproducible methodology for studying long-horizon agent memory.

Key Contributions

• A bounded-memory contract is introduced where each decision prompt is assembled from typed retrieval without appending a raw cross-decision transcript, keeping prompts bounded across runs and enabling independent ablation of memory layers; the contract is instantiated in the stochastic deck-building game Slay the Spire 2.
• Ablation experiments with a fixed-A0 configuration show that adding a triggered strategic skill layer (L5) to a no-store baseline increases the win rate from 3/10 to 6/10 games, providing directional evidence for the impact of explicit skill memory although the comparison is not statistically decisive (Fisher exact p ≈ 0.37).
• A reproducible testbed of 298 completed trajectories with condition tags, frozen memory and skill snapshots, prompt records, and analysis scripts is released, forming an agent design and validated methodology for studying how typed memory layers shape decisions in long-horizon LLM agents.

Introduction

Long-horizon LLM agents must decide what prior experience to surface in each decision prompt, yet typical designs either append full transcripts that balloon context windows or retrieve unstructured pieces, making the memory interface opaque and hard to audit. The authors target this gap by formalizing memory as a bounded, typed contract and instantiating it in the unsolved roguelike game Slay the Spire 2, where frontier models currently fail to win. Their agent, AgenticSTS, replaces raw transcript accumulation with five per-decision slots (protocol, state, retrieved rules, episodic summaries, and triggered skills), transforming memory from a growing history into a set of inspectable and individually ablatable components. This architecture lets researchers isolate which memory type matters most, and the authors release 298 condition-tagged trajectories, frozen slot snapshots, and analysis scripts to support reproducible comparisons of long-horizon agent memory design.

Dataset

The authors release a dataset of 298 complete game trajectories from the roguelike deck-builder Slay the Spire 2, collected under a bounded-memory LLM-agent interface. The data serves as a reusable evaluation surface for long-horizon planning and memory in language models.

• Composition and sources: each trajectory records the full text-based game state, the agent’s decisions, and metadata – target and reached Ascension level, outcome (victory/defeat), wall-clock duration, LLM-call counts, condition tag, and the active memory/scaffold setting. The runs were generated by LLM agents using the experimental framework described in the paper.

• Subset breakdown:

• Fixed‑Ascension 0 (A₀) balanced subset: 50 runs, 10 per experimental condition under a frozen configuration. Used as the primary head-to-head ablation benchmark.

• Cross‑backbone probes: additional trajectories that compare different underlying LLMs under otherwise identical settings.

• Auto‑mode ascension ladder: runs where the agent automatically advances to the next Ascension level after a victory or retries after a defeat, covering the full 11‑level ladder (A₀–A₁₀).

• Data usage: no training split exists; the dataset is purely for evaluation. The fixed‑A₀ subset isolates component effects at a single difficulty, while the auto‑mode runs track the agent’s ability to climb the ordinal difficulty ladder. The cross‑backbone trajectories support robustness checks across models.

• Processing and metadata: the raw trajectories are preserved as frozen stores (with long‑term memory layers L₄ and L₅ included) and per‑condition tags. For scoring, a derived analysis score is computed from the run outcome, floor reached, and number of act bosses cleared (full mapping in the appendix): 100 points for a victory; otherwise floor + (52/3) × bosses. This score is recalculated from trajectory fields, not copied from the game’s native archive, and serves as a secondary metric alongside win‑rate.

Method

The authors replace the conventional prompt-history interface with a bounded, typed retrieval contract. Instead of accumulating raw message turns over time, each decision prompt is composed on the fly from five knowledge layers. This design caps context size regardless of run length and makes the memory architecture itself a controlled variable for ablation.

For decision $d$d at state $s_d$sd​, the engine retrieves relevant items from the five layers and constructs a fresh user message $u_d = \pi\big(L_1,\, L_2(s_d),\, L_3(s_d),\, L_4(s_d),\, L_5(s_d)\big)$ud​=π(L1​,L2​(sd​),L3​(sd​),L4​(sd​),L5​(sd​)) which is sent to the LLM together with a static system prompt. The context budget is $O(|\mathrm{sys}| + s_{\mathrm{thread}} + \sum_i k_i \cdot s_i)$O(∣sys∣+sthread​+∑i​ki​⋅si​), where $k_i$ki​ and $s_i$si​ are retrieval caps and item sizes. Consequently, per-decision token cost does not grow with the number of decisions, unlike a transcript interface whose worst-case growth is $\Omega(d \cdot \bar{s})$Ω(d⋅sˉ).

The five layers are separated by mutability and role:

• $L_1$L1​ operator prompts: immutable role and protocol templates for each state type.
• $L_2$L2​ state-typed prompts: immutable schemas for combat, deckbuilding, map, event, and intermission decisions, including legal action formats.
• $L_3$L3​ game knowledge: enumerable rule data (cards, relics, events, enemies, intents) refreshed by game patches.
• $L_4$L4​ episodic memory: postrun summaries indexed by character, ascension, act, and enemy class, providing case-based recall.
• $L_5$L5​ skill library: triggered strategic guides distilled from logs, each with an explicit trigger, a prose policy, and a four-level write gate. Retrieval is triggered by condition matching rather than raw log similarity.

Only $L_4$L4​ and $L_5$L5​ can be written to after a run, constraining what information persists across decisions.

To further bound LLM usage, a dispatcher routes decisions to four model tiers: fast (trivial combat plans), strategic (ordinary decisions), analysis (postrun memory extraction), and evolution (skill distillation). Combat, the most frequent decision type, uses a local conversation object limited to at most three messages per round (combat_start, ok, and the latest state); earlier rounds are summarized into the typed state rather than appended. Together with fast-tier routing and mechanical handlers, this yields a median of 67 strategic LLM calls per run instead of one per in-game action.

Skill discovery populates $L_5$L5​ in two modes. In mistake-driven self-evolve, combat losses relative to per-enemy baselines trigger a pre-write A/B resample check (strict 2/3 agreement with zero harmful decisions), followed by a four-level write gate that applies cosine and Jaccard similarity filters, an LLM judge, and optional pruning. Most candidates are rejected or merged rather than added as new skills. In Mode B, a stub-template-filled authoring process fills five character-parametric templates for combat, boss, deckbuilding, map, and intermission decisions, with namespace isolation, a library lock, and warn-only validators. Mode B reaches parity with a human-authored seed library, allowing evaluation to separate the existence of a skill layer from the specific prose source.

Because context reaches the model through named slots, the architecture enables clean ablation: prompt strictness, rule retrieval, episodes, and strategic skills can be toggled independently. This bounded typed contract turns memory interface design into an evaluation object, decoupling it from an ever-growing transcript and allowing performance gains to be traced to specific layers.

Experiment

The evaluation setup decomposes performance at a fixed difficulty into five conditions, probes cross-backbone transfer of a frozen skill/episodic stack, and uses an auto-mode ladder to measure climb ceiling. The L₅ skill layer drives the largest observed gain, while episodic L₄ saturates at the single difficulty but enables longer-horizon climbing when postrun writes are active. Transfer of the frozen stack is backbone-sensitive, and a direct comparison with open-source accumulating-context agents shows that the bounded typed-retrieval contract wins more runs while consuming orders of magnitude fewer tokens, establishing the memory interface as an isolable and measurable design object.

A bounded typed-memory contract keeps the per-decision context flat (median ~5k tokens), whereas transcript-based replay agents see prompts grow from ~9k to 500k tokens in a single run. When evaluated on the same backbone, character, and ascension, the two publicly available transcript-accumulating agents require one to two orders of magnitude more tokens per unit of game progress and fall below even a no-store bounded baseline. This demonstrates the memory interface as an evaluable design object, with typed slots enabling layer-specific attribution and decoupling memory from accumulating state. Transcript-based agents’ per-call prompts swell from ~9k to 500k tokens within a run, while the bounded typed contract holds the strategic message flat at a ~5k median. The transcript-accumulating agents consume one to two orders of magnitude more tokens per point of game progress than even a no-store bounded baseline, falling far below it.

Skill-scaffolded agents consistently outperformed unscaffolded ones in this fixed ablation, with all three scaffolded configurations winning 6 of 10 games compared to 3 or 4 wins without scaffolding. The L5 skill layer accounted for the largest observed within-harness improvement, adding two wins over the prompt-only baseline. The bounded contract kept the strategic prompt size stable at around 5k tokens, contrasting sharply with transcript-accumulating approaches that grew to hundreds of thousands of tokens per call. Unscaffolded and prompt-only conditions achieved at most 4 wins out of 10 games, while every skill-scaffolded condition reached 6 wins. Adding the L5 skill layer produced a +2/10 win increase over the prompt-only baseline under identical prompt strictness. The bounded contract maintained a flat strategic message size of about 5k tokens, whereas external accumulating-context agents consumed one to two orders of magnitude more tokens per point of progress. The gap between unscaffolded and skill-scaffolded cells is directional only; Fisher exact tests with the available sample size do not yield statistical significance.

Cross-backbone transfer of a frozen Gemini-trained skill stack produces divergent results: Qwen's mean score improves substantially, while DeepSeek's declines, with neither backbone winning any matches. The transferred frozen skills remain competitive with hand-authored ones, demonstrating that skill stacks are backbone-sensitive and transfer is an empirical property rather than a guarantee. The same Gemini-trained L4+L5 stack lifts Qwen's mean score by a large relative margin but leaves its win count unchanged at zero. Applying the stack to DeepSeek reduces its mean score, showing that frozen skill transfer can hurt performance on certain backbones. Frozen template-filled skills achieve the same win estimate as hand-authored skills, confirming competitiveness inside the shared interface. The L4 layer does not alter the A0 win point estimate, indicating saturation there, though it continues to support longer-horizon ladder tasks.

The experiments compare bounded typed-memory contracts against transcript-accumulating agents, ablate skill scaffolding, and test frozen skill stack transfer across backbones. Transcript-based agents suffer extreme prompt inflation and require orders of magnitude more tokens per game progress, whereas the bounded contract keeps strategic prompts stable at about 5k tokens, demonstrating the memory interface as a critical design lever. Skill-scaffolded agents consistently outperform unscaffolded baselines, with the L5 layer driving the largest win-rate improvement, though differences are not statistically significant, and cross-backbone transfer shows that frozen skill stacks remain competitive but produce backbone-dependent outcomes, sometimes boosting mean score without gaining wins and sometimes degrading performance.