One-sentence Summary

Researchers from University of Washington, Stanford University, and other institutions propose SPADE, a self-play RL framework in which a single LLM plays an Environment Designer that writes executable Gym-style reset()/step() environments and a Reasoning Agent that learns to act in them, using hinted/unhinted reward regret to target feasible edge-of-capability tasks; SPADE improves fixed baselines by $+5.3$+5.3 across eight benchmarks, $+5.7$+5.7 on BFCL v4, and $+13.9$+13.9 on ACEBench-Agent.

Key Contributions

• The paper introduces SPADE, a self-play RL framework in which one LLM acts as an Environment Designer that writes executable Gym-style reset/step environments and a Reasoning Agent learns to act in them, unifying reasoning and multi-step tool use through a single stateful interface.
• A hint-based regret reward trains the Environment Designer to target environments at the Reasoning Agent's learning frontier by using the reward gap between acting with and without privileged hints, keeping environments feasible but challenging and grounded in minimax regret theory.
• Grounding environment design in a pretraining corpus and accumulated environment memory is critical; at 30B parameters on Qwen3 models, SPADE outperforms the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts tool use by +5.7 on BFCL v4 multi-turn and +13.9 on ACEBench-Agent.

Introduction

Agentic language models increasingly improve through interaction with environments that provide verifiable rewards, making the supply of adaptive training environments a central bottleneck. Prior work is limited by fixed environment pools, slow human curation, frozen synthetic generators, or self-play that produces sparse-reward tasks rather than full multi-turn environments. The authors introduce SPADE, a framework where one LLM plays two roles: an Environment Designer that generates complete Gym-style Markov decision processes as executable Python code, and a Reasoning Agent that learns from them. The Environment Designer is itself trained with reinforcement learning using a hint-based regret signal that targets solvable environments near the Reasoning Agent’s current capability frontier. This creates co-evolution between environment generation and agent capability, unifying single-turn and multi-turn tasks while keeping the training curriculum adaptive as the agent improves.

Dataset

The authors use synthetic tool-use environments as the data in this domain. Key details:

• Source: environments are generated by an Environment Designer grounded in a 15k-document code corpus. No benchmark tasks or data are shown to the Environment Designer.
• Composition: each generated environment includes simulated tools in OpenAI function-calling format, a backend state, and three to five natural-language user instructions that arrive one at a time. Each instruction has a check on the resulting state.
• Scale: generation uses k=8 because tool environments are more expensive to generate.
• Processing/validation: validation adds two checks beyond the game checks: a deterministic reset gate that exercises every success criterion under several seeds, and an LLM check that every success criterion can be met by some tool.
• Usage: the Reasoning Agent solves each environment by calling tools over multiple turns and is rewarded only when it completes every user instruction. The privileged hint is a step-by-step plan, so the agent must still find exact arguments through tool calls.
• Evaluation: the authors evaluate with BFCL v4 multi-turn, τ²-bench, and ACEBench-Agent, specifically the Agen category of ACEBench-en, with results reported per domain.

Method

SPADE is an end-to-end self-play framework where a single LLM, $\pi$π, alternates between generating executable training environments and learning to solve them. As shown in the framework diagram, the system comprises two main roles: the Environment Designer (ED) and the Reasoning Agent (RA). The ED conditions on an environment memory $M$M and a pretraining corpus $C$C to emit an executable environment $e$e and a privileged hint $h$h. The RA then plays the environment $e$e with and without $h$h. The return gap between these two rollouts constitutes the ED's hint-based regret reward $r_D(e)$rD​(e), while task correctness serves as the RA's reward. Both rewards update the shared policy $\pi$π via Group Relative Policy Optimization (GRPO).

The authors formulate adaptive environment self-play as a game where a single model $\pi_\theta$πθ​ acts in two roles via role-specific system prompts. In the Environment Designer role ($\pi_D$πD​), the model produces an executable environment $e \in \mathcal{E}$e∈E as a Python program implementing a Gym-style reset/step API. The transition function $T(s'|s,a)$T(s′∣s,a) and reward function $R(s,a)$R(s,a) are encoded in the step implementation. The ED also emits a privileged hint $h$h, which is task-relevant information such as a partial solution sketch or a key structural observation that makes the environment easier to solve. In the Reasoning Agent role ($\pi_A$πA​), the model interacts with $e$e via sequential actions. This code-as-environment representation unifies single-turn and multi-turn agentic settings under a single interface. Since both roles share parameters $\theta$θ, updates for either role affect the same policy.

The Environment Designer's reward for producing environment $e$e is defined by the hint-based regret:

where $\bar{r}_A(e|h)$rˉA​(e∣h) is the RA's average return over $G$G fresh rollouts sampled with the privileged hint $h$h in context, and $\bar{r}_A(e)$rˉA​(e) is the average over $G$G rollouts without it. High regret indicates an environment at the learning frontier (solvable with hints but not without), while low regret indicates either mastery or an intractable environment. This implements a minimax regret objective without a separate antagonist. The paired trajectories expose the behavioral source of the return gap, as the hint narrows the agent's search space or provides key structural observations directly.

To prevent the generator from mode-collapsing, the authors treat an external corpus as a mechanism to lengthen the invisible leash of novelty. Every round, the ED conditions on freshly sampled human corpus, which seeds an executable MDP. This corpus grounding sustains environment diversity, as demonstrated by the t-SNE visualization and Vendi scores, which show a significant drop in diversity when corpus grounding is removed. Additionally, a cross-episode memory buffer of previously generated environments, annotated with regret scores and skill tags, prevents the ED from re-posing mastered tasks. This memory ensures each round starts from what the RA currently finds hard, holding generated difficulty at the RA's frontier as that frontier moves.

The policy $\pi_\theta$πθ​ is trained via reinforcement learning with verifiable rewards (RLVR) using Group Relative Policy Optimization (GRPO). For each input prompt $x$x, GRPO samples a group of $G$G responses $\{y_1, \dots, y_G\} \sim \pi_\theta(\cdot|x)${y1​,…,yG​}∼πθ​(⋅∣x) and computes group-normalized advantages:

The policy is updated via clipped policy gradient with KL regularization:

To stabilize joint two-role training, advantages are normalized independently for each role. The ED update is delayed by $k$k rollouts to allow the difficulty anchor to score each environment's RA win rate over the full training window. The deployed ED reward blends floored regret with a flat-top difficulty anchor. Under this training recipe, the average gain over each backbone's base grows with model size, while fixed-environment GRPO stays near baseline, demonstrating the effectiveness of the adaptive curriculum in keeping the training signal challenging.

Experiment

SPADE is evaluated on game and tool-use environment design across three Qwen3 backbones, compared against fixed-environment and frozen-designer baselines. The game experiments show that adaptively generated environments improve held-out procedural reasoning, science, and code, while the tool-use experiments show gains on multi-step agentic benchmarks. Qualitative and ablation results indicate that corpus grounding sustains diversity, co-adaptation between the Environment Designer and Reasoning Agent is necessary for improvement, and hint-based regret rewards outperform an EMA-based alternative. Scaling analyses further show that benefits grow with model size and curriculum diversity.

Training a reasoning agent on diverse synthetic game environments improves held-out reasoning and code benchmarks at every backbone scale. The adaptive environment designer outperforms fixed-environment baselines, with gains over the base model growing as model size increases and procedural reasoning improving across all cognitive-skill categories while competition math is preserved. Curriculum diversity contributes substantially, since a reduced two-skill curriculum captures only part of the gains. SPADE improves average held-out performance at every backbone scale, and the gain over the base model grows from smaller models to the largest tested backbone. Fixed-environment baselines show much smaller average gains, and static environments provide a fixed signal that larger models fit quickly without continued benefit. Procedural reasoning improves across every cognitive-skill category, while competition math performance is preserved. The full six-skill curriculum produces stronger procedural, science, and code gains than a two-skill curriculum, indicating gains scale with curriculum diversity. At the largest tested backbone, SPADE reaches a suite average of 58.3, outperforming the strongest fixed-environment baseline by 5.3 points.

Synthetic tool-use environments generated by SPADE match dedicated data-synthesis systems and perform best on benchmarks requiring stateful, multi-step interaction. Gains are largest on ACEBench-Agent and BFCL v4 multi-turn, where task structure closely mirrors the generated environments, and SPADE at 30B-A3B surpasses reference synthetic-environment agents on both. The pattern indicates that structural training signal transfers where domain-specific data collection alone does not reach. SPADE at 30B-A3B leads the compared dedicated data-synthesis systems on BFCL v4 multi-turn and ACEBench-Agent. Tool-use gains track task structure, with the largest improvement on ACEBench-Agent, followed by BFCL v4 multi-turn and then τ2-bench. The generated environments improve every evaluated backbone, with the strongest relative gains on stateful, multi-call agentic tasks.

In the games setting, the full adaptive environment designer configuration achieves the highest suite-average performance among all variants tested. Removing corpus grounding or environment memory, or freezing the designer, lowers the best-checkpoint score, and even a frozen frontier-model designer does not close the gap. The comparisons show that adaptive training of the environment designer is central to the observed gains. The full adaptive configuration outperforms every partial and frozen-designer control on the unweighted suite average. Ablations that remove corpus grounding or environment memory, or that freeze the Environment Designer, all reduce best-checkpoint performance.

The experiments evaluate training reasoning agents in diverse synthetic game environments generated by an adaptive environment designer, comparing against fixed-environment and dedicated data-synthesis baselines. The adaptive approach improves held-out reasoning and code benchmarks at every backbone scale, with gains growing as model size increases and with greater curriculum diversity, while tool-use performance improves most on stateful, multi-step tasks. Ablations show that adaptive training of the environment designer is central, since removing corpus grounding, environment memory, or freezing the designer reduces best-checkpoint performance.