One-sentence Summary

University of Science and Technology of China et al. propose EvoPolicyGym, a benchmark of compact interactive RL environments that evaluates iterative policy improvement, where GPT-5.5 achieves the strongest aggregate rank score and top-two performance on all 16 environments, and trajectory-level diagnostics reveal that strong autonomous policy evolution hinges on discovering task-appropriate mechanisms and converting feedback into parametric tuning under a fixed interaction budget.

Key Contributions

• The paper introduces Autonomous Policy Evolution, a controlled evaluation setting where a harness-model agent iteratively edits an executable policy under a fixed interaction budget, learns from visible train feedback, and is evaluated on hidden validation-selected heldout performance.
• EvoPolicyGym instantiates this setting with a benchmark built from compact interactive RL environments; on the Core16 suite, GPT-5.5 achieves the strongest aggregate rank score and top-two performance across all 16 environments.
• Trajectory-level diagnostics from the benchmark show that agents allocate budget and convert feedback into parametric code changes, revealing that strong autonomous policy evolution depends on discovering task-appropriate mechanisms and refining policies under bounded feedback rather than isolated task wins.

Introduction

Modern coding agents and self-improving systems can iteratively refine outputs through tool use and reflection, yet evaluating this iterative capability is challenging because final success scores often hide trajectory-level problems such as overfitting to visible feedback, blind retries, and inefficient exploration. Existing benchmarks focus on one-shot patches, discrete unit-test outcomes under evolving specifications, episodic task completion, or domain-specific system construction, and they do not isolate the core ability to convert limited, sparse feedback into a generalizable reusable policy. The authors address this gap by formalizing Autonomous Policy Evolution and introducing EvoPolicyGym, a controlled benchmark in which agents repeatedly edit an executable policy under a fixed interaction budget, with only sandboxed train feedback visible and held-out generalization hidden. This design makes policy evolution itself the evaluated object, supporting both leaderboard comparisons and trajectory-level diagnostics that reveal how agents allocate budget, interpret feedback, and translate it into robust code improvements.

Method

The authors introduce EvoPolicyGym, a framework that frames autonomous policy evolution as an agent-driven optimization loop for executable decision policies. In this setup, a coding agent maintains a persistent policy workspace, submits candidate revisions for visible train episodes, receives server-generated rollout summaries and trajectories, and revises the policy under a fixed episode budget. The primary evaluation unit is a complete budget-constrained run, scored by the held-out return of its best validation checkpoint.

Before specifying the benchmark protocol, the framework defines three core objects: an environment, a policy system, and an episode. An environment exposes observations, accepts actions, and returns rewards through a standard reset and step interface. A policy acts as the decision rule choosing actions using observations and an internal state $h_t$ht​. A deterministic policy maps $(o_t, h_t)$(ot​,ht​) to an action and updated state $(a_t, h_{t+1}) = \mu(o_t, h_t)$(at​,ht+1​)=μ(ot​,ht​), while a stochastic policy samples from $\pi(\cdot \mid o_t, h_t)$π(⋅∣ot​,ht​). In EvoPolicyGym, this stateful mapping is implemented by an executable Python artifact written by the coding agent. An episode represents one complete policy-environment interaction, beginning with a reset and ending at termination, with the cumulative reward serving as the return.

The Autonomous Policy Evolution Protocol evaluates the coding agent's ability to optimize this executable policy system from environment feedback. A run begins with an environment, an initial workspace, and a fixed episode budget. At observed revision $i$i, the workspace state is $W_i$Wi​, the server-written feedback is $F_i$Fi​, and the remaining budget is $B_i$Bi​. The executable policy system is induced by the workspace as $P_i = \Phi(W_i)$Pi​=Φ(Wi​). The coding agent $\pi_\theta$πθ​, comprising a language model and its tool-using harness, observes $(W_i, F_i, B_i)$(Wi​,Fi​,Bi​) along with its accumulated history $\mathcal{H}_i$Hi​ to write patches. The run induces a sequence of observed-state transitions: $\pi_\theta(W_i, F_i, B_i, \mathcal{H}_i) \rightarrow (u_i, s_i, \mathcal{H}_{i+1}), \quad s_i \in \{\bot\} \cup \mathcal{C}(B_i)$πθ​(Wi​,Fi​,Bi​,Hi​)→(ui​,si​,Hi+1​),si​∈{⊥}∪C(Bi​) $W_{i+1} = \text{apply}(W_i, u_i), \qquad P_{i+1} = \Phi(W_{i+1})$Wi+1​=apply(Wi​,ui​),Pi+1​=Φ(Wi+1​) $(\Delta F_i, c_i) = S(B_i, P_{i+1}, s_i), \qquad B_{i+1} = B_i - c_i, \qquad F_{i+1} = F_i \cup \Delta F_i$(ΔFi​,ci​)=S(Bi​,Pi+1​,si​),Bi+1​=Bi​−ci​,Fi+1​=Fi​∪ΔFi​ Here, $u_i$ui​ is a workspace patch and $s_i$si​ is a submit command. The server operator $S$S returns new feedback $\Delta F_i$ΔFi​ and the charged episode count $c_i$ci​. After the run concludes, the server automatically performs hidden validation selection and held-out evaluation.

To understand how agents explore the policy implementation space, the authors distinguish between structural synthesis and parametric tuning. Structural synthesis creates task machinery such as perception, memory, or planning, while parametric tuning adjusts gains and thresholds within an existing controller. Environments are split into synthesis-dominant groups requiring richer machinery and tuning-dominant groups where simple controllers suffice. To compare performance across these groups, held-out scores are normalized on a per-environment random-to-best scale: $\text{norm}_{m,e} = \text{clip}\left(\frac{R_{m,e}^{\text{heldout}} - R_e^{\text{random}}}{R_e^{\text{best}} - R_e^{\text{random}}}\right)$normm,e​=clip(Rebest​−Rerandom​Rm,eheldout​−Rerandom​​)

The authors also compute a post-hoc diagnostic to track the evolution of the best-so-far hidden-validation score over the consumed episode budget. This reveals when useful candidate policies appear and how efficiently budget is converted into better policies. Vertical jumps in these trajectories correspond to the discovery of higher-quality candidate policies, while flat segments indicate periods without improvement.

Finally, the environment layer is separated from the agent protocol. Any Gymnasium-compatible episodic environment can be wrapped by an adapter that implements the necessary interfaces, converts observations and actions into compact schemas, and loads reproducible initialization splits. This design ensures that interaction, visibility, budget, and artifact semantics remain fixed as environment coverage expands.

Experiment

In the Core16 suite with a 128-episode budget and hidden validation, the experiments evaluate model-and-harness systems through autonomous policy code editing. GPT-5.5 achieves the most reliable coverage, ranking top two on all 16 tasks, while Claude Opus 4.7 leads on MiniGrid, and the performance gap widens substantially on synthesis-dominant tasks where successful agents discover task-specific control mechanisms via structural edits rather than mere parametric tuning. Post-hoc score trajectories and edit case studies reveal that high-performing runs tend to make early structural improvements and leverage rollout feedback to guide policy revisions.

Across the eight Gym/Box2D and MuJoCo environments shown, GPT-5.5 achieves the highest held-out return on five tasks and second place on the remaining three, illustrating the broad coverage that drives its top aggregate Core16 score. Claude Opus 4.7 wins ContinuousCar and Ant while placing second on every other task, demonstrating strong but slightly less widespread reliability. The random policy obtains far lower scores on all environments, confirming that the learned policies consistently outperform chance. GPT-5.5 posts the top return on Acrobot, BipedalWalker, CarRacing, Reacher, and Pusher, with a BipedalWalker score of 248.9 compared to -15.8 for the next best agent, Claude Opus 4.7. Claude Opus 4.7 takes first on ContinuousCar (98.8 vs. 95.2) and Ant (990.1 vs. 989.6) and is the runner-up on the other six tasks, while MiniMax-M3 wins only HalfCheetah but has much lower returns elsewhere.

GPT-5.5 and Claude Opus 4.7 consistently outperform other models on MiniGrid and Robotics/Driving tasks, with MiniMax-M3 and DeepSeek-V4-Pro often failing fundamental navigation goals. Qualitative diagnostics show that high-return runs on synthesis-dominant tasks are characterized by sustained structural code edits informed by visual rollout feedback, while tuning-dominant tasks require a viable policy skeleton before parameter optimization becomes useful. GPT-5.5 and Claude Opus 4.7 achieved near-perfect success on MiniGrid’s DoorKey, KeyCorridor, and ObstacleMaze, whereas MiniMax-M3 and DeepSeek-V4-Pro scored zero on those tasks. In the FourRooms task, Claude Opus 4.7 reached 0.701 and GPT-5.5 0.664, while MiniMax-M3 managed 0.403 and DeepSeek-V4-Pro only 0.079. On driving tasks, GPT-5.5 obtained the highest returns on Parking, FetchPush, and FetchPick, slightly ahead of Claude Opus 4.7, while DeepSeek-V4-Pro led only on Roundabout. GPT-5.5 was the sole agent to achieve positive performance on the tuning-dominant BipedalWalker environment, reaching a timeline-best score of 271, whereas all other models remained at negative scores. Agents that succeed in synthesis-dominant tasks like CarRacing tend to revise policy structures in response to visual failure evidence, in contrast to weaker runs that repeatedly modify mechanisms without escaping the wrong abstraction.

GPT-5.5 leads the aggregate reliability ranking with top-two placement in every environment and first-place finishes in nine, dominating three of four task families. Claude Opus 4.7 ranks second overall, driven by strong MiniGrid performance and broad top-two coverage, while other models achieve isolated wins but fall short on overall consistency. The leaderboard rewards sustained near-top performance across all heterogeneous tasks rather than a few local successes. GPT-5.5 places first or second in all 16 environments, winning nine of them, and leads the Gym/Box2D, MuJoCo, and Robotics/Driving families. Claude Opus 4.7 ranks second with five first places and 12 top-two finishes, and achieves the best MiniGrid family score. MiniMax-M3 wins HalfCheetah but its lower Gym/Box2D and MiniGrid scores drop its overall rank to third, illustrating how weak families hurt aggregate reliability. DeepSeek-V4-Pro secures a win on Roundabout yet records only one top-two placement overall, resulting in the lowest Core16 score among the tested agents. The uniform random policy's score comes almost entirely from shared rank credit on MiniGrid where all agents tied at zero return, confirming the metric penalizes weak coverage.

Policies selected for synthesis-dominant tasks exhibit substantially richer structural complexity than those for tuning-dominant tasks, with higher counts of functions, branches, loops, AST depth, and persistent state. The strongest agents produce the largest source bundles on synthesis tasks, but structural complexity alone does not guarantee strong performance. Synthesis-task policies from the highest-scoring agents contain over five times as many functions and four times as many loops as those from the lowest-scoring agent. Branches are the most abundant control-flow feature in synthesis policies, reaching 77.0 on average for one top agent, while tuning policies never exceed 9 branches. Tuning-dominant policies are consistently compact, with all agents averaging fewer than 9 functions, under 9 branches, and virtually no loops.

On synthesis-dominant tasks, Claude Opus 4.7 achieves the highest structural edit success rate (48%), while MiniMax-M3 struggles at 10% and nearly all parametric edits are absent. On tuning-dominant tasks, GPT-5.5 benefits more from parametric edits (61% success) than from synthesis edits (38%), whereas the other models' parametric success rates are similar to or lower than their synthesis rates. Claude Opus 4.7 records a 48% hit rate on synthesis edits for synthesis-dominant tasks, the highest among all agents, and makes no parametric edits on those tasks. On tuning-dominant tasks, GPT-5.5's parametric edits succeed 61% of the time, markedly outperforming its 38% synthesis edit success, while Claude Opus 4.7 and MiniMax-M3 see parametric success rates of 21% and 25% respectively. MiniMax-M3 has the lowest synthesis edit success on synthesis tasks (10%) and makes no parametric edits there, indicating a difficulty in discovering successful structural changes.

GPT-5.5 and Claude Opus 4.7 lead across diverse continuous control and navigation benchmarks, with GPT-5.5 achieving first or second place in all 16 environments and winning nine by excelling in both synthesis-dominant tasks that require rich structural code changes informed by visual rollout feedback and in the tuning-dominant BipedalWalker where only it leveraged successful parametric edits. Claude Opus 4.7 provides the next most reliable coverage, dominating MiniGrid and recording the highest structural edit success on synthesis tasks, while weaker agents like MiniMax-M3 and DeepSeek-V4-Pro often fail fundamental navigation goals and struggle to make effective structural modifications. Top-performing policies on synthesis tasks exhibit markedly greater complexity in functions, branches, and loops, but structural complexity alone does not guarantee success; instead, the ability to iteratively revise policy architectures in response to failure evidence distinguishes strong agents.