One-sentence Summary

Researchers from the University of Science and Technology of China, UC Berkeley, and collaborating institutions show that agent harness evolution can improve benchmark scores at the cost of coverage, reliability, and transferability, and they introduce EvoAudit, which revises the select, analyze, mutate, and gate stages into preservation-aware diagnostics to achieve the best performance.

Key Contributions

• Across four harness-evolution methods and general/coding tasks, repeated evaluations, controlled agent expansion, component interventions, and admission replays show that all sixteen evolved agents gain previously uncovered tasks while losing previously covered ones; in one general-task case, reliability rises by 5.2 points despite losing ten tasks, and reliability gains often increase deployment cost.
• Transfer is selective: coverage usually falls on the general-task target across both harness types and paradigms but rises on the coding target with the outer-layer-only harness, and stricter admission can discard useful alternatives, motivating preservation-aware evaluation, tested decision rules, and explicit applicability conditions.
• EvoAudit converts these diagnostics into revisions of the select, analyze, mutate, and gate operators and achieves the best performance; AGE improves reliability and coverage over its reference method in all four settings.

Introduction

Recursive self-improvement for language-model agents is often pursued through harness evolution, which keeps model weights fixed while persistently revising prompts, skills, tool interfaces, orchestration, or source code. Recent frameworks close the loop between execution, diagnosis, mutation, and evaluation and report benchmark gains, but aggregate scores can hide regressions, deployment cost, and weak generalization beyond the tasks used during evolution. The authors study four evolution methods across general and coding benchmarks and show that harness evolution tends to redistribute task success rather than uniformly improve it, with useful gains depending on productive starting agents, compatible changes, and admission decisions that preserve strong variants. From these findings they propose Audit-Guided Evolution (AGE), which revises diagnosis, mutation, selection, and admission to improve reliability and coverage over its reference method.

Method

The authors define an agent $A = (M, H)$A=(M,H) consisting of a frozen model $M$M and a harness $H$H. An evolution algorithm $E$E starts from a seed agent $A_0 = (M, H_0)$A0​=(M,H0​) and runs for $G$G generations. At each generation $g$g, a Parent is selected for modification, resulting in a set of proposed harnesses $\mathcal{C}_g$Cg​. Each $H \in \mathcal{C}_g$H∈Cg​ defines a Candidate agent $(M, H)$(M,H), and a rule $Q$Q selects the final Champion harness $H_\star$H⋆​ from the archived versions:

where $H_{<g}$H<g​ is the history of harness versions, $\mathcal{Z}_{<g}$Z<g​ is their evaluation evidence, $\mathcal{D}_{\mathrm{evo}}$Devo​ is the task set, and $B_{\mathrm{evo}}$Bevo​ is the search budget.

In the common evolution process, each recipe specifies how seven operators are implemented: select chooses a Parent, rollout collects execution evidence, analyze diagnoses it, and mutate proposes a Candidate. The validate operator optionally checks the proposal, gate decides admission, and record archives outcomes.

As shown in the figure below:

The authors propose EvoAudit, an agent-assisted workflow that runs diagnostics on archived artifacts and returns operator revisions with evidence and validation records. Installing its four revisions in A-Evolve gives Audit-Guided Evolution (AGE). The EvoAudit workflow consists of several stages: Audit-Prepare freezes inputs and contracts, Audit-Evaluate runs diagnostic experiments, Audit-Analyze grounds findings and hypotheses into an evidence report, Audit-Mutate revises operator implementations, Audit-Validate tests interfaces and regressions, and Audit-Gate admits revisions or reports failure.

As shown in the figure below:

In AGE, the operators are revised as follows:

Select: Regression-Aware Tie-Breaking. AGE retains the highest-score objective and revises only exact ties. Let $\mathcal{P}$P be the eligible Parents with finite recorded Validation scores $s(C)$s(C), and let $\mathcal{T} = \arg\max_{C \in \mathcal{P}} s(C)$T=argmaxC∈P​s(C). When comparable task-level evidence is available, selection minimizes Seed relative regressions within $\mathcal{T}$T:

where $\mathcal{D}_{\mathrm{val}}$Dval​ is the Validation task set and $\bar{r}_t$rˉt​ is the mean recorded reward on task $t$t.

Analyze: Behavior-Grounded Diagnosis. The analyzer collects the current rollout and, when available, one preceding rollout along the selected lineage, retaining the last 30 records. Each judge input contains chronological tool calls, successful and failed tool results, and the completion signal. The judge returns score, category, outcome, and failure_reason. Scores 0 to 3 indicate visible breakdown, 4 to 6 indicate partial or ambiguous progress, and 7 to 10 indicate visible completion.

Mutate: Mechanism-Linked, Dependency-Closed Changes. The mutation agent receives the selected behavior evidence, Parent workspace, prompt and skill inventory, and editing contract. Before choosing files, its instructions require the observed behavior, inferred mechanism, replacement behavior, activation conditions, and expected trade-off. The agent must inspect affected interfaces, make one coherent intervention, preserve useful behavior, and check for conflicting guidance.

Gate: Score-First, Cost-Aware Admission. AGE retains strict Parent-relative score improvement and adds a cost tie-break. For a Candidate $C$C and its matched Parent $P$P, with finite Validation scores, admission is:

where $c$c is the recorded USD cost of canonical Validation evaluation. The comparability predicate $Q$Q requires matching task-set hashes, finite strictly positive costs, and a direct Parent evaluation.

Experiment

The study evaluates four agent evolution recipes, two implementation-centric and two reflection-centric, using GPT-5.4 under two harnesses: source-editable MiniSWE and outer-layer-only Codex, across general banking and coding benchmarks. Key findings are that evolution redistributes task success rather than producing monotonic coverage gains, reliability and coverage often diverge, and generalization to held-out or external tasks varies by harness and paradigm. Mechanism analyses show that current scores do not reliably identify the best parent, specific trace-grounded changes outperform generic guidance, mutation benefits from preserving dependencies, and admission constraints have harness-dependent effects; expert review supports the evidence-guided revisions, and the combined AGE approach improves reliability and coverage over A-Evolve while losing fewer Seed-covered tasks.

In the reported MiniSWE harness results, AHE improved both reliability and empirical coverage over the Seed for τ3-Banking, with the largest gains among the listed variants, and also improved both metrics on Codex. Gains were not uniform across methods: A-Evolve slightly decreased average reliability on τ3-Banking while still improving coverage, and HyperAgents improved reliability more than coverage. Broader lineage counts likewise indicate that reliability and coverage gains are distinct outcomes, with some lineages moving the metrics in opposite directions. AHE showed the largest reliability and coverage improvements among the reported MiniSWE variants on τ3-Banking and also improved both metrics on Codex. A-Evolve was the only reported MiniSWE variant to reduce average reliability on τ3-Banking, despite a small coverage increase. Across all lineages, reliability and coverage gains were not always aligned: four lineages improved one metric while reducing the other, and another improved only reliability.

Task-level success changes are grouped into behavior categories that show distinct patterns of increase and decrease. State and dependency management appears most often with increased success, while exact grounding and completion handoff tend to appear more with decreased success. These mixed shifts indicate that coverage does not expand uniformly even when overall metrics improve. State and dependency management is the most common category linked to increased repeated success, with only a small presence among decreases. Exact grounding and completion and handoff show the reverse tendency, with decreases outweighing increases. The selected comparisons show simultaneous gains and losses across tasks, so aggregate improvements can mask task-level coverage churn.

Seed-relative gains vary across methods, splits, and transfer targets. Transfer average reliability is positive for all listed MiniSWE methods, while transfer coverage is positive only for GEPA and negative for the other MiniSWE methods. The listed Codex AHE result shows large training gains but no validation or test coverage gains and negative transfer on both reported outcomes. On the listed MiniSWE general-task rows, transfer coverage falls for three of four methods while transfer average reliability improves for all four. GEPA is the only listed MiniSWE method with positive transfer coverage, whereas the other MiniSWE methods record negative transfer coverage despite positive transfer reliability.

The experiment reports best parent selection rates for an Original rule and a Random baseline across general and coding benchmarks under two harnesses. Original selection falls below the random baseline on general tasks but exceeds it on coding tasks, with the same pattern across both harnesses. This indicates that current scores align with improvement potential more strongly in coding domains than in general domains. On general tasks, Original selects the best parent less often than random under both harnesses. On coding tasks, Original selects the best parent more often than random, exceeding 55% under both harnesses.

The evidence compares seed performance, isolated component additions, and complete champions across benchmarks, harnesses, and evaluation metrics. Complete champions generally achieve the highest reliability and pass rates, especially in coding-task settings, while individual components produce smaller or less consistent gains. A few general-task cases favor a subset over the full champion. Fifteen of sixteen specific component changes outperform their matching generic reminders in reliability, with at least one component improving in every evaluated case. Six of eight complete champions outperform either isolated component, covering all coding-task cases; in one coding case only the full champion improves over the seed.

The experiments compare MiniSWE variants on reliability and coverage, examine task-level behavior shifts under transfer, assess parent selection against random, and ablate isolated components versus complete champions. AHE yields the largest reliability and coverage gains, while A-Evolve reduces reliability; aggregate metric improvements often mask task-level churn, with state and dependency management improving but exact grounding and completion handoff declining. Transfer generally improves reliability while coverage declines for most methods, original parent selection outperforms random only on coding tasks, and complete champions usually provide the best pass rates, especially in coding domains.