One-sentence Summary

Researchers from the Beijing Academy of Artificial Intelligence, the Institute of Automation at the Chinese Academy of Sciences, and Renmin University of China introduce AgentSWE, a 25-task benchmark spanning agent creation, editing, and optimization that evaluates coding agents on held-out commission-style requirements, finding that none delivers faithfully and identifying overfitting to visible cases, hollow runtimes, and unsupported completion claims as recurring failure modes.

Key Contributions

• AgentSWE is introduced as a benchmark for agent software engineering spanning three lifecycle stages: Creation, Editing, and Optimization, with 25 tasks framed as commissions and evaluated on held-out cases through requirement-derived rubrics under hard pass/fail gates or, for Optimization, the source benchmark objective.
• Evaluations across models and harnesses find that no coding agent delivers faithful requirement satisfaction: the strongest average fidelity is 19.1 on Creation and 31.6 on Editing, and Optimization builders close no more than 5% of the remaining gap to the target on average.
• The accompanying empirical failure analysis identifies recurring failure modes such as overfitting to visible cases, hollow runtimes behind code that looks complete, and unsupported completion claims, charting where coding agents need to improve.

Introduction

Coding agents such as Claude Code and Codex already carry substantial engineering workloads, and non-engineer users increasingly delegate the construction of agents that wrap a model, tools, and state around a recurring job. This delegation matters because it depends on trust that the coding agent delivers the agent the user intends, but the creation, editing, and optimization lifecycle can fail in subtle ways: fitting only the visible cases, shipping runtimes that do no real work, and claiming completions that the records do not support. Existing benchmarks do not capture this gap. SWE-bench-style suites compare conventional code against a known reference, agent benchmarks evaluate finished agents rather than whether a coding agent could have built them, and meta-agent work measures benchmark-score improvements rather than faithful delivery. The authors introduce AgentSWE, a benchmark of twenty-five commission-style tasks spanning all three lifecycle stages. It gives builders only public development cases, freezes the delivered agent, and grades it on held-out cases with requirement-derived rubrics and validity gates, yielding a direct measurement of whether coding agents can faithfully create, modify, and optimize agents.

Dataset

Dataset composition and sources

• AgentSWE is a benchmark of 25 tasks: 10 Creation, 10 Editing, and 5 Optimization tasks.
• Creation tasks are sourced from open-source domain agents on GitHub, covering areas such as desktop automation, scientific PDF translation, and formal theorem proving.
• Editing tasks start from pinned repository snapshots of five general-purpose coding agents and five domain agents, including Claude Code, Aider, Open-Hands, and OpenClaw.
• Optimization tasks build on BrowseComp, τ³, TerminalBench, PinchBench, and OSWorld, keeping their native task definitions.

Key subset details

• Creation tasks: each source repository is reverse-engineered into a self-contained goal and a set of functional requirements, then extended with capability requests that broaden its purpose and raise compositional difficulty. The input consists of Markdown files stating the goal, interface contract, constraints, and resources. The initial artifact is empty.
• Editing tasks: commissioned capabilities come from design brainstorming or unresolved repository issues. The authors keep proposals that have a consequential user need, a bounded implementation surface, and behavior that execution can verify. Each proposal is written as a feature specification with an acceptance boundary and regression obligations already carried by the pinned snapshot.
• Optimization tasks: each task ships a starter agent as the initial artifact. The requirement names the objective, the files the builder may edit, and the runtime resources it may not change.

Evaluation package and statistics

• Each task separates builder-visible material, namely the requirement q, initial artifact a0, and development cases D, from reserved evaluation material.
• Creation and Editing tasks publish two development cases each and hold out six evaluation cases each.
• Optimization tasks draw a development split from the instance pool of the source benchmark and hold out a disjoint split.
• The full suite contains 150 development cases and 365 held-out cases.
• Held-out Creation and Editing cases exercise requirement clauses and capability interactions omitted from the development cases. Reserved held-out inputs, evaluator state, scoring contracts, credentials, and model routing are not exposed to the builder.

Processing and use

• Task materials are drafted by GPT-5.6 Sol from the analyzed source system, then verified and refined by the authors.
• The benchmark does not describe a conventional training split or mixture ratios. The builder agent receives q, the initial artifact, and development cases, and the delivered agent is graded on held-out cases.
• Creation and Editing tasks use a task-specific Result rubric weighted from functional requirements, with gates and ceilings for interface contract and constraint violations. Optimization tasks keep the native evaluator of their source benchmark. No image cropping strategy is described.

Method

The authors construct benchmark tasks through a unified drafting and verification workflow. Each task family fixes the requirement $q$q and the initial artifact $a_0$a0​ in its own way. In every family, GPT-5.6 Sol drafts the task materials from an analyzed source system, and the authors verify and refine each draft before it is finalized.

For creation tasks, the source systems are open-source domain agents on GitHub, covering areas such as desktop automation, scientific PDF translation, and formal theorem proving. Each repository is reverse-engineered into a self-contained goal and a set of functional requirements. These requirements are then extended with additional capabilities that broaden the original purpose and increase compositional difficulty. The resulting task is published as input Markdown files that state the goal, interface contract, constraints, and resources of $q$q. The initial artifact for creation tasks is empty.

For editing tasks, the benchmark extends a pinned repository snapshot $a_0$a0​ taken from five general-purpose coding agents and five domain agents, including Claude Code, Aider, Open-Hands, and OpenClaw. The commissioned capabilities come from design brainstorming or from unresolved issues in the repository. The authors keep proposals that have a consequential user need, a bounded implementation surface, and behavior that execution can verify. Each accepted proposal is written as a feature specification with an acceptance boundary and the regression obligations that $a_0$a0​ already carries.

For optimization tasks, the benchmark builds on BrowseComp, $\tau^3$τ3, TerminalBench, PinchBench, and OSWorld while preserving their native task definitions. Each task ships a starter agent as $a_0$a0​. The requirement names the objective, the files that the builder may edit, and the runtime resources that the builder may not change.

Experiment

The AgentSWE benchmark evaluates coding agents that build other agents across Creation, Editing, and Optimization tasks, with held-out cases and a frozen delivery protocol. The experiments show that all builders struggle: Creation agents rarely succeed, Editing agents deliver partial capabilities or nothing, and Optimization gains are inconsistent and often overfit visible development cases. Trajectory analysis identifies recurring failure modes including overfitting, hollow runtimes that fail when executed, and unsupported completion claims. Follow-up ablations find that a small number of development cases helps but more do not, extending budgets does not close the generalization gap, and results vary substantially across harness-model pairings and repeated builds.

Creation runs show rare, task-specific success, with most results low or zero and no builder reliably producing the requested agents. Editing yields only partial capabilities or no delivery at all, while optimization gains are small and inconsistent across tasks. Overall, held-out performance remains weak across all three lifecycle stages. Creation success is rare and concentrated, with most runs scoring zero or near zero and only a few runs exceeding 50. Editing either realizes partial capabilities or finishes without a candidate, and optimization gains depend strongly on task, with BrowseComp mostly improving and PinchBench mostly worsening.

Across three builder runs, held-out performance on PDF translation and PPTX generation improves when the builder is given one or two development cases, with the largest gains by two cases. With zero development cases, PDF translation scores zero on all runs; with four cases, performance does not improve further and PPTX generation declines relative to the two-case condition. Going from zero to two development cases raises mean held-out scores for both task families; PDF translation starts at zero with no development cases. Beyond two development cases, mean PDF scores plateau while PPTX scores fall from the two-case peak, and run-to-run spread widens for PDF.

Prompt priors affect tasks differently. A prescribed design prior leaves PDF translation performance nearly unchanged but substantially lowers PPTX generation performance. A self-testing prior yields only a modest PDF increase within run-to-run variation and keeps mean PPTX performance stable while reducing its spread. Strategy A has almost no effect on PDF translation but sharply reduces PPTX generation scores. Strategy B raises PDF scores only within run-to-run variation and narrows the variability of PPTX results without changing its mean.

Held-out Creation and optimization scores shift substantially across harness-model pairings, even when the model is held fixed. No single harness leads for every model or task, so the results support evaluating specific pairings rather than models alone. DeepSeek-V4-Flash performs much better under DSH than under Codex, with higher held-out Creation scores and far less negative optimization scores. For DeepSeek-V4-Pro, the harness effect is mixed: Codex produces a stronger PDF Creation score, while DSH produces a stronger GUI Creation score.

In AgentSWE-Lite, mean held-out results over repeated builds depend on both the builder and the task. DeepSeek-V4.1-Flash has the strongest Creation average, especially on PDF translation and database analytics, while GPT-5.6 Sol leads repository bug repair and DeepTutor editing. Qwen3.6 records zero held-out results on every listed task, and repeated builds of the same pairing can be highly variable. DeepSeek-V4.1-Flash leads the Creation mean, while GPT-5.6 Sol leads on repository bug repair and the DeepTutor editing task. Qwen3.6 scores zero held-out results on all listed Creation and Editing tasks, and repeated builds can vary widely within the same pairing.

The experiments evaluate agent-building across creation, editing, and optimization on held-out tasks including PDF translation, PPTX generation, BrowseComp, PinchBench, and AgentSWE-Lite. Overall held-out performance is weak and inconsistent, with no builder reliably producing requested agents, editing often yielding partial or no candidates, and optimization gains being small and task-dependent. Providing one or two development cases improves PDF translation and PPTX generation, but more cases plateau or hurt PPTX, while prompt priors and harness-model pairings affect outcomes differently by task and model; several builders, such as Qwen3.6, score zero and repeated builds are highly variable.