One-sentence Summary

Researchers at Shanghai Innovation Institute and Fudan University introduce SWE-bench Science, a repository-level benchmark of 119 tasks from 98 GitHub repositories across 20 scientific domains spanning issue-driven, expert-exploratory, and engineering-integration paradigms, where even the best agent, Claude Code with Opus-5 (max), achieves a pass@1 below 50%, and an analysis reveals four failure mechanisms alongside mixed effects of explicit scientific guidance on average repair performance and token efficiency.

Key Contributions

• The paper introduces SWE-bench Science, a repository-level benchmark with 119 tasks from 98 GitHub repositories across 20 scientific domains, organized into Issue-driven, Expert-exploratory, and Engineering-integration paradigms.
• The benchmark provides a Chain-of-Evidence Protocol that separates public and private tests and reports repair progress, exact success, and regression preservation, enabling distinction between visible-test performance and complete private-test correctness.
• Across eight coding-agent configurations, the best Pass@1 is 47.90% versus a 96.64% public score, and four recurring scientific failure mechanisms are identified. A paired ablation reveals that scientific guidance is not uniformly beneficial: well-grounded information improves average performance and token efficiency, whereas poorly aligned guidance induces anchoring and does not necessarily improve exact repair success.

Introduction

Scientific software is now a core component of the instruments that produce scientific claims, so a defective patch can corrupt not just a program output but the evidence behind a conclusion. Existing benchmarks evaluate coding agents on function synthesis, repository-level repair, or curated scientific programming problems, but they offer limited repository-level coverage across scientific domains and rarely examine why repairs fail in scientific contexts. The authors introduce SWE-bench Science, a repository-level benchmark with 119 manually verified tasks from 98 GitHub repositories spanning 20 scientific domains. They organize tasks into three paradigms (Issue-driven, Expert-exploratory, and Engineering-integration) and analyze unsuccessful repairs through four recurring failure mechanisms, including scientific abstraction deficits and incomplete system integration. A paired ablation further shows that providing scientific knowledge does not automatically improve repair success; its benefit depends on how the knowledge is organized and grounded in executable evidence.

Dataset

The authors introduce SWE-bench Science, a benchmark of 119 software engineering tasks drawn from 98 open-source GitHub repositories across 20 scientific domains. The tasks are designed to evaluate how well coding agents handle real scientific software challenges, spanning three capability dimensions.

• Dataset composition and scale

  • 52 Issue-driven tasks: repairing known historical defects in a low-distortion setting.
  • 49 Expert-exploratory tasks: autonomously investigating scientific discrepancies when the root cause is unknown.
  • 18 Engineering-integration tasks: understanding multi-module architecture and completing an end-to-end capability chain.
  • Repositories contain non-empty input code averaging 80,600 lines (range 174 to over 2 million lines). Reference patches add 118 lines on average and delete 45 lines, with large variance (1–1,035 added, 0–458 deleted).
  • The five largest domains provide 76 tasks (63.9%), while six domains contribute a single task each.

• Data sources and filtering

  • Real issues, pull requests, commit histories, and related scientific literature from candidate scientific-computing repositories.
  • Predefined screening removes overly simple fixes, tasks with unstable environments, severe solution leakage, or significant overlap with existing samples.

• Processing pipeline for raw tasks All tasks first pass through a unified four-stage construction protocol:

  1. Source sampling and screening – collect and filter repositories.
  2. Snapshot freezing and reproduction – freeze the source code before the defect, verify reproducibility in an isolated container.
  3. Public-material abstraction and information isolation – extract runnable public materials (repository, phenomenon description, reproduction script, domain background) while keeping patch locations and hidden assertions secret.
  4. Hidden oracle and counter-calibration – build hidden validation suites that check semantic correctness, reverse-checks for hard-coded or heuristic pseudo-fixes.

• Paradigm-specific task re-design After the raw task is built, it is adapted to one of the three paradigms:

  • Issue-driven: Degrade and isolate the defect, narrow the public symptom, apply anti-overfitting hidden validators that vary data scale, boundaries, and input order.
  • Expert-exploratory: Design a scientific scenario, extract an exploration point from theory or discrepancies, separate observable phenomenon from root cause, and validate through mechanism-generalization tests (changing parameter scales, topologies, etc.).
  • Engineering-integration: Analyze the pipeline chain, preserve full package structure and neighboring modules, and validate end-to-end via alternative paths, state resets, and inter-module behavioral contracts.

• Usage in the benchmark

  • Tasks are used to assess agents’ debugging, scientific reasoning, and systems-integration skills in realistic scientific codebases.
  • For 91 eligible tasks, the authors provide a paired ablation: one condition keeps the full task, the other witholds scientific auxiliary information (rationales, upstream fixes, expert guidance, paper excerpts) while retaining all repository cues. This isolates the marginal contribution of domain knowledge supplied alongside the code.

Method

The benchmark method combines a strict task schema with a unified construction protocol. In the standard condition, the agent is exposed to a repository snapshot frozen before the target change, with locked dependencies and runnable entry points but without Git history, remotes, future changelogs, build caches, or solution-linked artifacts. The agent also receives a problem statement, required scientific context $c_i^{req}$cireq​, consisting of local versioned definitions and constraints needed for a well-posed task, and public tests that support interactive debugging. Evaluator-only fields include private tests, contract labels, reference and alternative-valid patches, scientific rationale and localization support blocks, expected files, and metadata. These fields are mounted only after patch submission in a separate evaluator container and are not available through the agent workspace, environment variables, logs, or task metadata.

Tasks are generated through a unified Chain-of-Evidence Protocol with a four-stage raw-task construction process. In the first stage, source sampling and screening collects real issues, pull requests, commit records, and related literature from candidate open-source scientific-computing repositories. Screening removes overly simple fixes, unstable environment dependencies, severe solution leakage, and projects that overlap with existing samples. In the second stage, snapshot freezing and reproduction freezes the source-code snapshot $S_{bug}$Sbug​ immediately before the target defect or missing capability. The snapshot is executed in an isolated container to verify that the anomaly or capability gap can be reproduced reliably, ensuring that the observed failure originates from core algorithm or logic semantics rather than environmental noise. In the third stage, public-material abstraction and information isolation extracts the scientific invariants and data-flow constraints evaluated by each task. The public evaluation package includes the repository, a coarse-grained description of the observed phenomenon, a reproduction script, and the necessary domain-background documentation, while hiding the patch location and hidden assertions. In the fourth stage, hidden oracle and counter-calibration constructs hidden validation suites using semantic equivalence and boundary conditions. These validators check execution results under normal inputs and include reverse checks for hard-coded solutions, heuristic pseudo-fixes, and incomplete repairs.

Raw tasks are then redesigned into three task paradigms with differentiated evaluation goals. Issue-driven tasks repair known defects under low-distortion conditions. The construction process rolls the source code back to the state before the problematic commit, builds a minimal reproducible example, and narrows the upstream error into a coarse-grained auditable phenomenon without revealing solution-patch hints. Hidden validators vary data scales, boundary parameters, and input-order transformations to test whether the repair restores scientific semantics rather than merely fitting the public script.

Expert-exploratory tasks simulate real scientific investigation when the root cause is unknown. The authors first define a high-level scientific scenario, such as molecular representation consistency calibration or measurement-chain bias analysis. They then extract a core difference from theoretical materials, methodological literature, experimental phenomena, or result discrepancies. A runnable public workflow preserves the relevant domain background and observable anomaly while hiding the precise defective source-code location. Hidden validators change parameter scales, physical topologies, coordinate orderings, or boundary conditions to determine whether the agent inferred the underlying scientific mechanism.

Engineering-integration tasks evaluate cross-module completion of end-to-end scientific workflows. The construction process analyzes the repository's end-to-end call chain, including data loading, parameter interpretation, intermediate-representation construction, operator assembly, and numerical solving, and selects a functionality gap of appropriate scope. Public source code preserves the full package structure and neighboring modules, and public reproduction scripts show state comparisons at multiple stages. Hidden validators include alternative execution paths, state-reset tests, and inter-module behavioral contract tests to ensure architecture-level integration rather than a single passing public test.

The method also separates scientific auxiliary information to estimate the contribution of externally supplied domain knowledge. For each eligible task, paired conditions differ only in auxiliary scientific information, while the source-code snapshot, execution environment, public reproduction entry points, hidden validators, and required scientific context $c_i^{req}$cireq​ remain fixed. Scientific auxiliary information includes scientific rationales, upstream repairs, audit findings, paper excerpts, and expert guidance. Withholding this information removes scientific principles, equations or assumptions, expected properties, domain-specific diagnoses, and scientifically motivated repair strategies, while retaining repository-intrinsic engineering cues such as code structure, interfaces, traces, tests, and incomplete implementations. This design estimates the marginal contribution of scientific auxiliary information given the same repository evidence, rather than performance in a clue-free setting.

Experiment

The benchmark consists of 119 tasks constructed through a four-stage protocol and re-designed into Issue-driven repair, Expert-exploratory root-cause reasoning, and Engineering-integration multi-module workflows, with a paired ablation isolating auxiliary scientific information on 91 tasks. Eight coding agents were evaluated with public and private tests, showing that no single model leads all metrics: Claude-Opus-5 achieves the best overall Pass@1 and leads on Issue-driven and Expert-exploratory tasks, GPT-5.6-sol leads private repair metrics, and DeepSeek-V4-Pro leads public and Engineering-integration scores, though frontier pass rates remain below 50%. Failure analysis identifies scientific abstraction deficits, surface-level repairs, incomplete integration, and generalization failures, with Claude-Opus-5 producing the lowest categorized error count. Scientific auxiliary information improves DeepSeek-V4-flash more than GPT-5.6-sol, suggesting that weaker models may benefit more from external scientific guidance.

Science coding-agent benchmarks differ widely in scale and scope, with existing benchmarks covering one to sixteen scientific domains and task counts from a few dozen to several hundred, while SWE-bench Science covers 20 domains. Error profiles show complementary model strengths across scientific-error categories. Scientific information has model-specific effects, with DeepSeek-V4-flash gaining more in Pass@1 while GPT-5.6-sol declines slightly on Pass@1 despite modest public and private score gains. Existing benchmarks vary from one to sixteen scientific domains, while SWE-bench Science covers 20 domains. Task counts span from a few dozen to several hundred, and most benchmarks use expert-sourced research code or function programming tasks. Claude-Opus-5 has the lowest categorized scientific-error count and the fewest misguided-exploration or surface-level-repair errors, though it also has runtime or evaluation-path failures. DeepSeek-V4-flash has the fewest scientific-knowledge generalization errors, and DeepSeek-V4-Pro has the fewest scientific-knowledge or abstraction and incomplete-repair or system-integration errors. Adding scientific information produces a larger Pass@1 improvement for DeepSeek-V4-flash than for GPT-5.6-sol, whose Pass@1 decreases despite slight mean score improvements.

Among the evaluated models, GPT-5.6-sol led on private score and Pass@1, while Claude-Opus-5 achieved the highest overall and expert scores, and DeepSeek-V4-Pro attained a perfect public score along with top issue and engineering scores. Error analysis showed Claude-Opus-5 produced the fewest scientific errors, and experiments with scientific information indicated that weaker models gained more from external scientific guidance, whereas the strongest model experienced a slight Pass@1 decline. GPT-5.6-sol had the highest private score and Pass@1, but Claude-Opus-5 surpassed it in overall score and expert metric. DeepSeek-V4-Pro was the only model to reach a perfect public score and led in issue and engineering scores. Claude-Opus-5 made the fewest categorized scientific errors and the fewest misguided-exploration or surface-level-repair errors. Providing scientific information improved DeepSeek-V4-flash across all metrics while increasing token usage, but reduced Pass@1 for GPT-5.6-sol, suggesting weaker models benefit more from external scientific guidance.

Across the 119-task comparison, the coding agents show distinct error profiles. Claude-Opus-5 records the lowest categorized error total and the fewest exploration or surface repair errors, while DeepSeek-V4-Pro is lowest in knowledge or abstraction errors and in repair coverage or system integration errors. DeepSeek-V4-flash has the highest categorized error total but the fewest scientific generalization errors. Claude-Opus-5 has the lowest main error total, though it also has additional runtime or evaluation-path failures. DeepSeek-V4-Pro leads with the fewest knowledge or abstraction errors and the fewest repair coverage or system integration errors. DeepSeek-V4-flash shows a contrasting pattern, with many system integration errors but the fewest scientific generalization errors.

The experiments evaluate several coding agents on SWE-bench Science, a benchmark spanning 20 scientific domains and 119 tasks, and compare it with existing science coding benchmarks that cover one to sixteen domains. Error analysis shows complementary model strengths: Claude-Opus-5 has the fewest categorized scientific errors and exploration or surface-level repair errors, DeepSeek-V4-Pro has the fewest knowledge/abstraction and incomplete-repair or system-integration errors, while DeepSeek-V4-flash has the fewest scientific-knowledge generalization errors but more system-integration failures. When external scientific information is provided, DeepSeek-V4-flash improves across all metrics and Pass@1, whereas GPT-5.6-sol gains only modestly on public and private scores and declines slightly on Pass@1, indicating that weaker models benefit more from scientific guidance.