One-sentence Summary

The Apodex Team introduces FRONTIERCHALLENGE, a cross-domain benchmark of 300 end-to-end scientific workflows (97 released) spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment, and evaluates twelve frontier models with three scaffolds, finding that the best configurations complete only $20.6\%$20.6% of tasks and that high partial scores (up to $94.9$94.9 Avg. Score) and confident completion claims ($75.5\%$75.5% of non-passing trajectories) poorly predict full delivery, underscoring the need to assess workflow execution and deliverable completeness jointly.

Key Contributions

• Introduces FRONTIERCHALLENGE, a cross-domain benchmark of 300 end-to-end scientific workflows, and releases 97 tasks spanning six domains (quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment), each with fixed inputs, a deliverable contract, and a task-specific executable Grader.
• Proposes a contract-level evaluation framework using Pass Rate for full completion and Avg. Score for partial progress, and shows through evaluating twelve frontier models with three agent scaffolds that the best configuration achieves only a 20.6% Pass Rate despite Avg. Scores reaching 87.9, indicating that high partial scores do not imply complete delivery.
• Provides a failure analysis of 970 Claude Code trajectories, revealing that 75.5% of non-passing runs end with language claiming completion, and shows that tool errors occur in both passing and non-passing runs, establishing that final self-reports and error presence are weak indicators of successful scientific deliverable completion.

Introduction

Language models are evolving from text generators into agents that can plan, call tools, execute code, and modify files, with recent systems extending agentic support to multi-stage research workflows that require inspectable outputs. Existing benchmarks often evaluate a final answer, an interaction trace, or a single program, which does not fully capture whether an agent can complete heterogeneous scientific work that depends on several analytical stages and multiple required deliverables. To address this, the authors introduce FRONTIERCHALLENGE, a benchmark that asks whether an agent can independently execute a specified scientific workflow from input processing to final deliverables, satisfying a complete task contract. The benchmark includes 300 end-to-end workflows across six domains, with 97 tasks publicly released, and evaluates twelve frontier models with three agent scaffolds using Pass Rate as the primary metric. The central finding is a persistent gap between partial progress and complete delivery: the best configurations achieved only a 20.6% Pass Rate despite high average scores, with especially poor completion in analytical chemistry and electrochemistry. The authors contribute a cross-domain workflow benchmark, contract-level evaluation, and a failure analysis of 970 trajectories, showing that final self-reports and tool errors are weak indicators of successful delivery, and that reliable scientific agents will require explicit contract tracking and evidence-based completion checks.

Dataset

The authors build a dataset of 300 scientific workflows collected from professional practice in scientific and engineering settings. The tasks are designed to reflect realistic workflows that require domain knowledge, specialized software, experimental data, or engineering environments, rather than simple question answering or isolated coding exercises.

Dataset composition and sources

• The full collection contains 300 workflows, but the paper releases and evaluates 97.
• Tasks originate from analysis, computation, simulation, and research-delivery processes, not from expanded QA or standalone coding problems.
• Inputs begin from fixed public empirical data, public sequence or structural resources, or scientifically constrained synthetic data. These source types describe how workflows are instantiated, not that every task starts from direct lab measurements.

Key details for each subset

• The released 97 tasks are randomly selected from the subset whose official evaluation does not require GPU resources.
• The remaining 203 tasks form an internal held-out set, which includes workflows requiring GPU-based evaluation; none of these held-out tasks appear in the reported results.
• The released tasks comprise 74 Hard tasks and 23 Medium tasks.
• For analysis, the 97 tasks are organized into six reporting domains: quantum chemistry (20 tasks), molecular dynamics (16), materials characterization (22), analytical chemistry (23), life science (10), and electrochemistry (6).
• The tasks cover 21 workflow families and require heterogeneous deliverables, including scientific reports, structured data, figures, executable code, and simulation products.

Curation and filtering rules The authors apply four design principles during screening:

• Representativeness: workflows, software, and methods must reflect plausible professional practice.
• Complexity: tasks must require an end-to-end, dependency-aware process leading to a substantive deliverable, not a single command or local edit.
• Diversity: the collection must vary in scientific knowledge, workflow type, and difficulty, not repeat templates with changed inputs.
• Verifiability: outputs must be assessable through files, numerical values, quantitative measures, or explicit acceptance criteria.

They exclude tasks centered on isolated facts, single-step operations, purely subjective outputs, or those lacking materials for reproducible scoring. Every included task must have fixed inputs, a defined execution environment, a complete deliverable contract, and an executable evaluation procedure.

Packaging and metadata construction Each task is packaged as a self-contained unit with five aligned elements:

• A task description defining the scientific objective.
• Fixed inputs, including data and necessary context.
• Software and tools available in the execution environment.
• An output contract listing required deliverables.
• An evaluation procedure defining successful completion.

Each package also contains task metadata, agent-facing instructions, input data, expected-output or reference material, a stepwise scoring rubric, an executable Grader, and documentation for reproduction and scoring. Environment specifications, domain tools, auxiliary references, and execution traces are included when required. Agent-visible instructions and inputs are separated from evaluator-side references and scoring components.

How the data is used

• The 97 released tasks are used for the reported experiments, with a random selection from the GPU-free subset.
• The held-out 203 tasks are not included in any reported results.
• The six domains are descriptive slices of the released evaluation set, not probability samples of their fields. The authors note that performance differences across domains should not be interpreted as intrinsic rankings of disciplinary difficulty.
• The standardized packaging allows each task to be rerun under fixed conditions and scored consistently across evaluated systems.

Method

The authors construct the task collection by sourcing workflows from scientific and engineering practice, emphasizing realistic processes that require domain expertise, specialized software, experimental data, or engineering environments. The tasks derive from analysis, computation, simulation, and research-delivery pipelines rather than from expanded question answering or isolated coding exercises. The full collection contains 300 scientific workflows, of which 97 are released and evaluated in this paper.

Each task undergoes a curation process guided by four design principles. Representativeness ensures that the workflow, software, and methods reflect plausible professional practice. Complexity requires an end-to-end, dependency-aware process that culminates in a substantive deliverable, rather than a single command, tool call, or local edit. Diversity mandates variation in scientific knowledge, workflow type, and difficulty, avoiding mere template repetition with changed inputs or parameters. Verifiability demands that outputs be assessable through files, numerical values, quantitative measures, or explicit acceptance criteria, enabling repeatable automated evaluation. The authors also verify that every included task has fixed inputs, a defined execution environment, a complete deliverable contract, and an executable evaluation procedure. Tasks centered on isolated facts or single-step operations, tasks with purely subjective outputs, and tasks lacking materials for reproducible scoring are excluded before the final collection is formed.

After curation, each task is standardized into a self-contained package with five aligned elements: a task description defining the scientific objective; fixed inputs, including data and necessary context; the software and tools available in the execution environment; an output contract listing the required deliverables; and an evaluation procedure defining successful completion. Deliverables may include scientific reports, structured tables, diagnostic figures, executable analysis code, simulation products, or multiple artifacts that must remain mutually consistent. Completion therefore depends on the entire artifact bundle, not on whether the agent returns a plausible final answer.

Each package contains task metadata, agent-facing instructions, input data, expected-output or reference material, a stepwise scoring rubric, an executable Grader, and documentation for reproduction and scoring. Environment specifications, domain tools, auxiliary references, and execution traces are included when required by the workflow. Agent-visible instructions and inputs are separated from evaluator-side references and scoring components. This standardized organization allows each task to be rerun under fixed conditions and scored consistently across evaluated systems.

Experiment

The experiments evaluated twelve frontier models across three scaffolds on 97 scientific workflow tasks, finding that strict full completion was rare (Pass Rates from 3.1% to 20.6%) despite high average scores, indicating that models often produced artifact bundles missing at least one requirement. Domain-level performance varied substantially, with quantum chemistry and molecular dynamics showing the highest completion rates while analytical chemistry and electrochemistry/environment had near-zero or zero Pass Rates despite high average scores, revealing that aggregate rankings masked domain-specific strengths. Failure mode analysis showed that judge-assessed artifact shortfalls were common in non-passing submissions, final messages with completion language appeared in 75.5% of non-passing runs, and tool errors occurred in both passing and non-passing trajectories, demonstrating that self-reporting and raw errors are unreliable predictors of successful delivery.

The table compares the core design characteristics of several related benchmarks, indicating which features are central to each benchmark's primary design. Most benchmarks include executable evaluation, while features like fixed scientific inputs and cross-domain science vary across them. Only a few benchmarks incorporate all listed characteristics, such as CORE-Bench and PaperBench. All listed benchmarks include executable evaluation as a primary design characteristic. CORE-Bench and PaperBench are the only benchmarks that incorporate all six listed characteristics. Cross-domain science is a primary feature in only two benchmarks: HLE and BLADE. Scientific workflow core is a primary feature in most benchmarks, except HLE, ALE, and Frontier-Bench.

Strict task completion remained rare across all configurations, with pass rates ranging from 3.1% to 20.6% despite average scores often exceeding 80. High average scores did not guarantee complete delivery, as even the top configurations completed only about one-fifth of tasks. Performance varied by task difficulty, with medium tasks showing notably higher pass rates than hard tasks. The top pass rate was 20.6%, achieved by two configurations, while the lowest was 3.1%. Medium-difficulty tasks had pass rates up to 43.5%, whereas hard tasks peaked at 14.9%. Eight configurations scored above 80 on average, but none completed more than 20.6% of tasks under the strict criterion. Among similar scaffolds, one configuration outperformed another by about 2.7 points in average score and 2.1 percentage points in pass rate.

The evaluation compared benchmark designs and agent performance across configurations. Benchmark analysis showed that executable evaluation is universal, while CORE-Bench and PaperBench uniquely incorporate all six core features, and cross-domain science appears only in HLE and BLADE. Task completion results revealed a stark gap between average scores and strict completion, with pass rates ranging from 3.1% to 20.6% despite many configurations scoring above 80 on average. Medium-difficulty tasks achieved notably higher pass rates than hard tasks, and even the best configurations completed only about one-fifth of tasks under strict criteria.