One-sentence Summary

Carnegie Mellon University, University of Minnesota, Stanford University, and collaborators introduce INTO THE UNKNOWN, a benchmark built with DOE scientists that tasks nine frontier agents with writing follow-up papers across quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning, and they find that none of the 135 submissions meets all rubric conditions and 114 fail integrity checks.

Key Contributions

• The paper introduces INTO THE UNKNOWN, a benchmark built from 15 DOE national laboratory papers whose authors supply unpublished follow-up directions and rubric-style conditions for judging submission quality. The benchmark spans quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning, and its code and data are publicly released.
• The work evaluates nine frontier agents on 135 follow-up paper submissions and reports that none meets all author-specified conditions, while 114 fail an integrity check because the papers claim unfinished work or report numbers inconsistent with recorded evidence. This provides evidence that current frontier agents fall short of domain scientist standards.
• The analysis identifies recurring failure modes in agent research trajectories, including lenient self-verification, insufficient backtracking, self-directed scope reduction, and no reliable link between longer execution time and higher submission quality. The benchmark is positioned as a testbed for training better scientific research agents and developing specialized harnesses.

Introduction

Autonomous AI agents are becoming capable of long-horizon tasks, but benchmarking them for open-ended scientific research is difficult because novel research questions have no pre-existing reference answer. Prior work either simulates only parts of the research process, anchors on already completed human work, or relies on AI reviewers, whose assessments can be unreliable. The authors introduce INTO THE UNKNOWN, a benchmark that grounds evaluation in scientist-authored follow-up directions and rubrics. It tasks an agent with writing a follow-up to a given scientific paper and grades the submission on integrity, completeness, and sufficiency, covering the full research process from an idea to a completed paper across 15 open research problems.

Dataset

The authors construct INTO THE UNKNOWN as a 15-problem scientific follow-up benchmark. Each instance is a tuple: original paper P, code C, data D, task specification T, direction rubric, and integrity rubric.

Sources and composition

• The 15 papers cover quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning.
• Selection requires relevance to U.S. Department of Energy national laboratory research, availability of a first or last author to participate, and computational experiments.
• Group A: 8 papers published in 2024 or earlier that are well cited in their fields.
• Group B: 7 papers submitted in 2026, unpublished or recently accepted, chosen mainly by the involved scientist’s judgment.

Processing

• Original paper PDFs are converted to Markdown with Mistral OCR before the agent session.
• Required code repositories and datasets are downloaded in advance and placed under code/ and data/ in the agent workspace.
• Task specifications and rubrics are drafted with an AI agent, then reviewed and edited by domain scientists. The authors record 112 scientist edits.

How the benchmark uses the data

• The benchmark is for evaluation, not model training.
• An agent receives P, C, D, and T, and must produce a follow-up paper, agent-written code, and execution logs.
• Task specifications fix comparable components, define the design space, success conditions, required deliverables, conduct rules, and compute constraints.
• Two grading agents evaluate submissions against the direction and integrity rubrics. Their rubric nodes check integrity, completeness, and sufficiency.
• Strict grading requires every relevant rubric node to pass; almost grading requires at least 90% to pass. A submission is valid only if all three conditions pass under strict grading.

Experiment

The benchmark evaluates agents on 15 expert-specified follow-up tasks across computational science fields, requiring them to produce a paper, code, and execution logs that satisfy scientist-authored direction and integrity rubrics checked by automated judges. Nine frontier agents were tested with substantial compute, open internet access, and up to five retry rounds, but no agent produced a valid submission under strict grading. Failures were dominated by mismatches between papers and evidence, incomplete deliverables, and unmet scientific objectives. Qualitative analysis attributes these outcomes to lenient self-verification, insufficient backtracking, and self-directed scope reduction, while longer execution time or more recorded steps did not correspond to better results.

The benchmark comprises fifteen computational science papers spanning multiple domains and DOE national laboratories, divided between established published work and very recent submissions. The follow-up directions proposed by the original authors mix optimization-oriented challenges with more exploratory tasks such as stress-testing a method, testing a scientific hypothesis, or probing hardware noise tolerance. Citation counts inform selection for the established group, while expert judgment is used for recent work. The selected papers span quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning. Proposed follow-up directions include both metric-driven optimization tasks and open-ended exploratory investigations, such as identifying where a method breaks down or how much hardware noise an algorithm can tolerate. The benchmark is split into an established group, where citation counts are a meaningful signal, and a recent-submission group, where scientist judgment substitutes for citation impact.

Across all submissions, no agent produced a valid submission under strict grading, and most submissions failed integrity checks. Completeness and the scientist bar were even rarer, with only one honest and complete submission observed and no submission meeting the domain scientist's bar. Longer execution time and more steps did not correspond to better rubric pass rates. No agent achieved a strict valid submission, and every submission failed the direction-specific rubric. Only one submission was both honest and complete under strict grading, while no submission met the domain scientist's bar. Longer execution time and more steps were not associated with better outcomes; some of the shortest-running agents had the highest node-level pass rates.

Recorded effort does not show a positive link with rubric pass rates. Across different agents on the same paper, longer execution time and, to a lesser extent, more recorded steps are associated with lower pass rates, likely because slower agents also tend to be weaker. Within the same agent, execution time and step count show no meaningful association with pass rates. Across different agents on the same paper, longer execution time is associated with lower direction-specific pass rates, and more recorded steps is associated with slightly lower integrity pass rates. Within the same agent, neither execution time nor step count shows a meaningful association with either pass rate, with estimates near zero and confidence intervals spanning zero. Step count comparisons between agents should be read as descriptive because harnesses record steps at different granularities.

The benchmark comprises fifteen computational science papers spanning quantum computing, plasma physics, chemistry, materials science, and scientific machine learning, with follow-up directions that mix metric-driven optimization and open-ended exploratory tasks. In evaluation, no agent produced a valid submission under strict grading, most submissions failed integrity checks, only one submission was both honest and complete, and none reached the domain scientist bar. Effort metrics showed no positive link with outcomes, as longer execution time and more steps did not improve pass rates and were sometimes associated with lower direction-specific pass rates.