One-sentence Summary

OpenAI introduces GeneBench-Pro, an expanded benchmark of 129 multi-stage scientific problems spanning 10 primary domains and 21 terminal subdomains in genomics, quantitative biology, and translational biomedicine that forces AI agents to navigate multiple dependent inferential forks where a plausible wrong choice changes the downstream analysis; models such as GPT-5.6 Sol Pro reach a $31.5\%$31.5% pass rate but consistently exhibit a gap between noticing local diagnostic signals and acting on them, often selecting wrong estimators and persisting on initially plausible but incorrect analysis paths, underscoring the unreliability of long-horizon biological reasoning.

Key Contributions

• GeneBench-Pro is a benchmark of 129 multi-stage scientific analysis problems across genomics, quantitative biology, and translational biomedicine, with 82 problems reviewed by external domain experts to improve target identifiability and difficulty.
• Evaluations of over 60 model configurations show that GPT‑5.6 Sol reaches a pass rate of 28.7%, GPT‑5.6 Sol Pro reaches 31.5%, and models exhibit a gap where local diagnostic signals are detected but not propagated to correct analysis decisions.
• The benchmark is released in a stratified manner: 10 public problems, 50 held-out problems for third-party benchmarking by Artificial Analysis, and an internal holdout.

Introduction

Recent advances in AI agents have shown rapid progress in software engineering and long-horizon tasks, and genomic benchmarks have begun to move from simple knowledge tests toward more realistic biological workflows. However, the open-ended, multi-stage analyses that researchers routinely perform (where a wrong choice at any step can derail the entire conclusion) remain largely unexamined by existing evaluations. The authors introduce GeneBench‑Pro, a substantially hardened and expanded benchmark of 129 problems across genomics, quantitative biology, and translational biomedicine. Each problem requires an agent to navigate multiple dependent decision points with minimal guidance, reflecting the “notice‑act” gap that distinguishes expert from novice reasoning. By providing a tiered public release and evaluating over 60 model configurations, GeneBench‑Pro reveals that even frontier models struggle to reliably close the inferential loop, offering a precise measure of an emerging, high‑impact capability for automating scientific discovery.

Experiment

GeneBench-Pro evaluates frontier models on long-horizon, multi-stage genomic analyses within a containerized Linux environment without internet, using rigorous binary grading that requires correct final answers across all decision points. The benchmark reveals that current models can detect data irregularities and complete intermediate steps, yet they consistently fail to close the "notice-act" inferential loop needed for reliable end-to-end performance, a pattern that resembles expert-novice differences in scientific problem solving. This qualitative gap underscores that despite rapid progress, substantial capability advances in planning, self-revision, and uncertainty-aware control are still needed before models can automate the kinds of analyses that currently demand expert scientists, a goal with transformative potential for accelerating discovery.

GeneBench-Pro imposes design constraints that keep the scientific endpoint identifiable while preserving realistic ambiguity. The framework requires a recoverable ground truth, a single defensible answer supported by empirical constraints, and a clear numerical gap between correct and incorrect solutions, thereby avoiding common failures like unrecoverable parameters, underspecification, and grading artifacts. Problem specification rules further ensure that prompts define the scientific question without dictating the analysis method and that quality control thresholds do not introduce arbitrary evaluation outcomes. Agents are evaluated on quantities actually recoverable from the provided data, not on hidden data-generating parameters, so correct analyses are not wrongly penalized due to sampling variation. Each task supports exactly one defensible answer because the data or prompt includes empirical constraints that reasonably exclude alternative approaches, preventing the evaluation from measuring workflow preference instead of reasoning. Plausible but incorrect analyses and shortcut methods fail by clear margins in an ablation suite, ensuring that wrong solutions are not mistakenly graded as correct. Prompts specify the scientific question and the target estimand without prescriptive instructions, keeping the focus on open-ended reasoning rather than recipe-following. Quality control requirements use thresholds where nearby reasonable values produce the same graded outcome, so the benchmark measures recognition of qualitative problems rather than sensitivity to arbitrary cutoffs.

External review of benchmark problems uncovered several types of issues that prompted revisions to improve clarity and scientific rigor. Common adjustments included refining underspecified or ambiguous prompts, aligning grading with defensible alternative methods, updating ground truth to match realizable data, and fixing method implementations that did not match their stated task. Problems where a defensible alternative method existed, such as the admixture task where posterior-weighted ancestry fractions were scientifically reasonable, were updated to accept those alternatives rather than requiring a hard-call result. A genetic correlation benchmark was found to be grading an ad hoc LD-score estimator instead of true LDSC, and the reference implementation was corrected to match the method it purported to evaluate. When ground truth was not recoverable under the original prompt, like the dynamic pharmacogenomic treatment response, the benchmark was realigned to use the data-generating process's realized data and to make validator target and nuisance specifications explicit. An ambiguous prompt for a somatic target activation problem left room for interpreting high copy number without structural variant involvement, so the solver-facing materials were clarified by adding wording that specifies the causal structural variant mechanism.

GPT-5.6 Sol consistently recognizes and acts on subtle data complexities that GPT-5.5 overlooks or handles with simpler, less appropriate methods. Across pharmacogenomic survival, cell-type heritability, and peptide pQTL tasks, GPT-5.6 Sol correctly adjusts for time-varying confounding, reference-panel quality, and dose-response discordance, leading to more reliable analytical decisions. For pharmacogenomic time-to-event data, GPT-5.6 Sol used a marginal structural Cox model with stabilized inverse-probability weights to address treatment-confounder feedback, while GPT-5.5 applied a conventional counting-process Cox model that did not account for this feedback. When selecting an LD-score reference panel, GPT-5.6 Sol applied stricter quality filters including imputation-quality and chi-squared thresholds, yielding a refined SNP set, whereas GPT-5.5 chose the panel solely by allele-frequency concordance after basic filters. In the bridge-calibrated pQTL analysis, GPT-5.6 Sol identified peptides with discordant dose-response slopes and excluded them before averaging, while GPT-5.5 simply averaged all six peptides, including those with inconsistent patterns.

The GeneBench-Pro framework enforces design constraints that ensure each task has a single defensible ground truth answer and that grading isolates scientific reasoning rather than workflow habits, with failures designed to clearly separate correct from incorrect solutions. External review and revision cycles refined benchmark prompts and validator logic to eliminate ambiguity and confirmed that real-world complexities do not compromise grading reliability. Comparative testing showed that GPT-5.6 Sol consistently applied more appropriate statistical corrections and quality filters to handle subtle data confounds, whereas GPT-5.5 defaulted to simpler methods, demonstrating the benchmark's ability to detect meaningful differences in analytical reasoning.