Unlike most engineering tasks, scientific research is more iterative, open, and uncertain. Its core challenge lies not in the execution of the analysis process itself, but in a series of judgments driven by scientific intuition or "research taste": for example, what questions the data can support, which data should be included, what estimators or models to choose, whether the diagnostic results overturn the initial hypothesis, and under what strength of evidence can a conclusion be drawn.

GeneBench was proposed against this backdrop. It attempts not to evaluate the capabilities of a single algorithm, but rather whether the model can complete a full "reasoning loop" in a real scientific analysis process.This includes data understanding, quality control, method selection, statistical modeling, diagnostic correction, and final decision output—capabilities that are particularly crucial in modern life sciences.

On this basis,The OpenAI research team recently released an updated version of GeneBench, GeneBench-Pro.It covers a wider range of industry and academic fields, no longer limited to genomics, but extending to scenarios requiring multi-stage statistical inference, such as molecular and quantitative biology, pharmacogenomics, cancer biology, microbial genomics, and clinical translation. Compared to GeneBench, GeneBench-Pro adds 29 questions, removes 3 questions, and significantly redesigns 54 of the remaining 100 overlapping questions.

Of the 129 questions, 82 were reviewed by external domain experts whose feedback led to modifications to the hints and data, as well as the reconstruction of questions where the target was not identifiable. In the full 129-question evaluation, GPT-5.6 Sol achieved a pass rate of 28.71 TP3T at the highest inference strength, while GPT-5.6 Sol Pro achieved 31.51 TP3T in the separately reported GPT Pro run; GPT-5.5 achieved 12.01 TP3T, GPT-5.4 achieved 8.91 TP3T, and the best-performing non-GPT baseline model, Claude Opus 4.8, achieved 16.01 TP3T.

The related research findings, titled "GeneBench-Pro: Evaluating Multistage Statistical Reasoning in Genomics, Quantitative Biology, and Translational Biomedicine," have been published as a preprint on bioRxiv.

Research highlights:

* GeneBench-Pro is an update to the previously released GeneBench, introducing significantly more difficult versions of existing problems as well as problem sets from newly added domains.

* GeneBench-Pro is no longer limited to genomics, but has expanded to scenarios requiring multi-stage statistical inference, such as molecular and quantitative biology, pharmacogenomics, cancer biology, microbial genomics, and clinical translation.

* GeneBench-Pro introduces an external scientific review step: in this process, the complete design, relevant documentation, and analytical specifications for each question are submitted to external domain experts to verify their authenticity and scientific rationale.

Paper address:
https://hyper.ai/papers/GeneBench-Pro
The relevant datasets are available online:
https://hyper.ai/datasets/53157

GeneBench-Pro: Contains 129 questions covering 10 major areas.

GeneBench-Pro is a collection of 129 questions covering 10 main domains and 21 subdomains.This is used to measure whether an agent can identify and perform the necessary quantitative analysis from a potentially flawed dataset with minimal guidance, thereby estimating the target estimator (estimand). The figure below shows the domain coverage of the current evaluation suite.

Each GeneBench-Pro problem is packaged as an independent scientific analysis task.The agent is provided with an isolated working environment containing a minimum viable prompt, phased data files, and a standard scientific Python stack. The prompt only provides the scientific question/task and target estimates, but does not explicitly specify the analytical process to be performed. These data files are designed to simulate, as far as possible, the raw data obtained by real analysts from experimental or clinical systems, rather than cleaned toy datasets.

Each problem contains a series of interdependent decision nodes, so a wrong choice made at any stage will propagate backward, causing errors in subsequent analysis and ultimately failing to obtain the correct target result.

The agent runs in a realistic sandbox environment, with access to phased data files and general scientific computing libraries including numpy, pandas, scipy, scikit-learn, statsmodels, lifelines, matplotlib, and seaborn. It can also use standard genomic bioinformatics tools such as PLINK 2.0, pysnptools, bed-reader, bedtools, and pysam.

Core design goal: The agent obtains minimum feasible hints and explicit target estimates.

The reason open-ended scientific analysis is difficult to precisely benchmark is that real-world data often allows for multiple, all reasonable, analytical options. For example, quality control (QC) thresholds, model parameter settings, and reporting standards may differ among analysts, but these differences do not imply that only one approach is correct. If the results of a benchmark vary because an agent chooses a reasonable but different cutoff or convention, this difference may reflect only the arbitrariness of the benchmark design itself, rather than the quality of scientific reasoning.

To ensure that "failure" truly represents scientific error rather than natural decay,Researchers employed a constructive simulation problem in GeneBench-Pro: that is, the complete causal structure is known, and the data generation process is explicitly simulated.

Researchers quantify cascading structures using the "number of decision points"—key inference bifurcation points in the analysis process where reasonable but incorrect choices can lead to qualitative changes in subsequent results.

In practice, each problem is built starting from a real-world analytical model and a target estimator. These real-world analytical models are derived from literature and the experience of domain experts, aiming to reflect common and high-impact scientific problems and workflows, and deliberately avoiding direct reproduction of standard cases from textbooks or published papers to prevent the model from solving problems by memorization.

Then, data is generated through simulation so that the correct answer can be recovered from the phased data. Next, a minimum feasible hint is constructed, containing only the minimum information needed for the correct answer to be identified.

After the initial draft of the problem is completed, large-scale validation will be conducted. For paths that "seem reasonable but are wrong" in different stages of reasoning, ablation experiments will be used to examine them and verify whether their results are sufficiently distinguishable from the correct answer.

In addition, an independent scientific review will be conducted to assess the rationality of the methodology, the identifiability of the objectives, and the scientific validity.

The following table summarizes the main baseline constraints derived from these design requirements:

Example question: Residual risk in DRX1 carrier screening

The image below illustrates a GeneBench-Pro problem that simulates a carrier screening scenario for an autosomal recessive genetic disease. In this synthetic screening setting, DRX1 is the disease-causing gene: there is a reproductive risk in offspring only if both biological parents carry a reportable DRX1 allele.

The agent is provided with raw carrier screening data and needs to estimate the remaining carrier risk after a "screening result is negative" and combine this residual risk with the carrier frequency in the population of potential reproductive partners.

The difficulty of this problem lies in the fact that the answer cannot be simply estimated directly from the original carrier rate; a correct analysis must complete multiple inference steps sequentially:

·  First, identify reportable DRX1 carrier categories and distinguish copy number variation artifacts;

·  Next, the founder marker of the phased model is processed, and it is determined whether it should be considered a single haplotype.

·  Then, the sensitivity and false positive rate of the detection were estimated based on different covariates;

·  Next, based on the probability of no carrier detection for each category, the residual carrier risk under the condition of negative screening is calculated;

·  Finally, the frequency of partner carriers needs to be standardized to the complete list of partners, rather than being limited to the detected subset.

This example demonstrates GeneBench-Pro's core design goal is for the agent to obtain minimum feasible hints and explicit target estimates.However, success ultimately depends on whether it can recover the complete multi-stage quantitative analysis path from the data, rather than simply executing a pre-set process.

Results Showing: Significant improvement in capabilities, but the "reasoning loop" is not yet fully established.

Researchers evaluated GeneBench-Pro on a complete suite of 129 questions, covering 60 model configurations, including GPT-5.2, GPT-5.4, GPT-5.5, GPT-5.6 Luna/Terra/Sol and their corresponding GPT Pro variants, as well as non-GPT baseline models from Claude, Gemini, Grok, GLM, Kimi, DeepSeek, MiMo, Tencent, MiniMax, and Qwen. The figure below summarizes the unweighted pass rate calculated question-by-question:

Overall,The overall pass rate of the model remains low:

·  At the highest reported inference strength of each GPT mainline model, the unweighted average pass rate increased from 4.9% for GPT-5.2 (xhigh) to 8.9% for GPT-5.4 (xhigh), 12.0% for GPT-5.5 (xhigh), 16.5% for GPT-5.6 Luna (max), 23.3% for GPT-5.6 Terra (max), and 28.7% for GPT-5.6 Sol (max).

·  The separately reported GPT Pro results are as follows: GPT-5.2 Pro 8.5%, GPT-5.4 Pro 16.3%, GPT-5.5 Pro 20.5%, GPT-5.6 Luna Pro 23.6%, GPT-5.6 Terra Pro 28.5%, and GPT-5.6 Sol Pro 31.5%.

·  The results for non-GPT models range from approximately 0.6% to 16.0%, with Claude Opus 4.8 being the strongest non-GPT baseline model.

·  Within the GPT series, increasing inference strength has a significant impact: GPT-5.6 Sol has increased from 3.7% for none to 14.4% for low, 22.5% for medium, 24.4% for high, 26.8% for xhigh, and 28.7% for max.

at the same time,Significant unresolved issues remain—In the best GPT mainline model, the proportion of "0.% pass rate problems" decreased from 77.5% in GPT-5.2 to 67.4% in GPT-5.4, 64.3% in GPT-5.5, and 45.7% in GPT-5.6 Sol; while the proportion of "at least 50% pass rate problems" increased from 1.6% to 4.7%, 8.5%, and 30.2%. Therefore, even in the strongest mainline model, the baseline still mainly consists of high-difficulty problems, but stronger models can push more problems from a "complete failure" state to partial success or stable success.

Conclusion

GeneBench-Pro's core contribution lies not only in building a new biological benchmark, but also in its attempt to redefine how "scientific agent capabilities" are evaluated. Experimental results show that current state-of-the-art models possess a certain degree of "local scientific capability," such as identifying data anomalies, understanding statistical signals, or performing standard analytical procedures. However, this capability has not yet been stably extended to end-to-end scientific analysis; a significant gap remains between "identifying the problem" and "taking action."

From an application perspective, this breakthrough has significant potential value. In modern life sciences, the process from gene data to target screening, and from statistical signals to translational decisions, heavily relies on expert team collaboration and is costly. The paper points out that a GeneBench-Pro problem typically takes 10 to 40 hours to complete without human assistance, meaning that even partial automation could generate significant economic value in industrial research.

However, researchers also emphasize that the current model cannot reliably replace this process because it cannot consistently complete the entire reasoning chain. Future improvements may include stronger planning capabilities, self-correcting mechanisms, and uncertainty modeling capabilities.

References:

https://www.biorxiv.org/content/10.64898/2026.06.29.735386v2