One-sentence Summary

PhAI Labs, The Chinese University of Hong Kong, University of Michigan, et al. propose DiscoveryRSI, which recursively improves research agents by building evidence-grounded hypothesis trees through multi-agent evolutionary search, compiling investigations into executable Harbor tasks, and refining workflows via hypothesis credit optimization, harness replay, and rediscovery, thereby improving hypothesis quality on DFM-Bench across four life-science domains from $4.86$4.86 to $5.39$5.39 after two rounds.

Key Contributions

• The paper introduces DiscoveryRSI, a framework that defines valuable research hypotheses as evidence-grounded claims about consequential uncertainties, coupled with feasible investigations, explicit decision criteria, and alternative explanations. It develops hypothesis trees through multi-agent evolutionary search, compiles selected investigations into executable Harbor tasks, and recursively learns via hypothesis credit optimization, harness replay, and rediscovery.

• The paper also introduces DFM-Bench, which assesses hypothesis quality and cumulative research progress at the levels of individual hypotheses, revisions, development paths, and complete research runs.

• Across four life-science domains, automated evaluation shows DiscoveryRSI raising hypothesis quality from 4.86 for the base model to 5.39 after two self-improvement rounds, with gains in falsifiability and data resolvability. DiscoveryRSI-27B achieves a 6.3% post-cutoff hypothesis correspondence rate, compared with 2.5% for GPT-6 and 2.3% for Claude Opus 5, and receives higher expert-rated hypothesis importance than GPT-6 Luna.

Introduction

The authors focus on AI research agents that generate hypotheses, plan investigations, and execute computational experiments. This matters because scientific discovery depends on turning proposed explanations into testable studies and then using that experience to design better future research. Prior systems can propose and evaluate hypotheses, but they often treat the final proposal as the learning target, obscuring which revisions, evidence, or agent actions actually improved the work, and they struggle to make evolving hypotheses concrete enough to execute. The authors contribute DiscoveryRSI, a framework that builds research trees of testable hypotheses, compiles selected investigations into executable Harbor tasks, preserves full development histories, and uses hypothesis credit optimization, harness replay, and rediscovery to learn recursively from documented contributions. They also introduce DFM-Bench to measure hypothesis quality, revision value, and cumulative research progress.

Dataset

• The provided text describes evaluation data rather than a conventional training corpus. The authors use DFM-Bench to make scientific assessment objects explicit, including generated hypotheses, revisions, development paths, research runs, and contribution families. This allows evaluation of evidence use, experimentation, and research-process integrity alongside existing scientific benchmarks.

• The inspiration analysis uses generated hypothesis versions grouped by model. Reported group sizes are 257 for DeepSeek, 36 for GLM, 2,341 for GPT, and 131 for Claude. A pooled DiscoveryRSI set from two production runs contains 1,176 hypothesis versions, and the authors also report inspiration rates by domain, including therapeutic targets, population multi-omics, disease mechanisms, and clinical and population health.

• The literature reference corpus is built from 420 backward and forward citation traversals. This produces 17,790 citation edges and 16,440 resolved publications. The authors filter the resolved references using each model group’s recorded cutoff, so only later literature can qualify as an inspiration match.

• The filtering rule is strict. A post-cutoff publication is counted only if it provides an experimental design, comparator, measurement, or testable evidence gap relevant to developing or testing the generated hypothesis. The authors then assign an automated positive judgment to qualifying hypothesis versions and compute the inspiration rate as the proportion of generated versions with a positive judgment.

• Cropping is part of the processing pipeline. Retrieval uses the first 6,000 characters of a proposal, and the semantic audit reads only the first 2,500 characters. This can move scientific content away from the evaluated text when proposals contain extensive provenance or revision notes. The authors show this sensitivity with a diagnostic where querying by proposal title increases Claude’s retrieved proposal count from 15 to 95.

• The data are used for evaluation and diagnostic analysis, not for reported model training. In the supplied excerpts, there is no described training split, mixture ratio, or training use for these datasets. Their role is to support hypothesis generation comparison, literature citation analysis, inspiration-rate measurement, and auditing of how retrieval coverage and proposal format affect measured outcomes.

Method

The authors introduce a joint operational definition of valuable and testable research hypotheses for hypothesis driven research agents. A research hypothesis is represented as $\mathcal{H} = (c, \pi, d, a)$H=(c,π,d,a), where $c$c is a claim grounded in evidence, $\pi$π is an investigation plan, $d$d specifies decision criteria, and $a$a identifies relevant alternative explanations. It is considered valuable if resolving its uncertainty adds to current understanding, and testable if the plan is feasible under given constraints and its outcomes can support or weaken the claim.

As shown in the figure below, this definition is instantiated with a concrete proposal.

The figure illustrates a MIMIC IV proposal investigating whether repeated creatinine and urine output mismatch adds information about adverse outcomes after ICU discharge. The investigation plan defines the timeline, baseline models, and comparisons, while the decision criteria and alternative explanations explicitly bound the claim and interpretability of the results.

To turn hypothesis development into a recursive learning process, the authors propose DiscoveryRSI. This framework develops research trees of testable hypotheses through multiple agent evolutionary search and retains the evidence and decisions behind each proposal.

As shown in the figure below, the framework couples multiple agent hypothesis development with recursive self improvement.

In the hypothesis development phase, a Lead agent directs Generation and Evolution agents to create or revise hypotheses. These versions are evaluated by Critique and Progress agents. Selected hypotheses are compiled into executable Harbor tasks, which include concrete inputs, environments, required outputs, and verification checks, preserving the scientific commitments of the original proposal.

The retained research histories support three coupled recursive improvement mechanisms. First, Hypothesis Credit Optimization links assessments of research quality to documented contributions. Blinded Critique provides preferences, which are summarized using TrueSkill to estimate relative preference strength $z_h$zh​ and uncertainty. The probability of preference is modeled as:

Assessment coverage, uncertainty, validity, and progress determine a bounded hypothesis weight $w_i = \min(W_{\max}, \sum_h b_{ih})$wi​=min(Wmax​,∑h​bih​), where $b_{ih}$bih​ is the allocation from hypothesis $h$h after contributor sharing and duplicate suppression. This credit controls how strongly a history contributes to training. The authors train DiscoveryRSI 27B using a weighted loss function:

where $U_i$Ui​ contains eligible assistant token positions, $x_{iu}$xiu​ is the recorded context, and $\omega_i$ωi​ combines contributor credit with domain balancing.

Second, Harness Replay improves the research workflow while holding model weights fixed. By analyzing agent contribution and decision records alongside execution logs, the system identifies recurring failures such as repeated exploration or lost artifacts. Candidate harness edits are proposed and evaluated through quick runs and matched development evaluation to determine acceptance.

Finally, Rediscovery applies accepted model and harness updates to fresh investigations. The acceptance procedure evaluates qualifying updates separately and jointly, allowing the model, harness, both, or neither to change. The accepted system then returns to fresh questions, collecting new hypotheses and evidence, thereby completing the recursive learning cycle.

Experiment

The experiments use DFM-Bench to evaluate generated research at the levels of individual hypotheses, revisions, development paths, full runs, and scientific contributions, combining automated quality judgments, expert human review, and literature-based inspiration checks across biomedical and battery domains. Across iterative research rounds, hypothesis quality improves mainly through better testability, clearer evidence requirements, and more disciplined revision behavior, while models increasingly evolve existing hypotheses instead of restarting them. Expert reviewers recognize more favorable progress from revisions, although absolute proposal quality remains mixed and inspiration findings are descriptive, indicating promising but uneven scientific value.

The comparison evaluates how selected scientific-discovery benchmarks explicitly assess scientific outputs and research processes. Existing benchmarks provide complementary but narrower coverage, with OEB offering explicit assessment of evidence use and revision and partial coverage of development paths and whole-run allocation, while DFM-Bench is presented as covering all listed scientific and research-process dimensions. The comparison defines coverage through explicit assessment objects rather than the mere recording of intermediate actions. DFM-Bench is described as explicitly assessing hypothesis merit, contribution maturity, evidence use, revision, development paths, and whole-run allocation. OEB gives the strongest research-process coverage among the other listed benchmarks, while AgentIdeaBench gives the clearest prior hypothesis-merit assessment. The comparison distinguishes explicit coverage from partial or related coverage and emphasizes that intermediate actions must be assessment objects to count.

The experiment compares scientific-discovery benchmarks by evaluating which ones explicitly assess scientific outputs and research-process dimensions, such as hypothesis merit, evidence use, revision, development paths, and whole-run allocation. It finds that DFM-Bench provides the broadest explicit coverage, while OEB offers the strongest research-process assessment among the other benchmarks and AgentIdeaBench is clearest on prior hypothesis merit. The comparison emphasizes that only dimensions treated as explicit assessment objects should be counted, rather than merely recorded intermediate actions.