One-sentence Summary

PhAI Labs, CUHK, the University of Michigan, and the National University of Singapore introduce DiscoveryRSI and DiscoveryRSI-27B, which recursively learn valuable, testable hypotheses by evolving hypothesis trees, compiling selected investigations into executable Harbor tasks, and refining workflows through hypothesis credit optimization, harness replay, and rediscovery, while DFM-Bench evaluates hypothesis quality across four life-science domains, showing automatic scores rising from $4.86$4.86 for the base model to $5.39$5.39 after two self-improvement rounds, with gains in falsifiability and data resolvability, and expert review indicating higher importance and evidence-bounded plausibility than GPT-6 Luna.

Key Contributions

• The paper defines valuable and testable research hypotheses as evidence-grounded claims about consequential uncertainties, each paired with a feasible investigation, explicit decision criteria, and alternative explanations. This definition ties hypothesis generation to observable quantities, evidence constraints, and interpretation needed for scientific evaluation.

• The paper presents DiscoveryRSI, a framework that develops hypothesis trees through multi-agent evolutionary search, compiles selected investigations into executable Harbor tasks, and improves research agents through hypothesis credit optimization, harness replay, and rediscovery. It links scientific assessments to documented research contributions and uses accepted improvements to refine future investigation workflows.

• The paper introduces DFM-Bench to evaluate hypothesis quality and cumulative progress across hypotheses, revisions, development paths, and research runs, with results showing quality rising from 4.86 for the base model to 5.39 after two self-improvement rounds, especially in falsifiability and data resolvability. Across four life-science domains, DiscoveryRSI-27B also receives higher expert ratings for importance, plausibility, and evidence-boundedness than GPT-6 Luna, and it reaches 6.3% correspondence with research published after the model cutoff versus 2.5% for GPT-6 and 2.3% for Claude Opus 5.

Introduction

Scientific discovery agents can help researchers develop, test, and refine hypotheses, but their usefulness depends on turning proposals into executable investigations and on learning from the outcomes of those investigations. Prior systems, such as Co-Scientist, AI Scientist-v2, ScienceIDE, and ScienceBuddy, support hypothesis generation, experimental planning, or workflow improvement, yet they do not fully connect research assessments to the specific actions that contributed to them. This is challenging because a final judgment about a hypothesis may not reveal which proposal, control, revision, or evidence decision should be reinforced. The authors introduce DiscoveryRSI, a framework that treats hypotheses as joint objects containing claims, investigation plans, decision criteria, and alternative explanations, then evolves them through multi-agent search and compiles selected investigations into executable Harbor tasks. They also introduce DFM-Bench to evaluate hypothesis quality and cumulative research progress, and use hypothesis credit optimization, harness replay, and rediscovery to learn from documented research histories and improve later investigations.

Dataset

• Overall dataset role: The authors use the described data as an evaluation and analysis corpus, not as a model training dataset. It is used to judge scientific quality, revision value, development-path progress, research-run quality, and later scientific contribution.

• Primary source components: The dataset combines hypothesis records generated by research agents, revision and development-path records, agent action histories, research-run graphs, resource and opportunity records, and an external literature corpus linked through citations.

• Scientific assessment data:

  • Hypothesis records are scored on importance, novelty, grounding, falsifiability, data resolvability, feasibility, alternative explanations, and actionability.
  • Revision records compare each hypothesis with its predecessor and label the change as substantive progress, necessary repair, repetition, or insufficient evidence.
  • Development-path records follow ordered ancestry relationships and track evidence acquisition, coherent revision, scientific continuity, and continue or stop decisions.
  • Research-run records cover the full research graph, selection decisions, stopping decisions, exploration breadth and depth, cross-branch synthesis, and budget allocation.
  • Contribution records group proposals by scientific question and mark whether evidence adds a new distinction, negative result, supported finding, or uncertainty reduction.

• Model-generated hypothesis subsets:

  • The paper reports pooled DiscoveryRSI production runs with 1,176 hypothesis versions.
  • Other model groups include DeepSeek with 257 hypothesis versions, GLM with 36, GPT with 2,341, and Claude with 131.
  • These subsets are used to compare later inspiration to literature, not for model fitting or mixture-based training.

• Domain composition:

  • The evaluated DiscoveryRSI output spans therapeutic targets, population multi-omics, disease mechanisms, and clinical and population health.
  • The paper reports domain-level inspiration rates, including 16.9% for therapeutic targets, 7.4% for population multi-omics, 6.8% for disease mechanisms, and 2.3% for clinical and population health.

• Literature reference corpus:

  • The authors construct a shared reference corpus from 420 backward and forward citation traversals.
  • The corpus yields 17,790 citation edges and 16,440 resolved publications.
  • References are filtered using each model group’s recorded cutoff.
  • A later publication is judged inspirational only if it provides an experimental design, comparator, measurement, or testable evidence gap relevant to developing or testing the hypothesis.

• Inspiration evaluation data:

  • The inspiration set is built by matching generated hypothesis versions against retrieved publications.
  • The inspiration rate is the share of generated hypothesis versions receiving a positive automated judgment.
  • Across the pooled DiscoveryRSI set, 74 of 1,176 hypothesis versions are marked inspirational, giving 6.3%.
  • Reported model-level rates are 5.4% for DeepSeek, 2.8% for GLM, 2.5% for GPT, and 2.3% for Claude.

• Retrieval and filtering rules:

  • The evaluation applies cutoff filtering by model group, so only literature after the relevant cutoff is considered.
  • Retrieval gates decide which proposals are passed into later automated judgment.
  • For Claude, only 15 of 131 proposals pass the retrieval gate, or 11.5%.
  • For DiscoveryRSI, 355 of 1,176 proposals pass, or 30.2%.
  • Among retrieved proposals, the positive-judgment fractions are similar, 20.0% for Claude and 20.8% for DiscoveryRSI.

• Cropping and text-processing strategy:

  • Retrieval operates on the first 6,000 characters of a proposal.
  • Semantic audit operates on the first 2,500 characters.
  • The paper notes that Claude proposals have a median length of 41,134 characters, so provenance and revision notes can push scientific content outside the evaluated excerpt.
  • A diagnostic using proposal titles as queries raises Claude’s retrieved proposal count from 15 to 95 while keeping the corpus, scoring method, date rule, and overlap threshold fixed.

• Metadata construction:

  • Each hypothesis is tied to model group, domain, cutoff, predecessor version, ancestry position, and supporting action history.
  • Development paths record how hypotheses change over time.
  • Research runs record selection, stopping, exploration, synthesis, budget allocation, and resource usage.
  • Contributions are grouped by scientific question and annotated by maturity, separating proposed ideas from findings supported by completed investigations and independent checks.

• Use in the paper:

  • The data are used to calculate descriptive and benchmark-style quality scores.
  • No training split, mixture ratio, or model-training data pipeline is described in the supplied text.
  • The dataset supports evaluation across five levels, hypothesis-to-literature inspiration matching, and diagnostic analysis of retrieval sensitivity.

Method

The authors introduce DiscoveryRSI, a general framework that transforms hypothesis-driven research into a recursive learning process. Central to this approach is a joint operational definition of a valuable and testable research hypothesis, represented as $\mathcal{H} = (c, \pi, d, a)$H=(c,π,d,a), where $c$c is an evidence-grounded claim, $\pi$π is an investigation plan, $d$d specifies decision criteria, and $a$a identifies alternative explanations. A hypothesis is deemed valuable if resolving its uncertainty meaningfully advances current understanding, and testable if the investigation plan is feasible under declared constraints and yields observable outcomes that can support, weaken, or bound the claim.

As shown in the figure below:

This operational definition is instantiated through a structured investigation plan that defines the temporal scope, baseline models, and evaluation protocols. The decision criteria explicitly state which outcomes would support or weaken the claim prior to result interpretation, while the alternative explanations identify competing accounts and confounding factors that must be distinguished.

DiscoveryRSI develops these hypotheses across multiple episodes, retaining the work behind each version as linked research records. The framework employs a multi-agent evolutionary search involving roles such as Lead, Generation, Evolution, Critique, and Progress to grow research trees. Selected computational plans are then compiled into executable Harbor tasks, preserving the scientific commitments of the original hypothesis.

The core of the framework is its recursive self-improvement mechanism, which leverages accumulated research experience to update both the underlying model and the research workflow.

As shown in the figure below:

The recursive self-improvement step consists of two coupled mechanisms: hypothesis credit optimization and harness replay.

Hypothesis credit optimization links scientific assessments to documented contributions to provide supervision for model training. First, hypotheses are assessed using a TrueSkill model to estimate their relative preference strength and uncertainty. The resulting posterior parameters $(\mu_h, \sigma_h)$(μh​,σh​) describe the comparative assessment. Next, contributor credit is allocated to the specific research histories that produced a proposal, supplied a supporting investigation, or made a coordination decision. The accumulated credit for a history $i$i is calculated as:

where $b_{ih}$bih​ is the allocation from hypothesis $h$h after contributor sharing and duplicate suppression. Finally, the model is trained by reinforcing supported actions within these credited histories. The training objective for DiscoveryRSI-27B is defined as:

where $U_i$Ui​ contains eligible assistant-token positions, $x_{iu}$xiu​ is the recorded context, and $\omega_i$ωi​ combines contributor credit with domain balancing. This ensures the history's weight is distributed over its supported targets rather than every message.

Harness replay improves the research workflow while holding the model weights fixed. By analyzing agent contribution and decision records alongside execution logs, the system diagnoses recurring failures such as repeated exploration, failed tool use, or premature closure. These diagnoses motivate targeted harness edits. Candidate harnesses are proposed and evaluated through quick runs and matched development evaluation, considering scientific quality, exploration breadth, and reliability.

The final stage is rediscovery, which applies the accepted model and harness updates to fresh investigations. Let $\theta_k^+$θk+​ be the model candidate from hypothesis credit optimization and $\psi_k^+$ψk+​ the harness candidate selected through replay. The system for the next cycle is determined by:

The accepted system then returns to fresh questions, collecting new hypotheses and evidence, thereby closing the recursive learning loop.

Experiment

The experiments use DFM-Bench and an expert-supervised review protocol to assess DiscoveryRSI across biomedical, clinical, and battery research settings, with automated judging, behavioral analysis, and literature-based follow-up used to examine hypothesis quality, research behavior, human-perceived progress, and downstream inspiration. Learning from research histories primarily strengthens testable investigation design and evidence discipline, while shifting agents toward sustained refinement rather than repeated restarts. Human reviewers recognize more favorable revision trajectories, although individual proposals still show mixed design quality and occasional consequential flaws. The literature comparison indicates that some generated hypotheses correspond to concrete future research directions, supporting potential scientific utility despite limited prospective validation.

The table shows that DFM-Bench explicitly covers hypothesis merit, contribution maturity, evidence use, revision, development paths, and whole-run allocation, while existing benchmarks assess only overlapping subsets. AgentIdeaBench and DiscoveryBench provide the clearest scientific-assessment signals, BLADE and ScienceAgentBench mostly address evidence use, and OEB is the closest comparison for research-process dimensions, though development path and whole-run allocation remain partial. DFM-Bench is the only benchmark listed with explicit assessment across all six compared dimensions. Existing benchmarks tend to assess evidence use more often than revision, development paths, or whole-run allocation.

The comparison evaluates DFM-Bench against existing research-assessment benchmarks across hypothesis merit, contribution maturity, evidence use, revision, development paths, and whole-run allocation. The results show that existing benchmarks generally cover only partial subsets, with particular emphasis on evidence use and limited attention to revision, development paths, or whole-run allocation. DFM-Bench is distinguished as the benchmark with explicit coverage across all six dimensions, indicating broader validation of scientific research-process assessment.