One-sentence Summary

A team from Shanghai Jiao Tong University, Carnegie Mellon University, and other institutions introduces IdeaGene-Bench (IG-Bench), a benchmark that employs typed Idea Genome objects and GenomeDiff records to assess scientific lineage reasoning and lineage-grounded idea generation, finding that even the strongest LLM-based scientists achieve only $27.3\%$27.3% exact accuracy and that structured lineage context reshuffles rankings.

Key Contributions

• The IdeaGene framework represents scientific work as typed Idea Genome objects and uses GenomeDiff records to capture inheritance, mutation, loss, external import, and novel insertion under six evolutionary dynamics.
• IG-Bench contains 1,961 golden lineage traces, 1,085 curated Idea Genome objects, and 920 pairwise GenomeDiff records across 10 domains, and supports IG-Exam (42 task types for closed-form lineage reasoning) and IG-Arena (a lineage-conditioned Population-Evolution Score for idea generation).
• Experiments on 14 LLM-based systems show a compositional bottleneck, with the strongest model reaching 27.3% exact accuracy on lineage reasoning, and structured lineage context reshuffles system rankings rather than uniformly improving performance.

Introduction

LLM-based auto-research systems are typically evaluated on retrieval quality, factuality, writing fluency, or workflow competence, but these metrics overlook a critical question: when a proposal claims to extend a research direction, does it correctly inherit the parent mechanism, repair the stated limitation, and stay coherent with the lineage it builds on? Prior work treats scientific progress as topical proximity, so papers that share a task but not a mechanism can be conflated, and a generated idea can sound novel without carrying forward the core mechanism. The authors address this gap with the IdeaGene framework, which represents papers as typed Idea Genome objects and uses GenomeDiff records to trace inheritance, mutation, and other evolutionary dynamics. They instantiate it as IG-Bench, a benchmark with closed-form lineage reasoning tasks (IG-Exam) and a lineage-grounded generation arena (IG-Arena) scored by a heredity-variation-selection metric. Experiments on 14 systems show that plausible research text does not imply lineage competence: the best system reaches only 27.3% exact accuracy, and structured lineage context separates systems rather than uniformly helping them.

Dataset

The authors create IG-Bench, a benchmark built from curated scientific lineage traces. It consists of two evaluation subsets: IG-Exam for closed-form lineage understanding and IG-Arena for lineage-grounded idea generation.

• Overall composition and sources IG-Bench contains 1,961 golden lineage traces spanning 10 scientific domains: NLP, computer vision, multimodal learning, biology, chemistry, physics, materials, medicine, and mathematics. These traces are derived from landmark and frontier papers in each domain. Within the traces, 1,085 Idea Genome objects capture typed paper components, and 920 pairwise GenomeDiff records annotate how ideas evolve between papers.

• Key details for each subset

• IG-Exam: 1,029 evaluation instances across 42 task types. Each instance is a tuple (capability axis, anonymized context, answer space, gold target, metadata). The four task families test reading a single Idea Genome, tracing inheritance across multiple objects, explaining a transition under GenomeDiff criteria, and verifying proposed lineage claims. Scoring uses strict exact match; a lineage answer is considered unreliable if it identifies the right parent but assigns the wrong driver or object fate.

• IG-Arena: Used for pairwise battles in idea generation (details not expanded here).

• Data construction and processing Construction proceeds in four stages:

1. Seed collection – experts nominate landmark and frontier papers per domain.
2. Trace expansion – candidate predecessors and successors are added through citation links, semantic search, and domain curation, yielding lineage traces of 3–7 papers.
3. Genome extraction and diff alignment – multi-pass LLM-assisted extraction converts each paper into typed Idea Genome objects, which experts audit. Pairwise GenomeDiff records add object alignment, fate annotation, and driver labels.
4. Benchmark-level audit – programmatic checks and held-out annotators verify schema validity, answer contracts, temporal consistency, anonymization leakage, and trace coherence.

Quality assurance involved 50 graduate annotators who validated GenomeDiff labels, IG-Exam difficulty through stratified human solving, and IG-Arena battles. Inter-annotator agreement on dynamics labels reached 84.7% before adjudication.

• How the paper uses the data IG-Bench is used purely for evaluation. IG-Exam measures a model’s ability to understand and reason about scientific lineage in closed-form tasks; IG-Arena evaluates open-ended idea generation grounded in lineage. No training split is derived from the benchmark; it serves as a reusable artifact to test lineage understanding and generation capabilities.

Method

The authors introduce IdeaGene, a framework designed to make the inheritance structure of scientific work explicit and auditable. At its core, IdeaGene operationalizes the metaphor of an evolutionary lineage, enabling systematic evaluation of how ideas are inherited, mutated, lost, or recombined across papers. The framework consists of three tightly coupled components: Idea Genome objects, a genome extraction operator, and GenomeDiff alignments. Together, they transform lineage reasoning from a subjective narrative into a structured, evidence-grounded process.

The foundation of the framework is the Idea Genome, a typed, minimal, and lineage-relevant idea object extracted from a paper or proposal. Formally, a paper $p$p is represented as a set of Idea Genome objects:

$G(p) = \{g_i = (t_i, z_i, e_i, c_i)\}_{i=1}^{m_p}.$G(p)={gi​=(ti​,zi​,ei​,ci​)}i=1mp​​.

Each object $g_i$gi​ carries a functional role type $t_i \in \{\text{niche}, \text{mechanism}, \text{observation}, \text{limitation}, \text{delta}, \text{claim}\}$ti​∈{niche,mechanism,observation,limitation,delta,claim}, a content description $z_i$zi​, an evidence pointer $e_i$ei​ to the source text, and optional constraints $c_i$ci​. The role types encode the object's function in lineage reasoning: a niche defines the problem environment, a mechanism captures an inheritable method or design, an observation marks an empirical pattern, a limitation identifies a bottleneck, a delta specifies a design change relative to prior work, and a claim states the asserted outcome. This typing ensures that later alignment can distinguish, for example, an inherited mechanism from a repaired limitation.

Genome extraction is the abstraction operator $\mathcal{E}: p \mapsto G(p)$E:p↦G(p) that converts a paper into its Idea Genome. Unlike general summarization, the operator is constrained by four requirements: each object must be typed, evidence-grounded (pointing to a specific span, figure, or equation), minimally self-contained (small enough to be independently inherited or lost, yet complete enough to express a single functional idea), and lineage-relevant (its presence, absence, or modification must affect whether a successor is a coherent descendant). This disciplined extraction produces an auditable representation that directly supports downstream alignment.

Given a predecessor $p_s$ps​ and a successor $p_t$pt​, a GenomeDiff $\Delta_{s\to t}$Δs→t​ aligns the Idea Genome objects of $G(p_s)$G(ps​) to those of $G(p_t)$G(pt​) by type and semantic role. Source objects are classified as INHERITED, MUTATED, or LOST, while unaligned target objects are labeled NOVEL or EXTERNAL. Each alignment record also captures the primary transition driver, the relation to the surrounding task or domain setting, and an evidence-backed rationale. The authors construct these records through LLM-assisted extraction followed by expert audit, ensuring reliability.

A critical distinction in the alignment is between lineage and co-location. Shared benchmarks, datasets, or community conventions form an EcologyContext that explains why papers occupy the same research environment, but they do not by themselves establish descent. Genome continuity—the inheritance of mechanism-level objects—is what constitutes a lineage claim. Shared setting without driver inheritance is treated as niche competition; inherited mechanisms moving into a new setting are treated as adaptive radiation.

Finally, the framework classifies GenomeDiff patterns into operational evolutionary dynamics. The primary check is driver inheritance: without evidence for it, the relation is considered co-located rather than lineal. When evidence is present, ambiguous cases are resolved by a fixed priority rule: Hybridization before Speciation, Speciation before Niche Competition, and Adaptive Radiation before Mutation when the setting shift is the driver. These categories provide a consistent vocabulary for evaluating scientific lineages without claiming to exhaust every possible pattern of development.

Experiment

The IG-Bench benchmark evaluates AI scientists through IG-Exam (closed-form lineage understanding) and IG-Arena (lineage-grounded generation), using controlled information settings to isolate parametric ideation, paper-level context, and structured lineage context. IG-Exam reveals that lineage reasoning is compositionally difficult, with accuracy declining sharply from single-genome reading to full lineage verification, and tool scaffolds aid information gathering but not compositional consistency. IG-Arena shows that providing structured lineage improves proposal quality primarily by increasing heredity (grounding in parent mechanisms), while variation remains high across settings, and systems often produce plausible but lineage-incoherent ideas. Overall, the experiments indicate that paper-level evidence alone is insufficient, verification capability is tightly linked to generation quality, and current models overproduce surface plausibility relative to true lineage coherence.

IG-Bench is the only evaluation paradigm that represents ideas as an explicit Idea Genome, tracks evolutionary differences, and directly measures both understanding and generation. Prior paradigms operate on paper-level summaries, embeddings, or workflow traces, lacking a structured unit of idea inheritance. This gap motivates a benchmark that can distinguish lineage-grounded coherence from surface plausibility. IG-Bench uniquely provides an Idea Genome unit and explicit GenomeDiff tracking, unlike literature-grounded ideation (partial, implicit) and automated research agents (no fixed unit). Only IG-Bench evaluates both understanding and generation directly; other paradigms assess only factual or structural understanding, or generation without lineage grounding. Paper-level evidence and unordered summaries are insufficient to capture which idea components are inherited, mutated, or lost—lineage setting with GenomeDiff adds essential signal. Automated research agents lack a fixed unit of analysis and do not explicitly model differences, making their generation evaluation indirect.

Idea evolution can be classified into six dynamics based on whether a driver mechanism is inherited and whether the niche is shared. Lineage-grounded recombination, verified through GenomeDiff criteria, yields coherent population insertion, while current systems often produce plausible but lineage-incoherent ideas that lack proper inheritance of parent mechanisms. Adaptive radiation preserves a driver mechanism but applies it to a new task or domain, as seen when self-attention moved from token sequences to image patches. Hybridization combines driver objects from two distinct lineages, such as merging a CLIP-style visual encoder with an instruction-tuned language model. Lineage-setting Heredity strongly drives Population Insertion Score (PES), while Variation remains high across all dynamics, indicating that plausibility often outpaces coherent inheritance. Verification tasks that require identifying parent mechanisms and compatible Idea Genome objects are the hardest for systems, bridging understanding and generation quality. Retrieval-heavy research agents add little lineage-reasoning capability over a direct LLM, and multi-step pipelines can even reduce generation coherence.

The IG-Exam benchmark spans 42 task types and 1,029 instances across four capability axes: Genome Abstraction, Inheritance Tracing, Evolutionary Reasoning, and Lineage Verification. Lineage-grounded evaluation reveals that systems often produce plausible but lineage-incoherent proposals, with Heredity driving the gap in population insertion scores. Verification tasks prove hardest, and retrieval-heavy tool scaffolds amplify retrieval-dependent capabilities but leave compositional reasoning largely unchanged. T4 Lineage Verification tasks are the most challenging, requiring detection of intruders, wrong steps, missing links, citation conflicts, or parent mismatches. Lineage-conditioned PES gains, especially in Heredity, show that GenomeDiff structure carries signal beyond unordered paper summaries.

Lineage understanding remains extremely difficult for current systems, with the best direct model reaching only modest exact accuracy and tool-augmented harnesses providing limited gains. Improvements from tool scaffolding concentrate on retrieval-heavy subtasks like inheritance tracing, while compositional reasoning and verification—the capabilities most critical for generation—see little to no benefit. In open-ended generation, systems tend to produce plausible but lineage-incoherent ideas, as high variation persists while heredity drives the quality gap. The strongest direct LLM achieves 23.1% exact accuracy on IG-Exam, and the best tool-augmented harness reaches only 27.3%. Performance declines steeply from genome abstraction (T1) to lineage verification (T4), mirroring the added compositional burden. CLI harnesses substantially improve T2 Inheritance Tracing but show negligible gains on T3 Evolutionary Reasoning and T4 Lineage Verification. Research agents with retrieval-heavy workflows perform similarly to the base LLM, adding little lineage-reasoning capability. In generation, Heredity drives the PES gap while Variation stays high, indicating an overproduction of plausibility without lineage coherence.

IG-Bench evaluates idea evolution by representing ideas as an explicit Idea Genome and tracking differences via GenomeDiff, enabling direct measurement of both lineage-grounded understanding and generation. Experiments classify idea dynamics such as adaptive radiation and hybridization, and show that current systems often produce plausible but lineage-incoherent outputs, with Heredity driving the quality gap while Variation remains high. The IG-Exam benchmark spans four capability axes from genome abstraction to lineage verification, revealing that tool-augmented harnesses improve retrieval-heavy tasks but offer negligible gains on compositional reasoning and verification, and that the hardest verification tasks bridge understanding and generation quality. Overall, lineage understanding remains extremely challenging, and retrieval-heavy research agents add little lineage-reasoning capability beyond a direct LLM.