One-sentence Summary

Researchers from Yale University and the University of Chicago introduce a large-scale evaluation framework that reverse-engineers the prior works inspiring a paper, prompts LLMs to generate ideas from those priors, and uses a two-axis research-taste taxonomy to quantify the divergence, revealing that LLM ideas disproportionately concentrate on bridge-like opportunities and synthesis methods, yielding a narrower and systematically shifted distribution compared to human research taste.

Key Contributions

• An evaluation framework reverse-engineers related-work contexts from human research papers, enabling controlled comparison of human and LLM-generated ideas under shared literature-grounded inputs.
• A two-axis taxonomy profiles each idea by its opportunity pattern and research paradigm, quantifying distributional divergence between human and LLM ideation.
• Across model families and scientific domains, LLM-generated ideas concentrate disproportionately on bridge-like opportunities and synthesis-oriented methods. Human ideas span a wider range of opportunity patterns and paradigms, revealing a narrower and systematically shifted research taste.

Introduction

Large language models are now widely used to generate research ideas, yet the paper lack a systematic understanding of the types of ideas they produce compared to human researchers. Prior evaluations judge ideas individually on novelty or feasibility, but they do not capture whether an LLM’s outputs repeatedly favor the same narrow set of problem framings and contribution strategies. The authors address this gap by introducing a distributional view of research taste and a two-axis taxonomy that characterizes how an idea frames its research opportunity and what style of intellectual contribution it proposes. Through a controlled literature-grounded ideation task, they find that LLM-generated ideas concentrate heavily on bridging gaps and synthesizing existing work, while human ideas span a much wider range of opportunity patterns and methodological paradigms.

Dataset

The authors construct an evaluation corpus for research ideation, consisting of two subsets: Human Idea and LLM Idea.

• Sources and composition:

• Human ideas are extracted from published research papers in machine learning (ICLR, ICML, NeurIPS, 2023–2026) and Nature Communications (2023–2025), covering 71 scientific disciplines.

• For each paper, the original author’s idea is taken as the human endpoint. An LLM-assisted pipeline extracts a structured representation containing the innovation, departure from prior work, and key insight, then rewrites it into a proposal-style motivation and method.

• To form the input context, 4–8 highly relevant prior studies are reverse-engineered from the extracted idea and the paper’s related-work section. Only the titles and abstracts of these prior works are retained as input summaries.

• After filtering, the human-idea subset contains 11,683 valid examples.

• The LLM-idea subset is generated by prompting various LLMs (Claude, Gemini, GPT, DeepSeek, Qwen) with the same prior-work input to produce a new idea in the identical structured format (motivation and method).

• Usage in the paper:

• The corpus serves as an evaluation benchmark for ideation. The human-authored ideas act as reference targets, while the LLM-generated ideas are the outputs under assessment.

• No training split is applied; the data is used exclusively for comparing the quality of LLM-generated research ideas against human-written ones.

• Processing details:

• The extraction prompt asks for the paper’s innovation, departure from prior work, and key insight, then reformulates the result into a motivation-and-method proposal.

• Prior studies are identified through a reverse-engineering step that uses the extracted idea and the paper’s related-work section; only their titles and abstracts are kept, discarding full texts.

• The final input for both human and LLM ideas consists solely of these prior-work summaries, ensuring a consistent evaluation setting.

Experiment

The evaluation framework anchors both human and LLM ideas to the same prior literature, then labels each idea with a research-taste taxonomy covering opportunity patterns and method paradigms. Experiments reveal a consistent distributional gap: LLMs heavily favor bridge-and-synthesis ideas, while human researchers distribute more evenly across explanation, measurement, risk, and artifact contributions. Extended reasoning amplifies this template, and mechanism analyses show that LLMs repeatedly select high-frequency technical concepts and wrap them in safe integration operations, whereas human ideas more often replace, decouple, or formalize local mechanisms.

LLM-generated research ideas are distributionally far from human ideas, with a pronounced shift toward bridge-and-synthesis patterns. Models show lower entropy on both taxonomy axes, especially on the opportunity dimension, and even the best-matching model requires moving over a third of the probability mass to match the human distribution. This gap reflects a tendency for LLMs to frame contributions as connecting or unifying prior work, while human ideas distribute more evenly across explanation, measurement, optimization, and other contribution types. All tested LLMs produce more concentrated distributions than the human reference, with opportunity-pattern entropy ranging from 0.550 to 0.758 compared to 0.926 for humans. Bridge-and-synthesis opportunities account for only 12.1% of human ideas but 47.1–64.2% of LLM ideas, and synthesis/unification methods are 5.1% vs. 22.5–38.7%. The closest model on the opportunity axis, Gemini-3.1-Pro, has a total variation distance of 0.348, meaning over one-third of the distributional mass would need to shift to match human outputs. Richer full-paper context does not reduce the gap; both TVD and JSD increase for the two models tested, while entropy decreases or stays flat. Diagnostic scores reveal that LLM ideas tend to be more boilerplate and less specific, with Qwen models showing the highest surface-stitching flags, while Claude-Sonnet-4.6 achieves slightly higher bottleneck specificity than humans but remains distributionally distant.

Most model-generated proposals exhibit lower bottleneck specificity and higher boilerplate than human ideas, reflecting less precise mechanisms and more generic phrasing. The Qwen model shows the clearest degradation, with the highest surface-stitching score and flag rate, while Claude-Sonnet-4.6 is the sole model to slightly surpass the human baseline on both bottleneck specificity and boilerplate. Human proposals achieve a surface-stitching score of 0.00 and a flag rate of 0.0%, whereas Qwen3-8B reaches 0.58 and 20.6%, the worst among all sources. Claude-Sonnet-4.6 attains a higher bottleneck specificity (2.60) and lower boilerplate (0.37) than the human baseline (2.56 and 0.48), making it the only model to match or exceed human performance on these dimensions. All other models, including Gemini-3.1-Pro and GPT-5.4-mini, fall below human levels on bottleneck specificity and boilerplate, with GPT-OSS-20B recording the lowest specificity (2.07) among non-Qwen systems.

Replacing paper abstracts with model-generated full-paper summaries does not bring LLM idea distributions closer to human ones; total variation and Jensen-Shannon divergence increase on both opportunity and method paradigm axes, while normalized entropy drops. The gap widens rather than shrinks, and the same qualitative pattern of over-representing bridge-and-synthesis ideas persists. For both Qwen3-8B and DeepSeek-V4-Flash, TVD and JSD rise on both taxonomy axes when switching from abstract to full-paper context, indicating a larger divergence from the human reference distribution. Normalized entropy decreases under full context, showing that richer document information makes model outputs more concentrated, not more diverse.

Enabling thinking mode consistently shifts model outputs toward a more concentrated and less human-like idea distribution. Both Qwen3-8B and DeepSeek-V4-Flash exhibit higher bridge and synthesis template mass, increased total variation distance from human references, and lower entropy when reasoning is activated, indicating that extended reasoning sharpens the model's preferred ideation template rather than broadening its creative range. Adding reasoning increases bridge template mass by over 20 percentage points for Qwen3-8B and nearly 7 points for DeepSeek-V4-Flash, while synthesis mass rises similarly, revealing a stronger reliance on integration archetypes. Opportunity entropy drops substantially with thinking (e.g., from 0.658 to 0.481 for Qwen3-8B), and total variation distance from human distributions increases, confirming a sharper and less human-aligned idea distribution.

The evaluation framework compares LLM-generated research proposals to human ideas along taxonomy axes of opportunity and method paradigms, using divergence metrics and diagnostic scores for boilerplate, specificity, and surface-stitching. LLMs consistently over-represent bridge-and-synthesis patterns, leading to concentrated distributions that diverge considerably from human diversity; providing full-paper context or enabling thinking mode widens rather than narrows this gap. Although Claude-Sonnet-4.6 marginally outperforms the human baseline on bottleneck specificity and boilerplate, all models produce more generic, less diverse ideas and remain far from the human distribution in terms of contribution types.