One-sentence Summary

A framework proposed by Yale University and University of Chicago researchers reverse-engineers a paper’s inspirational prior works, prompts LLMs to generate ideas from them, and deploys a two-axis research-taste taxonomy to quantify the divergence, revealing that LLM-generated ideas are systematically narrower and disproportionately concentrated on bridge-like opportunities and synthesis methods compared to the broader distribution of human research ideas.

Key Contributions

• The paper introduces a large-scale evaluation framework that reverse-engineers a set of closely related prior works for each human research paper, then prompts LLMs to generate a new idea from those works, enabling controlled comparison of human and machine ideation under shared literature-grounded inputs.
• A two-axis research-taste taxonomy is proposed to profile each idea by its opportunity pattern and research paradigm, which is used to quantify the distributional divergence between human and LLM-generated ideas.
• Across multiple LLMs and scientific domains, findings show that LLM-generated ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, while human ideas span a broader range of opportunity patterns and methodological paradigms, revealing a narrower and systematically shifted ideation profile in current models.

Introduction

LLMs are increasingly used to generate research ideas, yet it remains unclear whether their outputs reflect the full diversity of human scientific reasoning. Most prior evaluations assess individual ideas on novelty or feasibility, but they overlook the possibility that an LLM might repeatedly produce the same types of problem framings and contribution strategies, even when each idea appears reasonable. The authors introduce a distributional perspective on research taste, comparing how human researchers and LLMs generate ideas under identical literature-grounded constraints. They show that current LLMs disproportionately generate integrative, synthesis-oriented proposals, while human ideas span a much broader set of opportunity patterns and methodological paradigms.

Dataset

The authors construct an Idea Corpus with two subsets designed to evaluate research ideation capabilities.

• Human Idea subset

• Sources: Research papers from ICLR, ICML, NeurIPS (2023–2026) and Nature Communications (2023–2025), covering 71 scientific disciplines.

• Extraction pipeline: An LLM-assisted process extracts the paper’s own idea (the human endpoint) into a structured representation that captures the innovation, departure from prior work, and key insight, then rewrites it as a proposal-style motivation and method.

• Context construction: For each idea, 4–8 highly relevant prior studies are reverse-engineered from the idea and the paper’s related-work section, and only their titles and abstracts are kept as input.

• Scale: 11,683 valid human ideas.

• LLM Idea subset

• Generation method: For each set of prior-work titles and abstracts from the human subset, an LLM is prompted to generate a new research idea in the same structured format (motivation synthesizing research gaps, and a high-level method).

• LLM families: Claude, Gemini, GPT, DeepSeek, and Qwen models are evaluated.

• Usage in the paper

• The corpus serves as an evaluation benchmark for idea generation. The authors compare human ideas against LLM-generated ideas using the same input context.

• No training split or mixture ratios are applied; the corpus is used purely for zero-shot evaluation.

• Processing details

• Cropping: The full paper content is reduced to only the titles and abstracts of prior works, stripping away the original paper’s text to form a level playing field.

• Metadata construction: The structured representation (motivation and method) is generated via the extraction prompt, and the prior-work list is built by combining information from the extracted idea and the paper’s related-work section.

Experiment

The evaluation anchors human and LLM research ideas to the same literature context and uses a taxonomy of opportunity patterns and method paradigms to compare their distributions. Experiments show that LLM outputs consistently concentrate on bridge-and-synthesis moves, diverging sharply from the more varied human ideas that often involve localized interventions such as replacing components or decoupling mechanisms. This gap persists under richer context and even worsens with extended reasoning, while diagnostic scores indicate that model-generated proposals tend to be less specific and more template-like. Mechanism analyses further reveal that LLMs repeatedly apply a safe integration recipe to high-frequency technical concepts, whereas human proposals more frequently modify, separate, or formalize narrower local structures.

LLM-generated ideas are substantially more concentrated than human ideas on both the opportunity and method axes, with the largest gap in how they frame why an idea is worth pursuing. Models overwhelmingly favor bridge-and-synthesis patterns, while human proposals distribute more evenly across explanation, measurement, optimization, and other contribution types, and this distributional mismatch persists even when models are given full paper context. LLMs strongly overproduce bridge-and-synthesis opportunity patterns: 47–64% of their ideas fall into this category, compared to only 12.1% of human ideas. Even the best model on the opportunity axis has a total variation distance of 0.348 from the human distribution, indicating that over a third of the probability mass would need to shift to match human idea diversity.

Model-generated proposals generally receive lower bottleneck specificity and higher boilerplate scores than human proposals, with Qwen3-8B exhibiting the strongest degradation across all dimensions. Claude-Sonnet-4.6 is an outlier, achieving slightly better bottleneck specificity and lower boilerplate than the human baseline while maintaining a negligible surface stitching score. Most model outputs have lower bottleneck specificity and higher boilerplate scores than human ideas, with Qwen3-8B showing the clearest degradation (highest surface stitching score, lowest specificity, and highest boilerplate). Claude-Sonnet-4.6 attains the best non-human diagnostic scores, slightly surpassing human proposals in bottleneck specificity and using less boilerplate, while surface stitching remains near zero.

Replacing abstracts with model-generated full-paper summaries did not reduce the distributional gap between LLM and human idea proposals; instead, total variation distance and Jensen-Shannon divergence increased on both taxonomic axes for both tested models, while normalized entropy decreased or stayed nearly flat. This indicates that richer context does not make model outputs more similar to the human reference distribution, and the pattern of overemphasizing bridge-and-synthesis ideas persists even with full-paper information. Full-paper context increased TVD and JSD on the opportunity axis for both models, with Qwen3-8B’s TVD rising from 0.376 to 0.430 and DeepSeek-V4-Flash’s from 0.368 to 0.400. On the method-paradigm axis, full-paper summaries also worsened distributional match, raising Qwen3-8B’s TVD from 0.338 to 0.400 and DeepSeek-V4-Flash’s from 0.213 to 0.236. Entropy decreased under full-paper context for both models on the opportunity axis (Qwen3-8B: -0.046, DeepSeek-V4-Flash: -0.005) and on the method axis for Qwen3-8B (-0.053), while DeepSeek-V4-Flash’s method entropy remained nearly unchanged (-0.007). The qualitative label distribution pattern remained the same: models continued to overproduce bridge-and-synthesis ideas and did not shift toward the broader human spread across explanation, measurement, and optimization-style contributions.

Enabling reasoning mode consistently increases bridge and synthesis template masses, reduces opportunity entropy, and raises total variation distance from the human distribution. This shift occurs across different model sizes, indicating that reasoning sharpens a model’s preferred ideation pattern rather than broadening its creative diversity. The result is a more concentrated and less human-like idea output. Thinking mode lifted bridge and synthesis template masses for both models, with the smaller model showing a larger absolute increase. Opportunity entropy dropped substantially, while total variation distance from the human distribution rose, making the output distribution sharper and less human-like. The same directional effect appeared in both Qwen3-8B and DeepSeek-V4-Flash, showing that reasoning sharpens ideation patterns regardless of model scale. Method-level total variation distance also increased, and method entropy decreased, further indicating a move away from human-like idea diversity.

The evaluation setup benchmarks LLM-generated research ideas against human proposals along taxonomic axes and diagnostic quality scores. The experiments reveal that LLMs heavily overproduce bridge-and-synthesis opportunity patterns, while human ideas distribute more evenly across explanation, measurement, and optimization types. Providing full-paper context or enabling reasoning mode further concentrates the model outputs and increases the distributional gap, rather than improving diversity. Diagnostic assessments show that most LLM-generated ideas have lower bottleneck specificity and higher boilerplate than human proposals, with Claude-Sonnet-4.6 as a notable exception.