One-sentence Summary

Researchers from Nanyang Technological University, Microsoft Research, and collaborators present ResearchStudio-Idea, a skill suite that distills 1,947 ICLR/ICML/NeurIPS papers (2021–2025) into 15 reusable ideation patterns, and IdeaSpark, its end-to-end skill that grounds problem-solving in evidence, retrieves conflicts, and audits proposals, with blind automated-judge evaluations showing it produces stronger, traceable research proposals than no-skill and generic-skill baselines while maintaining competitive novelty.

Key Contributions

• An analysis of 1,947 ICLR, ICML, and NeurIPS papers (2021–2025) distills 31 recurring ideation sub-patterns into 15 structured, reusable pattern cards that capture research contexts, bottleneck types, differentiation strategies, supporting precedents, and common failure modes.
• ResearchStudio-Idea is introduced as a skill suite composed of Paper-Search (multi-source literature search), Scoop-Check (prior-art collision checking), and IdeaSpark, which combines evidence grounding, pattern-guided generation, collision retrieval, and audit to produce traceable, auditable idea cards.
• Blind automated-judge evaluations show that IdeaSpark generates research proposals with higher judged quality than no-skill and generic-skill baselines while maintaining competitive novelty, indicating that patterns from large-scale conference outcomes can be operationalized for evidence-grounded ideation.

Introduction

LLM-based research agents can now retrieve literature, propose hypotheses, and draft reports, but the critical bottleneck has shifted to the early stage: organizing evidence into reusable skills that ground a defensible direction and audit its novelty before experiments begin. Prior work either automates the full research lifecycle, searches large candidate spaces with learned novelty signals, or induces patterns only from accepted papers, yet these approaches often lack a structured, outcome-grounded layer that connects literature gaps to actionable research strategies and exposes how similar attempts fail. The authors introduce ResearchStudio-Idea, a suite of three reusable skills: Paper-Search for multi-source literature grounding, Scoop-Check for claim-level prior-art collision checking, and IdeaSpark, which mines 1,947 papers from ICLR, ICML, and NeurIPS to induce ideation patterns from the contrast between accepted, high-citation, and rejected work and packages them as operational cards that guide both idea generation and audit.

Dataset

The authors construct a dataset of 1,947 machine learning papers from ICLR, ICML, and NeurIPS (2021–2025), collected via the OpenReview API and augmented with Semantic Scholar citation counts. Each paper carries title, abstract, author list, OpenReview id, decision, available review fields, meta-review (where present), and citation count. A paper may receive one or more of three labels designed to operationalize key contrasts in the work:

• Oral (1,014 papers): papers with a venue decision of Oral, marking program-committee preference.
• High-Cited (HC) (260 papers): the top‑30 most‑cited papers per venue-year (top‑10 for 2025), representing community adoption. The full set HC_all contains 49 papers that are also Oral, and a disjoint subset HC_disjoint (211 papers) is used when analyses require mutually exclusive Oral, HC, and Reject classes.
• Reject (722 papers): papers with an explicit Reject decision and a non‑trivial review set, providing a concrete failure signal.

After merging and deduplicating by OpenReview submission id, the dataset contains 1,947 unique papers. Metadata coverage varies: 1,891 papers have non‑empty abstracted fields (the abstract, introduction, and reviews) and are used for embedding; 1,074 carry a full abstract, 716 have parsed reviews, 682 have an extracted introduction, and 340 include a meta‑review. For card generation, a separate review‑recovery pass pulls review evidence from public OpenReview threads, and papers without recoverable review text are still included in signature extraction, with their review‑derived fields treated as low‑evidence auxiliary signals.

Processing steps include extracting the introduction from paper PDFs, parsing review threads, and deduplicating by OpenReview id. The 56 papers that lack non‑empty abstracted fields are dropped from the embedding pool. Author identities are retained but are not used as features. The dataset is further enriched by an LLM‑based domain induction pipeline: an LLM extracts research‑subarea tags from each paper’s title, abstract, and author‑supplied keywords; the tags are embedded and clustered, and a second LLM induces a 28‑domain taxonomy. The resulting per‑paper domain assignments cover 98.5% of papers.

The dataset is used to ground the extraction of strategy signatures, clustering, and pattern‑card generation. The three labels serve as the foundation for analyses that contrast committee preference, community adoption, and rejection. Internally, papers are referenced via identifiers like “ICLR 2022 0094,” which resolve to the dataset’s openreview id field.

Method

The authors convert each paper into a form suitable for strategy-level clustering by extracting a twelve-field innovation signature through a two-stage process. In Stage 1, they feed the paper's title, abstract, introduction, reviews, and meta-review into Claude Sonnet 4.6 to extract eight base fields. These include the innovation approach, key step, why non-obvious, trigger condition, reviewer praise, reviewer concern, acceptance signal, and contribution type. The first four fields are the most strategy-bearing. In Stage 2, a second pass rewrites these four fields into domain-agnostic equivalents. The prompt instructs the model to replace domain nouns with generic placeholders, adopt imperative phrasing, preserve the mechanism, and drop application detail. This abstraction prevents embeddings from latching onto domain-specific nouns and ensures the resulting clusters reflect reasoning strategies rather than topics.

For each paper, the four abstracted strategy fields are concatenated and embedded using OpenAI text-embedding-3-large (3,072 dimensions), followed by L2 normalization. The authors then apply UMAP to reduce the dimensionality to 10, followed by HDBSCAN clustering. They select a minimum cluster size of 10, which identifies 31 ideation sub-patterns. Each cluster is automatically labeled by an Opus 4.7 pass that reads the eight centroid-nearest representative papers to produce a label and a one-sentence description. A second pass generates a level-2 disambiguation card for each cluster, detailing the tactical pattern, step-by-step recipe, differentiation within the parent pattern, trigger conditions, tactical failure modes, and paper-agnostic examples.

To induce a higher-level taxonomy, the authors ask Claude Opus 4.7 to build a Level-1 ideation-pattern taxonomy from the 31 clusters, resulting in 15 distinct ideation patterns. Each pattern includes a definition, operational signature, and when-to-apply clause. The 15 patterns are not mutually exclusive; papers often execute several operators in composition.

The coverage and granularity of this hierarchy are visualized in the figure below.

The inner ring shows the 15 ideation patterns sized by paper count, the middle ring subdivides each pattern into its 31 constituent sub-clusters, and the outer band encodes each pattern's Oral acceptance rate. The distribution exhibits a long tail, with top patterns like "Audit and Pivot an Assumption" accounting for many papers, while others represent rare but coherent operators.

To understand why certain ideation pattern pairs combine often, the authors compute the centroid-cosine similarity between the 15 patterns in the embedding space.

The matrix reveals that ideation patterns are not orthogonal but form a dense neighborhood. Geometric proximity serves as evidence for plausible secondary pattern composition. For instance, "Audit and Pivot an Assumption" sits at high cosine similarity with "Substitute the Operator or Representation". The authors also perform paper-level multi-label tagging, finding that $k=2$k=2 is the empirical mode for the number of ideation patterns executed per paper.

The authors build IdeaSpark to turn this empirical substrate into a runnable research-ideation workflow. It is implemented as a two-tier skill: a runtime tier containing orchestration logic and prompts, and an evidence tier containing the induced pattern cards and corpus-derived failure-mode inventory. The workflow proceeds through several phases.

Phase 0 performs literature grounding by querying multiple sources to build a broad literature bundle and a full-text cache. Phase 1 identifies a literature-grounded bottleneck by arranging the neighborhood into a method-lineage tree to expose additive and subtractive gaps. Phase 2 separates pattern selection from candidate generation. The system selects the ideation pattern whose operational signature structurally closes the diagnosed gap and instantiates it as a concrete candidate mechanism.

Phase 3 acts as a quality gauntlet. It runs a focused literature search for mechanism-level collision and performs four corpus-anchored audit checks: gap-closure reject scan, recipe application, anti-pattern substantive verification, and paper-pointed threat. The audit returns a verdict to advance, revise, or abandon the candidate. Finally, Phase 4 expands the candidate into a final idea-card artifact bundle, runs an implementability audit, renders the output, and applies deterministic validators to ensure kill-switch integrity and expansion completeness. The system enforces faithfulness by grounding every load-bearing claim in retrieved records and using deterministic gates to prevent hallucination.

Experiment

The evaluation analyzes a corpus of 1,947 ML papers to extract 15 ideation patterns, revealing that acceptance hinges on execution quality at the sub-pattern level rather than on strategy choice, and that patterns like “Characterize a Limit, Then Surpass It” are robustly Oral across domains while others are domain-conditional. Ablations confirm that the pipeline’s domain-agnostic rewrites and paraphrase encoder better capture strategy similarity than topic-based alternatives, and a blind evaluation of IdeaSpark shows that its corpus-grounded workflow yields the highest quality ideas, while novelty alone is misleading due to a “novel-but-empty” failure mode.

The dataset contains 1,947 unique papers from ICLR, ICML, and NeurIPS (2021–2025), each labeled as Oral, High-Cited, or Reject, with 49 papers overlapping Oral and High-Cited. Mutually exclusive pools used in analysis are 1,014 Oral, 211 High-Cited (disjoint), and 722 Reject, enabling contrast between program-committee preference, community adoption, and failure signals. Oral and High-Cited labels overlap in only 49 papers, yielding a disjoint High-Cited pool of 211 after removing those overlaps. The dataset includes 722 Reject papers, providing a failure signal absent from accepted-only corpora. High-Cited counts are based on top-30 citations per venue-year, except for 2025 where only top-10 are used due to accumulating citations.

The HDBSCAN sweep over min cluster size shows that the smallest setting (10) yields the highest silhouette score (0.584) and the largest number of clusters (31), while leaving 47.7% of papers unclustered. Coarser settings produce fewer clusters and lower silhouette scores, but the unclustered fraction remains consistently high (40–49%), indicating these points are genuinely distant from density peaks. The unclustered label reflects geometric position between modal clusters, not absence of strategy content. The smallest min cluster size (10) achieves the strongest silhouette (0.584) and the most clusters (31), while coarser settings reduce both cluster count and silhouette score. The unclustered fraction stays persistently between 40% and 49% across all min cluster sizes, never dropping below 39.7%, showing these points are not fine-granularity artifacts. Increasing min cluster size from 10 to 40 cuts the number of clusters roughly in half (from 31 to 6) and lowers silhouette from 0.584 to 0.474.

A taxonomy of 15 reusable ideation patterns was induced from a large corpus of machine learning papers. The most frequent pattern, auditing and pivoting an assumption, dominates accepted papers, while other patterns such as unifying heterogeneous inputs show a more balanced mix of acceptance and high citation. When embedded in a structured workflow, these patterns substantially lift idea quality, but bare models can produce vacuous novelty, underscoring the need to evaluate both quality and novelty together. Audit and Pivot an Assumption is the most prevalent pattern among accepted papers, with a high oral count but relatively few highly cited papers, whereas Unify Heterogeneous Inputs into One Space has a nearly equal number of oral and highly cited papers, suggesting different patterns correlate with different forms of impact. IdeaSpark's quality advantage is driven by its corpus-grounded pattern cards and multi-phase workflow, not by the model backbone or retrieval alone, as shown by the same-backbone ladder where a generic auto-authored skill performed no better than the bare baseline.

In the embedding space, ideation patterns with high cosine similarity are more likely to be combined in oral presentations, as seen in the top three Oral-enriched pairs. The two most isolated patterns are Design a Confound-Isolating Diagnostic and Characterize a Limit, Then Surpass It, forming their own pockets not explained by sample size alone. Three patterns—Audit and Pivot an Assumption, Reframe as a Solvable Object, and Substitute the Operator or Representation—act as hubs, each serving as the nearest neighbor for three other patterns. Adjacent patterns in the embedding are often paired in practice: the top three Oral-enriched combinations all involve high-cosine neighbors like Audit and Pivot with Structural Prior Encoding or Operator Substitution. Design a Confound-Isolating Diagnostic is the most isolated pattern (lowest nearest-neighbor cosine) despite moderate support (n=86), because its confound-isolating measurement instrument signature does not compose in a standard way with other patterns.

Across all acceptance decisions, two ideation patterns is the most common count, with 59.2% of papers executing exactly two patterns. Only 7.2% of papers use a single pattern, while 33.6% apply three or more, revealing a substantial multi-pattern tail. The mean number of patterns is tightly clustered around 2.3 for Oral, Highly-Cited, and Reject papers, indicating that the sheer count of patterns does not distinguish acceptance outcomes. The mode of k is 2 for every acceptance class, with 59.2% of all papers executing exactly two ideation patterns. Over one-third of papers (33.6%) apply three or more patterns, a tail that the cluster-based approach (capped at k≤2) underrepresents.

Experiments draw on a dataset of 1,947 papers with acceptance and citation labels to contrast program-committee preference, community adoption, and failure signals. Clustering reveals that nearly half of papers lie between modal clusters, suggesting a continuous pattern space rather than discrete categories, while a taxonomy of 15 ideation patterns shows that the type of pattern, not the count, distinguishes accepted work, with certain hub patterns frequently co-occurring and others remaining isolated. Embedding these patterns into a structured workflow (IdeaSpark) substantially lifts idea quality, whereas bare models generate vacuous novelty, underscoring that grounded pattern use is critical.