Ai2 Study Finds LLM Novelty Judges Unreliable for Research Ideas

The Allen Institute for AI has released Old Ideas, Novel Problems, a study and benchmark assessing whether large language models can reliably judge the novelty of scientific research ideas. The release includes a benchmark, data, code, and an automated pipeline for building evaluation sets, extending Ai2’s broader work on AI systems for scientific discovery, including DataVoyager, AutoDiscovery, Theorizer and Asta. Ai2 said the key question is not only whether models can generate research ideas, but whether they can distinguish genuinely new directions from work that repackages existing research. That judgment is increasingly delegated to LLM based novelty judges, yet validation practices vary and are often tested only on human written papers rather than the AI generated ideas these systems are meant to evaluate. The study tested six judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. It found that small changes to prompts, reasoning budgets, retrieval settings or formatting can produce sharply different verdicts on the same ideas. In one case, changing GPT-5.4’s prompt on paired human and AI generated ideas reduced accuracy from 93.5 percent to 40.9 percent, below chance, and altered more than half the decisions. Prompt effects were less extreme on human authored ideas but still significant, with Claude Opus 4.6 shifting by 21 points. The benchmark used ICLR 2026 peer reviews to identify ideas with reviewer discussed evidence of novelty or lack thereof. It includes 154 high novelty human ideas and 145 lower novelty human ideas, plus a Human and Generated set pairing 154 human ideas with 154 ideas from a basic LLM generator. Expert review of 30 difficult pairs preferred all 30 human ideas. The study also found that more retrieval and reasoning did not consistently improve novelty judgments. An agentic GPT-5.6 Sol judge using high reasoning and live arXiv search scored 0.78 macro F1, behind a simpler Claude Opus 4.6 setup at 0.89 while costing about ten times more. Specialized systems performed worse than a simple prompt baseline, including Idea Novelty Checker and Sakana AI’s AI Scientist. Ai2 concluded that LLM novelty evaluation remains unreliable, especially for AI generated ideas. It urged future benchmarks to test judges on the kinds of outputs they will evaluate, measure prompt and model sensitivity, improve literature integration and evolve alongside scientific communities and model capabilities.