One-sentence Summary

A team from Princeton University, Cornflower Labs, the UK AI Security Institute, and other institutions propose shadow evaluations, in which frontier AI agents attempt to solve the central research question of an unpublished paper, and find that despite completing engineering tasks autonomously, the agents fail to make substantial research progress due to five recurring failure modes, indicating current agents cannot yet conduct open-ended AI research.

Key Contributions

• The paper introduces shadow evaluations, a new evaluation paradigm where an agent tackles the open-ended research questions of a high-quality unpublished paper and the original authors grade its output.
• Applying shadow evaluations to two NeurIPS 2026 submissions, frontier agents autonomously completed all engineering tasks but failed to make substantial progress on the core research questions, resulting in unambiguous rejection by the authors.
• The work identifies five recurring failure modes (poor judgment of publishability, uncreative responses to design flaws, ineffective backtracking, poor resource awareness, and instruction drift), replicates them with a second model and scaffold, and releases all artifacts including expert reviews, agent repositories, and logs.

Introduction

The question of whether AI agents can autonomously conduct AI research is central to forecasts of rapid self-improvement, yet existing evaluations focus narrowly on verifiable tasks with fixed metrics or rely on noisy blind peer review, leaving open-ended research skills largely unmeasured. The authors introduce shadow evaluation, a method in which a frontier agent is given the research question from a high-quality unpublished paper, along with generous compute and time budgets, and the original authors grade the resulting paper as conference reviewers. Applying this method to two NeurIPS 2026 papers, they find that even well-resourced agents fail to produce original work at a top-tier conference level, exhibiting poor judgment, limited awareness of resources, inability to creatively respond to feedback, and failure to backtrack from unpromising directions.

Method

The authors introduce a new evaluation protocol called shadow evaluations, designed to measure the research capabilities of AI agents on open-ended tasks with expert-level scrutiny. In contrast to automatic verifier-based benchmarks and blind peer review, shadow evaluations task an agent with answering a real, unpublished research question, then have the original authors of that question grade the agent’s output as if it were a conference submission. This design avoids the contamination risk of memorized training data, enables deep qualitative assessment of novelty and correctness, and mirrors the delegation scenario that many forecasts of recursive self-improvement envision.

The core workflow begins by selecting a research question from a paper that is not yet public. For the experiments, the authors used two such questions: one on the structure and controllability of LLM personas, and one on designing a distribution shift detector for tabular foundation models. The original researchers formulated the question for the agent without hinting at promising directions, set resource budgets sufficient to address the problem substantively, and later graded the resulting paper against the standards of top-tier AI venues. This setup positions the original authors as uniquely qualified reviewers, combining deep topic expertise with the ability to judge whether the agent made genuine progress on the same question they had spent months investigating.

The agent setup was kept as generic as possible to avoid overfitting to the specific tasks. The experiments ran Claude Opus 4.8 with extra-high reasoning on the OpenClaw scaffold. The agent received full access to a Linux virtual machine, could monitor its own API spend, compute budget, and remaining time, and was allowed to delegate work to subagents and maintain a running research log. A self-review subagent, restricted to viewing only the finished PDF and a NeurIPS review template, was instructed to evaluate the paper like a qualified referee. The agent was also pointed to three external AI reviewing tools for additional feedback. The scaffold was modified only for general machine-learning research usability, and every human intervention was documented. During the runs, only three interventions were required: a bug fix in the harness, a 24-hour deadline extension to let the agent iterate on a draft it had already identified as weak, and a request to rewrite the paper for readability. No intervention was specific to either research question.

This methodology addresses several weaknesses of prior evaluation approaches. Because the findings are not available online, the evaluation is uncontaminated. The in-depth, expert-led grading captures dimensions of research quality that automatic metrics or overloaded conference reviewers often miss. The design is repeatable: new unpublished papers can serve as future test cases, and the same template can be reused with stronger models and scaffolds. While the approach has limitations—non-blind reviewing may introduce bias, and each evaluation requires substantial expert time, constraining scale—the authors argue that shadow evaluations complement verifiable benchmarks and blind review, offering a distinct lens on AI R&D automation.

Experiment

The study evaluates whether frontier AI agents can autonomously produce novel AI research by using shadow evaluations: agents were given the same unpublished research questions as two real papers, and the original authors reviewed the output. In experiments with Claude Opus 4.8 and a robustness test with GPT-5.6 Sol, agents failed to generate papers meeting top conference standards, exhibiting premature commitment to weak hypotheses, inability to effectively incorporate feedback, and poor calibration of research quality, despite successfully managing engineering tasks like GPU debugging and large-scale experiments. The results indicate that while current AI agents can handle the engineering prerequisites of AI R&D, they lack the creative judgment, effective backtracking, and context awareness needed for genuine research contributions.

Expert reviewers evaluated two AI-generated research papers and found both to be unambiguous rejections, with low scores across quality, clarity, significance, and overall assessment. The reviews highlighted unprincipled experiment choices, poor reasoning, and dense writing, indicating that current agents struggle to produce coherent and well-justified research in open-ended AI R&D tasks. Both papers were rated as unambiguous rejections, with reviewers expressing high confidence in their judgments. The evaluations pointed to unprincipled experiment choices, weak justification over prior work, and dense, unclear writing as key weaknesses.

Previous work on evaluating AI research and development largely splits into automatically verifiable tasks scored by a fixed metric, and open-ended tasks judged by human peer review. Verifiable benchmarks like MLE-Bench and RE-Bench offer objective, repeatable evaluation but are limited to tasks reducible to a single number. In contrast, blind review of fully autonomous research, while allowing open-ended work, suffers from high stochasticity and cannot reliably distinguish genuine progress from lucky submissions. Most AI R&D evaluations use either automatic verifiers on narrow metrics or human blind review on open-ended research. Automatically verifiable tasks are objective and cheap to scale, but they restrict evaluation to problems with a single numeric score. Peer review at AI conferences is highly variable, with half of score variation being subjective and committees disagreeing on about a quarter of papers. Submitting many AI-generated papers and reporting only acceptances can inflate apparent success, as blind review does not require disclosing failed attempts.

Shadow evaluations assess AI agents on open-ended research questions using expert grading and uncontaminated tasks, but they are limited by small sample sizes, non-blind reviewing, and narrow question selection, which constrain the generalizability of findings. Agents were tested on unpublished research questions, preventing memorization from training data. Expert reviewers knew the papers were AI-generated, introducing potential bias from non-blind reviewing.

The agents' final self-reviews identified many of the same limitations as human experts, but they did not prioritize the most severe flaws. Human experts highlighted fundamental issues like hand-picked trait selection and unsupported generalizations, while the agents' self-reviews listed these alongside minor concerns, treating them with equal weight. This reveals a generator-verifier gap where the agents could recognize problems but could not effectively judge their relative importance. Human experts and agent self-reviews both noted the use of hand-picked, near-orthogonal traits, but the agents did not treat this as more damning than other, less critical issues. The agents' self-reviews echoed human criticisms, such as pointing out the absence of a positive control claim, yet they still did not reconsider the overall experimental approach.

Expert reviewers evaluated two AI-generated research papers in a shadow evaluation on unpublished questions to prevent memorization, though the review was non-blind. Both papers were unanimously rejected due to unprincipled experiment choices, weak justification, and dense writing, demonstrating that current agents cannot produce coherent open-ended research. The agents' self-reviews identified similar flaws but failed to prioritize the most critical issues, revealing a generator-verifier gap. These findings underscore the limitations of both AI research capabilities and existing evaluation methods, which are either too narrow or too stochastic.