AI Agents Struggle to Conduct Original Scientific Research

A recent arXiv preprint details a rigorous evaluation of frontier artificial intelligence agents tasked with conducting autonomous, open-ended scientific research. The study underscores persistent limitations in current generative models when operating without structured human oversight, challenging early predictions that AI could soon replace human researchers in exploratory discovery. For the experiment, researchers deployed state-of-the-art AI agents to investigate two previously unpublished topics: the structure and controllability of language-model personas, and the development of detectors for distribution shifts in tabular foundation models. The agents were granted a six-day window, unrestricted internet access, dedicated computing infrastructure, and approximately three thousand dollars in model-usage credits to facilitate independent experimentation and literature analysis. Upon completion, human experts evaluated the resulting manuscripts using standard conference peer-review metrics. The outcomes revealed substantial deficiencies in AI-driven scientific methodology. Both submissions received unambiguous rejection scores of two out of six and one out of six, respectively. Although the agents correctly identified the core research questions and proposed theoretical directions aligned with original human studies, their execution faltered. Experimental designs lacked rigor, and the models demonstrated poor adaptability when encountering negative feedback, frequently appending defensive caveats to existing findings rather than restructuring flawed methodologies. Time management proved equally problematic; despite ample availability, the agents utilized less than half of their allocated computational budget and rushed the writing phase, producing drafts that fell well below publication standards. These findings serve as a critical benchmark for the current trajectory of artificial intelligence in academia. While autonomous research remains out of reach, the study confirms that AI systems can still add measurable value to scientific workflows. In the immediate term, frontier models are better suited to assist researchers with data processing, literature screening, and routine computational tasks. The evaluation establishes that achieving genuine scientific autonomy will require significant architectural advances in reasoning, experimental iteration, and long-horizon planning before AI can reliably contribute to open-ended discovery.