Generative AI Shows Diagnostic Accuracy on Par with Non-Specialist Doctors, Study Reveals

Generative AI Matches Non-Specialist Doctors in Diagnostic Accuracy, Study Finds Credit: CC0 Public Domain The potential of generative AI in medical diagnostics has garnered significant interest, with numerous research papers exploring its capabilities. However, due to the varying evaluation criteria used in these studies, a comprehensive analysis was necessary to determine the practical utility of AI in real-world medical settings and to compare its performance with that of human doctors. A research team led by Dr. Hirotaka Takita and Associate Professor Daiju Ueda from Osaka Metropolitan University's Graduate School of Medicine conducted a meta-analysis of 83 research papers published between June 2018 and June 2024. These papers covered a broad spectrum of medical specialties and primarily focused on large language models (LLMs), with ChatGPT being the most frequently studied. The results, published in npj Digital Medicine, provide valuable insights into the diagnostic capabilities of generative AI. Medical specialists, on average, demonstrated a 15.8% higher diagnostic accuracy compared to generative AI. However, generative AI had an average diagnostic accuracy of 52.1%, and the most recent models often matched the performance of non-specialist doctors. Dr. Takita commented, "This research highlights that generative AI's diagnostic capabilities are on par with those of non-specialist doctors. It has the potential to support medical education and assist in diagnostics, particularly in regions with limited medical resources." Despite these promising findings, Dr. Takita emphasized the need for further research to fully validate AI's diagnostic abilities. Specifically, future studies should evaluate AI's performance in more complex clinical scenarios, using actual medical records, improving the transparency of AI decision-making, and verifying its effectiveness across diverse patient populations. The potential of generative AI in medicine is substantial, and this study provides a critical first step in understanding its role and limitations. As the technology continues to evolve, it may offer significant benefits in enhancing medical care and education, particularly in underserved areas.