Patients are increasingly turning to AI chatbots like ChatGPT for medical information, but a new study highlights the need for caution, especially in complex and rapidly evolving fields like blood cancer. Published September 3 in Future Science OA, the research evaluates the accuracy of ChatGPT 3.5—available at the time of the study in July 2024—on 10 real-world medical questions relevant to hematology-oncology. Four anonymous hematology-oncology physicians assessed the chatbot’s responses using a 5-point scale, where 1 meant "strongly disagree" and 5 meant "strongly agree." A score of 3 indicated a neutral response—neither clearly accurate nor inaccurate, but often incomplete or ambiguous. ChatGPT 3.5 performed best on general questions, such as those about common chemotherapy side effects and their management, earning an average score of 3.38. However, it struggled significantly with more specific and up-to-date topics, particularly newer therapies. For questions about emerging treatments like BCL-2 inhibitors, the average score dropped to 3.06. Notably, no answer received a perfect score of 5 from any reviewer. The study’s senior author, Dr. Justin Taylor of Sylvester Comprehensive Cancer Center at the University of Miami Miller School of Medicine, emphasized that while AI tools can be helpful, they should not replace professional medical advice. “I would warn patients to have some skepticism, especially about answers dealing with specific types of cancer and treatments, and always check with their doctor,” he said. Taylor noted that the limitations of ChatGPT 3.5 stem from its training data, which was cut off in 2021. This means it lacks knowledge of recent clinical advances, drug approvals, and evolving treatment guidelines—critical in a field where new therapies emerge frequently. Despite its shortcomings, the study acknowledges potential benefits. AI tools can help patients prepare for doctor visits by generating informed questions or guiding them toward reliable sources of information. They may also assist in reducing administrative burdens, as AI is already being used at Sylvester and the Miller School to summarize patient visits and streamline documentation. The university has also launched initiatives to integrate AI into medical education, including an elective course on AI for medical students and a self-paced course on AI ethics for Spanish-speaking healthcare professionals in Latin America. While this study focused on ChatGPT 3.5, Taylor anticipates that newer versions of the model will improve over time. He plans to re-evaluate performance with future iterations of the technology within the next one to two years. The research team included first author Tiffany Nong from Florida State University College of Medicine, along with collaborators from the University of Vermont and Florida Cancer Specialists. The findings underscore a key message: AI can be a useful supplement, but physician oversight remains essential for ensuring patient safety and accurate care in complex medical fields.