Experts Call for Patient-Centered AI Regulation to Prevent Discrimination and Enhance Trust in Health Care

A new commentary published in the Journal of the Royal Society of Medicine highlights significant concerns regarding current risk-based regulatory approaches to artificial intelligence (AI) in health care. These regulations, the authors argue, may insufficiently protect patients, leading to over- or undertreatment and potential discrimination against specific patient groups. The commentary, authored by Thomas Ploug, a professor of data and AI ethics at Aalborg University in Denmark, notes that while AI and machine learning systems have the potential to enhance clinical accuracy, they also pose challenges related to inaccuracy, opacity, and bias. These issues, the authors contend, are not adequately addressed by the European Union's AI Act, which was passed in 2025. Under the AI Act, medical AI is categorized as "high risk," triggering stringent controls on providers and deployers. However, the authors highlight that this risk-based framework fails to consider three critical aspects: individual patient preferences, systemic and long-term effects of AI implementation, and the disempowerment of patients in regulatory processes. Professor Ploug emphasizes that patients have diverse values and preferences when it comes to accuracy, bias, and the role AI should play in their care. "Regulation must move beyond system-level safety and account for individual rights and participation," he states. To address these gaps, the authors propose the introduction of specific patient rights related to AI-generated diagnoses or treatment plans. These rights include: The right to transparency: Patients should be informed about how AI is used in their care and the basis for AI-generated recommendations. The right to human oversight: Decisions made by AI should be subject to review and final decision-making by a healthcare provider. The right to opt-out: Patients should have the ability to decline AI-assisted care if they choose. The right to challenge AI outcomes: Patients should be able to contest AI-generated decisions and have them re-evaluated by human professionals. The authors caution that without immediate input and collaboration from healthcare stakeholders, including clinicians, regulators, and patient groups, these essential rights may be overlooked in the rapid advancement of AI in healthcare. "AI is transforming health care, but it must not do so at the expense of patient autonomy and trust," Professor Ploug asserts. "It is crucial to define and implement rights that will protect and empower patients in an AI-driven health system." This call to action underscores the need for a more nuanced and patient-centered regulatory approach to ensure that AI technologies benefit all patients equitably and ethically.