Agentic AI systems are rapidly gaining traction, especially in healthcare, where they enable complex decision-making through LLM-driven pipelines, retrieval-augmented reasoning, and multi-step workflows. These systems excel at synthesizing diverse data, reasoning through clinical scenarios, and generating actionable recommendations. However, a critical disconnect exists between how these systems operate and how they are evaluated. While agentic models often produce binary decisions—such as “yes, the patient has the disease” or “order the test”—the gold standard for evaluating medical prediction models remains the Area Under the Curve (AUC), a metric designed for continuous risk scores. AUC is widely used in clinical research because it effectively measures a model’s ability to rank positive cases above negative ones, even in imbalanced datasets. For example, in breast cancer screening where prevalence is low (around 5 in 1000), a model that always predicts “no cancer” would achieve high accuracy but fail catastrophically in practice due to unacceptably high false negatives. AUC avoids this pitfall by focusing on ranking performance rather than classification accuracy. Yet most agentic systems today output only binary outcomes, which collapses the score distribution into just two values—0 and 1. This severely limits the ability to compute a meaningful AUC, as the ROC curve becomes degenerate with few or no intermediate operating points. Without a continuous score, comparing agentic systems to traditional machine learning models using AUC becomes impossible. To bridge this gap, several practical methods can be used to derive continuous risk scores from agentic outputs: Extracting internal model log probabilities offers a robust, stable signal aligned with the model’s reasoning. When available, these logits provide a natural ranking of patient risk, enabling straightforward AUC computation. Asking the agent to explicitly output a probability—such as “risk_probability: 0.7”—is intuitive and widely applicable, especially with standard APIs. However, calibration remains a challenge; agents often produce overconfident or clustered probabilities unless prompted carefully with calibration examples. Monte Carlo sampling involves running the agent multiple times on the same input and estimating the frequency of positive predictions. This empirical probability captures uncertainty and can serve as a reliable continuous score, though it increases computational cost. For retrieval-augmented agents, similarity scores between the current patient and known positive cases can be used directly as risk scores. This leverages the agent’s underlying reasoning about clinical similarity. When agents output structured categories (e.g., low, medium, high risk), a small calibration model can be trained to map these outputs to continuous scores based on ground truth labels. Finally, if the agent allows tuning of parameters—such as aggressiveness thresholds—sweeping these settings across multiple runs generates a series of sensitivity and specificity values. Plotting these points yields an approximate ROC curve and a usable AUC. Each method has trade-offs in complexity, cost, and reliability, but all offer a path to meaningful evaluation. The key is to move beyond binary outputs and capture the agent’s internal confidence or risk assessment in a way that supports standard clinical evaluation metrics. In an era where agentic AI is reshaping medical decision support, aligning evaluation with established standards like AUC is essential. It ensures that new systems are not just innovative, but demonstrably better than existing approaches. By deriving continuous scores from agentic outputs, we can maintain scientific rigor, enable fair comparisons, and build trust in AI tools used in real-world clinical settings.