One-sentence Summary

Snell and Griffiths reformulate conformal prediction as Bayesian quadrature, offering a practical distribution-free uncertainty quantification method that replaces frequentist guarantees with interpretable Bayesian predictions of the likely range of test-time losses for high-stakes machine learning systems.

Key Contributions

• The paper reframes distribution-free uncertainty quantification for black-box predictors, including deep neural networks, from a Bayesian perspective by connecting conformal prediction, statistical prediction analysis, and Bayesian quadrature.

• The proposed method applies Bayesian quadrature to calibration losses to construct a nonparametric posterior over test-time losses, providing interpretable guarantees and a richer representation of likely loss ranges than frequentist bounds.

• Two common frequentist methods appear as special cases of the approach, and the Bayesian formulation makes assumptions explicit, permits incorporation of additional knowledge, and reduces overly optimistic guarantees.

Introduction

Deep learning systems are increasingly used in high-stakes domains such as medicine and finance, where models must perform well in aggregate while also meeting fairness requirements and limiting the probability of harm. This makes it important to assess whether a black-box predictive model is suitable for deployment, especially when training details are unavailable or proprietary.

Prior distribution-free methods such as conformal prediction can provide finite-sample performance guarantees, but they are rooted in frequentist statistics. This makes it difficult to incorporate available domain knowledge, such as data distribution assumptions or model-specific constraints, and they often emphasize expected loss across hypothetical datasets rather than the full distribution of outcomes relevant to observed data.

The authors develop a Bayesian framework for uncertainty quantification by modeling uncertainty in quantile values and constructing a nonparametric characterization of possible deployment distributions from a calibration set. This approach connects conformal prediction to statistical prediction analysis and Bayesian quadrature, recovers split conformal prediction and conformal risk control as special cases, and provides a richer, more interpretable view of performance guarantees while still allowing distribution-free analysis.

Method

The authors begin by casting uncertainty calibration as a decision problem. Let $z=(z_1,\ldots,z_n)$z=(z1​,…,zn​) be calibration data, with $z_i=(x_i,y_i)$zi​=(xi​,yi​), and let a future test point follow the same distribution. A control parameter $\lambda$λ, such as a score threshold, is chosen from the calibration data and induces a loss $L(\theta,\lambda)$L(θ,λ) under the unknown state of nature $\theta$θ. The risk of a decision rule is

and the distribution-free objective is to find a rule whose worst-case risk is controlled:

Any rule satisfying this condition is called $\alpha$α-acceptable. Split conformal prediction corresponds to choosing $\lambda$λ so that the miscoverage loss remains below the target level. The threshold is obtained from the ordered nonconformity scores:

Conformal risk control generalizes this idea by allowing a monotonic non-increasing individual loss $\ell(z_{\mathrm{new}},\lambda)$ℓ(znew​,λ) and selecting

The authors then reinterpret these constructions through Bayesian decision theory. Instead of treating $\theta$θ as known while data are unknown, they condition on the observed calibration losses and reason about the posterior distribution of the risk. If the individual losses can be written as an expectation over a loss quantile function $K(t)$K(t), then

This suggests a Bayesian quadrature procedure: place a prior over quantile functions, update it using observed loss values, and integrate the expected loss. As shown in the figure below:

The standard Bayesian quadrature view places a prior over the quantile function and updates it after observing both loss values and their quantile levels. In conformal prediction, however, the quantile levels corresponding to the calibration losses are not directly observed. The authors therefore combine a right rectangular integration rule with distribution-free results on quantile spacings.

To remove the dependence on a prior over quantile functions, they upper-bound the posterior expected loss using only the ordered calibration losses. Let $\ell_{(1)}\le \cdots \le \ell_{(n)}$ℓ(1)​≤⋯≤ℓ(n)​ be the ordered observed losses, and set $\ell_{(n+1)}=B$ℓ(n+1)​=B for a known loss upper bound. The worst-case posterior expectation satisfies

Thus the problem reduces to assigning weights to the observed ordered losses. The uncertainty over the quantile locations is handled through probability integral transform properties. If losses are i.i.d. with continuous distribution, then the spacings between consecutive ordered quantile levels follow a Dirichlet distribution:

The authors define a random variable

They show that $L^{+}$L+ stochastically dominates the posterior expected loss under any prior:

Therefore, for a target probability $\beta$β, the critical value

provides a valid upper confidence bound on the risk. The left part of the next figure illustrates why considering only the average expected loss can underestimate the risk, whereas the proposed approach incorporates variability from unknown quantile levels:

In practice, the distribution of $L^{+}$L+ can be approximated by sampling Dirichlet weights and forming weighted combinations of the ordered losses. This yields a posterior-like distribution over plausible expected losses for each candidate decision parameter $\lambda$λ. The density of $L^{+}$L+ shifts as $\lambda$λ changes, allowing the method to evaluate how conservative the resulting risk bound is for different candidate thresholds:

The existing conformal procedures appear as special cases when only the expectation of $L^{+}$L+ is used. Since

conformal risk control corresponds to selecting the smallest $\lambda$λ satisfying

For split conformal prediction, the loss is $\ell_i=1-\mathbb{1}\{s_i\le \lambda\}$ℓi​=1−1{si​≤λ}. If $\lambda=s_{(k)}$λ=s(k)​, then

so the condition $\mathbb{E}(L^{+})\le \alpha$E(L+)≤α is satisfied whenever

Thus the approach preserves the familiar conformal thresholds while extending the analysis from a single expected-loss estimate to the full upper-bounding distribution of possible expected losses.

Experiment

Experiments on synthetic loss distributions, synthetic heteroskedastic prediction intervals, and MS-COCO multilabel classification evaluate repeated calibration splits by measuring how often the chosen risk threshold is exceeded and how large the resulting prediction sets or intervals become. The synthetic settings validate that the posterior-based highest posterior density rule provides a stronger data-conditional risk guarantee than marginal conformal risk control, reducing excessive-risk trials while still controlling failure rates. On MS-COCO, the proposed method better balances risk control and efficiency than baselines, keeping false-negative-rate risk within the allowed bound while producing shorter prediction sets than more conservative approaches.

Across repeated trials, CRC exceeds the target risk threshold much more often than RCPS or the proposed method. RCPS shows almost no threshold violations, while the proposed method also keeps violations extremely rare and is described as offering shorter prediction intervals than the baselines. Together, these results support a practical balance between empirical risk compliance and prediction efficiency. CRC has the highest frequency of threshold exceedance among the compared decision rules. RCPS controls the risk conservatively, with nearly zero observed violations. The proposed method maintains very low violation frequency while targeting smaller prediction sets or intervals.

The synthetic heteroskedastic experiment shows a tradeoff between risk-threshold violations and prediction interval length. Split conformal prediction and conformal risk control often exceed the target risk threshold, while producing shorter intervals, whereas RCPS essentially avoids violations but requires much longer intervals. The proposed method with beta = 0.95 keeps violations low while producing intervals shorter than RCPS. The proposed method balances failure-rate control and interval length more effectively than the compared baselines. RCPS is highly conservative in excess-risk violations but incurs a larger interval-length cost. The method achieves low violations without yielding intervals as long as the strictly conservative RCPS baseline.

The results show a tradeoff between risk-threshold exceedance and prediction set size. CRC uses the smallest prediction sets but exceeds the target risk threshold most frequently, while RCPS avoids threshold exceedance in the reported trials but produces the largest prediction sets. The proposed method with beta = 0.95 reduces frequent threshold exceedance relative to CRC while keeping prediction sets smaller than those of RCPS. CRC yields the smallest prediction sets but the highest relative frequency of threshold exceedance. RCPS avoids threshold exceedance in the reported trials but at the cost of larger prediction sets. The proposed method balances risk control and set compactness, exceeding the threshold much less often than CRC while using shorter sets than RCPS.

Across repeated synthetic experiments comparing risk-control methods, CRC tends to satisfy thresholds with compact intervals or sets but frequently exceeds the target risk, whereas RCPS is highly conservative and nearly eliminates violations at the cost of substantially larger intervals or sets. The proposed method achieves a more favorable balance by maintaining very low threshold exceedance while producing shorter intervals and smaller prediction sets than RCPS. Overall, these results show that the method provides a practical tradeoff between empirical risk compliance and predictive efficiency.