One-sentence Summary

Researchers from CNR institutes and the University of Pisa propose PISA, a prompt-adaptive cost-aware single-model LLM router, and PISA-E, its ensemble extension, both using soft cluster memberships with model-specific predictors to estimate individual model capabilities; PISA-E jointly selects ensemble composition and cardinality under an adjustable cost-quality preference, surpassing the strongest competitors by $2.09$2.09 and $2.87$2.87 percentage points under in-distribution and out-of-distribution conditions while reducing average inference cost by approximately $36\%$36% and $25\%$25% at selected lower-cost operating points.

Key Contributions

• The paper introduces PISA, a prompt-adaptive and cost-aware single-model router that combines soft cluster memberships with model-specific predictors to estimate prompt-dependent capabilities of individual models.
• The paper presents PISA-E, an ensemble-routing extension that jointly selects ensemble composition and cardinality under an explicit cost-quality preference adjustable at inference time without retraining, using a tractable ensemble-success surrogate and efficient candidate selection without ensemble-specific training.
• Across four LLM routing benchmarks under in-distribution and out-of-distribution conditions, PISA achieves the highest macro-average cost-performance hypervolume among the evaluated single-model routers. PISA-E achieves the highest macro-average task performance among the evaluated ensemble approaches when prioritizing response quality, surpassing the strongest competitors by 2.09 and 2.87 percentage points under ID and OOD conditions, and at selected lower-cost operating points exceeds the strongest ensemble competitors’ macro-average task performance while reducing average inference cost by approximately 36% and 25%, respectively.

Introduction

Large language models are increasingly deployed in web-based services such as search, recommendation, and intelligent agents, where providers must balance response quality against inference cost across heterogeneous user requests. Prior routing methods select a single model per prompt, which cannot exploit the complementary strengths of multiple LLMs, while existing ensembling approaches often fix ensemble cardinality, optimize only part of the cost-quality tradeoff, or rely on classification-specific aggregation or sequential execution. The authors introduce PISA, a prompt-adaptive and cost-aware single-model router, and PISA-E, an ensemble routing extension that reuses individual model capability estimates to select both which and how many models to invoke for each prompt. A key advantage is that the cost-quality preference can be adjusted at inference time without retraining or additional hyperparameter optimization.

Method

The authors propose a shared capability-estimation framework that supports both single-model and ensemble routing, referred to as PISA and PISA-E. This unified pipeline estimates the prompt-dependent capabilities of individual large language models and uses these estimates to perform cost-aware selection.

As shown in the figure below:

Shared Front-End and Capability Estimation The pipeline begins by mapping an input prompt $q$q to an embedding $x = f(q)$x=f(q). To capture semantic similarities among prompts, the authors partition the training embeddings using X-means clustering to determine the number of clusters $K$K and their centroids $\mu_k$μk​. Instead of relying on hard assignments, the framework represents each query through soft memberships over the learned centroids. After $L_2$L2​-normalizing the embeddings and centroids, these memberships are computed from cosine distances as:

$$u_k(x) = \frac{\exp(-\beta d_k(x))}{\sum_{j=1}^K \exp(-\beta d_j(x))}, \quad d_k(x) = 1 - x^\top \mu_k$$uk​(x)=∑j=1K​exp(−βdj​(x))exp(−βdk​(x))​,dk​(x)=1−x⊤μk​

where $\beta > 0$β>0 controls the concentration of the assignments. This representation allows the model to exploit information from multiple neighboring regions rather than relying on a single cluster.

Using the membership vector $u(x)$u(x), the framework learns an independent logistic predictor for each candidate model $m \in \mathcal{M}$m∈M. The estimated capability of model $m$m for a given prompt is:

$$\hat{p}_m(x) = \sigma(w_m^\top u(x) + b_m)$$p^​m​(x)=σ(wm⊤​u(x)+bm​)

where $\sigma(\cdot)$σ(⋅) is the logistic sigmoid, and the parameters are fitted using L2-regularized logistic regression. The inference cost for each model is estimated from its average observed training-set cost and normalized to the range $[0, 1]$[0,1].

Cost-Aware Single-Model Routing (PISA) For single-model routing, the authors formulate the decision as selecting the model that maximizes a cost-aware utility function:

$$\hat{m}(q) = \arg\max_{m \in \mathcal{M}} \left[ \alpha \hat{p}_m(q) + (1 - \alpha)(1 - \bar{c}_m(q)) \right]$$m^(q)=argm∈Mmax​[αp^​m​(q)+(1−α)(1−cˉm​(q))]

where $\alpha$α controls the trade-off between predicted response quality and inference cost.

Prompt-Adaptive Cost-Aware Ensembling (PISA-E) To extend routing to model subsets, PISA-E evaluates candidate ensembles $E \subseteq \mathcal{M}$E⊆M with sizes up to $k_{\max}$kmax​. Directly estimating the performance of every combination is intractable, so the authors construct an ensemble-level utility estimate from the individual capabilities. They use the probability that a strict majority of the selected models produce correct responses as a tractable surrogate. Assuming conditional independence, this strict-majority success probability is computed using a Poisson-binomial dynamic program.

The ensemble cost is normalized by the sum of the $k_{\max}$kmax​ largest average model costs to allow fair comparison across different cardinalities. To avoid a combinatorial search, PISA-E employs a greedy candidate-construction procedure. Models are first ranked by their individual utilities. Candidate ensembles are then constructed as progressively larger prefixes of this ranking, restricted to odd cardinalities up to $k_{\max}$kmax​. This restriction is justified because an even-cardinality ensemble can always be improved by removing one model, which increases the strict-majority success probability while lowering the cost.

Once the optimal ensemble $\hat{E}$E^ is selected, its constituent models are queried in parallel. The final response is determined using self-weighted BERTScore, a capability-aware extension of minimum Bayes risk decoding. The framework normalizes the capability estimates over the selected ensemble to compute weights:

$$w_i(x, E) = \frac{\hat{p}_i(x)}{\sum_{j \in E} \hat{p}_j(x)}$$wi​(x,E)=∑j∈E​p^​j​(x)p^​i​(x)​

Each candidate response $y_i$yi​ is scored by weighting its BERTScore similarity to all other responses in the ensemble:

$$S_i(x, E) = w_i(x, E) \sum_{j \in E} w_j(x, E) \text{BERTScore}(y_i, y_j)$$Si​(x,E)=wi​(x,E)j∈E∑​wj​(x,E)BERTScore(yi​,yj​)

The response with the highest score is selected as the final output.

Experiment

The experiments evaluate PISA and PISA-E on four LLM routing benchmarks under in-distribution and out-of-distribution conditions, comparing them against single-model routers and ensemble methods using task performance and inference cost. PISA achieves the best aggregate cost-performance tradeoffs among single-model routers, while PISA-E attains higher macro-average task performance at lower inference cost than competing ensemble approaches and adapts ensemble cardinality per prompt, with adaptive cardinality helping especially in out-of-distribution settings. Component analysis indicates that the Poisson-binomial estimator paired with self-weighted BERTScore is the most effective configuration, and ablations confirm that prompt-dependent model ranking independently improves cost-performance tradeoffs.

This evaluation compares single-model routers on normalized hypervolume across four benchmarks in ID and OOD settings. PISA achieves the highest macro-average normalized hypervolume in both settings, ahead of EmbedLLM-MF and the other baselines. Its advantage is broad, with the best result in most benchmark-setting combinations, although it does not lead every individual benchmark. PISA records the highest macro-average normalized hypervolume among single-model routers in both ID and OOD settings. PISA ranks first in six of eight benchmark-setting combinations and leads RouterBench, FusionBench, and SPROUT in the ID results shown. EmbedLLM-MF is the strongest competing router on aggregate and leads the Track-2 benchmark in ID, while MLP consistently trails the field by a wide margin.

On ID benchmarks, PISA-E attains higher macro-average task performance than the strongest competing ensemble approaches while requiring lower inference cost. Under OOD conditions, it also improves performance relative to the best competitor with lower cost. Varying the cost-performance tradeoff parameter yields large inference cost reductions with moderate performance declines. PISA-E outperforms SelectLLM by 0.60 percentage points on ID macro-average performance while reducing inference cost by approximately 36 percent. With quality prioritized, PISA-E reaches the highest macro-average performance among deployable approaches in both ID and OOD settings, exceeding the strongest competitors by 2.09 and 2.87 percentage points respectively.

On held-out OOD task groups, PISA-E achieves higher macro-average performance than the strongest prior ensemble, DFPE, while also reducing inference cost. The highest-quality PISA-E configuration reaches the best deployable macro performance, and lower-cost configurations still slightly outperform DFPE. Across its tradeoff range, PISA-E spans a large cost-performance curve while Oracle remains far above the deployable approaches. PISA-E outperforms DFPE on OOD macro average by 0.48 percentage points at about 25% lower query cost. Lowering PISA-E's tradeoff parameter from the highest-quality setting reduces inference cost by roughly 106-fold while macro average falls from 63.24% to 50.54%.

PISA-E shows a controllable cost-performance trade-off through its alpha parameter. Lower alpha settings reduce the number of models per query and increase singleton usage, lowering inference cost but also lowering task performance. Raising alpha to 1.0 yields the highest reported performance in both ID and OOD settings, but at substantially higher cost, especially under OOD conditions. At alpha=0.3, PISA-E uses about 1.3 models per query with singleton rates around 82 to 84 percent, achieving the lowest reported costs but the lowest reported performance among the evaluated PISA-E settings. From alpha=0.3 to 1.0, ID performance improves from 70.75% to 75.27% and OOD performance improves from 50.54% to 63.24%, while ID cost rises roughly 21-fold and OOD cost rises roughly 106-fold.

Adaptive prompt-dependent ensemble routing achieves the highest macro-average hypervolume under both ID and OOD conditions, with a larger improvement over the best fixed-cardinality alternative on OOD benchmarks. Fixed cardinality can remain competitive or even better on specific benchmarks, such as RouterBench where single-model selection records higher hypervolume. Hypervolume generally declines as the fixed number of selected models increases, suggesting cardinality is more effective as a prompt-dependent decision. Adaptive routing achieves the highest macro-average hypervolume in both ID and OOD settings. The advantage of adaptive routing over fixed-cardinality alternatives is larger under OOD conditions. Fixed single-model selection outperforms adaptive routing on RouterBench, so adding models is not universally beneficial. Larger fixed cardinalities tend to produce lower hypervolume, with steep declines at the largest ensemble sizes.

The experiments evaluate PISA and PISA-E on in-distribution and out-of-distribution routing benchmarks, including single-model router comparisons, ensemble cost-performance tradeoffs, and adaptive prompt-dependent routing. PISA ranks highest overall among single-model routers, while PISA-E outperforms prior ensemble methods with lower inference cost and offers a controllable tradeoff between cost and task performance through its alpha parameter. Adaptive prompt-dependent routing achieves the best aggregate performance, especially in out-of-distribution settings, although fixed single-model selection can be better on specific benchmarks, and larger fixed ensembles generally reduce hypervolume.