One-sentence Summary

Researchers from the University of Illinois Urbana-Champaign and collaborators propose LLMRouter, a unified sequential-decision formulation and open-source infrastructure that includes xRouteBench for standardized development and evaluation of LLM routers, supports more than 16 modular routers, and demonstrates that learned routers achieve a $14.6\%$14.6% relative improvement over the strongest fixed-model baseline, with ranking reversals under tighter cost constraints and consistent personalization gains.

Key Contributions

• The paper introduces a unified formulation of LLM routing as a sequential decision process, characterizing routers by context encoders, model encoders, scoring functions, decision rules, and learning signals, and organizes existing methods into single-turn, multi-turn, and personalized routing families.
• It develops an automated pipeline that constructs routing supervision by running candidate models across benchmarks and evaluates routers on response quality and inference cost, producing the xRouteBench benchmark spanning generic LLM, memory-augmented, vision, time-series, and personalized routing scenarios.
• The paper provides LLMRouter, an open-source modular library with more than 16 built-in routers unified behind one interface, and reports a systematic study showing learned routers achieve a 14.6% relative improvement over the strongest fixed-model baseline, lightweight routers rank better under tighter cost constraints, and user-conditioned routing delivers consistent personalization gains.

Introduction

As large language models have proliferated into a heterogeneous ecosystem with widely varying costs and task-specific capabilities, routing each query to an appropriate model has become essential for cost-effective deployment, quality matching, and user-specific adaptation. Prior routing research has produced many approaches, but they are developed under different formalisms, separate codebases, incompatible interfaces, and inconsistent evaluation stacks, making it hard to identify what actually drives performance. Existing benchmarks also precompute responses for fixed pools and focus on single-turn routing, lacking standardized cost-aware evaluation for multi-turn and personalized settings as well as supervision pipelines for new tasks or candidate models. The authors introduce LLMRouter, a unified infrastructure that formalizes routers through common components, supports single-turn, multi-turn, and personalized methods behind one interface, automates routing supervision and evaluation over configurable candidate pools, and provides the xRouteBench benchmark spanning multiple scenarios under one protocol.

Dataset

Dataset description: xRouteBench

Composition and sources

• The benchmark contains 4,767 instances across five tracks.
• Generic LLM Tasks mix knowledge and commonsense QA from MMLU, MMLU-Pro, ARC-Challenge, OpenBookQA, CommonsenseQA, BoolQ, HellaSwag, and SQuAD; mathematical reasoning from GSM8K, MATH, and AIME; and code generation from MBPP and HumanEval.
• Memory uses long-horizon conversational QA from LoCoMo and LongMemEval, where input history dominates token cost.
• Vision uses Geometry3K and MathVista for image-grounded mathematical reasoning and Charades-Ego for egocentric video understanding.
• TimeSeries uses TSRBench, with each time series rendered as both text and image so the router also selects a modality encoding.
• Personalized uses open-ended prompts from Chatbot Arena and MT-Bench, tied to user personas and scored by a persona-conditioned LLM judge.

Processing and schema

• All tasks are constructed through the LLMRouter data engine and share a common query schema, supervision format, and evaluation protocol.
• Queries are sampled from source benchmarks, normalized into a unified schema, and split into training and test sets.
• Each non-text asset is converted by a transformation script into a self-contained textual query, with an optional pointer to the source image, video, or time series. This lets text-only and multimodal candidates receive the same textual input.
• No separate filtering rules are reported beyond source sampling and normalization.
• No cropping strategy is described; non-text assets are transformed into text queries with optional source pointers.

Response collection and scoring

• Each query is dispatched to every candidate in a candidate pool declared in a configuration file.
• Responses are collected together with their token counts.
• Every response is scored with its task metric and priced from its token counts.
• The result is a dense query-model matrix of performance and cost, used as routing supervision and as the test bed.
• Built-in metrics include exact and close matching, multiple-choice accuracy, token-level F1, mathematical answer verification, and execution-based code evaluation. Optional LLM-based judging is also supported.
• Multi-turn and agentic decomposition and aggregation calls are priced into trajectory cost.

How the data is used

• The training/test split comes from query curation, although mixture ratios are not specified in the described section.
• During evaluation, a test query goes only to the candidate selected by the router, rather than to the whole candidate pool.
• The query-model matrix provides supervision that jointly captures response quality and inference cost.
• For the personalized track, supervision is preference feedback rather than pointwise correctness.

Method

The authors formulate LLM routing as a sequential decision process to unify diverse routing strategies. At step $t$t, the router observes a state $s_t = (q, u, h_t)$st​=(q,u,ht​), comprising the input query $q$q, optional user context $u$u, and accumulated interaction history $h_t$ht​. The router takes an action $a_t \in \mathcal{M} \cup \{\perp\}$at​∈M∪{⊥}, where dispatching to a candidate model $m$m appends the response to the history, and the terminating action $\perp$⊥ aggregates responses into the final answer. The goal is to find a policy $\pi$π that maximizes the performance-cost trade-off:

where $\operatorname{perf}(y \mid q)$perf(y∣q) aggregates task-specific quality metrics, $c(\tau)$c(τ) sums the monetary or token cost of the routing trajectory, and $\lambda$λ controls the trade-off.

Refer to the framework diagram for an overview of this unified formulation.

Under this formulation, a router is characterized by five key components. The context encoder $E_q$Eq​ maps the routing state to a representation, which can be embedding-based (e.g., sentence embeddings or graph nodes) or text-based (e.g., verbalized prompts). The model encoder $E_m$Em​ profiles candidates using static metadata, historical behavior, learned embeddings, or verbalized descriptions. The scoring function $g$g measures compatibility between the encoded state and candidates, while the decision rule $d$d converts scores into routing actions, such as greedy selection or cost-aware thresholds. Finally, the learning signal $L$L fits these components toward the optimal policy using pointwise labels, pairwise comparisons, or trajectory-level rewards.

To operationalize this formulation, the authors introduce LLMRouter, an executable system that integrates data construction, training, inference, and evaluation.

The system architecture is illustrated in the figure below:

The infrastructure is organized into six modules centered around a query-model matrix:

1. Data Engine: Implements a three-stage pipeline consisting of Query Curation, Response Collection, and Metric Scoring and Pricing. This transforms declared tasks and candidate pools into a dense query-model matrix that serves as both routing supervision and a test bed.
2. Router Library: Provides a unified MetaRouter interface for over 16 routers. Users subclass MetaRouter to implement routing logic, encapsulating the context encoder, model encoder, scoring function, and decision rule.
3. Trainer: Decouples training from routing via a BaseTrainer. It defines the learning signal as a pointwise loss, pairwise loss, or trajectory-level reward to optimize the router for the weighted objective.
4. Route Engine: Drives inference by dispatching queries to selected candidates and aggregating responses for multi-turn policies until a termination action is produced.
5. Evaluation: Scores routers on identical test queries, candidate pools, and metrics, sweeping the trade-off weight $\lambda$λ to trace the performance-cost frontier.
6. Deployment: Exposes routers as OpenAI-compatible servers for live single-agent and multi-agent traffic.

To support standardized evaluation across diverse scenarios, the Data Engine constructs xRouteBench. All tasks share a common schema and evaluation protocol, converting non-text assets into textual queries to separate routing from perception.

The task distribution of the constructed benchmark is shown in the figure below:

The benchmark spans five tracks: Generic LLM Tasks for conventional single-turn text settings, Memory for long-horizon conversational QA where token cost is governed by history, Vision for image and video inputs, TimeSeries for modality selection, and Personalized for preference-based feedback. This design ensures that routing decisions are evaluated across regimes where input-token costs, modality capabilities, and user preferences fundamentally differ.

Experiment

The paper introduces an automated LLM routing evaluation pipeline and the xRouteBench benchmark, which spans generic, memory, vision, time-series, and personalized tasks under a unified cost-aware protocol. Experiments compare more than sixteen routers across eighteen models, including quality-only, cost-weighted, real-user, and multi-agent deployment settings. The findings show that no single router dominates all tasks, multi-turn routing does not consistently outperform single-turn routing, and router rankings shift substantially with cost preferences. User context improves personalization but simulated preferences transfer only partially to real users, while learned per-agent routing in multi-agent systems generally beats always selecting the largest model.

The table decomposes three router families into a common sequential decision framework by specifying routing state, encoders, action rule, and learning signal. Single-turn routers use only the query and fit per-candidate quality-cost rewards, while multi-turn routers include interaction history and optimize full trajectories. Personalized routers add user context and learn from pairwise preference comparisons among model responses. Single-turn routing conditions on the query alone and selects the candidate with the highest score. Multi-turn routing conditions on the query and accumulated history and is optimized with episode-level return. Personalized routing further conditions on user context and is trained on pairwise preferences. All three router families share the same underlying objective of trading off response quality against inference cost.

Under the performance-only setting, no single router dominates across the evaluated tracks: RouterDC performs best on Generic LLM Tasks, SVMRouter performs best on LoCoMo, and GraphRouter achieves the best average without consistently outperforming all other routers. Learned single-turn routers can improve over rule-based baselines, while the largest model alone delivers only mediocre performance relative to learned routes. Multi-turn routing does not reliably outperform a single routing decision and can introduce redundant computation. Winner routers vary by task, with RouterDC leading on Generic LLM Tasks and SVMRouter leading on LoCoMo. GraphRouter attains the best average on xRouteBench but does not consistently outperform other routers across all tasks. Learned routes can dominate always calling the largest model, since many queries solved by smaller or cheaper models are missed by the largest model. Multi-turn routing does not provide consistent gains over single-turn routing, and one well-chosen route is often sufficient.

On the personalized routing track, user-context-aware routers perform best, with GMTRouter leading PersonalizedRouter and the strongest user-agnostic router, EloRouter. The gap between the two personalized methods indicates that how user context is encoded and integrated matters beyond simply using it. Static and user-agnostic baselines generally trail the personalized approaches. GMTRouter and PersonalizedRouter rank first and second, confirming that conditioning routing decisions on user context improves persona-judge accuracy. EloRouter is the best user-agnostic router but still trails both personalized methods, while simple selection baselines such as largest-LLM and smallest-LLM score much lower. The small accuracy gap between GMTRouter and PersonalizedRouter suggests that the way user context is modeled remains important.

On held-out real-user sessions, PersonalizedRouter achieves the highest accuracy at matching pairwise preferences, while the fine-tuned CausalLM router ranks last. Several learned routers outperform static model-size baselines, and simulated rankings do not fully transfer to real feedback, as GMTRouter drops to sixth. PersonalizedRouter leads held-out routing accuracy by a clear margin. The fine-tuned CausalLM router is the weakest at agreeing with human preferences. GMTRouter falls from the top of simulated rankings to sixth on real-user sessions, indicating that deployment validation matters.

Across coordination topologies, independently routing each node with learned routers generally improves performance over always selecting the largest model. MFRouter achieves the highest average, closely followed by SVMRouter and kNNRouter, while GraphRouter is the only learned router that trails the largest-model baseline on average. The Graph topology tends to produce the strongest single-topology results. Six of seven learned routers beat the largest-model baseline on average, with MFRouter leading the group. The Graph topology yields the strongest per-topology scores for several learned routers, while GraphRouter is a notable exception and underperforms the baseline overall.

The experiments evaluate single-turn, multi-turn, and personalized routers within a common sequential decision framework that trades response quality against inference cost. Performance-only results show no single router dominates across Generic LLM Tasks and LoCoMo, learned single-turn routes can improve over rule-based and largest-model baselines, while multi-turn routing gives inconsistent gains. Personalized routing benefits from user context, with personalized methods outperforming user-agnostic baselines in simulated settings, and held-out real-user sessions further reveal that simulated rankings transfer imperfectly. In multi-node coordination, most learned routers improve over always selecting the largest model, with graph topology often yielding the strongest per-topology results, though GraphRouter is a notable exception.