One-sentence Summary

Google Research, Google DeepMind, and Mistral AI propose speculative cascading, which implements cascade deferral rules via speculative execution to combine the cost-quality trade-offs of cascades with the speed guarantees of speculative decoding, outperforming both baselines on Gemma and T5 models across language benchmarks.

Key Contributions

• Introduces speculative cascading, an inference strategy that implements cascade-style deferral through speculative execution so a smaller model handles easy inputs while a larger model is invoked via draft-and-verify steps only when needed.
• Characterizes an optimal deferral rule for speculative cascades and employs a plug-in approximation to make the rule practical for token-level interleaving between two language models.
• Experiments with Gemma and T5 models on language benchmarks show better cost-quality trade-offs than baselines using standard cascading or speculative decoding.

Introduction

Large language models deliver strong quality on many natural language tasks, but they also create high inference latency, motivating methods that combine small and large models to improve efficiency. Prior efficient inference strategies offer complementary strengths: cascades use deferral rules to route only hard inputs to a larger model, while speculative decoding drafts tokens with a small model and verifies them in parallel to approximate the larger model’s distribution. The authors leverage both ideas by designing speculative cascades that implement deferral rules through draft and verification steps, allowing the larger model to decide in parallel whether to accept or defer. Their approach produces better cost and quality trade-offs than standard sequential cascades and standard speculative decoding.

Method

The authors leverage a speculative cascading framework that combines the quality trade-offs of cascaded generation with the parallel execution efficiency of speculative decoding. The system uses two language models: a small drafter $q$q and a large verifier $p$p. At a prefix $x_{<t}$x<t​, the drafter produces a candidate next-token distribution $q_t(\cdot)$qt​(⋅), while the large model is invoked in parallel scoring mode over draft tokens generated by the small model. The resulting output distribution is controlled by a deferral or target distribution function that decides when to rely on $q$q and when to rely on $p$p.

A token-level cascade starts from the standard deferral idea: for each prefix $x_{<t}$x<t​, compute $q_t(\cdot)$qt​(⋅) and sample from $p_t(\cdot)$pt​(⋅) only if the small model is insufficiently confident. For example, Chow's rule can be written as

where $\alpha$α is a deferral threshold. If the rule does not defer, the cascade samples from $q_t(\cdot)$qt​(⋅); if it defers, it samples from $p_t(\cdot)$pt​(⋅). More generally, an oracle-style diff rule can compare the confidence of the two models:

This diff rule is not directly realizable in a strictly sequential token-level cascade because evaluating it requires the large model to be invoked at every step. It is useful, however, as an oracle for understanding the best cost-quality trade-offs achievable by cascading.

As shown in the figure below:

The curves illustrate that cascade-based generation can occupy a wide range of cost-quality trade-offs. The fraction of generated tokens for which the system defers to the larger model acts as the inference cost, and different deferral rules trace out different quality curves. In some regions, cascade rules can match or exceed the quality of the larger model, which is generally not possible with standard speculative decoding if the target remains the larger model's distribution. However, sequential cascades may require many deferrals because each deferral typically involves running the large model once to generate the next token, after which control returns to the small model.

To address this inefficiency, the authors generalize speculative decoding by introducing a target distribution $\pi = \mathbb{T}(q,p)$π=T(q,p) derived from the drafter and verifier distributions. The small model first drafts a block of tokens. The large model then scores the full prefix in parallel. For each draft token $x_t$xt​, the method compares $q(x_t)$q(xt​) against the target probability $\pi(x_t)$π(xt​). If $q(x_t) \le \pi(x_t)$q(xt​)≤π(xt​), the token is accepted. Otherwise, it is rejected with probability

Upon rejection, a replacement token is sampled from the normalized residual distribution built from

This procedure encompasses standard speculative decoding when $\pi = p$π=p, but it also supports richer target distributions that intentionally deviate from the large model to obtain different quality and cost trade-offs.

The speculative cascade instantiates this idea by representing a deferral-based cascade as a target distribution. If a deferral rule $r(x_{<t})$r(x<t​) chooses between $q_t$qt​ and $p_t$pt​, the resulting next-token distribution can be written as

Equivalently, the system can use a distribution-level decision function $\delta(q,p) \in \{0,1\}$δ(q,p)∈{0,1} and define

This formulation allows speculative execution of rules that were previously oracle-like. In particular, the large model can be run in parallel over drafted prefixes, making it possible to evaluate rules that depend on both $q_t$qt​ and $p_t$pt​, such as the diff rule, without the sequential overhead of a traditional cascade.

A key module of the method is the cost-aware deferral rule. In speculative cascading, deferral incurs cost not only because the large model is called, but also because deferral changes the target distribution and therefore affects the probability that a drafted token is rejected. For a target distribution

the rejection probability of a token drawn from the drafter is

where

is the total variation distance between the two distributions. When the models agree closely, deferral causes little rejection and is relatively cheap. When they disagree strongly, deferral increases the chance of a rollback.

The authors formulate deferral as a loss-versus-cost trade-off. A natural Lagrangian objective is

The optimal rule then has the form

Under a 0-1 loss, the expected losses can be replaced by confidence estimates from the models, yielding a practical plug-in rule:

This rule is more adaptive than Chow's rule because the penalty for deferring depends on how different the drafter and verifier are at the current prefix.

The framework also includes a token-specific interleaving module. Standard cascaded deferral decides whether to use $q_t$qt​ or $p_t$pt​ based on distribution-level confidence, such as the maximum token probability. That decision may be too coarse, because the sampled draft token itself can be poor even when the drafter's distribution is sharply peaked. To address this, the authors propose token-specific deferral rules

where the decision depends on both the prefix and the candidate token $v$v. The target distribution becomes

with

serving as a normalization term. This construction keeps the small model's probability mass on tokens deemed acceptable and shifts mass to the large model for tokens considered risky.

Several plug-in variants approximate this token-specific rule. One simple version uses the drafter's probability of the candidate token:

Another version evaluates the candidate token under the large model:

A multiplicative variant is

For example, with TokenV1, the target distribution keeps the drafter's distribution on tokens whose $q_t$qt​ probability is above the threshold and replaces the remaining mass with a re-scaled version of $p_t$pt​. This gives a more fine-grained interleaving of the two models and can provide a broader range of cost-quality trade-offs than directly using a fixed lossy speculative target, especially under low temperature, $P$P-sampling, or greedy decoding.

Experiment

Experiments compare token level cascades, standard speculative decoding, and proposed speculative cascades across translation, summarization, reasoning, coding, and question answering tasks using T5 and Gemma models under different sampling and few shot settings. The trade off analysis shows that cascades can target task label distributions rather than only matching the large model, and can sometimes exceed large model quality, while speculative decoding prioritizes large model fidelity and parallel execution. Across T5 and Gemma settings, speculative cascades, especially with token specific deferral, generally provide more flexible and better cost quality tradeoffs than sequential cascades or standard speculative decoding, though they use more total compute.

The described inference strategies compare token target distributions that either match the larger model or blend small model and large model probabilities through a deferral decision. Deferral rules depend on drafter confidence, model disagreement, or verifier advantage, while execution can be sequential, oracle based, or speculative. This framework supports speculative cascades that aim to improve cost quality tradeoffs while retaining much of the larger model quality. Standard speculative decoding targets the larger model distribution, while lossy variants relax exact matching. SpecCascade methods use mixed small model and large model probabilities controlled by binary deferral decisions. In Gemma few-shot evaluation, token based SpecCascade often matched or exceeded larger model quality with lower rejection rates.

The experiment compares speculative cascading, sequential cascading, and speculative decoding on translation and summarization tasks, using both latency to match a larger model and best quality under the larger model's latency. Speculative cascading shows the clearest benefit on translation, where it can provide strong quality within the latency budget and sometimes lower latency while matching large-model quality. The trade-off is task dependent, with sequential cascades or lossy speculative decoding being better on parts of the summarization results. Speculative cascading offers the strongest relative advantage on WMT, especially when quality is constrained by the large model's latency. Lossy speculative decoding can provide the highest quality under the latency budget, while speculative cascading remains close and competitive.

The evaluation compares speculative cascading with sequential cascading and standard or lossy speculative decoding across translation and summarization tasks, focusing on quality under latency constraints rather than isolated metrics. Across few-shot Gemma settings and task evaluations, speculative cascades generally preserve most of the large model quality while reducing rejection and offering favorable cost quality tradeoffs. The benefits are strongest for translation, especially when the large model's latency is the limiting factor, whereas summarization can favor sequential cascades or lossy speculative decoding depending on the budget.