One-sentence Summary

Ben-Gurion University of the Negev, Kensho Technologies, and MIT conduct a controlled evaluation of seven tokenization inference methods across BPE, WordPiece, Unigram, and SentencePiece, showing that greedy inference performs surprisingly well for common subword tokenizers while the contextually-informed SaGe method outperforms all others on morphological alignment.

Key Contributions

• This work isolates subword tokenizer inference from vocabulary construction, comparing seven inference methods across four tokenizer algorithms and three vocabulary sizes to clarify how decoding choices affect English token sequences.
• The paper introduces an intrinsic evaluation suite for English subword tokenization that combines morphological, cognitive, and information-theoretic measures, providing a benchmark for assessing segmentation quality.
• The results show that greedy inference performs strongly across the tested vocabularies, SaGe yields state-of-the-art morphological alignment, and token-count-minimizing methods score highest on cognitive metrics.

Introduction

The authors focus on subword tokenization in modern NLP systems, including large language models, where an inference method maps raw text into token sequences. This step matters because the chosen mapping shapes how downstream models represent and process language. Prior work has mostly studied vocabulary construction, vocabulary size, multilingual vocabularies, and vocabulary effects, while inference methods have received less systematic study and are often obscured in popular tokenization tools. The authors address this gap by running a controlled experiment that isolates inference methods across four tokenizers and introduces an intrinsic evaluation benchmark spanning morphological, information-theoretic, and cognitive metrics. They show that greedy inference performs well across vocabularies, SaGe achieves strong morphological alignment, and methods minimizing token count perform well under cognitive measures.

Dataset

• The authors create and release an intrinsic benchmark for evaluating English subword tokenizers, built from word-level datasets and information measures expected to relate to downstream language modeling or machine translation performance.

• The benchmark has three main data components:

  • Morphological alignment data: sourced from LADEC, MorphoLex, MorphyNet, and DagoBert, and augmented with morpheme segmentation, novel blend structure detection, and compound separation data. Word counts for each resource are provided in Table 2.
  • Cognitive plausibility data: drawn from Beinborn and Pinter (2023), consisting of human lexical decision task records with response times and accuracy across four linguistic setups.
  • Corpus-level segmentation statistics: computed on a reference corpus to measure token distribution properties and segmentation behavior.

• For morphological alignment, the authors compare tokenizer-generated word segments with gold-standard morphological segmentations using the metric from Creutz and Linden (2004), and report macroaveraged F1 scores across resources.

• For cognitive plausibility, the authors measure how tokenizer outputs correlate with human response times and accuracy, and report the average absolute correlation across the four setups.

• For distribution and segmentation statistics, the authors report Rényi efficiency, average tokens per word, and the proportion of pretokens that are segmented differently from the default. They omit token character-length distribution because it does not change with segmentation strategy.

• The data are used for intrinsic tokenizer evaluation rather than training: there is no described training split, mixture ratio, cropping strategy, or additional metadata construction beyond the existing task fields and gold-standard annotations.

Method

The authors decouple tokenization into two relatively independent components: vocabulary construction and inference. A fixed subword vocabulary $\mathcal{V}$V is learned from pre-training data, while an inference method determines how a word, or pretoken $w$w, should be segmented into subword tokens. Formally, a segmentation is defined as:

where each $t_i \in \mathcal{V}$ti​∈V, and the concatenation of all $t_i$ti​ reconstructs $w$w. This separation allows the authors to keep the vocabulary constant and compare different inference strategies using intrinsic metrics. The central assumption is that tokenization quality depends not only on the learned vocabulary, but also on the algorithm used to apply that vocabulary during inference.

The method is organized around three broad classes of inference algorithms: greedy methods, merge rule based methods, and likelihood based methods. Each class defines a different optimization strategy for selecting a valid segmentation.

Greedy inference methods generate the segmentation incrementally by selecting one token at a time. The first variant, longest prefix, scans the pretoken from the left and selects the longest vocabulary token that matches the current prefix. It then removes that token and repeats the process on the remaining string. This strategy aligns naturally with the way many languages are morphologically segmented, especially when affixes follow prefixes.

The second greedy variant, longest suffix, scans from the right and selects the longest vocabulary token that matches the current suffix. This method is used primarily as a diagnostic baseline. Because English words often begin with prefixes, a suffix-first strategy can produce segmentations that are less linguistically aligned. Comparing longest prefix and longest suffix therefore helps reveal how much linguistic structure influences the effectiveness of the inference method.

The third greedy variant, longest token, searches the entire pretoken for the longest vocabulary substring contained in it. That substring is added to the output segmentation, and the remaining character sequences are recursively segmented. This approach approximates a word by its most informative subword pieces and tends to preserve morphological structure more flexibly than strictly left-to-right prefix selection.

Merge rule based inference methods begin with the character sequence of the word and repeatedly apply merge rules learned during vocabulary construction. In the deterministic setting, the algorithm recursively applies the first applicable merge rule according to the order in the trained merge list. This is the standard inference mode used in byte pair encoding. The segmentation is fully determined by the order and validity of the available merges.

The authors also test a stochastic variant called dropout. Instead of always applying a merge rule when it is applicable, this strategy applies each valid merge with probability $p$p. This introduces regularization at inference time. Rare tokens are more likely to be exposed during training, giving their embeddings more opportunities to learn useful representations. The method can improve downstream behavior, especially in generation-oriented tasks, by reducing over-reliance on very frequent subword tokens.

Likelihood based inference methods treat tokenization as a probabilistic optimization problem. Each token is associated with a likelihood value, and the goal is to find a segmentation whose total likelihood is maximized. In the default Unigram language model setting, the likelihood of a segmentation is the product of the individual token likelihoods. The segmentation is therefore chosen by maximizing the joint probability of the selected token sequence.

A simpler likelihood based objective assigns the same constant likelihood to every token. This pushes the inference process toward segmentations with fewer tokens, because fewer token choices produce a higher total score under the constant-likelihood model. This objective is closely related to the minimal-segmentation preference used in PathPiece. Although it does not use the learned likelihood values directly, it still operates within a likelihood-style framework by converting an efficiency preference into an optimization target.

Together, these modules form a modular tokenization pipeline. The authors first construct or assume a fixed vocabulary, then apply different inference algorithms to produce segmentations from the same vocabulary. This design makes it possible to isolate the effect of inference behavior from the effect of vocabulary learning, allowing systematic analysis of how segmentation strategy influences token quality, morphological alignment, and downstream token usage.

Experiment

The experiments establish an intrinsic benchmark for subword tokenizers, using morphological alignment, cognitive plausibility, and token distribution measures to validate how segmentation affects linguistic structure, human processing, and efficiency. Across BPE, UnigramLM, WordPiece, and SaGe trained on MiniPile, they show that inference strategy materially changes tokenizer behavior and that default methods are not always optimal. The overall findings reveal tradeoffs, with SaGe favoring morphology, compact segmentations improving token economy and some cognitive signals, and likelihood based methods often underperforming despite their stated goals.

The examples show that tokenizer inference choices can change word segmentations even when the underlying vocabulary is fixed. Default modes for BPE, UnigramLM, and SaGe can produce segments that cross morpheme boundaries, while alternative inference modes can yield more morphologically aligned cuts. BPE merge based inference can join fragments such as am too early, obscuring larger word parts. UnigramLM likelihood based inference can favor cross morpheme forms such as nprecedented. SaGe inference can blur prefix and suffix boundaries, but alternative segmentation modes can give clearer morphological splits.

The table catalogs morphological resources assembled for an English subword tokenizer benchmark, showing that the collection spans small specialized datasets and much larger word-level lexicons. Each entry records size, source, and license, revealing that the resources come from different origins and carry mixed licensing terms. The listed resources support evaluation of morphological alignment and cognitive plausibility by clarifying which word-level data underlie the benchmark. The benchmark pool consists of English morphological resources with widely varying scale. Specialized resources are smaller than the broad morphological lexicons included in the benchmark. Licensing is mixed, including Creative Commons options and citation-based terms where a license is not specified.

On the 40k vocabulary intrinsic benchmark, the default decoding strategy is frequently surpassed by alternative inference methods on at least one metric. Longest token and least tokens approaches tend to offer strong morphological alignment, cognitive plausibility, or token efficiency, while merge rule based methods expose sensitivity to early merges crossing morphological boundaries. Dropout improves Rényi efficiency relative to likelihood based methods, but at the cost of larger decoding differences from the default. Default decoding is not consistently best across metrics Least tokens performs well on token count and cognitive measures Dropout favors Rényi efficiency over likelihood based inference

The experiments evaluate subword tokenization on a fixed English benchmark built from diverse morphological resources, focusing on how different decoding strategies affect word segmentation rather than vocabulary size. Across cases and the intrinsic benchmark, default inference often produces segments that cross morphological boundaries, while alternative modes can yield clearer morphological splits, better token efficiency, and stronger cognitive plausibility. Merge based methods show sensitivity to early merges, likelihood based approaches may favor less aligned forms, and dropout can improve information efficiency but increases deviation from default segmentations. Overall, tokenizer choices at inference time materially influence morphological alignment and segmentation quality.