One-sentence Summary

Cohere introduces methods that combine tokenizer analysis, model weight-based indicators, and prompting techniques to automatically detect under-trained glitch tokens such as _SolidGoldMagikarp that induce unwanted model behavior, revealing their prevalence across diverse LLMs and offering insights for improving efficiency and safety.

Key Contributions

• This paper develops a detection workflow for glitch or undertrained tokens by combining direct tokenizer analysis, model embedding-weight indicators, and prompting tests.
• Analysis across a diverse set of large language models finds severely undertrained tokens in about 1 percent of vocabularies in some cases, with tokenizer-data mismatch, unreachable tokens, and unused byte fallback tokens highlighted as key issues.
• The work provides practical recommendations for LLM tooling, including consistent pre-processing across tokenizer training, model training, and inference; checking unreachable tokens by encoding and decoding the vocabulary; and using embedding-based sanity checks after model training.

Introduction

Large language models still depend heavily on separate tokenizers, most commonly byte-pair encoding, even though the rest of the model is trained end to end on large data. This split creates a real operational risk: tokens that are rare or absent during training can trigger hallucinations, broken outputs, extra inference cost, and potential guardrail bypasses, especially as models use tools and process external data.

Earlier attempts to move away from tokenization or identify “glitch” tokens have been limited. Byte-level approaches are often slower, Unigram-based tokenization is not widely adopted, and existing detection methods are not consistently reliable across models.

The authors address this gap by introducing automated, efficient, and theoretically grounded techniques that use model embedding weights and tokenizer configuration to identify under-trained and untrained tokens. They apply these methods across many open-weight models, explore closed-source extensions, and release a Hugging Face-compatible analysis tool.

Method

The authors leverage a three stage detection pipeline for identifying under trained tokens in decoder only language models. The method begins by analyzing the tokenizer, then uses weight space indicators to rank candidate problematic tokens, and finally verifies candidates through targeted prompting. This design separates architectural signal from behavioral confirmation, allowing the authors to detect tokens that are present in the vocabulary but have not learned reliable representation or generation behavior.

The first module performs tokenizer analysis by inspecting the vocabulary and checking whether each token can be decoded and retokenized consistently. The authors define several categories that require special treatment. Partial UTF 8 sequences are tokens that encode only part of a multi byte Unicode character, often falling in fallback byte ranges such as $0x80$0x80 to $0xFF$0xFF. Unreachable tokens are those that are never produced by the tokenizer when a decoded string is re encoded, typically due to tokenizer configuration errors or conflicts between trained and manually added tokens. Special tokens are manually defined control symbols identified by patterns such as <...> and [...]. Partial UTF 8 sequences and unreachable tokens are excluded from the main automatic verification pipeline because they cannot reliably produce valid prompt strings for testing.

The second module computes model architecture dependent indicators to rank likely under trained tokens. The authors distinguish between models with tied embeddings and models with untied embeddings. In tied embedding architectures, the input embedding matrix and output embedding matrix are the same. In both tied and untied architectures, all rows of the output embedding matrix participate in the language modeling head during training, so even rarely seen or unused tokens receive gradient updates. The authors observe that under trained tokens tend to move together in output embedding space, forming a shared direction relative to the mean output vector of the model.

To exploit this behavior, they first choose a reference set of known unused or highly under trained embedding indices $t_{\mathrm{ref}}$tref​. This set can include placeholder token IDs such as <unused_token123> or embedding rows outside the tokenizer vocabulary. They compute a reference mean embedding vector

$$u_{\mathrm{ref}} = \frac{1}{|t_{\mathrm{ref}}|} \sum_{i \in t_{\mathrm{ref}}} E_{\mathrm{out}, i},$$uref​=∣tref​∣1​i∈tref​∑​Eout,i​,

where $E_{\mathrm{out}}$Eout​ is the output embedding matrix. Candidate tokens are then ranked by cosine distance from this reference vector. For a matrix $A$A and vector $x$x, the cosine distance is

$$C(A, x)_i = 1 - \frac{A_i \cdot x}{\|A_i\| \, \|x\|}.$$C(A,x)i​=1−∥Ai​∥∥x∥Ai​⋅x​.

The authors also compute Euclidean distances between output embeddings and $u_{\mathrm{ref}}$uref​, using

$$L_2(A - u_{\mathrm{ref}})_i = \|A_i - u_{\mathrm{ref}}\|,$$L2​(A−uref​)i​=∥Ai​−uref​∥,

and test more complex bias corrected variants. Their experiments indicate that the simpler cosine distance formulation is sufficient for ranking candidates.

For models without tied embeddings, the authors use a different signal. Input embeddings for tokens that never appear in a training batch are only weakly updated, mainly through weight decay or initialization effects. If weight decay is applied to the input embedding matrix, embeddings corresponding to under trained tokens tend toward zero; otherwise they may remain near their initial low norm values. Therefore, for untied models they rank tokens using

$$L_2(E_{\mathrm{in}})_i = \|E_{\mathrm{in}, i}\|.$$L2​(Ein​)i​=∥Ein,i​∥.

This input embedding norm indicator does not require a manually selected reference set of unused tokens, although it may not reliably identify control tokens that appear only in inputs.

The third module verifies candidate tokens through prompting. The indicators provide a ranking but not a definitive threshold, and the relationship between weight space scores and model behavior is noisy. To reduce false positives, the authors select the top 2 percent of tokens according to the chosen indicator, excluding partial UTF 8 sequences and unreachable tokens. They then construct repetitive prompts designed to give normal tokens high output probability. A candidate token is verified as problematic if its maximum output probability in the verification prompt remains very low, specifically below $1$1 percent, meaning $p < 0.01$p<0.01. This step converts a heuristic score into behavioral evidence.

The authors validate the indicators against both training data statistics and verification outcomes. For the open OLMo v1.7 model, they compare the proposed indicators with token frequencies from training. The embedding based indicators correlate strongly with how often a token appears in training, not only near the extreme under trained end but across a very wide range of token frequencies. As shown in the figure below:

The verification step further confirms that the indicators predict model behavior. Tokens with low indicator scores tend to have very low maximum output probabilities in verification prompts, while normal tokens cluster at high probability. The default 2 percent candidate threshold captures most verifiable under trained tokens while limiting computational cost. As shown in the figure below:

The authors also compare alternative indicators across models. Different indicators often reveal similar structure, including a secondary concentration of under trained tokens near low norm or low cosine distance values. In some models, added tokens form separate clusters, and density plots show a bimodal distribution with a distinct peak near zero for problematic tokens. This supports the use of these indicators as a practical screening mechanism before behavioral verification. As shown in the figure below:

Experiment

The experiments evaluate simple token-level indicators across open-weight language models by comparing them to model behavior and, where available, training-data statistics, showing that low output probabilities or embeddings effectively identify under-trained and rare tokens. Across diverse tokenizers and model families, they find that under-training commonly arises from unused single-byte tokens, intermediate BPE fragments, and special-token or pre-tokenization artifacts, while models whose tokenizers align closely with training data tend to show fewer issues. Additional black-box tests on closed models transfer these insights through targeted prompts, revealing repetition failures, hallucinations, or unexpected special-token behavior that supports the predicted token gaps. Overall, the results validate the proposed indicators as practical diagnostics for auditing model tokenizers and detecting training deficiencies.

The results show that under-trained tokens are detectable across diverse open-weight language models, but their prevalence varies by model family, tokenizer vocabulary, and special-token handling. Verified candidates typically form a small share of the full vocabulary yet can appear at different rates among tested candidates, with examples ranging from concatenated fragments to unusual strings that confuse output. These patterns support transferring checks to closed-source models through prompts that reveal repetition failures or abnormal special-token behavior. Verified under-trained tokens appear across many model families, but their number varies widely rather than tracking model size in a simple way. Many confirmed examples look like unusual fragments, concatenated strings, or special-character sequences that likely receive little useful training. Closed-source model tests show related failure patterns, including inability to repeat certain strings and occasional emission of unexpected special tokens.

The experiments evaluate detection of under-trained tokens across multiple open-weight language models and then extend checks to closed-source models through prompted behavioral tests. They find that verified candidates are generally rare but inconsistent across model families, tokenizer vocabularies, and special-token handling, rather than scaling predictably with model size. The confirmed tokens often resemble concatenated fragments, unusual strings, or special-character sequences, and closed-source models exhibit related failure modes such as repetition breakdowns and unexpected special-token emission.