One-sentence Summary

Researchers at Kyoto University and RIKEN propose representing language models as log-likelihood vectors computed on a predefined text set, where squared Euclidean distance approximates the Kullback-Leibler divergence of text-generation probabilities, enabling scalable comparison of over 1,000 models at linear computational cost.

Key Contributions

• The paper introduces a scalable comparison method for autoregressive language models based on log-likelihood vectors computed on a predefined text set, with features derived directly from cross-entropy loss and no costly pairwise text generation.
• It establishes that these log-likelihood vectors can serve as model coordinates whose squared Euclidean distance approximates the Kullback-Leibler divergence between text-generation probability distributions.
• Experiments on more than 1,000 language models construct a model map, show that nearby models have similar text-generation distributions, and demonstrate that the resulting features can predict benchmark performance, detect data leakage, and validate the relationship between model coordinates and KL divergence.

Introduction

Language models are increasingly central to AI systems, so understanding how they relate, specialize, and differ is important for choosing, comparing, and managing them. Prior studies often compare models using outputs, activations, leaderboards, or probability differences across prompts, but these approaches can require costly text generation or pairwise evaluations that do not scale well to large model collections.

The authors leverage language models as probability models and represent each model as a log-likelihood vector over a fixed text corpus. This creates model coordinates that capture geometric relationships among probability distributions, where squared Euclidean distance approximates KL divergence. The main contribution is an efficient, scalable method for comparing many language models without generating text, enabling model maps, similarity analysis, data leakage detection, and benchmark performance prediction.

Dataset

• The authors construct a text dataset from the Pile Uncopyrighted corpus, after removing five categories of copyrighted material.

• The final dataset contains 10,000 texts, and each text is tagged with a category label from the Pile.

• To create the dataset, the authors start from the first 1,000,000 texts in the Pile Uncopyrighted corpus and split them into 1024-byte chunks encoded in UTF-8.

• When decoding errors occur, the authors remove bytes one at a time until the chunk decodes successfully.

• Chunks smaller than 256 bytes are discarded, leaving about 5.7 million valid chunks.

• The authors randomly sample 10,000 valid chunks to form the final text dataset.

• The sampled texts have an average length of 972.3188 bytes.

• The authors use this text dataset to compute model coordinates for language models, rather than for model training.

• The study uses a total of 1,018 language models: 1,000 models selected from causal language models on Open LLM Leaderboard v1, plus 18 models from the DeepSeek language model series.

• No training split, validation split, or dataset mixture ratio is described for the text data; it functions as a sampled input corpus for coordinate computation.

Method

The core method maps each language model into a finite-dimensional coordinate system by evaluating its text-generation probabilities on a fixed set of texts. Rather than comparing model parameters directly, the authors treat a language model as a generator of probability distributions over texts and define its position from how well it assigns likelihood to a common probe corpus. The resulting geometry makes distances between points correspond to an approximation of the Kullback-Leibler divergence between the corresponding language models.

The pipeline begins with an autoregressive language model $p_i$pi​ over a token vocabulary $\mathcal{V}$V. A text $x$x is a token sequence $x=(y_1,\ldots,y_n)$x=(y1​,…,yn​), and the model defines the joint probability

where $y_0$y0​ is the beginning-of-sequence token. Given a corpus $D=(x_1,\ldots,x_N)$D=(x1​,…,xN​), the model is scored on each text through its log-likelihood

For each model $p_i$pi​, the authors form the log-likelihood vector

and stack the vectors for $K$K models into the log-likelihood matrix

This matrix converts each model into a profile over the same set of texts.

The probe corpus is sampled to represent multiple text domains. Texts are extracted from the uncopyrighted portion of the Pile, converted into fixed-size byte chunks, filtered by length, and randomly sampled to form the final dataset used for coordinate computation. Each sampled text retains a category label, providing a diverse set of prompts for evaluating language models. This domain diversity helps ensure that the coordinate differences reflect behavior across different text types rather than overfitting to one corpus style. As shown in the figure below:

After obtaining $L$L, the method applies double centering to remove global offsets. First, each row of $L$L is centered by subtracting the mean log-likelihood of the corresponding model:

This gives the centered log-likelihood vector $\boldsymbol{\xi}_i$ξi​. Second, the matrix of centered vectors is centered column-wise by subtracting the average centered vector across models:

The vector $\mathbf{q}_i$qi​ is the primary model coordinate. It removes both the model-specific average log-likelihood and the text-specific average across models, isolating relative pattern differences in how the models distribute probability over the corpus.

The authors show that this coordinate system has a meaningful probabilistic interpretation. The Kullback-Leibler divergence between two language models is

Under the assumption that the texts are drawn from an underlying distribution $p_0$p0​ and that $p_i$pi​ and $p_j$pj​ are reasonable approximations of $p_0$p0​, the divergence can be approximated by the variance of the log-likelihood difference:

Using the dataset and the double-centered coordinates, this becomes

Thus, if the coordinates are scaled as $\mathbf{q}_i/\sqrt{N}$qi​/N​, the squared Euclidean distance between two points approximates twice the KL divergence between the corresponding models.

The original log-likelihood vectors can also be interpreted geometrically. Since $\boldsymbol{\ell}_i$ℓi​ differs from $\mathbf{q}_i$qi​ by centering terms, the squared distance in the raw log-likelihood space decomposes as

The first term captures the centered distributional difference between models, while the second term captures the difference in their overall average log-likelihoods. In this view, $\mathbf{q}_i$qi​ represents horizontal axes related to distributional positioning, and $\bar{\ell}_i$ℓˉi​ can be viewed as a global scale or height dimension in a visualization.

Experiment

The experiments build a log-likelihood-based model-coordinate map from thousands of texts and over a thousand language models, using it to visualize relationships among model families, specializations, text preferences, benchmark performance, model size, and creation date. Regression analyses show that these coordinates strongly predict leaderboard scores and mean log-likelihood, indicating that model positions encode task-relevant capability structure beyond simple perplexity. Additional validations confirm that distances in log-likelihood space approximate KL divergence, and that linear weight interpolation preserves meaningful structure in coordinate space, supporting its use for exploring model behavior.

Across the nearest-neighbor comparisons, models that share a base lineage or fine-tuning history appear closest under KL divergence measured in bits per byte. The code model's nearest neighbors are largely other code-oriented or coding-family models, while the larger base model's closest neighbors are mostly variants within the same family. These patterns support the use of log-likelihood coordinates to reveal semantic and architectural relationships among many language models. Name-related models tend to have small KL divergence distances Code-oriented models cluster near other code-generation models Family variants rank closer than cross-family models

Ridge regression using model coordinates predicts benchmark scores with consistently high linear and rank correlations across individual tasks. The strongest predictions occur for mean log-likelihood, while the six-task average remains very high. These results indicate that the coordinate representation retains enough performance-related structure, even though the mean is double-centered. All evaluated benchmarks show strong predictive alignment between model coordinates and scores. Mean log-likelihood is predicted more closely than individual benchmark scores. The six-task average also has high Pearson and Spearman correlations. GSM8K has the comparatively lowest correlations among the individual tasks.

Mean log-likelihood, closely related to perplexity, shows only moderate associations with benchmark scores across the evaluated tasks. These correlations are weaker than the strong predictive performance associated with model coordinates. The strength of the relationship varies by benchmark, with some tasks showing substantially lower agreement than others. Mean log-likelihood and benchmark scores show moderate overall correlations rather than near-perfect agreement. TruthfulQA and GSM8K show especially weak associations with mean log-likelihood. HellaSwag and Winogrande show comparatively stronger correlations with mean log-likelihood.

Nearest-neighbor analysis using KL divergence in bits per byte shows that models with shared lineages, coding roles, or name similarities are closest, indicating that log-likelihood coordinates capture architectural and semantic relationships. Ridge regression on these coordinates predicts benchmark scores strongly, with mean log-likelihood and the six-task average predicted most reliably, although GSM8K is comparatively less predictable. By contrast, mean log-likelihood alone correlates only moderately with benchmark scores, with weaker agreement for TruthfulQA and GSM8K and stronger agreement for HellaSwag and Winogrande.