One-sentence Summary

Researchers from Konan University, RIKEN, and Waseda University reformulate Zipf's meaning-frequency law by quantifying meaning counts as contextual diversity derived from directions of contextualized word vectors in language models, which enables examination over a wider variety of words and corpora and reveals that autoregressive LMs require much more parameters than masked LMs to observe the law.

Key Contributions

• The work reformulates Zipf’s meaning-frequency law as a relationship between word frequency and contextual diversity, measured from the directions of contextualized word vectors generated by a language model, without using dictionary sense counts. This formulation enables analysis over more word types and corpora than dictionary-based studies.
• The dictionary-free formulation is applied to broader language data, including historical and learner corpora, and reports that the law holds for a wider range of words than prior studies could evaluate.
• Experiments connect the reformulated law to language-model scale and architecture, showing that small language models do not exhibit the law and that autoregressive language models require about 14 times more parameters than masked language models to do so, suggesting the law as a sanity check for lexical capability.

Introduction

The research addresses Zipf’s meaning-frequency law, which states that more frequent words tend to have more meanings, following a power law. This relationship matters for computational linguistics, lexical semantics, and language modeling because it links word usage to word sense diversity in natural language data.

Prior work relied on dictionaries or manual annotations to count word meanings. This caused unstable meaning counts across resources, missing or unbalanced senses, and limited vocabulary coverage, often excluding function words, high frequency words, and inflected forms. It also made large-scale studies difficult and raised questions about how to compare words with the same number of senses but different usage distributions.

The authors leverage contextualized word vectors from language models to reformulate meaning count as contextual diversity. This removes dictionary dependence, connects the law to Harris’s distributional hypothesis, and lets them test the law on much broader word sets and corpora. They also show that the new formulation can serve as a diagnostic for the lexical capability of language models.

Dataset

• The authors analyze canonical language corpora rather than a single new training set. The main sources are:

  • English: the British National Corpus (BNC) and the cleaned 2000s subcorpus of the Corpus of Historical American English (CCOHA).
  • Japanese: the Aozorabunko Corpus of Japanese literature and the Balanced Corpus of Contemporary Written Japanese (BCCWJ).
  • Multilingual biblical data: 27 corpora in 24 languages from five language families.
  • Detailed corpus sizes and additional source information are deferred to Appendix A.2 and are not specified in the excerpt.

• Each corpus subset is processed with a task-appropriate BERT encoder to obtain contextual word vectors:

  • English corpora use the final hidden layer of bert-large-uncased.
  • Japanese corpora use the final hidden layer of cl-tohoku/bert-large-japanese-v2.
  • Bible corpora use the final hidden layer of bert-base-multilingual-uncased.

• The authors do not use all tokens or vocabulary items directly. They filter target words using two corpus frequency conditions:

  • frequency count greater than 100;
  • frequency rank better than 20,000.
  • For words represented by multiple subwords, middle and final subword pieces are excluded.
  • The retained words include high-frequency words, function words, and inflected or conjugated forms.

• The primary derived dataset variables are:

  • f: word frequency in the corpus;
  • v: contextual diversity estimated from contextual word vectors. These variables are used to fit power-law regressions and test the meaning-frequency or context-frequency relationship.

• The paper does not report a training split, mixture ratio, cropping strategy, or rich metadata schema for these corpora. The data are used analytically: word vectors are extracted from BERT, filtered tokens are counted and measured, and frequency-diversity relationships are evaluated.

• The canonical corpora are used to test whether word frequency and contextual diversity follow a power law. For the Bible corpora, the reported average fit is an R^2 of 0.85 and an average slope coefficient of 0.043.

• The authors also construct three synthetic control datasets from the CCOHA 2000s subcorpus:

  • a shuffled corpus where word order is randomized but vocabulary and word frequencies remain unchanged;
  • a uniformly sampled corpus where words are drawn uniformly from the original vocabulary to match the original corpus size;
  • a virtual random-vector corpus where unit vectors are generated on a high-dimensional sphere, with generated vector counts matching word frequencies in the CCOHA 2000s subcorpus.

• These synthetic datasets are used only to compute f and v and to compare human language corpora against non-linguistic statistical controls.

Method

The method proposes a distributional reformulation of the meaning frequency law. Instead of using the number of meanings listed in a dictionary, it treats each context in which a word occurs as evidence for a potentially distinct meaning. The central idea is that differences in word meaning are reflected by differences in contextual usage, and that these differences can be measured through the directions of word vectors. This shifts the analysis from symbolic meaning counts to continuous vector geometry.

The first key module converts word usage into directional variation on a unit sphere. For a given word type, the method uses context-dependent or contextualized vectors to represent each occurrence of the word in its local environment. Because semantic similarity between vectors is commonly evaluated through cosine similarity, the vector norm is ignored and only the direction matters. Therefore, the method normalizes each word vector to unit length. Under this representation, a word whose context vectors point in nearly the same direction is semantically concentrated, whereas a word whose vectors spread across many directions exhibits more diverse contextual usage.

To quantify this directional spread, the method models the normalized vectors of a word type as samples from a von Mises-Fisher distribution. This distribution is defined over unit vectors on a high-dimensional sphere and is parameterized by a mean direction and a concentration parameter. For a word type, let $\mathbf{x}$x denote a normalized word vector. The distribution is written as:

Here, $\boldsymbol{\mu}$μ is the mean direction of the vectors, with $\|\boldsymbol{\mu}\|=1$∥μ∥=1, and $\kappa \geq 0$κ≥0 controls how tightly the vectors cluster around that mean direction. A large value of $\kappa$κ indicates that the vectors are strongly concentrated along one direction, implying relatively stable contextual behavior. A small value of $\kappa$κ indicates that the vectors are broadly distributed, implying greater contextual variability.

Because the method aims to measure meaning diversity rather than concentration, it uses the reciprocal of the concentration parameter as a proxy for contextual diversity:

This inverse relationship is important. If $\kappa$κ is large, $v$v is small, meaning the word is used in a narrow set of contexts. If $\kappa$κ is small, $v$v is large, meaning the word occurs in more varied contexts. The scalar $v$v thus serves as the operational replacement for the traditional dictionary-based meaning count.

To estimate $\kappa$κ from observed vectors, the method uses an approximate maximum likelihood expression. For a word type with $d$d-dimensional normalized vectors, let $l$l denote the norm of the mean vector of those unit vectors. Then:

This estimator connects directly to the geometry of the vectors. When all vectors point in nearly the same direction, their mean vector has a norm close to one, so $l$l is close to $1$1, $\kappa$κ becomes large, and $v$v becomes small. When vectors are widely dispersed, $l$l decreases, $\kappa$κ decreases, and $v$v increases. Thus, the computation of $v$v reduces the whole set of context vectors for a word to a single scalar measure of semantic or contextual diversity.

With this quantity in place, the meaning frequency law is reformulated as a power law between word frequency and contextual diversity. Let $f$f be the frequency of a word in the corpus and let $v$v be its contextual diversity score. The proposed relationship is:

This expresses the hypothesis that more frequent words tend to acquire or exhibit a wider range of contextual meanings. To test whether the relationship follows a power law and to estimate the exponent $\alpha$α, the method takes the logarithm of both sides:

where $c$c is a constant related to corpus size. A positive value of $\alpha$α indicates that higher frequency is associated with greater contextual diversity, while the fit quality and statistical significance of the slope coefficient determine whether the law is supported in a given corpus.

The workflow applies this formulation through a frequency binned regression pipeline. Raw word frequency is counted across the corpus, and word vectors are extracted for the word types under analysis. The vectors are then normalized to unit length. For each word type, the mean vector and its norm $l$l are computed, and $\kappa$κ is estimated using the approximate formula above. From this, the contextual diversity score $v$v is obtained as $1/\kappa$1/κ.

To reduce noise from individual words and stabilize the regression, words are grouped into bins according to their frequency ranks. Average frequency and average contextual diversity are computed within each bin, and these binned averages are regressed using the log-linear equation. A typical setting uses frequency-rank bins of size $100$100, so that words with consecutive ranks are pooled together before regression. The final evaluation relies on the slope coefficient $\alpha$α, the coefficient of determination $R^2$R2, and a Student $t$t test for whether the slope coefficient is nonzero. In this way, the method converts high-dimensional vector variation into a single word-level statistic and then tests whether that statistic scales with word frequency according to the meaning frequency law.

Experiment

The experiments test the new formulation of the meaning frequency law by estimating word frequency and contextual diversity from BERT word vectors across canonical, historical, learner, random, and model size settings. The law is robust in standard corpora, although lower frequency cases in older and learner texts show deviations that indicate limited sensitivity to word meanings. Randomized controls show that the pattern depends on structured human language rather than superficial frequency order, and it weakens in smaller or autoregressive models unless they are large enough. Overall, the findings support a broad regularity across languages and suggest the law can serve as a diagnostic of a model's ability to encode meaning.

The table compares the number of target words used for analysis in English and Japanese across studies. The current English study uses a larger target vocabulary than the earlier studies, especially Bond et al. In Japanese, the current study uses a much larger target vocabulary than the previous study listed. The English target vocabulary in this study is larger than those in the compared studies. The Japanese target vocabulary is substantially larger than the previous Japanese count.

Larger masked language models show a clearer meaning-frequency relationship, while smaller BERT models exhibit weaker or even negative frequency-diversity trends. Autoregressive GPT-2 medium does not show this relationship despite having a parameter count comparable to or larger than BERT-base, and only a much larger autoregressive model recovers the trend. The results suggest that model size and bidirectional context help distinguish subtle differences in word meaning. Meaning-frequency trends become clearer as BERT model size increases. Small BERT models show weak or negative frequency-diversity slopes. Autoregressive GPT-2 medium fails to show the trend despite a model size comparable to BERT-base.

The table shows that larger BERT models produce larger positive slope coefficients, while smaller models produce weak or negative values. These models also show stronger associations between contextual diversity and WordNet sense counts and higher MNLI accuracy. The strong correlations between slope coefficient and both measures suggest the slope reflects lexical quality and downstream performance. Slope coefficients decrease from large to tiny BERT models and become negative for the smallest models. Correlations between contextual diversity and WordNet meanings increase with model size. Slope coefficient is strongly positively associated with MNLI accuracy.

The experiments compare target vocabulary coverage across studies and evaluate how model size, architecture, and bidirectional context relate to meaning-frequency patterns. Larger BERT models show clearer positive frequency-diversity trends, whereas smaller models and GPT-2 medium yield weak, negative, or absent relationships, indicating that sufficient capacity and bidirectional context support finer-grained meaning distinctions. These slope patterns also align with stronger associations between contextual diversity and sense inventory and with downstream language understanding, suggesting that they reflect lexical quality and practical NLP capability.