One-sentence Summary

Researchers from FAIR at Meta, Google DeepMind, Cornell University, and NVIDIA propose a method to decompose language model memorization into unintended memorization and generalization, estimating that GPT-family models have an approximate capacity of $3.6$3.6 bits-per-parameter, and demonstrating that once capacity fills, grokking occurs as unintended memorization decreases, yielding scaling laws relating model capacity and data size to membership inference.

Key Contributions

• A compression-based method formally separates unintended memorization from generalization, enabling per-sample entropy measurement of the bits a model stores about a dataset.
• GPT-family models have an approximate capacity of 3.6 bits per parameter, with memorization filling before a grokking phase where unintended memorization decreases and generalization begins.
• Scaling laws relating model capacity and data size to membership inference are derived from training hundreds of transformer models with 500K to 1.5B parameters.

Introduction

Modern large language models are trained on datasets that vastly exceed their parameter counts, intensifying the question of whether and how they memorize individual training samples. Prior definitions typically rely on extraction—testing whether a model can regenerate a specific string given a prompt—or on membership inference, but these methods cannot cleanly separate genuine recall of a datapoint from the model’s ability to generalize patterns learned across many examples. The authors recast memorization as a compression problem: a sample is considered memorized if the model can encode it in fewer bits than a generic compression scheme. They further decompose this measure into unintended memorization (information retained about a particular dataset instance) and generalization (knowledge about the underlying data-generating process), yielding a practical, instance-level framework that disentangles the two and reveals how memorization scales with model capacity and data volume.

Method

The authors leverage information theory toestablish a statistical framework for understanding memorization in machine learning models. They begin by defining memorization through the lens of mutual information, distinguishing between intended memorization, which corresponds to generalization, and unintended memorization. To practically measure these quantities at the instance level, where entropy-based definitions fall short due to the singleton nature of trained models and datasets, the authors transition to Kolmogorov complexity. This approach defines the information content of a string as the length of its shortest representation in a given computational model.

To estimate Kolmogorov complexity in practice, the authors employ compression algorithms, specifically arithmetic coding, which ties compression rates to model likelihoods. They approximate the conditional Kolmogorov complexity $H^K(x \mid \hat{\theta})$HK(x∣θ^) using the negative log-likelihood of the data under the target model, and $H^K(x \mid \hat{\theta}, \theta)$HK(x∣θ^,θ) using the maximum likelihood between the target and a reference model. This reference model is typically a larger model of the same architecture trained on a much wider data distribution.

Building on this, the authors formalize the concept of model capacity for memorization. Capacity is defined as the maximum amount of memorization a learning algorithm can store across its parameters. When a model reaches its capacity, the total memorization no longer increases with dataset size. This capacity is computed by training models to saturation on varying dataset sizes and measuring the maximum unintended memorization.

The authors further explore the relationship between model capacity, dataset size, and membership inference success rates. They observe that for a fixed model capacity, membership inference follows a roughly sigmoidal form with respect to dataset size. The intuitive explanation is that membership inference is easy for large models overfit to tiny datasets, but becomes increasingly difficult as the dataset size grows, eventually decaying toward random performance. The authors model this relationship using the following equation: $\text{Membership}_{F_1}(\theta, \mathcal{D}) = \frac{1}{2} \left(1 + c_1 \sigma \left(c_2 \left(\frac{\text{Capacity}(\theta)}{|\mathcal{D}|} + c_3\right)\right)\right)$MembershipF1​​(θ,D)=21​(1+c1​σ(c2​(∣D∣Capacity(θ)​+c3​))) where $\sigma(x) = \frac{1}{1 + e^{-x}}$σ(x)=1+e−x1​.

The authors fit this scaling law using a non-linear least squares solver on empirical data, demonstrating that the sigmoidal function accurately predicts membership inference scores across different model sizes and dataset configurations.

As shown in the figure below:

This empirical model provides evidence for why membership inference attacks fail on models trained on extremely large datasets, as the predicted performance approaches 0.5 when the tokens-per-parameter ratio is sufficiently high.

Experiment

The evaluation first measures Transformer capacity by training on uniformly sampled synthetic tokens, revealing that models memorize roughly 3.5 bits per parameter and reach a clear plateau once their capacity is saturated. Switching to perfectly deduplicated text, the study disentangles unintended memorization from generalization, showing that extraction rates converge to test-set extraction and that double descent occurs exactly when the dataset size exceeds the model's capacity. Membership inference experiments on both synthetic and text data demonstrate that loss-based attacks become ineffective as the training set grows, and the derived sigmoidal scaling law predicts that membership inference converges to chance performance for models trained on sufficiently large corpora.

Model capacity in bits increases with parameter count but bits-per-parameter declines as width or depth grows, meaning larger models store less information per parameter. Switching from half precision (bfloat16) to full precision (float32) yields only a small increase in bits-per-parameter, with the average remaining around 3.5 to 3.8 bits-per-parameter. These capacity estimates inform scaling laws that relate model size and dataset size to membership inference risk. Widening a single-layer model from 32 to 256 hidden units causes fp32 bits-per-parameter to drop from 4.23 to 3.12, and a similar decline is seen in the two-layer configuration. The gap in bits-per-parameter between bfloat16 and float32 remains small across all widths and depths, indicating that doubling numerical precision provides only a marginal increase in effective capacity per parameter.

For GPT-2 Medium and XL, the scaling law accurately predicts the dataset sizes needed to achieve target membership inference F1 scores, with observed values within 1.5 percentage points of predictions. The largest error occurs at F1 0.75, where the sigmoidal relationship is steepest, while extreme scores of 0.55 and 0.95 are more precise. The results explain why loss-based membership inference becomes near-random as dataset size grows. Observed membership inference F1 scores deviated by no more than 1.5 percentage points from the scaling law predictions, with the greatest inaccuracy at the 0.75 target due to the steepness of the sigmoid curve. The predictions confirm that sufficiently large datasets push membership inference F1 toward 0.5 (random guessing), supporting the observation that models trained on massive corpora resist such attacks.

The experiments first quantify model capacity in bits per parameter, revealing that larger models (more width or depth) store less information per parameter and that moving from half to full precision yields only a marginal capacity increase. These capacity estimates inform scaling laws linking model and dataset size to membership inference risk. Validation on GPT-2 Medium and XL shows the scaling law accurately predicts dataset sizes for target F1 scores, with deviations under 1.5 percentage points, and confirms that sufficiently large training sets push membership inference F1 toward random guessing.