One-sentence Summary

Stanford University and Columbia University propose a method to provably detect test set contamination in black-box language models by comparing the likelihood of canonically ordered benchmarks against shuffled orderings, exploiting the tendency of memorization to make specific orderings disproportionately likely, and use the test to audit four popular models with sensitivity down to 1.4 billion parameters and 1000 examples.

Key Contributions

• The paper introduces a black-box statistical test for benchmark contamination that uses only log probabilities and exploits the fact that a clean language model should not strongly prefer a benchmark's original example order. The test flags possible contamination when the likelihood of a canonically ordered test set is significantly higher than that of shuffled orderings, without requiring access to pretraining data or model weights.

• The paper develops shard-based tests that construct an exact null distribution from the exchangeability of benchmark examples, giving false-positive guarantees and improving detection precision and sensitivity. Canary experiments show detection in challenging settings, including models with 1.4 billion parameters, test sets of 1000 examples, and datasets that appear only a few times in the training corpus.

• The paper applies the audit to four publicly accessible language models and finds little evidence of pervasive test-set contamination. The audit demonstrates third-party testing using only log probabilities.

Introduction

Large language models are often trained on broad web data that may accidentally include evaluation benchmarks, making it harder to tell whether strong scores reflect true generalization or memorized test examples. Prior contamination audits often depend on provider disclosures, targeted filtering, n gram overlap, prompting, or heuristic log probability signals, which can be unreliable, limited to known benchmarks, or unable to provide strong statistical evidence. The authors leverage dataset exchangeability and only model log probability queries to build a black box statistical test: if a model shows a strong preference for a benchmark’s public ordering over shuffled orderings, that can indicate the dataset was seen during training. Their main contribution is a sharded hypothesis test with controlled false positive rates that can provably identify benchmark contamination in small and partially duplicated datasets.

Method

The authors leverage a likelihood based statistical framework to detect verbatim test set contamination in large language models. The central premise is that a clean dataset should not have an unusually high log probability under the target model compared with randomly permuted versions of itself. If the model has memorized the canonical ordering, the model is expected to assign a substantially higher log probability to the original sequence than to many shuffled variants.

The method can be understood as a sequence of modules. First, a candidate dataset is assembled into a token sequence $X$X. The model $\theta$θ scores the sequence using the log probability of its canonical ordering,

The authors then compare this quantity against log probabilities computed under random permutations of the same dataset. The null hypothesis is that the dataset is exchangable, meaning that its ordering carries no information relevant to the model’s log probability. Under this hypothesis, the likelihood of the original ordering should not be systematically larger than the likelihoods of permuted orderings.

A direct approach is a permutation test. For each permutation $\pi$π, the authors compute

They then estimate the probability that the canonical ordering has lower log probability than a randomly permuted version:

Because enumerating all permutations is intractable, this expectation is replaced by a Monte Carlo estimate using $m$m sampled permutations. The resulting p-value estimator is

The test rejects the null hypothesis of clean data when $\hat{p}$p^​ is smaller than a chosen significance level $\alpha$α. This formulation is conceptually simple and gives finite-sample validity under standard permutation testing theory. However, it has important computational and statistical limitations. The runtime scales with the number of permutations and the sequence length, approximately $O(m|X|)$O(m∣X∣) in terms of log probability evaluations. More importantly, the smallest attainable p-value is bounded below by $1/(m+1)$1/(m+1), which can be too large for applications requiring very strict significance thresholds or multiple hypothesis testing corrections.

To improve the practical utility of the test, the authors propose a sharded likelihood comparison procedure. Instead of asking whether the canonical ordering has a higher likelihood than almost all permuted orderings, the test asks whether it has a higher likelihood than the average permuted ordering. This is a weaker but more practically achievable condition. It does not require the original ordering to occupy an extreme tail of the permutation distribution. Instead, it only requires that the model consistently favors the canonical ordering over randomly shuffled versions on average.

The sharded method first partitions the examples in the dataset into contiguous shards. If the dataset contains $X_1, \dots, X_n$X1​,…,Xn​, the authors group adjacent examples into shards such as

where each shard contains at least $k = n/r$k=n/r examples for $r$r total shards. These shards are designed to be independent units for statistical aggregation. The key idea is that random permutations are applied within shards, while comparisons are made between the original ordering and Monte Carlo estimates of the average permuted ordering.

For each shard, the method computes a likelihood comparison statistic:

Here, $\operatorname{Mean}_\pi$Meanπ​ denotes the average log probability over multiple random permutations. A positive value of $s_i$si​ indicates that the model assigns higher log probability to the canonical ordering than to shuffled variants, which is evidence against the null hypothesis.

The shard statistics are then aggregated across all shards using their mean,

Finally, the authors test whether $s$s is consistent with a zero-mean quantity using a one-sided t-test. If the test strongly rejects the null hypothesis that $s$s is zero, the method concludes that the model appears to assign an abnormally high likelihood to the original dataset ordering, which can indicate contamination.

This sharded likelihood comparison test retains the core intuition of the permutation test while changing the statistical mechanism. It converts a single rank-based comparison into a collection of mean-based comparisons followed by aggregation. This design reduces the burden on the tail of the permutation distribution. The test does not require the canonical log probability to exceed nearly every permuted log probability. It only requires that the canonical ordering consistently receive a higher log probability than the average permuted ordering.

From a computational perspective, the sharded method still requires $O(m|X|)$O(m∣X∣) log probability evaluations when $m$m permutations are used. However, the resulting p-values are not constrained by the same finite-sample lower bound that limits the naive permutation test. In practice, the p-values produced by the sharded test can decay much more rapidly, making the method more useful when researchers need small significance values or apply stringent evaluation protocols.

The authors also provide a false positive guarantee for the sharded test. Under the null hypothesis, assuming an i.i.d dataset and finite second moments of the relevant log probabilities, the aggregate statistic is expected to behave like a zero-mean random variable. The shard-level statistics are independent because the examples are split into disjoint shards. By the central limit theorem, the aggregate statistic converges to a Gaussian distribution, allowing a one-sided t-test to produce asymptotically valid p-values. This gives the method a formal calibration property under the clean-data assumption, although the guarantee is asymptotic rather than finite-sample.

Experiment

The experiments first validate the contamination test on a model trained with known benchmark examples inserted at varying duplication rates, showing that the method detects substantial duplication efficiently and can identify small amounts of contamination, while sharding and permutation choices remain robust. Ablations indicate detection improves with moderate sharding and more permutations, with diminishing returns beyond moderate permutation counts. When applied to public models and benchmarks, the procedure generally finds no pervasive verbatim contamination, except for borderline indications in one model-benchmark pair, and produces mild heuristic evidence for contamination in MMLU after handling non-exchangeability.

Training a 1.4B language model with small injected benchmark subsets shows that the contamination test becomes strongly significant when the test set appears multiple times in pretraining. At low duplication counts, the permutation and sharded likelihood p-values remain generally non-significant, but detection becomes very clear once duplication reaches ten or more. The results suggest the test can identify modest contamination in realistic model and dataset sizes. Detection strength rises sharply with duplication count. Small injected sets of about 1000 examples produce significant p-values at ten or more duplications. Single-copy injections do not provide reliable contamination signals.

The contamination tests show mostly nonsignificant p-values across the evaluated open models and public benchmarks, with no strong evidence of pervasive test set contamination. A low result for ARC on Mistral stands out, but the authors note it is near the boundary of statistical significance and should be interpreted cautiously. For MMLU, filtered and aggregated results produce small p-values for LLaMA2 and Mistral, consistent with reports of mild contamination. Most p-values for the evaluated models and benchmarks are nonsignificant. The BioMedLM negative control also yields nonsignificant values, suggesting the tested benchmarks are not broadly triggering false positives. Filtered MMLU tests indicate possible mild contamination for LLaMA2 and Mistral.

Experiments first validate contamination detection by injecting benchmark examples into a 1.4B model, showing that signals emerge sharply only after substantial duplication, while single copies remain unreliable. Across public open models and benchmarks, the tests mostly yield nonsignificant results, indicating no widespread test set leakage, though possible mild contamination is observed for MMLU on LLaMA2 and Mistral and an ARC edge case is noted. A negative control on BioMedLM supports that the method does not broadly produce false positives.