One-sentence Summary

The authors develop a random matrix theory framework that attributes the consistency of diffusion models trained on disjoint data subsets to shared Gaussian statistics, uncovering a self-consistent noise renormalization $\sigma^2 \mapsto \kappa(\sigma^2)$σ2↦κ(σ2) that overshrinks low-variance directions and deriving variance formulas that predict cross-split disagreement by extending deterministic-equivalence tools to fractional matrix powers, with validation on UNet and DiT architectures.

Key Contributions

• Diffusion models trained on nonoverlapping data splits produce strikingly similar outputs given the same noise seed; this consistency is largely explained by shared Gaussian statistics that already predict much of the generated images.
• A random matrix theory framework quantifies how finite datasets shape the expectation and variance of the linear denoiser and sampling map. It reveals that sampling variability renormalizes the noise level to overshrink low-variance directions, and that cross-split variance decomposes into anisotropy across eigenmodes, inhomogeneity across inputs, and scaling with dataset size.
• Deterministic-equivalence tools are extended to fractional matrix powers to derive closed-form predictions for entire sampling trajectories. The theory is validated on UNet and DiT architectures in their non-memorization regime, identifying where and how sample deviations emerge across data splits.

Introduction

Diffusion models exhibit a striking consistency across training runs: when trained on the same data distribution but with non-overlapping splits, different architectures, or repeated initializations, they often map the same noise seed to highly similar outputs. This contrasts with other generative frameworks like GANs and VAEs, where latent spaces are rotationally ambiguous. The phenomenon suggests that diffusion models recover universal statistical structure of the data manifold, raising fundamental questions about generalization and memorization. Prior work lacked a rigorous theoretical explanation for this consistency. The authors address this gap by leveraging random matrix theory (RMT). They show that a linear denoiser already predicts cross-split agreement, and that finite-sample variability enters through a renormalized noise scale that explains overshrinkage of low-variance modes. The analysis derives a variance law that factors deviation into anisotropic eigenmode contributions, input-dependent inhomogeneity, and a global scaling with dataset size. By extending deterministic equivalence to fractional matrix powers, the authors enable analysis of full sampling trajectories, and they validate that these RMT principles qualitatively govern consistency in deep CNN and DiT models beyond the linear regime.

Method

The authors leverage random matrix theory toanalyze the consistency of diffusion models trained on independent data splits. The core technical tool is deterministic equivalence, which allows the empirical covariance matrix to be replaced by a deterministic surrogate in the large-dimensional limit. This approach leads to a self-consistent equation for a renormalized noise scale $\kappa(\lambda)$κ(λ), where the stochastic effects of the sample covariance are absorbed into a scalar, leaving the population covariance unchanged.

In terms of expectation, the authors demonstrate that finite data effectively renormalize the noise scale in the population denoiser. This mechanism acts as an adaptive Ridge penalty, causing the finite-sample denoiser to shrink low-variance directions more aggressively toward the dataset mean. When extending this analysis to the full diffusion sampling map, which involves fractional powers of covariances, the authors utilize an integral representation to derive the deterministic equivalence. The resulting expectation reveals a systematic overshrinkage toward the dataset mean along lower eigenmodes, thereby reducing the generated variance in these directions.

To understand the fluctuation and consistency of the denoiser across different dataset realizations, the authors decompose the variance into three interpretable components: anisotropy, inhomogeneity, and global scaling with dataset size. The anisotropy factor shows that uncertainty is maximized along eigenmodes whose variance matches the renormalized noise. The inhomogeneity factor indicates that uncertainty is amplified for inputs displaced along high-variance modes.

The authors validate these theoretical predictions through numerical simulations of the linear diffusion sampling map. As illustrated in the figure, the expected scaling along eigenmodes confirms the overshrinkage effect on lower eigenmodes compared to the ideal scaling. The cross-split mean squared error demonstrates the anisotropy of consistency, showing larger deviations along the top eigenspaces, and the inhomogeneity effect, where samples displaced along high-variance modes exhibit greater disagreement. Furthermore, the decomposition of the mean squared error across eigenbands reveals that lower-variance modes require substantially larger dataset sizes before the cross-split deviation decays, highlighting that fine details need more data to achieve consistency across training splits.

Experiment

Diffusion models trained on independent data splits yield near-identical samples, a consistency largely captured by a linear Gaussian predictor and arising from shared covariance statistics. Finite-sample analysis shows that limited data renormalize noise scales, causing overshrinkage toward the mean and uneven denoiser agreement: uncertainty peaks along eigenmodes whose variance matches the renormalized noise and for inputs displaced along high-variance directions. These predictions extend to deep networks, where the same bias and spectral variance patterns emerge once the dataset size moves beyond the memorization regime, confirming that linear random matrix theory captures key aspects of diffusion model behavior under finite data.