HyperAIHyperAI

Command Palette

Search for a command to run...

Posterior Matching Distillation

Hidir Yesiltepe Julius Berner Arash Vahdat Pinar Yanardag

Zusammenfassung

Diffusionsdestillation beschleunigt die Bildund Videogenerierung, indem der iterative Sampling-Prozess eines vortrainierten Lehrermodells in wenige Netzwerkauswertungen komprimiert wird. Verteilungsbasierte Verfahren erreichen dies, indem sie die Ausgabeverteilung des Schülers an die des Lehrers angleichen. Dieses Matching wird häufig durch Reverse-KL-Ziele gesteuert, deren modussuchende Verzerrung dominante Lehrermoden auf Kosten der Diversität über die Stichproben hinweg bevorzugt. Wir stellen Posterior Matching Distillation (PMD) vor, ein neues Mitglied der Familie der verteilungsbasierten Destillation, das modusüberdeckendes Posterior-Sampling mit Generierung in beliebig vielen Schritten kombiniert. Bei verschwindendem Populationsverlust rekonstruiert PMD die vollständige Posteriorverteilung sauberer Bilder des Lehrers auf jeder trainierten Rauschstufe. Sein stochastischer Schüler unterstützt beliebige Sampling-Budgets innerhalb des trainierten Zeitschrittgitters ohne weiteres Training. Um diese Posteriorverteilungen zu lernen, gleicht PMD exponentiell umgewichtete Mittelwerte von Schülerstichproben, bedingt auf einen festen verrauschten Zustand, an die entsprechenden Posterior-Mittelwerte des Lehrers an. Unter Gaußscher Korruption ist der exponentiell gekippte Posterior-Mittelwert eine einzige Lehrerauswertung an einem verschobenen Eingang. Wir beweisen, dass das Angleichen gekippter Mittelwerte über eine offene Umgebung die vollständige Posteriorverteilung eindeutig identifiziert. PMD überwacht somit die gesamte bedingte Verteilung durch eine Familie von Mittelwert-Nebenbedingungen, ohne ein zusätzliches Schüler-Score-Netzwerk. Experimente mit SDXL und FLUX.2 [klein] 4B zeigen, dass PMD bei beiden Modellen den besten FID unter den 4-Schritt-Verfahren erreicht und gleichzeitig wettbewerbsfähige oder führende Leistungen bei Metriken zur Prompt-Ausrichtung und Wahrnehmungsqualität erzielt.

One-sentence Summary

Researchers from Virginia Tech and NVIDIA introduce Posterior Matching Distillation (PMD), a distribution-based diffusion distillation method that matches exponentially reweighted student-sample averages to teacher posterior means and thereby uniquely identifies the full posterior without an auxiliary student-score network, achieving the best 4-step FID on both SDXL and FLUX.2 [klein] 4B.

Key Contributions

  • The paper introduces Posterior Matching Distillation (PMD), a distribution-based distillation method that combines mode-covering posterior sampling with any-step generation. At zero population loss, PMD recovers the teacher's full posterior of clean images at each trained noise level, and its stochastic student supports any-step sampling within the trained timestep grid without further training.
  • PMD supervises the full conditional posterior by matching exponentially reweighted averages of student samples conditioned on a fixed noisy state to corresponding teacher posterior means. Under Gaussian corruption, each exponentially tilted posterior mean is obtained from a single frozen-teacher evaluation at a shifted input, and matching these tilted means over an open neighborhood is proven to uniquely identify the full posterior without an auxiliary student-score network.
  • Experiments on SDXL and FLUX.2 [klein] 4B show that PMD attains the best FID among 4-step methods on both models while remaining competitive or leading on prompt-alignment and perceptual-quality metrics.

Introduction

Diffusion and flow models are powerful generative models for images, video, and text, but their iterative denoising process typically requires many neural network evaluations. Distillation reduces this cost by training a one- or few-step student, yet prior work differs in what it preserves: trajectory-based methods compress the teacher’s sampling path, while distribution-based methods such as Distribution Matching Distillation and Moment Matching align generated marginals or conditional means. These approaches can still miss conditional structure, for example DMD’s approximate reverse-KL objective can be mode-seeking and reduce diversity, and Moment Matching compares means at a re-noised student prediction rather than at the original noisy input. The authors propose Posterior Matching Distillation (PMD), which directly matches the teacher’s conditional posterior at each noise level using exponentially tilted means. A key Tilt=Shift identity allows each target to be obtained from a single frozen-teacher evaluation at a shifted noisy input, avoiding expensive teacher posterior sampling while providing constraints that uniquely identify the full multimodal conditional posterior.

Method

The authors introduce Posterior Matching Distillation (PMD) to distill a diffusion teacher into a few-step student by matching the full conditional denoising posterior rather than just the posterior mean. Under the Gaussian corruption process, the conditional distribution over clean images for a noisy observation ztz_tzt​ is given by q(x∣zt)∝q0(x)exp⁡(−∥zt−αtx∥2/2σt2)q(x \mid z_t) \propto q_0(x) \exp(-\|z_t - \alpha_t x\|^2 / 2\sigma_t^2)q(x∣zt​)∝q0​(x)exp(−∥zt​−αt​x∥2/2σt2​). At high noise levels, this posterior can be broad or multimodal, making it crucial to model the conditional uncertainty for few-step generation.

To capture information beyond the posterior mean, the authors consider an exponential tilt of the teacher's conditional distribution using a tilt vector λ∈Rd\lambda \in \mathbb{R}^dλ∈Rd. The tilted distribution is defined as qλ(x∣zt)=eλ⊤xq(x∣zt)Eq(x∣zt)[eλ⊤x]q_\lambda(x \mid z_t) = \frac{e^{\lambda^\top x} q(x \mid z_t)}{\mathbb{E}_{q(x \mid z_t)}[e^{\lambda^\top x}]}qλ​(x∣zt​)=Eq(x∣zt​)​[eλ⊤x]eλ⊤xq(x∣zt​)​. As shown in the figure below:

The exponential reweighting factor eλ⊤xe^{\lambda^\top x}eλ⊤x changes the relative prominence of the posterior's modes, giving greater weight to samples aligned with λ\lambdaλ. Different tilts emphasize different modes of the same posterior. Computing the means of these tilted distributions would typically require sampling and reweighting. However, the authors establish the Tilt=Shift Identity, which states that for a fixed noise level, exponentially tilting the posterior is exactly equivalent to shifting the noisy input to the denoiser: qλ(x∣zt)=q(x∣zt+ρtλ)q_\lambda(x \mid z_t) = q(x \mid z_t + \rho_t \lambda)qλ​(x∣zt​)=q(x∣zt​+ρt​λ), where ρt=σt2/αt\rho_t = \sigma_t^2 / \alpha_tρt​=σt2​/αt​. Consequently, the mean of the tilted posterior is obtained directly from the frozen teacher denoiser as Eqλ(x∣zt)[x]=Dt(zt+ρtλ)\mathbb{E}_{q_\lambda(x \mid z_t)}[x] = D_t(z_t + \rho_t \lambda)Eqλ​(x∣zt​)​[x]=Dt​(zt​+ρt​λ).

The authors prove that matching these tilted means over an open neighborhood of λ=0\lambda = 0λ=0 is sufficient to identify the entire conditional distribution. Let GθG_\thetaGθ​ denote a stochastic student that predicts a clean image x^=Gθ(zt,u,t)\hat{x} = G_\theta(z_t, u, t)x^=Gθ​(zt​,u,t) with random input u∼N(0,I)u \sim \mathcal{N}(0, I)u∼N(0,I). PMD requires the student's exponentially weighted mean to match the teacher's tilted mean:

Eu[x^eλ⊤x^]Eu[eλ⊤x^]=Dt(zt+ρtλ)\frac{\mathbb{E}_u[\hat{x} e^{\lambda^\top \hat{x}}]}{\mathbb{E}_u[e^{\lambda^\top \hat{x}}]} = D_t(z_t + \rho_t \lambda)Eu​[eλ⊤x^]Eu​[x^eλ⊤x^]​=Dt​(zt​+ρt​λ)

If this holds for every tilt in the neighborhood, the student recovers the teacher's full conditional posterior pGθ(⋅∣zt)=q(⋅∣zt)p_{G_\theta}(\cdot \mid z_t) = q(\cdot \mid z_t)pGθ​​(⋅∣zt​)=q(⋅∣zt​).

To operationalize this, the authors define the posterior matching residual r(zt,λ)r(z_t, \lambda)r(zt​,λ) as the difference between the student's and teacher's tilted means. The Posterior Matching Loss minimizes the expected squared residual over noise levels, noisy inputs, and tilts:

LPMD(θ)=Et,zt,λ[∥r(zt,λ)∥22]\mathcal{L}_{\text{PMD}}(\theta) = \mathbb{E}_{t, z_t, \lambda} \left[ \| r(z_t, \lambda) \|_2^2 \right]LPMD​(θ)=Et,zt​,λ​[∥r(zt​,λ)∥22​]

This objective matches a property of the student's conditional distribution rather than forcing individual predictions to equal the teacher's mean.

During training, PMD is trained on noisy states encountered during the student's own sampling process. At each iteration, a training noise level ttt is selected, and ztz_tzt​ is obtained through backward simulation from Gaussian noise zTz_TzT​, with preceding steps detached from the gradient graph. Given ztz_tzt​, a tilt vector λ\lambdaλ is sampled, and the frozen teacher is evaluated at the shifted input zt+ρtλz_t + \rho_t \lambdazt​+ρt​λ to obtain the teacher's tilted posterior mean. The student generates MMM independent predictions x^(m)=Gθ(zt,u(m),t)\hat{x}^{(m)} = G_\theta(z_t, u^{(m)}, t)x^(m)=Gθ​(zt​,u(m),t). The student's tilted mean is estimated using weighted Monte Carlo samples, where weights are derived from a softmax over the dot products λ⊤x^(m)\lambda^\top \hat{x}^{(m)}λ⊤x^(m). The final student objective augments the PMD loss with an auxiliary non-saturating logistic GAN loss:

Lstudent=LPMD+γGANLGAN\mathcal{L}_{\text{student}} = \mathcal{L}_{\text{PMD}} + \gamma_{\text{GAN}} \mathcal{L}_{\text{GAN}}Lstudent​=LPMD​+γGAN​LGAN​

where the discriminator operates on caption-conditioned features extracted by the frozen teacher.

Finally, the authors design PMD to support any-step sampling without training separate students for different step counts. Because PMD learns the conditional posterior of clean images at each noise level rather than explicit transitions between noise levels, the student can predict a clean image from the current noisy state. If further steps are required, this prediction is re-noised to the next selected level using the Gaussian corruption schedule, and the student is evaluated again.

Experiment

The experiments evaluate Posterior Matching Distillation on SDXL and FLUX.2 4B text-to-image models, primarily with four sampling steps, against several few-step baselines and an undistilled reference. Qualitative comparisons show PMD generates more varied subject appearance, scene composition, and backgrounds while preserving prompt semantics, and it supports diverse sampling at different inference step counts. The results also indicate PMD achieves strong distributional quality and prompt adherence, and intra-prompt diversity experiments consistently favor PMD across feature-space, perceptual, and Vendi metrics. Ablations further show that coarse tilt sampling improves diversity and that the stochastic student draw budget affects diversity metrics during training.

On FLUX.2 [klein] 4B, PMD achieves the best FID among four-step methods and leads in HPSv3, PickScore, and CLIP-S. The FLUX.2 reference model posts higher GenEval, DPG-Bench, and OneIG-EN than PMD, revealing a trade-off in prompt-alignment metrics. Moment Matching trails the other four-step approaches on most reported quality and fidelity measures. PMD has the best FID and the highest HPSv3, PickScore, and CLIP-S among the four-step FLUX.2 methods shown. The FLUX.2 reference model outperforms PMD in GenEval, DPG-Bench, and OneIG-EN, while PMD improves FID over that reference.

PMD achieves the strongest diversity results on PartiPrompts among the compared methods, with the highest scores on Vendi, LPIPS, DINOv2, and DreamSim. The gains are most pronounced for DINOv2, DreamSim, and Vendi diversity, while LPIPS improves more modestly. Reported comparisons across SDXL and FLUX.2 indicate a consistent advantage in intra-prompt diversity under fixed prompts and noise. PMD records the highest diversity values across all four metrics for SDXL, surpassing LCM-LoRA, SDXL-Turbo, SDXL-Lightning, Flash, PCM, and DMD2. The relative improvement over the strongest baseline is largest for DINOv2 and DreamSim, while LPIPS shows a smaller gain. A similar diversity advantage is reported for FLUX.2, where PMD improves over baselines across all metrics.

The experiments evaluate four-step text-to-image methods on FLUX.2 and SDXL using quality, prompt-alignment, and diversity benchmarks. PMD achieves the best FID and top HPSv3, PickScore, and CLIP-S among four-step FLUX.2 methods, while the FLUX.2 reference model scores higher on GenEval, DPG-Bench, and OneIG-EN, indicating a trade-off in prompt-alignment metrics; Moment Matching generally trails. On PartiPrompts, PMD consistently yields the highest intra-prompt diversity across Vendi, LPIPS, DINOv2, and DreamSim for both SDXL and FLUX.2, with the largest gains in DINOv2, DreamSim, and Vendi and more modest LPIPS improvement.


KI mit KI entwickeln

Von der Idee bis zum Launch – beschleunigen Sie Ihre KI-Entwicklung mit kostenlosem KI-Co-Coding, sofort einsatzbereiter Umgebung und bestem GPU-Preis.

KI-gestütztes kollaboratives Programmieren
Sofort einsatzbereite GPUs
Die besten Preise

HyperAI Newsletters

Abonnieren Sie unsere neuesten Updates
Wir werden die neuesten Updates der Woche in Ihren Posteingang liefern um neun Uhr jeden Montagmorgen
Unterstützt von MailChimp