HyperAIHyperAI

Command Palette

Search for a command to run...

사후 분포 정합 증류

Hidir Yesiltepe Julius Berner Arash Vahdat Pinar Yanardag

초록

확산 증류는 사전 학습된 교사의 반복적 샘플링 과정을 소수의 네트워크 평가로 압축하여 이미지 및 비디오 생성을 가속화한다. 분포 기반 방법은 학생의 출력 분포를 교사의 분포에 정합함으로써 이를 달성한다. 이러한 정합은 흔히 역 KL 목적함수에 의해 추진되는데, 그 모드 추구 편향은 샘플 전반의 다양성을 희생하면서 지배적인 교사 모드를 선호하게 만든다. 본 논문에서는 분포 기반 증류 계열의 새로운 구성원인 사후 분포 정합 증류(PMD)를 소개한다. PMD는 모드 포괄적 사후 샘플링과 임의 단계 생성을 결합한다. 모집단 손실이 0일 때, PMD는 각 학습된 잡음 수준에서 깨끗한 이미지에 대한 교사의 전체 사후 분포를 복원한다. 확률적 학생은 추가 학습 없이 학습된 시간 단계 격자 내에서 임의의 샘플링 단계 예산을 지원한다. 이러한 사후 분포를 학습하기 위해, PMD는 고정된 잡음 상태를 조건으로 하는 학생 샘플의 지수 재가중 평균을 대응하는 교사 사후 평균에 정합한다. 가우시안 손상 하에서 지수 기울임 사후 평균은 이동된 입력에서의 단일 교사 평가로 얻어진다. 우리는 열린 근방에서 기울임 평균을 정합하는 것이 전체 사후 분포를 유일하게 식별함을 증명한다. 따라서 PMD는 보조 학생 점수 네트워크 없이 평균 제약 조건군을 통해 전체 조건부 분포를 지도한다. SDXL 및 FLUX.2 [klein] 4B에 대한 실험은 PMD가 두 모델 모두에서 4단계 방법 중 최고의 FID를 달성하면서, 프롬프트 정렬 및 지각 품질 지표에서 경쟁력 있거나 선도적인 성능을 유지함을 보여준다.

One-sentence Summary

Researchers from Virginia Tech and NVIDIA introduce Posterior Matching Distillation (PMD), a distribution-based diffusion distillation method that matches exponentially reweighted student-sample averages to teacher posterior means and thereby uniquely identifies the full posterior without an auxiliary student-score network, achieving the best 4-step FID on both SDXL and FLUX.2 [klein] 4B.

Key Contributions

  • The paper introduces Posterior Matching Distillation (PMD), a distribution-based distillation method that combines mode-covering posterior sampling with any-step generation. At zero population loss, PMD recovers the teacher's full posterior of clean images at each trained noise level, and its stochastic student supports any-step sampling within the trained timestep grid without further training.
  • PMD supervises the full conditional posterior by matching exponentially reweighted averages of student samples conditioned on a fixed noisy state to corresponding teacher posterior means. Under Gaussian corruption, each exponentially tilted posterior mean is obtained from a single frozen-teacher evaluation at a shifted input, and matching these tilted means over an open neighborhood is proven to uniquely identify the full posterior without an auxiliary student-score network.
  • Experiments on SDXL and FLUX.2 [klein] 4B show that PMD attains the best FID among 4-step methods on both models while remaining competitive or leading on prompt-alignment and perceptual-quality metrics.

Introduction

Diffusion and flow models are powerful generative models for images, video, and text, but their iterative denoising process typically requires many neural network evaluations. Distillation reduces this cost by training a one- or few-step student, yet prior work differs in what it preserves: trajectory-based methods compress the teacher’s sampling path, while distribution-based methods such as Distribution Matching Distillation and Moment Matching align generated marginals or conditional means. These approaches can still miss conditional structure, for example DMD’s approximate reverse-KL objective can be mode-seeking and reduce diversity, and Moment Matching compares means at a re-noised student prediction rather than at the original noisy input. The authors propose Posterior Matching Distillation (PMD), which directly matches the teacher’s conditional posterior at each noise level using exponentially tilted means. A key Tilt=Shift identity allows each target to be obtained from a single frozen-teacher evaluation at a shifted noisy input, avoiding expensive teacher posterior sampling while providing constraints that uniquely identify the full multimodal conditional posterior.

Method

The authors introduce Posterior Matching Distillation (PMD) to distill a diffusion teacher into a few-step student by matching the full conditional denoising posterior rather than just the posterior mean. Under the Gaussian corruption process, the conditional distribution over clean images for a noisy observation ztz_tzt​ is given by q(x∣zt)∝q0(x)exp⁡(−∥zt−αtx∥2/2σt2)q(x \mid z_t) \propto q_0(x) \exp(-\|z_t - \alpha_t x\|^2 / 2\sigma_t^2)q(x∣zt​)∝q0​(x)exp(−∥zt​−αt​x∥2/2σt2​). At high noise levels, this posterior can be broad or multimodal, making it crucial to model the conditional uncertainty for few-step generation.

To capture information beyond the posterior mean, the authors consider an exponential tilt of the teacher's conditional distribution using a tilt vector λ∈Rd\lambda \in \mathbb{R}^dλ∈Rd. The tilted distribution is defined as qλ(x∣zt)=eλ⊤xq(x∣zt)Eq(x∣zt)[eλ⊤x]q_\lambda(x \mid z_t) = \frac{e^{\lambda^\top x} q(x \mid z_t)}{\mathbb{E}_{q(x \mid z_t)}[e^{\lambda^\top x}]}qλ​(x∣zt​)=Eq(x∣zt​)​[eλ⊤x]eλ⊤xq(x∣zt​)​. As shown in the figure below:

The exponential reweighting factor eλ⊤xe^{\lambda^\top x}eλ⊤x changes the relative prominence of the posterior's modes, giving greater weight to samples aligned with λ\lambdaλ. Different tilts emphasize different modes of the same posterior. Computing the means of these tilted distributions would typically require sampling and reweighting. However, the authors establish the Tilt=Shift Identity, which states that for a fixed noise level, exponentially tilting the posterior is exactly equivalent to shifting the noisy input to the denoiser: qλ(x∣zt)=q(x∣zt+ρtλ)q_\lambda(x \mid z_t) = q(x \mid z_t + \rho_t \lambda)qλ​(x∣zt​)=q(x∣zt​+ρt​λ), where ρt=σt2/αt\rho_t = \sigma_t^2 / \alpha_tρt​=σt2​/αt​. Consequently, the mean of the tilted posterior is obtained directly from the frozen teacher denoiser as Eqλ(x∣zt)[x]=Dt(zt+ρtλ)\mathbb{E}_{q_\lambda(x \mid z_t)}[x] = D_t(z_t + \rho_t \lambda)Eqλ​(x∣zt​)​[x]=Dt​(zt​+ρt​λ).

The authors prove that matching these tilted means over an open neighborhood of λ=0\lambda = 0λ=0 is sufficient to identify the entire conditional distribution. Let GθG_\thetaGθ​ denote a stochastic student that predicts a clean image x^=Gθ(zt,u,t)\hat{x} = G_\theta(z_t, u, t)x^=Gθ​(zt​,u,t) with random input u∼N(0,I)u \sim \mathcal{N}(0, I)u∼N(0,I). PMD requires the student's exponentially weighted mean to match the teacher's tilted mean:

Eu[x^eλ⊤x^]Eu[eλ⊤x^]=Dt(zt+ρtλ)\frac{\mathbb{E}_u[\hat{x} e^{\lambda^\top \hat{x}}]}{\mathbb{E}_u[e^{\lambda^\top \hat{x}}]} = D_t(z_t + \rho_t \lambda)Eu​[eλ⊤x^]Eu​[x^eλ⊤x^]​=Dt​(zt​+ρt​λ)

If this holds for every tilt in the neighborhood, the student recovers the teacher's full conditional posterior pGθ(⋅∣zt)=q(⋅∣zt)p_{G_\theta}(\cdot \mid z_t) = q(\cdot \mid z_t)pGθ​​(⋅∣zt​)=q(⋅∣zt​).

To operationalize this, the authors define the posterior matching residual r(zt,λ)r(z_t, \lambda)r(zt​,λ) as the difference between the student's and teacher's tilted means. The Posterior Matching Loss minimizes the expected squared residual over noise levels, noisy inputs, and tilts:

LPMD(θ)=Et,zt,λ[∥r(zt,λ)∥22]\mathcal{L}_{\text{PMD}}(\theta) = \mathbb{E}_{t, z_t, \lambda} \left[ \| r(z_t, \lambda) \|_2^2 \right]LPMD​(θ)=Et,zt​,λ​[∥r(zt​,λ)∥22​]

This objective matches a property of the student's conditional distribution rather than forcing individual predictions to equal the teacher's mean.

During training, PMD is trained on noisy states encountered during the student's own sampling process. At each iteration, a training noise level ttt is selected, and ztz_tzt​ is obtained through backward simulation from Gaussian noise zTz_TzT​, with preceding steps detached from the gradient graph. Given ztz_tzt​, a tilt vector λ\lambdaλ is sampled, and the frozen teacher is evaluated at the shifted input zt+ρtλz_t + \rho_t \lambdazt​+ρt​λ to obtain the teacher's tilted posterior mean. The student generates MMM independent predictions x^(m)=Gθ(zt,u(m),t)\hat{x}^{(m)} = G_\theta(z_t, u^{(m)}, t)x^(m)=Gθ​(zt​,u(m),t). The student's tilted mean is estimated using weighted Monte Carlo samples, where weights are derived from a softmax over the dot products λ⊤x^(m)\lambda^\top \hat{x}^{(m)}λ⊤x^(m). The final student objective augments the PMD loss with an auxiliary non-saturating logistic GAN loss:

Lstudent=LPMD+γGANLGAN\mathcal{L}_{\text{student}} = \mathcal{L}_{\text{PMD}} + \gamma_{\text{GAN}} \mathcal{L}_{\text{GAN}}Lstudent​=LPMD​+γGAN​LGAN​

where the discriminator operates on caption-conditioned features extracted by the frozen teacher.

Finally, the authors design PMD to support any-step sampling without training separate students for different step counts. Because PMD learns the conditional posterior of clean images at each noise level rather than explicit transitions between noise levels, the student can predict a clean image from the current noisy state. If further steps are required, this prediction is re-noised to the next selected level using the Gaussian corruption schedule, and the student is evaluated again.

Experiment

The experiments evaluate Posterior Matching Distillation on SDXL and FLUX.2 4B text-to-image models, primarily with four sampling steps, against several few-step baselines and an undistilled reference. Qualitative comparisons show PMD generates more varied subject appearance, scene composition, and backgrounds while preserving prompt semantics, and it supports diverse sampling at different inference step counts. The results also indicate PMD achieves strong distributional quality and prompt adherence, and intra-prompt diversity experiments consistently favor PMD across feature-space, perceptual, and Vendi metrics. Ablations further show that coarse tilt sampling improves diversity and that the stochastic student draw budget affects diversity metrics during training.

On FLUX.2 [klein] 4B, PMD achieves the best FID among four-step methods and leads in HPSv3, PickScore, and CLIP-S. The FLUX.2 reference model posts higher GenEval, DPG-Bench, and OneIG-EN than PMD, revealing a trade-off in prompt-alignment metrics. Moment Matching trails the other four-step approaches on most reported quality and fidelity measures. PMD has the best FID and the highest HPSv3, PickScore, and CLIP-S among the four-step FLUX.2 methods shown. The FLUX.2 reference model outperforms PMD in GenEval, DPG-Bench, and OneIG-EN, while PMD improves FID over that reference.

PMD achieves the strongest diversity results on PartiPrompts among the compared methods, with the highest scores on Vendi, LPIPS, DINOv2, and DreamSim. The gains are most pronounced for DINOv2, DreamSim, and Vendi diversity, while LPIPS improves more modestly. Reported comparisons across SDXL and FLUX.2 indicate a consistent advantage in intra-prompt diversity under fixed prompts and noise. PMD records the highest diversity values across all four metrics for SDXL, surpassing LCM-LoRA, SDXL-Turbo, SDXL-Lightning, Flash, PCM, and DMD2. The relative improvement over the strongest baseline is largest for DINOv2 and DreamSim, while LPIPS shows a smaller gain. A similar diversity advantage is reported for FLUX.2, where PMD improves over baselines across all metrics.

The experiments evaluate four-step text-to-image methods on FLUX.2 and SDXL using quality, prompt-alignment, and diversity benchmarks. PMD achieves the best FID and top HPSv3, PickScore, and CLIP-S among four-step FLUX.2 methods, while the FLUX.2 reference model scores higher on GenEval, DPG-Bench, and OneIG-EN, indicating a trade-off in prompt-alignment metrics; Moment Matching generally trails. On PartiPrompts, PMD consistently yields the highest intra-prompt diversity across Vendi, LPIPS, DINOv2, and DreamSim for both SDXL and FLUX.2, with the largest gains in DINOv2, DreamSim, and Vendi and more modest LPIPS improvement.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp