One-sentence Summary

the paper propose score entropy, a loss that extends score matching to discrete spaces, and build Score Entropy Discrete Diffusion (SEDD), which reduces perplexity by $25\text{-}75\%$25-75% over prior diffusion methods, outperforms GPT-2 on language modeling, and generates faithful text without distribution annealing.

Key Contributions

• Score entropy discrete diffusion (SEDD) is introduced as a discrete diffusion model for language, parameterized by the concrete score and trained with a novel score entropy loss that extends score matching to discrete token spaces.
• The method replaces high-variance implicit score entropy and prior heuristic training objectives with a principled denoising score entropy objective, enabling more efficient and scalable training.
• On standard language modeling tasks, SEDD reduces perplexity by 25-75% relative to existing language diffusion models, outperforms GPT-2, achieves about 6-8x better un-annealed generative perplexity than GPT-2, and can match quality with 32x fewer network evaluations while supporting controllable infilling.

Introduction

Deep generative models now support powerful text generation and question answering, but autoregressive transformers remain the main competitive approach despite slow token-by-token sampling, limited controllability, and reliance on sampling tricks such as nucleus sampling. Earlier diffusion-based language models, especially discrete diffusion methods, have struggled because their training objectives are often unstable or poorly suited to discrete data, and they usually lag behind autoregressive models in likelihoods and sample quality without heavy empirical fixes. The authors introduce Score Entropy Discrete Diffusion models, which learn the reverse diffusion process through probability ratios using a new score entropy loss, yielding a principled and scalable method that improves language modeling, enables high-quality unannealed sampling, and supports flexible generation and infilling.

Method

The authors leverage a score entropy parameterization to train discrete diffusion models by learning positive, ratio-based scores rather than directly modeling normalized transition probabilities. The central object is a score network $s_\theta(x,t)$sθ​(x,t) whose coordinate $s_\theta(x,t)_y$sθ​(x,t)y​ is intended to approximate the concrete score ratio

$$s_\theta(x,t)_y \approx \frac{p_t(y)}{p_t(x)}, \qquad y \neq x.$$sθ​(x,t)y​≈pt​(x)pt​(y)​,y=x.

This means the model learns how much more likely, or less likely, an alternative state $y$y is compared with the current state $x$x under the noisy marginal $p_t$pt​. Because these ratios are strictly positive, the training objective is designed as a score entropy loss that naturally incorporates positivity and convexity. For a distribution $p$p and nonnegative weights $w_{xy}$wxy​, the score entropy takes the form

$$\mathcal{L}_{\mathrm{SE}} = \mathbb{E}_{x \sim p} \left[ \sum_{y \neq x} w_{xy} \left( s_\theta(x)_y - \frac{p(y)}{p(x)} \log s_\theta(x)_y + K\left(\frac{p(y)}{p(x)}\right) \right) \right],$$LSE​=Ex∼p​​y=x∑​wxy​(sθ​(x)y​−p(x)p(y)​logsθ​(x)y​+K(p(x)p(y)​))​,

where

$$K(a) = a(\log a - 1)$$K(a)=a(loga−1)

acts as a normalizing constant ensuring nonnegativity. The loss is built from a Bregman divergence with a negative logarithmic generator, which gives it desirable properties such as nonnegativity and convexity. Under full support and positive weights, the minimizer recovers the ground truth score ratios, meaning the learned scores satisfy

$$s_{\theta^*}(x)_y = \frac{p(y)}{p(x)}.$$sθ∗​(x)y​=p(x)p(y)​.

The method extends this objective to scalable discrete diffusion by using a denoising version of score entropy. When the intermediate distribution $p$p is generated by perturbing a clean data distribution $p_0$p0​ through a transition kernel $p(\cdot \mid x_0)$p(⋅∣x0​), the score entropy can be equivalently trained through

$$\mathcal{L}_{\mathrm{DSE}} = \mathbb{E}_{\substack{x_0 \sim p_0 \\ x \sim p(\cdot \mid x_0)}} \left[ \sum_{y \neq x} w_{xy} \left( s_\theta(x)_y - \frac{p(y \mid x_0)}{p(x \mid x_0)} \log s_\theta(x)_y \right) \right].$$LDSE​=Ex0​∼p0​x∼p(⋅∣x0​)​​​y=x∑​wxy​(sθ​(x)y​−p(x∣x0​)p(y∣x0​)​logsθ​(x)y​)​.

This formulation is practical because a Monte Carlo step samples a clean point $x_0$x0​, perturbs it to obtain $x$x, and then evaluates the score network once at $x$x. The output $s_\theta(x)$sθ​(x) contains the score estimates for many possible perturbations $y$y, so the training step does not require repeated evaluations for each candidate state.

In diffusion modeling, the marginal $p_t$pt​ is itself a perturbation of the base distribution $p_0$p0​ through a time-dependent forward transition. Therefore, the method uses the tractable conditional transition $p_{t \mid 0}$pt∣0​ inside the loss, replacing $p(x \mid x_0)$p(x∣x0​) and $p(y \mid x_0)$p(y∣x0​) with $p_{t \mid 0}(x_t \mid x_0)$pt∣0​(xt​∣x0​) and $p_{t \mid 0}(y \mid x_t)$pt∣0​(y∣xt​). This yields a diffusion-weighted training objective that directly optimizes the model along the forward diffusion trajectory.

The same score-based construction also supports likelihood-based training and evaluation. The learned score network defines a parameterized reverse transition matrix by replacing true score ratios with learned ones. For off-diagonal entries, the reverse rate is

$$\overline{Q}_t^\theta(y,x) = s_\theta(x,t)_y Q_t(x,y), \qquad x \neq y,$$Q​tθ​(y,x)=sθ​(x,t)y​Qt​(x,y),x=y,

with diagonal entries chosen so that each row sums to zero. The induced reverse-time density satisfies a differential equation of the form

$$\frac{d p_{T-t}^\theta}{dt} = \overline{Q}_{T-t}^\theta p_{T-t}^\theta, \qquad p_T^\theta = p_{\text{base}}.$$dtdpT−tθ​​=Q​T−tθ​pT−tθ​,pTθ​=pbase​.

This gives an upper bound on the negative log likelihood of a data point:

$$-\log p_0^\theta(x_0) \le \mathcal{L}_{\mathrm{DWDSE}}(x_0) + D_{\mathrm{KL}} \left( p_{T \mid 0}(\cdot \mid x_0) \, \Vert \, p_{\text{base}} \right),$$−logp0θ​(x0​)≤LDWDSE​(x0​)+DKL​(pT∣0​(⋅∣x0​)∥pbase​),

where $\mathcal{L}_{\mathrm{DWDSE}}$LDWDSE​ is a diffusion-weighted denoising score entropy. This is particularly useful for language modeling because likelihood and perplexity remain central evaluation metrics.

For sequence data, the state is factorized into $d$d tokens, each taking one of $n$n values. A full transition matrix would be exponentially large, so the method uses a sparse, structured transition kernel that perturbs tokens independently. Nonzero transitions only change one token at a time:

$$Q_t(x^1 \ldots x^i \ldots x^d, x^1 \ldots \widehat{x}^i \ldots x^d) = Q_t^{\text{tok}}(x^i, \widehat{x}^i).$$Qt​(x1…xi…xd,x1…xi…xd)=Qttok​(xi,xi).

Because of this sparsity, only score ratios between sequences with Hamming distance one are needed. The score network is therefore implemented as a sequence-to-sequence map

$$s_\theta(x,t) \in \mathbb{R}^{d \times n},$$sθ​(x,t)∈Rd×n,

where the entry

$$s_\theta(x,t)_{i,\widehat{x}^i} \approx \frac{ p_t(x^1 \ldots \widehat{x}^i \ldots x^d) }{ p_t(x^1 \ldots x^i \ldots x^d) }$$sθ​(x,t)i,xi​≈pt​(x1…xi…xd)pt​(x1…xi…xd)​

compares the current sequence with the sequence formed by replacing token $i$i with $\widehat{x}^i$xi.

The forward transition for a whole sequence factorizes because tokens are perturbed independently:

$$p_{t \mid 0}^{\text{seq}}(\widehat{\mathbf{x}} \mid \mathbf{x}) = \prod_{i=1}^d p_{t \mid 0}^{\text{tok}}(\widehat{x}^i \mid x^i).$$pt∣0seq​(x∣x)=i=1∏d​pt∣0tok​(xi∣xi).

For token transitions, the method sets

$$Q_t^{\text{tok}} = \sigma(t) Q^{\text{tok}},$$Qttok​=σ(t)Qtok,

where $\sigma(t)$σ(t) is a noise schedule. Defining the cumulative noise

$$\overline{\sigma}(t) = \int_0^t \sigma(s)\,ds,$$σ(t)=∫0t​σ(s)ds,

the token transition probabilities can be computed from matrix exponentials without numerical integration:

$$p_{t \mid 0}^{\text{tok}}(\cdot \mid x) = \text{the } x\text{-th column of } \exp\left(\overline{\sigma}(t) Q^{\text{tok}}\right).$$pt∣0tok​(⋅∣x)=the x-th column of exp(σ(t)Qtok).

For large vocabularies, explicitly storing and exponentiating a full token transition matrix is infeasible. The method therefore uses structured token transition kernels. A uniform transition kernel couples all token values symmetrically and has entries proportional to all-ones off-diagonal terms. An absorbing transition kernel introduces a mask token, allowing each non-mask token to transition into the mask state while the mask state remains absorbing. These structures make the transition probabilities and score-weighted losses cheap to compute, allowing training to proceed with speed and memory usage comparable to standard autoregressive sequence modeling.

Once trained, the learned scores are used to simulate the reverse diffusion process. A simple Euler-style reverse step would only update one token per step, which is inefficient for long sequences. The method instead uses a $\tau$τ-leaping strategy that updates all token positions simultaneously. At each step, token $i$i is updated according to a local probability influenced by the learned score:

$$\delta_{x_t^i}(x_{t-\Delta t}^i) + \Delta t\, Q_t^{\text{tok}}(x_t^i, x_{t-\Delta t}^i) s_\theta(\mathbf{x}_t,t)_{i,x_{t-\Delta t}^i}.$$δxti​​(xt−Δti​)+ΔtQttok​(xti​,xt−Δti​)sθ​(xt​,t)i,xt−Δti​​.

This makes each position independently sampleable while the full sequence condition remains in the score estimate.

The method further improves this sampling scheme by using a discrete analogue of Tweedie’s theorem. Since the learned scores approximate ratios of the current marginal, they provide information about the optimal reverse-time denoising distribution. The resulting Tweedie $\tau$τ-leaping update replaces the basic local transition probability with

$$\left( \exp(-\sigma_t^{\Delta t} Q^{\text{tok}}) s_\theta(\mathbf{x}_t,t)_i \right)_{x_{t-\Delta t}^i} \exp(\sigma_t^{\Delta t} Q^{\text{tok}}) (x_t^i, x_{t-\Delta t}^i),$$(exp(−σtΔt​Qtok)sθ​(xt​,t)i​)xt−Δti​​exp(σtΔt​Qtok)(xti​,xt−Δti​),

where

$$\sigma_t^{\Delta t} = \overline{\sigma}(t) - \overline{\sigma}(t-\Delta t).$$σtΔt​=σ(t)−σ(t−Δt).

This update retains the independent token-wise structure of $\tau$τ-leaping while incorporating the optimal denoising interpretation provided by the score model. When the learned scores are perfect, this sampling rule is optimal among $\tau$τ-leaping strategies in terms of minimizing divergence from the true reverse transition.

The same score formulation also enables arbitrary prompting and infilling without retraining a conditional model. For a set of filled indices $\overline{\Omega}$Ω and unfilled indices $\Omega$Ω, the conditional distribution

$$p_t(\mathbf{x}^\Omega \mid \mathbf{x}^{\overline{\Omega}} = \mathbf{y})$$pt​(xΩ∣xΩ=y)

can be handled through Bayes’ rule. Conditional score ratios reduce to unconditional score ratios evaluated on sequences completed with the fixed context:

$$\frac{ p_t(\mathbf{x}^\Omega = \mathbf{z}' \mid \mathbf{x}^{\overline{\Omega}} = \mathbf{y}) }{ p_t(\mathbf{x}^\Omega = \mathbf{z} \mid \mathbf{x}^{\overline{\Omega}} = \mathbf{y}) } = \frac{ p_t(\mathbf{x} = \mathbf{z}' \oplus_\Omega \mathbf{y}) }{ p_t(\mathbf{x} = \mathbf{z} \oplus_\Omega \mathbf{y}) }.$$pt​(xΩ=z∣xΩ=y)pt​(xΩ=z′∣xΩ=y)​=pt​(x=z⊕Ω​y)pt​(x=z′⊕Ω​y)​.

Here, $\oplus_\Omega$⊕Ω​ denotes concatenation of the unfilled pattern and the fixed context. During sampling, filled positions are frozen and only unfilled positions are updated, allowing the same learned unconditional score network to perform flexible conditional generation.

Experiment

The experiments evaluate SEDD on language modeling and generation using transformer-based diffusion variants trained with matched data, tokenizers, and model scales. Likelihood results on character-level and word-level corpora show strong competitiveness, with SEDD Absorb surpassing prior diffusion baselines and approaching or exceeding GPT-2 perplexity on several zero-shot tasks. Generation evaluations further validate high-quality unconditional sampling, fast step-budget tradeoffs, and strong infilling and conditional prompting without explicit autoregressive induction. Overall, the results indicate that SEDD improves both modeling quality and generation versatility over existing discrete diffusion methods.

For zero-shot unconditional language modeling, the SEDD model with absorbing transitions provides the strongest perplexity among the compared diffusion baselines. It matches or surpasses GPT-2 on most standard datasets, supporting the claim that a non-autoregressive diffusion model can reach perplexity quality comparable to a modern autoregressive model. The comparison also shows a clear advantage over prior language diffusion approaches. Absorbing SEDD gives better perplexity than uniform SEDD and earlier language diffusion baselines. Absorbing SEDD outperforms GPT-2 on a majority of zero-shot tasks while remaining competitive on the others.

On text8 bits-per-character language modeling, SEDD delivers the strongest non-autoregressive result while remaining close to leading autoregressive-based approaches. It improves substantially over prior discrete diffusion models such as D3PM and outperforms other non-autoregressive baselines. SEDD achieves the best non-autoregressive performance on text8. SEDD trails autoregressive and autoregressive-based flow methods only slightly. SEDD substantially improves over D3PM despite sharing discrete diffusion principles.

SEDD achieves the strongest diffusion model perplexity on the One Billion Words dataset and is close to the autoregressive transformer benchmark. Because the diffusion results are upper bounds, this suggests that SEDD can effectively match the likelihood quality of the autoregressive model while substantially outperforming prior diffusion baselines. SEDD produces lower perplexity than all compared diffusion baselines, with a large margin over methods such as D3PM. SEDD is within a very small perplexity gap of the autoregressive transformer, indicating competitive likelihood-based performance. This result is presented as evidence that a non-autoregressive language model can match a well-known autoregressive model on perplexity.

The comparison evaluates small-model unconditional text generation with perplexity, showing that SEDD samples outperform GPT-2 samples. SEDD reaches competitive quality with far fewer network evaluations, and longer sampling step budgets provide further improvements. The generated examples also show a trend from fragmented GPT-2 text to more coherent SEDD text. SEDD small and medium models outperform GPT-2 small and medium models on generated-text perplexity. Using more SEDD sampling steps follows a predictable quality versus step count tradeoff and improves results. Example outputs show SEDD producing more coherent text than the corresponding GPT-2 samples.

The qualitative samples show that conditional generation can use prompt text at the beginning, end, middle, or in separated fragments. The related experiments indicate that this flexible prompting is compatible with competitive language generation quality, including strong standard prompting results against compared baselines. This is notable because the approach does not use distribution annealing or an explicit left-to-right architectural bias. Prompt tokens can be placed at the front, end, middle, or in split positions Both standard generation and infilling with beginning and end context are supported Standard prompting is competitive with, and in one case better than, the compared baselines

The experiments evaluate SEDD through zero-shot unconditional language modeling on standard corpora and small-model text generation, comparing its perplexity and sampled outputs against GPT-2, autoregressive baselines, and prior discrete diffusion models. Across these settings, absorbing SEDD is shown to outperform earlier diffusion baselines and approach or match autoregressive performance on likelihood, while providing more coherent generated text than GPT-2 with fewer network evaluations. Conditional experiments further demonstrate that flexible prompt placement and infilling are supported without architectural left-to-right bias, yielding competitive generation quality.