One-sentence Summary

Amazon Inc. presents the first comprehensive evaluation of generative (autoregressive, masked language modeling, discrete diffusion) and discriminative (encoder) transformer architectures for text classification, revealing that the classical "two regimes" phenomenon manifests distinctly across architectures and training paradigms while offering practical guidance on model selection under real-world constraints such as latency and data limitations.

Key Contributions

• The paper provides a comprehensive evaluation of modern generative and discriminative architectures for text classification, covering autoregressive models, masked language modeling, discrete diffusion, and encoders. It examines accuracy, sample efficiency, calibration, noise robustness, and ordinality.

• The comparison includes pseudo-generative transformer variants, including masked language modeling with label suffixes and append-label formulations, showing that minimal changes to label placement or loss bring standard architectures into a comparable setup.

• Experiments reveal that the classical trade-off between sample complexity and asymptotic error manifests differently across model families and training paradigms. Practical guidance follows: 1-layer encoders are efficient, robust, calibrated, and ordinal for latency-sensitive settings; 12-layer masked language models perform best in data-rich offline tasks across F1, calibration, and ordinal metrics but are less robust to token dropping; discrete diffusion is strong for low-resource noisy cases, especially at 6 layers; and autoregressive models are preferred when calibrated probability outputs are critical.

Introduction

The authors study the discriminative versus generative debate for text classification, a core NLP task used in sentiment analysis, topic classification, and emotion detection. Transformer-based discriminative classifiers are strong but often require large labeled datasets, while generative classifiers can be more data efficient by modeling the joint data distribution, making them attractive for low-resource settings.

Prior work compared these approaches mainly on tabular data or with older neural architectures such as LSTMs, and modern transformer-era comparisons remain limited. Previous studies also focused mostly on classification accuracy and sample size, leaving important deployment questions underexplored, such as model scaling, calibration, robustness to noisy inputs, and preservation of ordinal relationships between classes.

The authors contribute the first large-scale transformer-era comparison of discriminative encoders and generative text classifiers, including auto-regressive models, discrete diffusion, and masked language modeling. They evaluate these approaches across multiple benchmark datasets, model sizes, and training settings, then provide practical recommendations for selecting models based on data availability, latency, robustness, and probability calibration requirements.

Dataset

• The authors evaluate models on a text classification benchmark suite of nine datasets: AG News, Emotion, Stanford Sentiment Treebank (SST2 and SST5), Multiclass Sentiment Analysis, Twitter Financial News Sentiment, IMDb, and Hate Speech Offensive.

• The datasets are used to test robustness across domains and task types, including sentiment analysis, movie review classification, news categorization, and social media or financial message analysis.

• The benchmark covers different classification settings, from binary tasks to fine-grained multiclass tasks, and includes short sentence-level inputs as well as longer paragraph-level passages.

• In the provided excerpt, the dataset is described as an evaluation suite rather than a training mixture; the authors use it to assess model performance across multiple classification domains.

• The excerpt does not provide per-subset sizes, train validation test splits, filtering rules, schema definitions, class distributions, other statistics, cropping strategy, or metadata construction details; these are said to be provided in Appendix C.

Method

The authors formulate text classification as a comparison between discriminative and generative language modeling paradigms. Given a dataset $\mathcal{D}=\{(X_i,y_i)\}_{i=1}^{N}$D={(Xi​,yi​)}i=1N​, where $X_i=x_i^1,\ldots,x_i^n$Xi​=xi1​,…,xin​ is an input sequence from vocabulary $\mathcal{V}$V and $y_i\in \mathcal{Y}$yi​∈Y is a discrete class label, discriminative models directly learn the conditional distribution $P(y\mid X)$P(y∣X), while generative models first learn a joint or pseudo joint distribution over the sequence and label and then infer the label by evaluating candidate label likelihoods or masked label predictions.

For the discriminative architecture, the method uses a standard Transformer encoder setup. The encoder $f_\theta$fθ​ maps the input text to a $d$d-dimensional hidden representation,

$$h_i=f_\theta(X_i).$$hi​=fθ​(Xi​).

A linear classification head $W\in \mathbb{R}^{|\mathcal{Y}|\times d}$W∈R∣Y∣×d converts this representation into class scores. The predicted label distribution is obtained with a softmax layer,

$$\hat{y}_i=\operatorname{softmax}(Wh_i),$$y^​i​=softmax(Whi​),

and training minimizes the cross entropy objective

$$\mathcal{L}_{\mathrm{enc}}=-\sum_{i=1}^{N}\log P(y_i\mid X_i).$$Lenc​=−i=1∑N​logP(yi​∣Xi​).

This branch represents the conventional encoder based classification pipeline, where the label is not generated as part of the sequence but is predicted from a final text embedding.

The generative side includes three main variants: Masked Language Modeling, Auto Regressive modeling, and discrete text diffusion. All of them cast classification as a sequence modeling or sequence reconstruction problem, with the label token inserted into the input template.

For Masked Language Modeling, the input is first expanded into a structured sequence. During training, the authors construct

$$X_i'=[\text{CLS}]\,X_i\,[\text{SEP}]\,\text{"The label is"}.$$Xi′​=[CLS]Xi​[SEP]"The label is".

The label token $y_i$yi​ is appended, yielding a sequence in which both original text tokens and the label token are present. Standard masking is then applied to a fraction of the tokens, typically $15\%$15%, and the model predicts the masked tokens using bidirectional context. The training objective is

$$\mathcal{L}_{\mathrm{mlm}} = -\sum_{i=1}^{N} \sum_{j\in \mathcal{M}_i} \log P(x_i^j\mid X_i' \oplus y_i^{\backslash j}),$$Lmlm​=−i=1∑N​j∈Mi​∑​logP(xij​∣Xi′​⊕yi\j​),

where $\mathcal{M}_i$Mi​ is the set of masked positions and $X_i'\oplus y_i^{\backslash j}$Xi′​⊕yi\j​ denotes the input with only position $j$j masked. Because the masked prediction objective can be interpreted as a denoising or pseudo likelihood surrogate, the authors place MLM within the broader generative family. At inference, the label is removed and replaced by a mask:

$$X_i'=[\text{CLS}]\,X_i\,[\text{SEP}]\,\text{"The label is"}\,[\text{MASK}].$$Xi′​=[CLS]Xi​[SEP]"The label is"[MASK].

The model predicts the token probability for the masked label position. The output space is then restricted to the valid label vocabulary $\mathcal{V}_{\mathcal{Y}}$VY​, and the corresponding token probabilities are renormalized to produce class probabilities.

For Auto Regressive classification, the method trains a causal language model to maximize the likelihood of the full label plus input sequence. Training uses the next token prediction loss

$$\mathcal{L}_{\mathrm{gpt}} = -\sum_{i=1}^{N} \sum_{j=1}^{L_i} \log P(x_i^j\mid y, x_i^1,\ldots,x_i^{j-1}),$$Lgpt​=−i=1∑N​j=1∑Li​​logP(xij​∣y,xi1​,…,xij−1​),

where $L_i$Li​ is the sequence length. At inference, the authors perform one forward pass for each candidate label $y\in \mathcal{V}_{\mathcal{Y}}$y∈VY​ by prepending that label to the input sequence and computing the log likelihood of the text conditioned on the label. The final prediction is

$$\hat{y} = \arg\max_{y\in \mathcal{V}_{\mathcal{Y}}} \log P(X\mid y).$$y^​=argy∈VY​max​logP(X∣y).

This corresponds to a full likelihood evaluation over all possible labels. The authors also discuss a pseudo generative Auto Regressive variant in which the label is appended after the input rather than prepended. In that setting, only a single forward pass is needed, and the model generates the label token directly. Label placement matters primarily for causal architectures with left-to-right attention, while it has no comparable theoretical effect for bidirectional models such as MLM or diffusion based models.

The text diffusion branch defines a template sequence for each input-label pair:

$$X_i = X_i\,[\text{SEP}]\,\text{"The label is"}\,y_i.$$Xi​=Xi​[SEP]"The label is"yi​.

Instead of gradually adding Gaussian noise as in image diffusion, the forward process corrupts discrete text by replacing tokens with pure noise, represented by the special symbol $[\text{MASK}]$[MASK]. This corruption is modeled through a continuous Markov process with transition matrices $Q_t$Qt​,

$$\frac{d p_t}{d t}=Q_t p_t, \qquad p_0=p_{\text{data}},$$dtdpt​​=Qt​pt​,p0​=pdata​,

where each token position is independently corrupted over timesteps $t\in[0,T]$t∈[0,T] until the sequence becomes completely masked.

The reverse process learns to reconstruct the original tokens from the corrupted sequence. It models score ratios of the form

$$s_\theta(x,t)_z=\frac{p_t(z)}{p_t(x)},$$sθ​(x,t)z​=pt​(x)pt​(z)​,

where $x$x and $z$z are vocabulary tokens. The reverse-time dynamics are defined as

$$\frac{d p_{T-t}}{d t} = s_\theta(x,t)_z Q_{T-t}p_{T-t}.$$dtdpT−t​​=sθ​(x,t)z​QT−t​pT−t​.

To train the score model in a tractable way, the authors use Denoising Score Entropy,

$$\mathcal{L}_{\mathrm{DSE}} = \mathbb{E}_{x_0\sim p_0,\,x\sim p(\cdot\mid x_0)} \left[ \sum_{z\neq x} w_{xz} \left( s_\theta(x)_z - \frac{p(z\mid x_0)}{p(x\mid x_0)} \log s_\theta(x)_z \right) \right],$$LDSE​=Ex0​∼p0​,x∼p(⋅∣x0​)​​z=x∑​wxz​(sθ​(x)z​−p(x∣x0​)p(z∣x0​)​logsθ​(x)z​)​,

where $p$p is treated as a perturbation of a base density $p_0$p0​ and $w_{xz}>0$wxz​>0 are weighting terms. The objective provides an upper bound on the negative log likelihood through an ELBO style relation,

$$-\log p_0^\theta(x_0) \leq \mathcal{L}_{\mathrm{DWDSE}}(x_0)+\text{constant}.$$−logp0θ​(x0​)≤LDWDSE​(x0​)+constant.

At inference, the label token in the diffusion template is masked, the model predicts the corresponding token from the fully masked or partially masked sequence, and the output is constrained to valid label tokens in $\mathcal{V}_{\mathcal{Y}}$VY​.

Experiment

The experiments evaluate five text classification paradigms on nine benchmarks across data sizes and model scales, while also testing robustness to token corruption and calibration or ordinality. They show that the best approach depends strongly on model capacity and data amount: compact discriminative ENC is more effective for small models, generative diffusion and autoregressive methods help in low-data regimes, and pseudo-generative MLM excels when large models have abundant data. Noise testing reveals that diffusion models are generally robust and autoregressive models are more sensitive, while calibration results show discriminative and pseudo-generative models can be reliable in deployment-oriented settings. Pretraining further favors discriminative models and reduces the data-size advantages observed for generative approaches.

The comparison frames discriminative and generative text classification approaches as different trade-offs in data needs, model size, sample efficiency, score structure, calibration, and noise robustness. ENC is favored on several score-quality properties and requires less data and smaller models, while AR and DIFF are more sample efficient and DIFF is notably robust to noise. Experimental context adds that AR is the more reliable generative option in low-data settings for calibration and ordinal behavior, and that pretraining reduces the typical low-data advantage of generative models. ENC combines lower data and model-size demands with stronger ordinality, unimodality, and calibration in the table comparison. AR and DIFF offer higher sample efficiency, while DIFF also provides strong robustness to noise but weaker score ordination and calibration. In low-data training, AR is more reliable than pseudo-AR for calibration and more ordinal than DIFF, while pretrained ENC outperforms AR across regimes.

Under random token dropping, higher values indicate that a model tolerates more dropped tokens before a given weighted-F1 drop, and all methods require more noise as the tolerated drop grows. Among 6-layer methods, DIFF is the strongest robustness signal at moderate and larger drop thresholds, while AR is also competitive. In the 12-layer comparison, ENC remains above AR-pseudo across the tested thresholds and is more robust than its 6-layer variant, suggesting architectural scaling can shift the advantage toward ENC. DIFF provides the most robust 6-layer performance under random token dropping. ENC appears more robust when scaled from 6 to 12 layers than AR-pseudo.

Under random token substitution, the results show that architecture choice affects robustness to input noise. ENC consistently requires more noise before weighted-F1 falls by a given amount, while AR-pseudo is less robust and drops from its peak at lower noise levels. This suggests ENC is a more reliable choice when the test data may contain token-level perturbations, especially in the smaller model setting. ENC shows the highest noise tolerance across weighted-F1 drop thresholds. AR-pseudo is more sensitive to random token substitution than ENC.

The experiments compare discriminative and generative text classifiers, including ENC, AR, and DIFF, in terms of data efficiency, model size, calibration, ordinal score behavior, and robustness to token-level noise. ENC is favored for score quality, calibration, and lower data and model-size demands, while AR and DIFF provide better sample efficiency and DIFF is especially robust to random token dropping in smaller models. Scaling ENC from six to twelve layers improves its robustness relative to AR-pseudo, and ENC is generally more tolerant than AR-pseudo to random token substitution, making pretrained ENC a strong choice when both score structure and noise reliability matter.