One-sentence Summary

Researchers from the University of Rome, Tor Vergata and Amazon propose GAN-BERT, which extends BERT-like fine-tuning with unlabeled data in a semi-supervised generative adversarial setting, reducing the required annotated examples to only 50–100 while maintaining good performance on several sentence classification tasks.

Key Contributions

• GAN-BERT extends BERT fine-tuning with a semi-supervised generative adversarial setting in which a generator produces fake examples and BERT serves as the discriminator.
• The method reduces annotation requirements, achieving good sentence classification performance with only 50 to 100 labeled examples and results comparable to fully supervised settings with fewer than 200 labeled examples.
• The semi-supervised adversarial schema consistently improves over BERT without adding inference cost, since the generator is used only during training.

Introduction

Transformer-based models such as BERT achieve strong NLP performance after fine-tuning on large annotated datasets, but their accuracy degrades significantly when only a few hundred labeled examples are available, especially for classification tasks with many categories. Since manual annotation is expensive, semi-supervised generative adversarial networks are an attractive alternative, yet their use in NLP has been limited to a kernel-based GAN that operates over fixed precomputed embeddings rather than updating the representation space. The authors introduce GAN-BERT, which places BERT in a semi-supervised GAN framework as the discriminator, uses unlabeled data to improve representations, and enables results comparable to fully supervised training with fewer than 200 labeled examples.

Method

The authors leverage Semi-supervised GANs (SS-GANs) to enable semi-supervised learning within a GAN framework. In this setup, a discriminator is trained over a $(k + 1)$(k+1)-class objective. True examples are classified into one of the target $(1, \dots, k)$(1,…,k) classes, while generated samples are classified into the $(k + 1)$(k+1)-th class. Formally, let $D$D and $G$G denote the discriminator and generator, and $p_d$pd​ and $p_\mathcal{G}$pG​ denote the real data distribution and the generated examples, respectively. The loss function of $D$D is defined as $L_D = L_{D_{sup}} + L_{D_{unsup}}$LD​=LDsup​​+LDunsup​​, where:

$L_{D_{sup}}$LDsup​​ measures the error in assigning the wrong class to a real example among the original $k$k categories. $L_{D_{unsup}}$LDunsup​​ measures the error in incorrectly recognizing a real unlabeled example as fake and failing to recognize a fake example. The generator loss $L_G$LG​ combines a feature matching loss and an unsupervised loss, encouraging $G$G to produce examples whose intermediate representations are similar to real ones.

To adapt this for natural language processing, the authors introduce GAN-BERT, which extends a pre-trained BERT model by integrating SS-GAN layers during the fine-tuning stage. Given an input sentence, BERT produces vector representations, and the $h_{CLS}$hCLS​ representation is adopted as the sentence embedding for target tasks.

As shown in the figure below:

The architecture adds a discriminator $D$D for classifying examples and a generator $G$G acting adversarially on top of BERT. The generator $G$G is a Multi Layer Perceptron (MLP) that takes a 100-dimensional noise vector drawn from $N(\mu, \sigma^2)$N(μ,σ2) as input and produces a vector $h_{fake} \in \mathbb{R}^d$hfake​∈Rd. The discriminator is another MLP that receives a vector $h_* \in \mathbb{R}^d$h∗​∈Rd, which can be either $h_{fake}$hfake​ produced by the generator or $h_{CLS}$hCLS​ for unlabeled or labeled examples from the real distribution. The last layer of $D$D is a softmax-activated layer outputting a $k + 1$k+1 vector of logits.

During the training process, the system optimizes two competing losses, $L_D$LD​ and $L_G$LG​. In the forward step, when real instances are sampled ($h_* = h_{CLS}$h∗​=hCLS​), $D$D classifies them into one of the $k$k categories. When $h_* = h_{fake}$h∗​=hfake​, it classifies the example into the $k + 1$k+1 category. During back-propagation, unlabeled examples contribute only to $L_{D_{unsup}}$LDunsup​​, meaning they are considered in the loss computation only if they are erroneously classified into the $k + 1$k+1 category; in all other cases, their contribution to the loss is masked out. Labeled examples contribute to the supervised loss $L_{D_{sup}}$LDsup​​. Examples generated by $G$G contribute to both $L_D$LD​ and $L_G$LG​. When updating $D$D, the BERT weights are also modified to fine-tune its inner representations using both labeled and unlabeled data. After training, the generator $G$G is discarded, retaining the original BERT model for inference without additional computational cost.

Experiment

The experiments compare GAN-BERT with a fine-tuned BERT-base model on topic classification, question classification, sentiment analysis, and natural language inference, using progressively larger labeled sets and additional unlabeled examples. GAN-BERT consistently improves over BERT when only few labeled examples are available, while BERT often diverges at 1% of labels, and the benefit is more pronounced for tasks with many categories. The advantage also appears on SST-5 and is systematic on MNLI up to about 0.5% of annotated examples, after which both models perform similarly.