Contrastive Learning

summary

Contrastive learning is a technique that enhances the performance of vision tasks by using the principle of contrasting samples against each other to learn properties that are common between data classes and properties that distinguish one class from another.

This learning model, which mimics the way humans learn about the world around them, has shown promising results in the deep learning literature and has become increasingly important in the field of computer vision research.

What is Contrastive Learning?

Contrastive learning is a machine learning paradigm in which unlabeled data points are juxtaposed against each other to teach the model which points are similar and which are different.

That is, as the name implies, samples are pitted against each other, and samples belonging to the same distribution push towards each other in the embedding space. Conversely, those belonging to different distributions compete with each other.

The Importance of Contrastive Learning

Supervised learning is a machine learning technique that trains a model by using a large number of labeled examples. The quality of the data labels is critical to the success of supervised models.

However, obtaining such high-quality labeled data is a tedious task, especially in areas such as biomedical imaging, where expert doctors are required to annotate the data. This is both expensive and time-consuming. Supervised Learning ML Project 80%'s time is invested in obtaining and cleaning data for model training.

Therefore, recent research in deep learning focuses on reducing the supervision requirements in model training. To this end, several methods have been proposed, such as semi-supervised learning, unsupervised learning, and self-supervised learning.

In semi-supervised learning, a small amount of labeled data and a large amount of unlabeled data are used to train the deep model. In unsupervised learning, the model tries to understand unstructured data without any data labels.

Self-supervised learning (SSL) takes a slightly different approach.

As with unsupervised learning, unstructured data is provided as input to the model. However, the model annotates the data on its own, and the labels predicted with high confidence are used as ground truth in future iterations of model training.

This continuously improves the model weights to make better predictions. The efficacy of SSL methods compared to traditional supervised methods has attracted the attention of several computer vision researchers.

One of the oldest and most popular techniques used in SSL is contrastive learning, which uses “positive” and “negative” examples to guide a deep learning model.

Contrastive learning has since been further developed and is now used in both fully supervised and semi-supervised settings and improves the performance of existing state-of-the-art techniques.

A framework for contrastive learning

There are three popular working mechanisms of contrastive learning frameworks proposed by deep learning and computer vision researchers in recent literature.

1.SimCLR

The SimCLR model, developed by Google Brain, is a framework for contrastive learning of visual representations. SimCLR is proposed to solve self-supervised and semi-supervised learning problems through contrastive learning.

Its basic working principle is to utilize contrastive loss in latent space to maximize the consistency between different enhanced versions of the same sample.

2. NNCLR

Most instance-discriminative contrastive learning algorithms train the encoder to be invariant to predefined transformations of the same instance.

While most methods consider different views of the same image as positive values for the contrastive loss, the nearest neighbor contrastive learning (NNCLR) framework developed in this paper attempts to use positive values from other instances in the dataset, i.e., use different images from the same class instead of augmenting the same image.

3. PCL

Prototype Contrastive Learning (PCL) is an unsupervised representation learning method that connects contrastive learning with clustering. PCL learns low-level features for instance discrimination tasks and also encodes the semantic structure discovered through clustering into the learned embedding space.

References

【1】https://www.v7labs.com/blog/contrastive-learning-guide#ten-contrastive-learning-frameworks