Super-resolution Topic | 3 Methods, 4 Tutorials, 10 Datasets, Get the Core Knowledge Points in One Article

December 2010,Su Guangda, a professor at the Department of Electronic Engineering at Tsinghua University, received an unusual call, a police officer from the Jungar Criminal Police Team of the Inner Mongolia Autonomous Region asked Su Guangda for help with a blurry facial image of the suspect.

"This image was taken by a roadside surveillance camera. The pixels are very low and completely unrecognizable to the naked eye," Su Guangda recalled. At that time, he used super-resolution technology to put this blurry image into the software they developed and reconstructed it. Based on the reconstructed high-definition image, the Junggar police quickly identified the suspect and solved the murder case.

In fact, this case is not an isolated one. When the police are investigating a case, they can obtain images of the suspect through surveillance cameras to speed up the investigation. However, the details of the photos taken by surveillance cameras are often blurred after being enlarged, which greatly reduces the efficiency of the police in obtaining key evidence.

With the development of super-resolution technology, these problems have turned around and are gradually being solved.

This article will start with the concept, introduce the mainstream super-resolution algorithms and their advantages and disadvantages, and demonstrate their applications in different scenarios such as public security, medical diagnosis, satellite remote sensing, digital entertainment, etc. In addition, the HyperAI Super Neural official website has launched a large number of public tutorials and public data sets related to super-resolution, welcome to try it out!

What is super-resolution?

Super-Resolution (SR), in short,It is to upgrade the low resolution (LR) image to high resolution (HR) through an algorithm.Compared with low-resolution images, high-resolution images have greater pixel density and richer texture details, and are more reliable.

This technology can overcome or compensate for problems such as imaging blur and poor image quality caused by the limitations of the image acquisition system or the acquisition environment itself, and provides important support for subsequent image processing such as feature extraction and information recognition.

Classification of super-resolution algorithms

At present, super-resolution methods are mainly divided into three categories: interpolation-based methods, reconstruction-based methods and learning-based methods.

* Interpolation-based methods

The interpolation method increases the size of the image by inserting new pixels around the original pixels of the image and assigning values to these pixels to restore the image content and achieve the effect of improving the image resolution.
*Pixel: The most basic unit element of an image, i.e. a dot. Each pixel has its own color value. The more pixels there are per unit area, the clearer the image.

* Reconstruction-based approach

Reconstruction-based super-resolution algorithms usually use multiple low-resolution images taken in the same scene as input, and then analyze the frequency domain or spatial domain relationship of these images. The reconstruction process is guided and constrained by introducing prior information, and a single high-resolution image is reconstructed.
*Frequency domain: refers to the characteristics of the signal in the frequency domain.
*Spatial domain: refers to the distribution of signals in space.
*Prior information: This information is available "before the experiment" and can generally be understood as domain knowledge.

* Learning-based methods

Learning-based super-resolution methods usually use a large amount of training data to learn the mapping relationship between low-resolution images and high-resolution images, predict the high-frequency detail information lost in low-resolution images, and thus generate super-resolution images.

Shallow learning-based methods mainly include machine learning, manifold learning, sample learning, and sparse coding, which can be used in situations where the amount of data is small.

Deep learning-based methods can be divided into SR methods based on convolutional neural networks, SR methods based on residual networks (ResNet), and SR methods based on generative adversarial networks (GAN).

The advantages and disadvantages of the above three methods are as follows:

Nowadays, deep learning has become the mainstream method in the field of super-resolution.

In 2014, Dong et al. first applied deep learning to the field of image super-resolution reconstruction.The SRCNN (Super-Resolution Convolutional Neural Network) network model was proposed.The result was published on arXiv under the title "Image Super-Resolution Using Deep Convolutional Networks". Since then, a wave of deep learning has been set off in the field of super-resolution reconstruction.

Paper address:

https://arxiv.org/pdf/1501.00092

As the earliest model to apply deep learning methods to image super-resolution, SRCNN achieved a PSNR value far exceeding that of traditional methods by using only three convolutional layers.

Specifically, a low-resolution image is input, the image is enlarged to the target size using bicubic interpolation, and then a three-layer convolutional neural network is used to fit the nonlinear mapping between the low-resolution image and the high-resolution image, and finally the reconstructed high-resolution image is output.

*PSNR value: Peak signal-to-noise ratio. The larger the value, the better the output HR image quality.

SRCNN has become an important milestone in the field of image super-resolution due to its simplicity and high efficiency.Since then, super-resolution technology based on deep learning has developed rapidly, from the early super-resolution technology based on convolutional neural networks (CNN) to the recent technology based on generative adversarial networks.

Deep learning + super-resolution: diverse applications from public security to digital entertainment

Demand drives technology development, and technology iteration helps applications to be implemented. Nowadays, super-resolution technology has been widely used in public security, medical diagnosis, satellite remote sensing, entertainment media and other fields.

* Public security field

Public surveillance videos are often blurred and have low resolution due to factors such as weather and distance. The application of super-resolution technology can help police extract clear key information such as faces and license plate numbers, which is helpful for solving cases.

Feng Shunli from Jilin Police College used ESRGAN and BSRGAN networks to analyze portraits and natural scenery in different environments, and explored the feasibility of applying super-resolution technology in public security and courts.

The result was published in the Journal of the Armed Police Academy under the title "Application of Image Super-Resolution Reconstruction Technology in Police Work".

Paper address:

https://www.faxin.cn/lib/Flwx/FlqkContent.aspx?gid=F805122&libid=040106

First, the researchers reproduced, trained and tested the ESRGAN and BSRGAN models to obtain the optimal model parameters. Then, they used the trained models to perform super-resolution reconstruction on low-quality images such as portraits and natural scenery, and obtained the reconstruction results of ESRGAN and BSRGAN.

The researchers compared the reconstructed images from ESRGAN and BSRGAN with the original images.The results show that the reconstructed portraits in frontal, oblique angles and complex scenes have significant improvements in visual quality and fidelity.

In the comparison of natural scenes, BSRGAN reconstruction effect is better than ESRGAN.It can effectively remove unknown complex noise from the original low-quality image and generate clear edges and fine details.

* Medical diagnosis field

Due to the limitations of imaging equipment and complex clinical environments, images obtained in the medical field often have insufficient resolution, which directly affects doctors' accurate diagnosis and treatment decisions of diseases.

Peng Bo and others from Sichuan People's Hospital used the Generative Adversarial Network (SRGAN) for super-resolution reconstruction of natural images as the basic method.The network structure was changed by reducing 2 input channels and deleting 1 residual block, the network loss function was improved, and a new blur processing data set was added to reconstruct a medical ultrasound image with clear edges and no artifacts that was magnified 4 times.

This achievement has been published in the Journal of Terahertz Science and Electronic Information Technology under the title "Super-resolution Reconstruction of Ultrasound Images Based on Generative Adversarial Networks".

Paper address:

http://www.iaeej.com/xxydzgc/article/abstract/202305015

The researchers compared the improved SRGAN with three other algorithms.The results show that the reconstruction results of the improved SRGAN are smoother overall and the texture edges are clearer.

* Satellite remote sensing field

In recent years, remote sensing satellite images have been widely used in environmental monitoring, resource exploration, disaster warning and military fields. However, factors such as atmospheric changes, transmission noise, motion blur and under-sampled optical sensors have severely restricted the clarity of remote sensing satellite images. Super-resolution technology can improve the quality and availability of satellite remote sensing data by processing and enhancing low-resolution images.

Yan Bodhi and others from Harbin Institute of Technology introduced the detailed feature information of local cluster target areas in remote sensing images into the sampling and reconstruction of complete remote sensing images, extracted image features of different scales through multi-layer neural networks, and fused and reconstructed these features through residual learning.This method can significantly improve the detail effect of the global remote sensing image and optimize the resolution of the cluster target area with the help of the pixel information of the local image.

The achievement, titled "Super-resolution reconstruction of remote sensing cluster target areas combined with local high-definition images", has been published in the Journal of Nanjing University of Aeronautics and Astronautics.

Paper address:

https://jnuaa.nuaa.edu.cn/njhkht/article/html/202306002

The comparison chart presented in the study shows that this method is significantly better than other existing methods in terms of visualization effect.It is suitable for urban and outdoor scenes and shows good results.

* Digital entertainment

Anime is made up of multiple still images connected together, and the resolution of the still images will affect the clarity of the final animation. However, existing hand-drawn or digital drawings cannot guarantee high resolution of the first draft, which is quite unfriendly to the user's visual experience. By applying super-resolution technology, these low-resolution images can be converted into higher-resolution images, which can present more details and textures, making the images of anime works more vivid and realistic.

Bilibili has launched an animation quality restoration model called Real-CUGAN (Real Cascaded-U-Net-style Generative Adversarial Networks).

First, the researchers used the model to cut the animation frames into blocks, and used the image quality scoring model to score and filter the candidate blocks, ultimately obtaining a training set of millions of high-quality animation image blocks.

Then, through a multi-stage degradation algorithm, the high-definition image blocks are downsampled to obtain low-quality images, allowing the model to learn and optimize the reconstruction process from low-quality images to high-quality images. After training, the model can process real low-quality two-dimensional images into high-definition images.

The comparison chart of different algorithms is as follows:

Super-resolution open source datasets and tutorials

at present,HyperAI's official website has launched a large number of super-resolution resources.Includes public tutorials and datasets.

* Public Tutorials

1. APISR Animation Image Super Resolution Enhancer

APISR is an open source project for improving the resolution of anime images and videos. It can restore and enhance low-quality, low-resolution anime images and video sources, and handle various image degradation problems (blur, noise, compression artifacts, etc.).
Online operation address:https://hyper.ai/tutorials/31383

2. SUPIR-AI Image Restoration Tutorial

SUPIR can upscale low-resolution images to high resolution while maintaining image details and realism and handling various complex degradation situations.
Online operation address:https://hyper.ai/tutorials/30940
3. Image Super-Resolution Using Latent Diffusion

This tutorial shows how to use the latent diffusion image super-resolution model using the diffusers library.
Online operation address:https://hyper.ai/tutorials/26207
4. Image super-resolution using efficient sub-pixel CNN

This tutorial implements super-resolution on the BSDS 500 dataset using an efficient sub-pixel model.
Online operation address:https://hyper.ai/tutorials/25044
* Public datasets
1. MSU Super-resolution Dataset

The dataset contains indoor and outdoor videos as well as animations, all of which have low SI/TI values and simple textures. It aims to minimize possible compression artifacts and restore details.
Direct use:https://hyper.ai/datasets/20401
2. MSU Video Super-Resolution Dataset

This dataset is used for video super-resolution tasks and contains the most complex content in detail restoration tasks: faces, text, QR codes, license plates, textures without patterns, and small details. The video includes different types of motion and different types of weakening: bicubic interpolation, Gaussian blur, and downsampling.
Direct use:https://hyper.ai/datasets/17212
3. DRealSR Super-resolution Image Dataset

This dataset is a super-resolution image dataset about the real image degradation process, alleviating the limitations of traditional simulated image degradation. The dataset covers indoor and outdoor scenes, such as advertising posters, plants, offices, and buildings.
Direct use:https://hyper.ai/datasets/20446
4. TextZoom Super-Resolution Dataset

The dataset consists of pairs of low-resolution (LR) and high-resolution (HR) images. These images are taken in the wild by cameras with different focal lengths. Each image pair is annotated with a case-sensitive string (including punctuation), bounding box type, and original focal length.
Direct use:https://hyper.ai/datasets/19550
5.SR-RAW Image Super-resolution Dataset

This dataset contains RAW sensor data with ground truth high-resolution images taken at high optical zoom levels, where each sequence contains 7 (some contain 6) images taken at different focal lengths.
Direct use:
https://hyper.ai/datasets/19743
6.Set5, Set14 Low complexity single image super-resolution dataset

This is a low-complexity single image super-resolution dataset based on non-negative neighborhood embedding. The training set is used for single image super-resolution reconstruction, reconstructing high-resolution images based on low-resolution images to obtain more detailed information. It is widely used in computer vision and graphics, medical imaging, security monitoring and other fields.
Direct use:
https://hyper.ai/datasets/5382
7.DIV2K Single Image Super-Resolution Dataset

This dataset contains 1000 low-resolution images with different degradation types, which can be used to reconstruct high-resolution images from low-resolution images.

Direct use:https://hyper.ai/datasets/15624
8.S2-NAIP US super-resolution remote sensing dataset

This dataset contains paired images of NAIP and Sentinel-2 in the continental United States, providing high-precision data support for surface monitoring, resource management, and environmental change assessment.
Direct use:
https://hyper.ai/datasets/30427
9. Sun-Hays 80 Dataset Super-resolution Image Dataset

This dataset uses global scene descriptions to compare and search for related scenes in image databases. The scenes provide ideal example textures to constrain image sampling problems.
Direct use:https://hyper.ai/datasets/5391
10. Urban100 super-resolution reconstruction dataset

The dataset contains challenging urban scenes with details in different frequency bands. Downscaling the real images using bicubic interpolation can obtain LR/HR image pairs to obtain training and testing datasets.
Direct use:
https://hyper.ai/datasets/5385
The above is a summary of super-resolution resources on the HyperAI official website. If you see a good project, please leave a message in the background to recommend it to us! In addition, we have also established a tutorial exchange group, welcome friends to join the group to discuss various technical issues and share application results~
Scan the QR code to join the tutorial exchange group