Use Real-ESRGAN to Save the Quality of Landline Images and Make Your Own HD Animation Resources

Contents at a glance: Real-ESRGAN is an upgrade of ESRGAN, with three main innovations: proposing a high-order degradation process to simulate actual image degradation, using spectral normalization U-Net
The discriminator increases the ability of the discriminator and uses purely synthetic data for training. Keywords: Real-ESRGAN Super-resolution video restoration
This article was first published on WeChat official account: HyperAI

Compared with the new animations with excellent and exquisite pictures, the old animations have poor picture quality and low resolution due to the technical and equipment limitations of the times.The classic cartoons from childhood are still taken out and watched over and over again by the audience.

Whenever classic anime videos are restored in 4K, the number of views on video websites remains high.The high-quality images and classic content are enough to make "double fans ecstatic".

4K restored videos on Bilibili are extremely popular

This tutorial introduces how to use Real-ESRGAN to super-optimize anime videos and restore video quality.The tutorial can be run on the cloud platform OpenBayes and has no dependency on device configuration. Easily enjoy the happiness brought by 1080P video.

Real-ESRGAN: A blind super-resolution model for the love of the second dimension

In traditional animation production, animators first draw each frame by hand, then use a camera to capture the frame and scan it into a computer for digital processing.The quality of the shooting equipment, the compression of the uploaded animations to the video platform, and the unpredictable noise are complex factors. It will affect the image effect of the animation.

The causes of image degradation in the real world are very complex, which makes non-blind super-resolution algorithms, such as ESRGAN, not very effective in restoring images.Therefore, blind super-resolution is needed to perform super-resolution enhancement on low-resolution images of unknown degradation type.

Blind super-resolution is mainly divided into two types of methods: explicit modeling and implicit modeling.

Explicit Modeling

The blur kernel and noise information are parameterized, and the image degradation process, including noise, blur, downsampling and compression, is estimated through prior knowledge. However, simply combining several degradations cannot fit the image degradation in the real world well.

Implicit Modeling

Instead of relying on any explicit parameters, it leverages additional data to implicitly learn a latent super-resolution model via the data distribution.

The authors of Real-ESRGAN refer to explicit modeling as first-order modeling. First-order degradation modeling is difficult to fit complex degradations.The authors proposed a high-order degradation model. In this model, the n-order model contains n repeated degradation processes, each of which follows the classical model:

x = Dn(y) = (Dn ◦ · · · ◦ D2 ◦ D1)(y)

The authors used a second-order degradation process in the paper.This keeps it simple while solving most practical problems.

Real-ESRGAN is trained entirely using synthetic data. When generating high-definition and low-definition data pairs, the model downsamples the input image by 4 times (subsampled or reduced image), and then continues to downsample by 1 or 2 times.

Real-ESRGAN uses the same structure as ESRGAN

To reduce the amount of calculation,The author innovatively proposed the Pixel Unshuffle operation. Reduce the input resolution and increase the number of channels.

When generating high-definition and low-definition data pairs, the paper uses a blurred kernel for convolution, then downsamples the image by a factor of r, adds noise, and finally performs JPEG compression.These operations mimic the real-life situation where images are compressed multiple times during transmission.

Real-ESRGAN uses multiple image degradation methods

Compared with ESRGAN, Real-ESRGAN handles blurry images better and received the Honorable Paper Nomination Award at ICCV AIM 2021.

See the code for details

Paper link

Real-ESRGAN in action: Making old animations clearer

This tutorial will demonstrate how to use the Real-ESRGAN algorithm on OpenBayes to achieve image enhancement and make old animation videos clearer.

Full Tutorial

Step 1: Environment Preparation

# !git clone https://github.com/xinntao/Real-ESRGAN.git
%cd Real-ESRGAN
!pip install basicsr
!pip install facexlib
!pip install gfpgan
!pip install ffmpeg-python
!pip install -r requirements.txt
!python setup.py develop

Step 2 Reasoning

# ! python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n RealESRGANv2-animevideo-xsx2 -s 4 -v -a --half --suffix outx2
! python inference_realesrgan_video.py -i /openbayes/home/results.mp4 -n RealESRGANv2-animevideo-xsx2 -s 4 -v --half --suffix outtsx2
# 参数
# -i, --input: 输入视频
# -n, --model_name: 使用的模型名字
# -s, --outscale: 放大尺度
# -v, --video: 将增强的帧转换回视频中
# -a, --audio: 将输入的音频复制到增强的视频中
# --half: 推理半精度
# -suffix: 输出视频的后缀

Step 3 Visualization

from IPython.display import HTML
from base64 import b64encode

def show_video(video_path, video_width = 600):

  video_file = open(video_path, "r+b").read()

  video_url = f"data:video/mp4;base64,{b64encode(video_file).decode()}"
  return HTML(f"""<video width={video_width} controls><source src="{video_url}"></video>""")

# 输入视频
show_video('inputs/video/onepiece_demo.mp4')

# 增强后的视频
show_video('results/onepiece_demo_outx2.mp4')

Full Tutorial

Video explanation of specific processing effects and tutorials,Click to view

That’s all for this tutorial. What was your childhood dream? Clone the tutorial “Real-ESRGAN Super Resolution of Anime Videos” on OpenBayes and make your own clear videos.

Note: High-definition self-made content is only for personal learning use

Reference Links:
https://zhuanlan.zhihu.com/p/431612275
https://zhuanlan.zhihu.com/p/558893171

-- over--