They Used Convolutional Neural Networks to Discover the Secrets Hidden in Famous Paintings

The famous art treasure "Ghent Altarpiece" is undergoing a massive restoration to ensure that people today can feel the brilliant light emanating from this great masterpiece. With the advancement of technology, the introduction of high-resolution digital technology, and the gradual increase in data information, AI technology can also be used to restore paintings. So what surprises can AI technology bring to the restoration of the "Ghent Altarpiece"?

Keywords:World famous paintings, painting restoration, image generation

The Ghent Altarpiece (also known as The Love of the Mystic Lamb) is a national treasure of Belgium, the most influential oil painting in the world to date, and the most popular masterpiece in the art field.

It was completed by the Van Eyck brothers between 1415 and 1432, which took 18 years to complete. It is a huge folding painting consisting of 20 inner and outer panels. When fully opened, it is 3.5 meters long and 4.7 meters wide.

But precisely because of its great fame, this painting has encountered many misfortunes since its release.

Moreover, in addition to human damage, it is also like many world famous paintings that cannot escape the erosion of dust, moisture, etc., causing the picture to change color and lose its luster. Therefore, the demand for the restoration of the painting is becoming more and more urgent.

Recently, University College London and Duke University used artificial intelligence to decode high-resolution X-ray images of the Ghent Altarpiece. Their research results were published in the journal Science Advances:"Artificial Intelligence for Art Investigation: Meeting the Challenge of Separating X-ray Images of the Ghent Altarpiece".

This research is expected to improve our understanding of artistic masterpieces and provide new opportunities for art investigation, conservation and display. 

The famous painting is scarred and the restoration project is huge

An altarpiece is a composite painting painted on a wooden board and placed in front of the altar of a church for decoration. The Ghent Altarpiece is based on the Bible and depicts classic scenes from it, covering more than 100 characters. This work is alsoThe pioneering work of modern oil painting.In their creation, the Van Eyck brothers experimented with oil coloring for the first time, using new paints, rosin and emulsions to keep the picture fresh and lasting. In addition, it has become a treasure of the generation because of its material and texture rendering, as well as its exquisite and complex composition.

Since its first exhibition in 1432, the Ghent Altarpiece has been stolen seven times and involved in 13 different forms of crimes, making it the most stolen famous painting. To this day, one panel is still missing.

In addition, like other art treasures, it has been experiencing the test of time. The original painting has been destroyed by years of dust and erosion, and its luster and picture have been compromised.

In oil painting, varnishes made from natural resins, or even synthetic materials, will degrade and yellow over time, becoming cloudy, blurring the surface and greatly changing the tonal balance and layering of the image.

Therefore, the Ghent Altarpiece has been cleaned and restored many times. During one restoration, the staff were surprised to find that the painting had three layers and the repainted area reached 70 %, which means that the painting has undergone two unrecorded major restorations.

After a detailed evaluation, the museum began a massive restoration project in October 2012. Experts also decided to change the restoration goal from "restoring the color of the picture" to "restoring the original appearance of the painting."

This work requires a lot of manpower and financial resources and is expected to be completed as early as 2020. The restoration process includes removing surface dust and varnish, restoring the color of the painting, and also includes a variety of scanning imaging and the process of building electronic models.

Needless to say, this is a difficult task. The development of AI technology is contributing to restoration work.

Restoration Challenge: Interpreting X-ray Images

In painting restoration, there are already some mature machine learning cases, including material recognition of different layers of the picture, digital image restoration of cracks, hidden design and visual presentation.

In the restoration of the Ghent Altarpiece, researchers from University College London and Duke University recentlyUsing deep learning, we solved the technical problems in X-ray image scanning. X-ray radiography (XR) is an important method in painting restoration. It uses the principle that different materials absorb X-rays to different degrees and can reveal hidden details in paintings.

In the restoration work of the Ghent Altarpiece, establishing a complete X-ray image is also an important step in the restoration. X-ray images can aid in restoration work by showing hidden cracks in the clearcoat or cover coat, paint wear or other structural areas.

Additionally, scanned images can reveal to staff the artist’s working methods, revealing the physical structure of the canvas or panel and its support, as well as the different coatings used in its creation.

But for works such as the Ghent Altarpiece, the interpretation of its X-ray images faces a severe challenge.

In the Ghent Altarpiece, several panels have patterns painted on both sides. X-ray scans produce complex three-dimensional absorption patterns, but they are ultimately recorded as two-dimensional files, so what emerges is a mixed, complex image.

This includes multiple information, such as the paint surface morphology, the structure showing the oak support, the wood grain, the location of the wooden nails, as well as the cracks and wear in the paint layer. To obtain valuable information, it is necessary to separate the individual layers to be analyzed from this chaotic information.

This poses a huge interpretation problem, and the challenge is how to separate the mixed double-sided X-ray image into individual X-ray images of the corresponding "single-sided" paintings.

The solution: Convolutional neural networks

To solve this problem, a research team from University College London and Duke University, with support from the Royal Institute of Cultural Heritage of Belgium and the Simons Foundation, launched a study proposing aA self-supervised framework based on convolutional neural network (CNN),Cracked it perfectly.

The mixed image obtained by X-ray is actually a nonlinear combination of the images on both sides of the panel. The core of this study is to use modeling toFind the mapping between the final mixed pattern and each side information,With the help of some details, the X-ray signals are separated.

So their idea turns into, in a fully supervised source setting, from a training set containing mixed and individual signals (perhaps different paintings drawn by the same artist in a similar style),Let the algorithm analyze and learn the correspondence between the mixed image and the source signal.

To this end, they designed aSelf-supervised neural networks,The model learns to achieve this separation by learning how to convert an RGB image into an X-ray image, which is then “reconstructed” as a virtual image of a single panel, and then by minimizing the error of the reconstructed X-ray image and comparing the difference with the original mixed X-ray image.

Instead of using a large amount of labeled data, the whole process uses high-resolution images (allowing the creation of a large number of input patches) and trains the network based on implicit labels.

In detail, they constructedSeven-layer CNN,Each convolutional layer contains batch normalization and rectified linear unit (ReLU) activation layers. The network structure is inspired by the pix2pix structure (pix2pix uses conditional adversarial networks for image-to-image conversion). After training, the model can input mixed X-ray images and output two separate panel images.

The new method, applied to two separate test image sets, reproduced the individual details of Adam and Eve in the panel with far greater clarity than expected. 

Artificial Intelligence Makes Art Treasures Last Longer

Through their research, the problem of separating X-ray patterns was perfectly solved, and even the researchers did not expect that such amazing clarity could be achieved through the use of deep learning methods.

One researcher excitedly compared the process to exploration in physics: through experimental attempts, unexpected results were found that had no relevant theoretical explanation.

Next, the research team hopes to try this method on some other famous masterpieces: "We hope to see similar AI approaches help reveal other hidden features in paintings, such as discovering early hidden designs."

As for the famous "Ghent Altarpiece", I believe that with the support of AI technology, it will surely shine with its original brilliance more quickly.