Restore damaged paintings in just hours with an AI-generated 'mask'

Art restoration takes steady hands and a discerning eye. For centuries, conservators have restored paintings by identifying areas needing repair, then mixing an exact shade to fill in one area at a time. Often, a painting can have thousands of tiny regions requiring individual attention. Restoring a single painting can take anywhere from a few weeks to over a decade.

In recent years, digital restoration tools have opened a route to creating virtual representations of original, restored works. These tools apply techniques of computer vision, image recognition, and colour matching, to generate a "digitally restored" version of a painting relatively quickly.

Still, there has been no way to translate digital restorations directly onto an original work, until now.

In a paper appearing in Nature,  researchers present a new method he's developed to physically apply a digital restoration directly onto an original painting.

The restoration is printed on a very thin polymer film, in the form of a mask that can be aligned and adhered to an original painting. It can also be easily removed. A digital file of the mask can be stored and referred to by future conservators, to see exactly what changes were made to restore the original painting.

Because there's a digital record of what mask was used, in 100 years, the next time someone is working with this, they'll have an extremely clear understanding of what was done to the painting. "And that's never really been possible in conservation before."

As a demonstration, the researchers applied the method to a highly damaged 15th century oil painting. The method automatically identified 5,612 separate regions in need of repair, and filled in these regions using 57,314 different colours. The entire process, from start to finish, took 3.5 hours, which  is about 66 times faster than traditional restoration methods.

As with any restoration project, there are ethical issues to consider, in terms of whether a restored version is an appropriate representation of an artist's original style and intent. Any application of his new method should be done in consultation with conservators with knowledge of a painting's history and origins.

There is a lot of damaged art in storage that might never be seen. Hopefully with this new method, there's a chance we'll see more art.

Digital connections

Digital tools can significantly speed up the restoration process. Researchers have developed artificial intelligence algorithms that quickly comb through huge amounts of data.

The algorithms learn connections within this visual data, which they apply to generate a digitally restored version of a particular painting, in a way that closely resembles the style of an artist or time period. However, such digital restorations are usually displayed virtually or printed as stand-alone works and cannot be directly applied to retouch original art.

All this made the researchers think: If we could just restore a painting digitally, and affect the results physically, that would resolve a lot of pain points and drawbacks of a conventional manual process.

'Align and restore'

For the new study, the researchers developed a method to physically apply a digital restoration onto an original painting, using a 15th-century painting that they acquired . Their new method involves first using traditional techniques to clean a painting and remove any past restoration efforts.

This painting is almost 600 years old and has gone through conservation many times. In this case there was a fair amount of overpainting, all of which has to be cleaned off to see what's actually there to begin with.

Then they scanned the cleaned painting, including the many regions where paint had faded or cracked. They then used existing artificial intelligence algorithms to analyze the scan and create a virtual version of what the painting likely looked like in its original state.

Then, they developed software that creates a map of regions on the original painting that require infilling, along with the exact colors needed to match the digitally restored version. This map is then translated into a physical, two-layer mask that is printed onto thin polymer-based films. The first layer is printed in colour, while the second layer is printed in the exact same pattern, but in white.

In order to fully reproduce colour, you need both white and colour ink to get the full spectrum. If those two layers are misaligned, that's very easy to see. So they also developed a few computational tools, based on what they know of human colour perception, to determine how small of a region they can practically align and restore.

They used high-fidelity commercial inkjets to print the mask's two layers, which they carefully aligned and overlaid by hand onto the original painting and adhered with a thin spray of conventional varnish. The printed films are made from materials that can be easily dissolved with conservation-grade solutions, in case conservators need to reveal the original, damaged work. The digital file of the mask can also be saved as a detailed record of what was restored.

For the painting that they used, the method was able to fill in thousands of losses in just a few hours. 

The researchers estimate that the new method can be orders of magnitude faster than traditional, hand-painted approaches. If the method is adopted widely, they emphasize that conservators should be involved at every step in the process, to ensure that the final work is in keeping with an artist's style and intent.

It will take a lot of deliberation about the ethical challenges involved at every stage in this process to see how this can be applied in a way that's most consistent with conservation principles.

The researchers are setting up a framework for developing further methods. As others work on this, they will end up with methods that are more precise.