In 2019, the Rijksmuseum in Amsterdam embarked on a multiyear restoration of Rembrandt van Rijn’s 1642 masterwork, “The Night Watch.” The painting, which had weathered vandalism and a major trimming in 1715 that lopped off its top and left portions, began undergoing a restoration in public view: a glass case was constructed around the canvas in the museum’s gallery, allowing visitors and an online audience to witness restorers at work. More recently, Operation Night Watch, as it is known, turned its focus to a central facet of the conservation project: reconstructing the missing parts of the painting, long lost since the 18th century.
To do so, the Rijksmuseum referenced a copy of the work, made by Dutch painter Gerrit Lundens between 1642 and 1655, and called on high-tech scanners and artificial intelligence (AI), training neural networks that could, quite simply, “paint like Rembrandt.” The missing segments were then printed and mounted to recompose the painting. And last month, three centuries after its debut, the Rijksmuseum unveiled the restored and reconstructed “Night Watch,” presenting a full view of the piece as Rembrandt might have painted it.
“The Night Watch,” which was altered in 1715 to fit into Amsterdam’s City Hall (above), was reconstructed by the Rijksmuseum (below) using AI and a copy of the work by Dutch painter Gerrit Lundens. Image: Rijksmuseum
The Rijksmuseum’s high-profile use of AI to study, preserve, and restore art cements the technology’s expanding role in museums’ conservation efforts.
Art restoration is a challenging craft — practically, ethically, and theoretically. “There are two schools of thought in [art] restoration,” says Lisa Rosen, Director of Fine Art Restoration in New York City, who’s practiced the craft since 1986. “In Italy, where I learned over many years, they teach you not to falsify or recreate the missing area exactly. You have to go down a tone so that your retouching is noticeable. And here in America, the galleries and people that sell art don’t want to see the difference; they want it to be perfect, like an invisible mend.”
There’s no set methodology for restoring art, as each piece has its unique history, and has aged and been damaged in individual ways. “Each piece is so very different, and you get to know each work very intimately,” Rosen says. Technology, though, is giving the process a boost. “Lately, [art restoration professionals] have been using microbes to clean marbles in Italy,” she adds. “They set up a room, brought different samples of microbes in, and put them on top of the marbles that were filthy and stained. These microbes ate away at the stains and cleaned it perfectly.”
Museums’ in-house teams of conservation and restoration specialists are also finding themselves newly armed with an array of high-tech conservation tools. From 3D imaging and reluctance transformation imaging that virtually moves light across scanned image surfaces to infrared heating tools that enable zero-contact conservation, technology is rapidly empowering professionals in art restoration sectors.
Installation of the missing pieces alongside the original “Night Watch.” Image: Rijksmuseum / Reinier Gerritsen
Robert Erdmann, Senior Scientist at Rijksmuseum and Professor at the University of Amsterdam, tells Jing Culture and Commerce that restoration professionals should see these technological advancements, particularly AI, as tools, not threats, to the industry. “AI is not replacing anyone,” Erdmann says. “I’m not replacing a traditional restorer, because I’m not doing restoration. I am making tools to make them all more powerful at what they do. We’re bringing the power of the computer and the image to help museum professionals do their work better, and not replacing anyone at all.”
Below, Erdmann and Casper van der Kruit, Press Officer of Rijksmuseum, expand on the museum’s thinking behind Operation Night Watch.
Why was the restoration of “The Night Watch” important to the Rijksmuseum? And why conduct the process in public?
Robert Erdmann: The “Night Watch” is the centerpiece of the Rijksmuseum; it’s the biggest painting that we have from Rembrandt. It is probably the most important painting that we have in the Rijksmuseum. So when people go to the museum, they expect to be able to see it. We can satisfy both of these needs at the same time by doing our work in front of the public. This is an opportunity to educate the public about how much effort and expertise and how many people and amazing technologies go into the behind-the-scenes work at the Rijksmuseum. It will make them appreciate more how important these works are to us. In many cases, for young people, it will show them the possibility of additional careers in the museum related to things like science and engineering.
Casper van der Kruit: We want to be transparent on everything we do. This is the most important painting we have, and we do not want to take it away, do everything in secrecy, and then bring it back — ta-dah, here it is. No, we want the whole world to see every step we take to do the conservation of this important painting. Nothing is hidden from the public; we want to be as transparent as possible to our audience on what we are doing, especially because this is such an important painting.
“The reconstruction of the missing pieces is a public face of how artificial intelligence is being used here,” says Robert Erdmann, Senior Scientist at Rijksmuseum. Image: Rijksmuseum / Reinier Gerritsen
Could you outline how AI was used in Operation Night Watch?
Erdmann: In the Rijksmuseum, the reconstruction of the missing pieces is a public face of how artificial intelligence is being used here. For example, when we take hundreds of photographs of an artwork so they’re very high resolution, the way these photographs get fused together into one big photo, that is with the help of artificial intelligence. We take high resolution, visible photos of a painting, but we also have a number of additional techniques that we can use to look at paintings. For example, an infrared camera, can see through the paint and see what’s what’s behind the paint, because oil paint is transparent to infrared light. So we can see past versions that may have been painted over.
This gives us a large collection of different kinds of images: visible image, infrared image, ultraviolet image, macro X-ray fluorescence which tells us chemical information. And artificial intelligence helps us to fuse these together so they’re very perfectly aligned with each other, and then interpret these multiple pieces of information about every pixel into something meaningful.
How else do you foresee AI aiding in museums’ restoration projects?
Erdmann: When it comes to the scientific side of the museum, I anticipate very rapid increase in the use of artificial intelligence to help us do our work, especially in relation to images. And in the slightly more distant future, I anticipate it being a very important tool for art historians who may help who may use artificial intelligence to help them find similar patterns of the visual visual composition that exists throughout our huge collection.
What are some initial steps a museum can take to adopt AI technology into their institutions?
Erdmann: I think academia is where this is most likely to be to be fruitful. If I were a museum professional, and I didn’t know anything about AI, but I was excited about the possibilities, I would call the computer science department of my local university and say, “Hey, here are some problems that I have. I work with a museum, would you like to collaborate?” It’s been my experience that in the academic world, people are very excited to work in the museum world, because it’s working with famous works of art.
The biggest barrier is that these two professional worlds have completely different vocabularies. They don’t necessarily understand each other at first, so it requires a lot of effort on on both sides to learn to communicate effectively. I think both of these worlds are eager to collaborate with each other, but that communication is the most is the biggest hurdle.
