The last clue that exists of Medicine, Jurisprudence and Philosophy, the so-called Paintings of the Faculty, by Gustav Klimt, is from 1945. The three great works were in a castle in Austria that, the day before the end of World War II, the Nazis had burned before the Red Army confiscated an artistic heritage that they themselves had plundered all over Europe. In that fire more works by the Austrian artist disappearedIt is not known how many, what seems more than proven is that these three pieces from the end of the 19th century, measuring more than four meters, succumbed to the flames. The only trace left were black and white photographs until recently the Google Arts & Culture and the Belvedere Museum of Austria the pieces and their brilliant colors have been resurrected thanks to Artificial Intelligence.
The work has been done by an algorithm that has been trained by specialists from both institutions after exhaustive chromatic research. The starting point, since of the three pieces only black and white images remained, was a fragment of color that is still preserved from the Medicine. From there, 80 images of Klimt paintings were collected and distributed in different institutions. With this first information, the robot would learn through trial and error a bias towards the colors of its work. The experts resorted to works such as Beethoven Frieze, where golden snakes appear similar to those surrounding the three women at the forefront in Jurisprudence, they explain in the Google Institute.
The before and after the coloring of the algorithm of the box of the ‘Jurisprudence’. Courtesy of Arts & Culture Google Institute
But an algorithm of this type needs more food to continue learning, about 5,000 images on average to assimilate an object, so they also showed him a million photos of things in the real world, including people, animals and buildings; and 91,749 works of art already stored by Google Arts & Culture (the platform houses pieces and documentation from more than 2,500 cultural institutions from 80 countries). “This allows the machine learning model to assimilate the limits of objects, textures and compositions common in works of art,” say Google experts.
The before and after of the work ‘Medicine’ by Klimt. Courtesy of Arts & Culture Google Institute
Not only did they use images, Klimt experts such as Franz Smola, from the Belvedere Museum, did a research work gathering academic and journalistic documents of the time. The specialist used, for example, excerpts from the journalist Ludwig Hevesi’s chronicles on the Jurisprudence which was exhibited in 1903: “Three terrifyingly beautiful avenging goddesses, with golden snakes in their hair.” Six years later, in another chronicle, Hevesi returns to treating the painting: “… a luxurious hell, where the golden instruments of torture are encrusted with diamonds and the martyrs bleed rubies.” The precision in the details of these texts and, above all, in the description of the colors has been as decisive for learning the robot as the images. “To make the paintings historically accurate, we guided the algorithm with Dr. Smola’s research. If we know that a certain object has a specific color, we add that color directly to black and white photos ”, they explain at the Google Institute.
The before and after of Klimt’s ‘Philosophy’. Courtesy of Arts & Culture Google Institute
With all this information, Emil Wallner, a Google engineer, spent almost six months programming the Artificial Intelligence code with which the algorithm works so that it could generate color predictions according to Klimt’s work. In the next step in the process, Smola and the lab team reentered. The robot does not manually color the paintings, but instead does a statistical analysis of Klimt’s existing artwork and learns to mimic the coloring style. So, using the color references generated by the algorithm, the specialists carefully inserted the colors into the three Klimt paintings.
“The result was surprising because we were able to color even the parts about which we had no information,” explains Smola. “When I first saw the green sky of Philosophy, I exclaimed: ‘what is this?’ I was amazed because I assumed it would be blue. It was a special emotion, something that I will never forget ”, accompanies Wallner. The sky was emerald green, as some newspaper documents that described greenish tones in that part of the painting had already advanced.
Google engineer Emil Wallner and Klimt expert Franz Smola at the Google lab in Paris Google Arts
In this way they revived the works with which the painter caused a scandal – another throughout his career – in the academic institution. In 1894, when the University of Vienna commissioned Klimt and artist Franz Matsch for the works, they asked for allegorical paintings depicting these three disciplines. An opportunity that the artist took advantage of to criticize “the narrow-mindedness of the Austrian state and society.” To achieve this, he unfolded his imagination through mythological concepts such as naked women trapped in snakes, children, pregnant women, skeletons, spectral beings with long hair, all brilliantly finished in Klimt’s characteristic gold. The result was the offense and anger of the experts who described the pictures as pornographic and offensive.
A technician uses Google cameras to photograph Klimt’s’ Judith II ‘at the Galleria Internazionale d’Arte Moderna in Ca’ Pesaro in Venice. Google Arts
Researchers at the Gustav Klimt Foundation recall that after being required on several occasions to review his creations, the painter finally removed the pieces, returned the money they had given him and sold them to Serena Lederer, a wealthy Jew and Klimt art collector living in Vienna, who was a victim of Nazi plunder. His collection ended up in the castle that the SS officers burned down. Now revive on the internet. The result has become a 360º augmented reality experience that is part Klimt vs. Klimt – The Man of Contradictions.