Artificial intelligence will recognize art thanks to citizen science

“Our great challenge is to provide artificial intelligence (AI) with common sense”, explains the researcher at the Barcelona Supercomputing Center (BSC), Joaquim More. This expert in computational linguistics verified to what extent machines are incapable of contextualizing by performing an image recognition test with the work Saint George killing the dragon by Raphael. “The helmet identified him as a motorcyclist’s helmet and the dragon at the foot of the horse as a puppy,” he recalls.

The reason for the interpretation errors that the AI ​​makes comes from the lack of data, as Joaquim Moré laments: “it recognizes people riding vehicles because it has more data from today’s daily life, but not a dragon or a horse to wear.” in the context of the moment of the work. “This is what we are trying to solve with the project. Saint George on a Bike. We are going to put a cultural layer on the AI ​​so that it makes the proper identifications”, she reveals.

Thanks to this initiative, which has the ambition that machines interpret the images, it is the first time that artificial intelligence provides descriptions of images of cultural heritage taking into account the temporal context of the creation of works of art.

The project Saint George on a Bikewhich is coordinated by the BSC in collaboration withEuropeana Foundationtrains these Artificial Intelligence models using natural language processing algorithms and deep learning. “The key is to collect very powerful databases. However, we got into the project without knowing that there was hardly any art, since there are some very limited data sets, it is not extensive or of good quality”, warns the BSC researcher and coordinator of Saint George on a BikeMaria-Cristina Marinescu.

In order for the machines to have the amount of data necessary to train the AI ​​model, they resorted to citizen science, which is common in projects in other fields such as the environment. They went to the portalcrowsourcing Zooniverse, which still has active the Saint George on a Bike project campaign, where more than 700 volunteers have already signed up. One of those people who collect descriptive annotations is the researcher from the NOVA University of Lisbon Giulia Sonetti, who in addition to citizen science is also a social volunteer of the association Ageless bike, with which he picks up elderly people from residences to take them by bicycle. “I like to collaborate and more if it is to collect scientific data,” she celebrates.

Another of the Saint George on a Bike volunteers is the artist Júlia Pérez, who has previously worked on other projects related to the technological world, such as the generation of virtual three-dimensional models of sculptures. “In Fine Arts they teach us the requirements for a visual construction, with elements such as the light that structures a painting. At the beginning it was difficult for me to describe the image without taking these elements into account. All this information that people perceive involuntarily and that produces an emotion in us are layers that may be incorporated into the AI ​​in the future, although for now we limit ourselves to describing the images”, admits Pérez. BSC researcher Maria-Cristina Marinescu clarifies that, for now, they focus on “training a model so that, when you show it another photo other than a work of art, it can accurately describe what appears in it, based on the data that we have introduced from other descriptions, but not its symbolism

The system manages to make inferences using a language model. Moré gives an example: “we resort to ingenuity, so that he manages to deduce that if we have a person mounted on a horse in a painting, it is a horseman. If he also carries a sword, he is a knight.”

To collect these descriptions, BSC researchers have focused on images of 16,000 works from the 12th to 18th centuries, which they have obtained from museums and additions. They have focused on that period, “mainly because there is no abstract painting,” says Marinescu. “We already have more than 11,000 descriptions of 3,000 works of art. Volunteers go at a rate of 150 entries a day. It is important to have several different descriptions of each image, in order to have more quality data”, highlights the project coordinator.

The project will have multiple applications. “In addition to generating rich, quality data on cultural heritage so that people can access it openly, it will help museums better organize their heritage and be able to carry out analyzes and launch creative initiatives,” says its coordinator Maria -Cristina Marinesco. “Also so that artists can look for iconographies to make groundbreaking works iconographically. For example, Jesus Christ never appears eating”, adds BSC researcher Joaquim Moré.

In order for citizens to understand the possibilities offered by the use of artificial intelligence for image detection, the extraction of relationships between thousands of images and the opportunity to organize virtual exhibitions based on the relationship between works of art, the Saint George on a bike project has published an informative video.

Beyond art, “the same system can be explored for other uses, since it represents a first step towards contextual image recognition, from which artificial intelligence will be able to go beyond what it currently does with capchas, which are full of traffic lights, cars and little else”, concludes Moré.

You can follow THE COUNTRY TECHNOLOGY in Facebook y Twitter or sign up here to receive our newsletter semanal.

50% off

Exclusive content for subscribers

read without limits

---