Mouth masks embarrass facial recognition


Now that everyone is hiding behind a mouth mask, facial recognition is in doubt: the error margins of the algorithms are skyrocketing. But they can learn quickly enough to recognize covered faces.

The mouth mask has become widely established in our country in no time. In the province of Antwerp it is mandatory everywhere outside. But what if you stroll over the Meir with your mouth mask on and you want to use your iPhone? Newer iPhones work with FaceID, where an algorithm in the smartphone ‘recognizes’ your face by matching it with a previous photo of yourself. Only then will the iPhone unlock.

Is that still possible now that part of your face is covered with a mouth mask? The American National Institute of Standards and Technology, an agency that focuses on innovation, addressed this question. It studied 89 commercial facial recognition systems. The experiment went as follows: 6 million photos were given a digital version of a mouth mask, in nine different shapes and sizes. None of those 89 algorithms had been trained to handle such a mask.

The algorithms wanted to do their job. They tried to recognize the person with a mouth mask by matching it with an earlier photo of the same person. The results were not great. The best algorithms failed to recognize the person in 5 percent of cases, the worst even in half of the cases. In previous experiments without a mouth mask, the margin of error among the toppers was 0.3 percent. “Mouth masks destroy the algorithms that work on face recognition,” concluded the reputed American tech magazine The Verge.

You have to assume that the limit for an algorithm is wherever it is for humans.

Wiebe Van Ranst

Postdoctoral researcher in computer vision

Flemish experts in artificial intelligence are not surprised by those results. “The emergence of the mouth mask is a disaster for the world of facial recognition,” says Jonathan Berte, the CEO of Ghent-based AI company Robovision, which is not active with the technology for ethical reasons. Artificial intelligence is the idea that technology will someday approach human intelligence. Face recognition belongs to that field. But if an algorithm is so confused with a mouth mask, isn’t artificial intelligence far from human?

Berte explains this. ‘Face recognition is a learning model that starts from the data it gets: position of mouth, nose, chin, hairline … When nose and mouth disappear, a crucial part of the data used to test against previously stored data disappears. face.’ And the algorithm is not trained for part of the data to disappear, says Wiebe Van Ranst, postdoctoral researcher at EAVISE, a KU Leuven group on computer vision. ‘An algorithm can learn from large data sets. But there simply are no large datasets of photos with mouth masks. ”


They can get there. The research reveals a major weakness of self-learning algorithms: they are not prepared for change. This is inherent to the system of self-learning algorithms. You train them with datasets in which they search for patterns that they then reproduce themselves. That is ‘supervised learning’, currently the main focus in the field. But Berte notes a shift. “Companies such as Google and Facebook are increasingly investing in algorithms that learn without supervision.”

5 percent

margin of error
Even with the best algorithms, the margin of error skyrocketed: from 0.3 percent to 5 percent.

Nevertheless, it remains feasible for the time being to disrupt the algorithm regularly. A year ago, Van Ranst, together with professor Toon Goedemé and master’s student Simen Thys, succeeded in fooling an algorithm that could detect a person. That person just had to carry a sign with a ‘patch’ on it, a kind of image that another algorithm had determined would confuse the detector to the maximum. The survey made the world press.

People can sometimes be too fast at technology, but technology is also advancing rapidly. Innovation often comes from the major technology giants. Apple is an example. That company recently obtained a patent for facial recognition based on the location of blood veins, rather than appearance. In twins, for example, facial features can be completely the same. But blood veins are unique, they don’t lie.

Van Ranst is convinced that the mouth mask problem is therefore not insurmountable. ‘You have to assume that the limit for an algorithm is wherever it is for humans. I can probably recognize someone with a mask on, but not with a balaclava. That will also apply to an algorithm. ”


Please enter your comment!
Please enter your name here