Short Wave - Why Tech Companies Are Limiting Police Use of Facial Recognition

Episode Date: February 18, 2021

In June 2020, Amazon, Microsoft and IBM announced that they were limiting some uses of their facial recognition technology. In this encore episode, Maddie and Emily talk to AI policy analyst Mutale Nkonde about algorithmic bias — how facial recognition software can discriminate and reflect the biases of society and the current debate about policing has brought up the issue about how law enforcement should use this technology.See pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences.NPR Privacy Policy

Hey, everybody, Emily Kwong here. Today we have an episode about the fraught nature of facial recognition technology. It's from late June. When news broke that Amazon, Microsoft, and IBM, were putting limits on how this technology is used by law enforcement. Facial recognition sometimes gets it wrong. And this issue was first brought forward by black computer scientists and researchers. And we want to highlight them today.

Okay, here's the show. You're listening to Shortwave from NPR. All right, Maddie, let's start with the basics. What do you know about facial recognition technology? Well, I remember being very weirded out by it when I could suddenly unlock my cell phone with my face. Right, me too. That's called a one-to-one search. Your phone is basically saying, aha, yes, this is Maddie's face, we shall unlock.

But it also kind of creeps me out. And I will say, Emily, sometimes it doesn't recognize my face in the morning, which is rude. So here's the thing about facial recognition. One, it is imperfect. And two, it's completely unregulated. There are no federal laws or standards dictating how these technologies should and shouldn't be used. Innovations in AI have basically moved way faster than policies to regulate them. Right.

And I guess now we're seeing facial recognition being tested in doctors' offices. to help diagnose patients, and in shopping malls, to look at patterns of how people move around, that kind of stuff. Right. And local and state law enforcement agencies have been using facial recognition technology for years to identify people through what's called one to many searches, taking, let's say, a photo of a suspect or grainy security camera footage, and seeking to match the image within these massive photo databases,

made up of mugshots, passport and visa pictures, and driver's license images. This technology has helped agencies solve cases, identify victims, but civil liberties groups say it's a violation of privacy and prone to discrimination. And that's what I want to talk about today. Growing evidence that facial recognition identification systems are riddled with gender and racial bias. All of these systems work better on lighter skin faces than darker skin faces. They all overall worked better on male-identified faces than female-identified faces. That's MIT researcher Joy Bula Mweeney, speaking with NPR's Bobby Allen earlier this month.

In 2018, Joy and fellow researcher Timnit Gaboru were among the first to provide evidence of algorithmic bias in facial recognition software. It was groundbreaking work, raising questions about just how accurate these systems really are. So today on the show, algorithmic bias, how even facial recognition software can discriminate and reflect the biases of society. And how current debate about policing has opened the door for a national dialogue about how this technology should be used by law enforcement. All right, Emily Kwong, so we're talking about this announcement from a string of tech companies that they are going to put limits on their facial recognition technology, especially when it comes to law enforcement. Amazon, Microsoft, and IBM. Yes. On June 8th, IBM said it would discontinue general purpose, facial recognition or analysis

software altogether. Get out of the business completely. And it made an impression. After IBM's big letter, Amazon announced a one-year moratorium on sales of their very popular software recognition, spelled with a K, to law enforcement, to give Congress time to, quote, implement appropriate rules. So a one-year ban. Yes. Microsoft took it a step further, saying it wouldn't sell products to law enforcement at all until a federal law is in place. Here's Microsoft President Brad Smith speaking to the Washington Post. We need to use this moment to pursue a strong

national law to govern facial recognition that is grounded in the protection of human rights. And for Motali and Kande, who has been pushing for regulation changes in tech for years, this was a big deal. When these words were coming out of Silicon Valley, she felt all of the feelings. My initial was, thank God. Thank God. I was happy. I was pleased. I was optimistic. I was short of breath. I was exhausted. Montali is the CEO of AI for the people, a fellow at both heart. Harvard and Stanford universities. For her, these announcements shifted the conversation, but that's about it. So I'm pleased. It's got us incredibly far, but we're by no means out of the woods. Not out of the woods, because for all of the advancement in facial recognition,

these systems still get it wrong. They'll incorrectly match folks, what's called a false positive, or fail to associate the same person to two different images. So a false negative? Yeah, and what's vexing is these areas. are happening more often when the machines are analyzing dark-skinned faces. And that can disproportionately affect already marginalized communities prone to unconscious bias at the hands of law enforcement, leading to false accusations, arrests, and much worse. So until there's action on this, Mutali said words just aren't enough. Gotcha. So, okay, Emily, let's unpack this a little bit.

Let's talk about how bias gets into facial recognition systems in the first place. I'd love that. Okay, so it starts right with how these systems learn to do their jobs, a process known as machine learning. So to make facial recognition systems, engineers feed algorithms large amounts of what's called training data. In this case, that would be pictures of human faces. Yes.

The way machines learn is that they repeat a task again and again and again and again and again. Developing a statistical model for what a face is supposed to look like. So if you wanted to teach the algorithm to recognize a man, you'd put in like millions of pictures of men. You got it. The machine will then measure the distance between the eyes on each picture, the circumference of the nose, for example, the ear to eye measurement. And over time, the machine starts to be able to predict whether the next image it's seeing is, quote, a man, which sounds okay, right? here comes the but. But the machine is only as smart as its training data.

So remember Joy Bulamweeney, who I mentioned at the top of the episode? Yeah, the one at MIT. Yes. So she and her colleague Timit Gabriel developed a way to analyze skin color in these training sets. And the two they looked at were overwhelmingly composed of lighter-skinned subjects, 79% for IJB-A and 86% for adients. These are two common data sets that were, largely, as Joy put it, pale and male. So basically, the training data used to create these algorithms is not diverse, and that's how that bias gets in.

The diversity of human beings is not always being represented in these training sets. And so faces outside the system's norm sometimes don't get recognized. Here's Matale explaining what the research meant to her. That goes back to this other issue. of not just hiring, but a bigger issue of there's no one in the team to say that you haven't put all the faces, you know, you haven't put all the digital images of what all human beings could look like in the way that they show up in society in order to recognize these faces. So after realizing how unbalanced these training sets were, Joy and Timit decided to create

their own with equality in race and gender to get a general idea of how facial AI systems performed with a more diverse population. So basically, they fed it more diverse pictures to look at. Yeah, it was kind of interesting. They used images from the top 10 national parliaments in the world with women in power. Yeah, specifically picking African and European nations. And they tested this new data against three different commercially available systems for classifying gender.

One made by IBM, the second by Microsoft, and the third by Face Plus Plus. And in running these tests, Joy and Timit found clear discrepancies along gender and racial lines, with darker, skinned faces getting misclassified the most. Here's Mutali again. So one of the things that Joy Bullywami's amazing work looks at is the coloration between short hair and gender. So many, many, many black women with afros were mislabeled as men, misgendered, because the system had trained itself to recognize short hair as a male trait. And this research project, Maddie, produced a massive ripple effect.

Further studies, legislation, in December, the National Institute of Standards and Technology, or NIST, published a big paper of its own, testing 189 facial recognition algorithms from around the world. And they found biases, too. Looking at one global data set, some algorithms in their study, produced 100 times more false positives with African and Asian faces compared to Eastern European ones. And when tested using another data set of mugshots from the U.S., the highest false positives were found among American Indians with higher rates in African American and Asian populations. Again, depending on the algorithm.

Wow. Yeah, that is not what you want from your data. And I'm guessing white men benefited from the highest accuracy rates. Yes, they did. Now, the NIST study did conclude that the most accurate algorithms demonstrated far less demographic bias. But for Mutale, this evidence of bias raises a bigger question about the ethics of relying on AI systems to classify and police people at all. The problem with AI systems and machine learning is that they're really, really, really good at standard routine tasks. And the issue with humans is that we are not standard. we're not routine. We're actually massively messy.

Right. We're not all the same. But when a police officer searches a face in the system, they're not making an arrest based on just that match alone, are they? Oh, absolutely not. Yeah, it's a tool for identifying potential suspects. But if you think about how there's already implicit bias in policing, critics a facial recognition are basically saying it doesn't make sense to embrace technologies riddled with bias too. Right, if all this research has shown these tools are capable of misidentifying black people. We cannot use biometric tools that discriminate against a group of people who are already discriminated against within the criminal justice system, but policing most specifically. Maddie, when I first spoke to Mutale in March, she was open to moratoriums on facial recognition like Amazon is doing,

buying time for these systems to improve or regulations to be put in place. But the protests have changed her views. Because why am I being moderate when what we need to do is completely reimagine how we interact with technology? So now, she wants to see facial recognition banned from law enforcement use, which some cities in the U.S. have done. Mutale has tried to push for legislation to outlaw discrimination in technology before. But it seems like now people are paying attention and have a language for talking about structural racism that they just didn't have before. Whether white or America listen to me or not, I was going to continue with this work. I believe that technology should be an empowering force for all people.

And that's my work. But now having old and new, not just allies, but co-conspirators, right? I'm so happy because I didn't think it would happen in my lifetime. And it's happening. I'm delighted. Okay, Emily Kwong, I appreciate you. Thanks for reporting this out. You're welcome, Maddie.

This episode was produced by Britt Hansen and fact-checked by Burley McCoy. Our editor was Viet Le. I'm Maddie Sofeyer. And I'm Emily Kwong. Thanks for listening to Shortwave from NPR. In recent mass shootings, people have been targeted for who they are or who they worship. But on June 28, 2018, people were targeted for the job they do at a newspaper. Listen to the news series from NPR's embedded about the survivors at the Capitol Gazette.