Science Friday - How AI Is Influencing Decisions In Police Departments And Courtrooms. Sept 13, 2019

Episode Date: September 13, 2019

Facial recognition technology is all around us—it’s at concerts, airports, and apartment buildings. But its use by law enforcement agencies and courtrooms raises particular concerns about privacy, fairness, and bias, according to some researchers. Some studies have shown that some of the major facial recognition systems are inaccurate. Amazon’s software misidentified 28 members of Congress and matched them with criminal mugshots. These inaccuracies tend to be far worse for people of color and women. We'll talk about how AI is guiding the decisions of police departments and courtrooms across the country—and whether we should be concerned. Plus: Scientists were threatened with firings after the National Weather Service projections for Hurricane Dorian contradicted President Trump’s tweets, and more of the biggest science stories from the week. Finally, wind turbines are great at producing green energy. But when they reach they end of their life-span, their parts are incredibly difficult to recycle.   Subscribe to this podcast. Plus, to stay updated on all things science, sign up for Science Friday's newsletters.

This is Science Friday. I'm Ira Flato. In the weeks following the National Weather Service, contradicting President Trump's prediction about the path of Hurricane Dorian, also known as hashtag SharpieGate, there have been reported threats of firings and suppression of science at the agency. Here to fill us in is Sophie Bushwick, technology editor at Scientific American. Welcome back. Sophie, always good to see you. Thank you. So let's talk about this. What happened in the weeks after Dorian? How has it? has that administration and agency handled this? Well, the president and his support staff have doubled down on the idea that Alabama really was at risk from Hurricane Dorian. And then this was contradicted when the National Weather Service office in Birmingham tweeted that Alabama was in no danger, which makes a lot of sense in terms of preparation.

You don't want people making runs on stores or thinking they have to evacuate if there's no danger. And then the New York Times has recently reported that the secretary, of Commerce then threatened Noah, the National Weather Service is part of Noah, and they threatened to fire the political appointees if they didn't issue a corrective to the message. And so as a result, there was an unsigned memo that Noah released, sort of saying that the Alabama office shouldn't have tweeted out that Alabama wasn't going to be in danger. So they were really asking politics that triumph over science in this case. That's correct. And this really kind of points out the danger of a

post-truth era, right? Like, it's one thing if you're arguing about the size of a crowd. And it's another thing when you're talking about people's lives at risk. I mean, knowing accurately where a hurricane is going to strike and what areas are in danger and which areas are not is really vital to saving people's lives and also to preventing damage. And if you can't rely on that, and if you so distrust of the weather service, that's potentially extremely dangerous. So because for the next time you may think it's a hoax also. Exactly, exactly. If you can't trust an accurate hurricane forecast, then you're not going to take action in the best possible way.

All right. Let's move on to another story. Another interesting story about a test for PTSD. Yes, this one's really interesting because there still today is a lot of stigma around post-traumatic stress disorder. Many people don't, if they're experiencing symptoms, they might not want to admit it or to go seek the treatment they need. So the military obviously has a lot of soldiers are often exposed to traumatic situations, and they want to have some way of screening for PTSD that wouldn't just rely on people's self-reporting. And so they've investigated, they supported research that has found a potential blood test that could be used to screen for PTSD.

So basically, they chose about a million different biomarkers. So biomarkers are just things that could be like heart rate or proteins or molecules. in the blood, and they compared these biomarkers in two groups of veterans. It was about 150 people. All of them had served overseas and had experienced a potentially traumatic event in battle. And about half of them had symptoms of PTSD and the other half did not. So by comparing these biomarkers in the two groups, they narrowed down to a list of 28 different biomarkers, so 27 substances in the blood and heart rate that could be used to differentiate

between the groups. And was it pretty accurate? It was about as accurate as some other screenings. It was about 81% of the time it could distinguish between these groups when they tested it. And the idea is that this wouldn't be, you wouldn't just take this test and be like, oh, I have PTSD. But it would be, oh, this person potentially has PTSD and it would allow that person to then go into a psychological evaluation or undergo further screening. And it's not quite ready then. No, it's not quite ready. First of all, They only tested it in a quite small group, right, like about 150 people initially, and then they tested it on a different smaller group. And the second thing is it was only men in this group, and it was only veterans.

So if you wanted to use it in, say, the general population or in women, you'd want to do a lot more testing. And indeed, the research group does plan to test it further. Interesting. Let's move on to the Indian Space Organization, space research organization, launched the lunar lander, but they sort of lost contact with it, right? It's a little sad story. It's a little sad, but there's a silver lining.

So the lander was supposed to land. It would be the first lander to land near the south pole of the moon. And unfortunately, when it was about two kilometers above the surface, the Indian Space Research Organization lost contact with it. And so they thought it might have crashed. But just recently, so this was a two-part mission, there was an orbiter as well as a lander. and the orbiter photographed a part of the surface of the moon,

and the Indian Space Research Organization has said they found where the lander is, and now they're trying to reestablish contact. But, you know, the Indian program doesn't get a lot of press here. They're really very serious about space, aren't they? Yes, absolutely. They've sent an orbiter to Mars. They're only the, they've landed a previous mission. They sent an orbiter around the moon,

and it crash landed at the end of its mission. and this was their attempt at a soft landing. And if they'd achieved it, they would have only been the fourth nation to ever have a soft landing on the moon. Let's keep our fingers crossed for reestablishing. I will. Finally, your last story is shocking. I'm shocked because it's about electric eels. Scientists found a new electric eel species?

So basically what happened was we had just assumed that electric eels were all one species. And this study took about 107 different electric eels, and they compared their head shapes and their DNA. and they say, no, there's actually three different species we're looking at here. And one of them, my favorite, electroforest voltai, I think. I might be getting the genus name wrong, but this species is the most shocking of all. It can release 860 volts in a shock. For comparison, a defibrillator that can shock your heart back to life is about 1,000. Do we know why it's so big and how big it got that, you know?

So basically the way that electric eels produce these shocks are with organs made up of electric current producing cells. And the idea is if you have more cells, they act kind of like batteries in a series and they can strengthen and increase the output of the shock. Very interesting, Sophie, Sophie, Sophie, Bushwick, a technology editor at Scientific American. And now it's time to check in on the state of science. This is KERNO. St. Louis Public Radio News. Iowa Public Radio News. Local science stories of national significance.

If you've driven by a wind turbine, you've probably noticed that they are huge, getting bigger. They're often several hundred feet tall, and the blades can be as tall as the Statue of Liberty. You heard me right. So when a turbine is decommissioned, most of the parts can be recycled or reused, but not the blades. And this is causing a problem for wind farms. and landfills. What happens to these giant turbines at the end of their lifespans?

My next guest is here to tell us how the Midwest deals with discarded turbine blames. Christina Stella is a reporter for Harvest Public Media and NET. That's Nebraska Educational Telecommunications. In Lincoln, welcome, Christina. Hey, Ira. Thanks for having you. You're welcome. There's a lot of wind energy developing happening in the Midwest. Tell us a little bit about what it's like in Nebraska. Are there lots of turbines going up there? Well, I mean, a lot of states in the middle of the country have really been embracing wind development, especially over the last 10 years. But Nebraska has been a little slower on the uptake.

So across our state, we've got about 1,000 turbines operating across 25 farms. And to give you a little context, that's pretty low compared to our neighbors like Kansas and Iowa, which each have between 3 and 5,000, Iowa having closer to 5,000 turbines. So these, as I mentioned, these turbines, they have these big blades on them. What are they made out of and what is the problem with disposal of them after life? Yeah, so the problem really is twofold, okay? They're really strong and they're really big, which you already said. So turbine blades are made from a mix of resin and fiberglass, and so the resin is what makes them really durable, while the fiberglass gives them a little bit of bend, sort of like an airplane wing.

So they're really built to survive any kind of weather pattern. No, go ahead, I'm sorry. Go ahead. So, but what that means is that it takes really high-powered equipment to break them down, and it costs a lot of money and energy to do that. So a lot of times, blades will just get cut up into three or so pieces and hauled off, and that poses quite a space issue for landfills. So there's really the, you have to have a special landfill to take them in, not just any landfill. Yeah, I mean, a lot of municipal landfills are just not as interested in taking on wind projects because they just create so much weight. You know, to give you a sense of the size at the wind farm that I visited in Kimball, Nebraska,

one turbine blade was 127 feet long. And that's an old model. These days, the blades are upwards of 300 feet long. And so if each turbine has three blades and a farm can have tens of turbines, you can imagine how quickly that adds up at the landfill. I can't. You talked to Rob Van Fleet, who was a scrapper in Nebraska, someone who disposes of these blades. and we have a clip from him.

So if you're a small utility or municipality, let's say, and all of a sudden hundreds of blades start to come to your landfill, you don't want to use up your capacity for your local municipal trash for wind turbine blades. So what do they do then? What happens when a wind turbine is decommission? Are there any regulations for what to do with the blades? Well, right now in the U.S., there's actually no federal regulations for wind turbine disposal.

I mean, a couple of states have adopted some policies requiring companies to provide what's called an end-of-life plan. But a lot of the time, it's really local governments asking for those plans as the farm is being developed. But the thing about those plans is that they don't tend to include where the waste actually goes, just kind of how turbines will be taken down and how the land is going to be restored. So a lot of the time, they do end up just being taken to whatever landfills will accept them. Now, in Europe, the EU has tightened their waste management guidelines really to try and force companies to divert wind turbine waste from landfills. And sometimes that looks like selling older parts to markets in Eastern Europe and

Asia. But here, because the industry is still pretty young, we are still, and companies are still trying to answer that question and find a better solution. You did talk to a couple of people who are thinking of new ways of getting rid of these blades. I'm thinking of Cindy Langstrom and Casper and Carl England. Tell us about what they said. So Cindy has actually established a special turbine blade disposal project. And in my story, she talked about how it was too expensive for them really to actually crush up the blades. And so she cuts them up into three pieces. And then she takes two of them and stuffs them into the third. And then they sort of store that at their landfill in Wyoming. And Carl is actually, he's

He's worked on a process that involves grinding up the blades and then making something called a polypellate out of them. He sort of takes that material and mixes them with thermoplastics. And so you can make all kinds of cool stuff like decking materials and piping and all that. Creative ways. We're going to have to come up with some more of them. Thank you, Christina Stella, reporter, NET and Harvest Public Media in Lincoln, Nebraska. We're going to take a break. And when we come back, we're going to talk about facial recognition.

I'm Ira Flater. This is Science Friday. from WNYC Studios. This is Science Friday. I'm Ira Flato. Three U.S. cities, I'm talking about Oakland, San Francisco, and Somerville, Massachusetts, have banned their police departments from using a form of artificial intelligence called facial recognition. It analyzes a person's facial features and checks its database of faces and comes up within identity match.

The idea that your face is now being recorded and stored, can be upsetting. Presidential candidate Bernie Sanders is demanding a national ban on this technology, and other candidates are calling for greater scrutiny of how it's being used. This hour, we're going to be talking about it ourselves. We're taking our own look at how police departments, law enforcement agencies, and courtrooms are using facial recognition and other forms of AI, from emotion detection algorithms to risk assessments and asking whether these technologies are really accurate and fair. And that's what I'm going to be asking you.

What do you think about using facial recognition and other forms of AI in the criminal justice system? Give us a call. Our number, 844724-8255. 844-724-8255. Or you can tweet us. Tweet us at SciFRI. Let me start out with the facial recognition technology.

first. Jennifer Lynch is the surveillance litigation director at the Electronic Frontier Foundation. Welcome to Science Friday. Thank you. Well, just to start off, give me a definition of what facial recognition is. Sure. Facial recognition or face recognition is a technology that allows you to identify or verify the identity of somebody based on specific features of their face. Usually that's performed on digital images, but it can also be performed on video. And when a law enforcement, when law enforcement is using facial recognition, where are these photos coming from? Well, for the most part, if law enforcement is using face recognition, the photographs are

coming from mugshot databases. So about 14 different states partner with the FBI and share their mugshot photographs with the FBI, and then they have access to the FBI's photos. But what we also know is that many states, you know, is that many states, also include face recognition in their driver's license databases. So about 43 states in the United States include face recognition in their driver's license databases, and of those 20 to 30 states are sharing that information with the cops as well. That's really very interesting.

Could I find out if I'm in a database? Well, you know, I think that if you live in one of the 43 states that has face recognition in their driver's license database, then you're likely in that database if you have a driver's license. If you have a passport, you could also be in the state department's face recognition database. And if you've ever been arrested for a crime, there's a good chance that you're in the FBI's mugshot database. But there's no central place on the internet that I can look up my name and see if I'm in a database someplace. Nope, there's no central place on the internet. And so I think that's really challenging for Americans right now, because based on a study out of Georgetown, we learned a couple years,

ago that pretty much 50% of Americans are already in a government face recognition database. But it's hard to figure out which database you're in, who has access to that information, and whether you can actually get yourself out of that face recognition database. Does law enforcement have to disclose how they're using your facial recognition or you in the database? Well, I think there's a good argument that under public records laws, they should have to disclose that. we have a right to information about what data the government has on us under the Privacy Act, which is a federal law. But, you know, we should also be able to contact our local and state police departments and ask them whether they have information on us as well. And that includes

face recognition. Do we have a sense of what could be next and how it's used? Well, we do have a sense of that. And so for the most part, what we're seeing now with face recognition is that law enforcement is trying to use face recognition on static images. So that might be trying to identify somebody in a Facebook post or an Instagram post, or trying to identify somebody who refuses to identify themselves against the mugshot database. But what we're seeing on the near horizon is the use of face recognition on the back end of cameras like surveillance cameras and body cameras. So I think we will start to see that very soon in cities across the country unless we see cities start to pass bans or moratorium like we've seen in San Francisco, Oakland, and Somerville. What about facial recognition, for example, used in the public by commercial uses?

Like if you're walking to a mall and you go into a store, do they take a picture of your face and go into a database or possibly figure out what you're shopping for? Yeah, well, we don't have any federal privacy laws right now that require stores or malls to disclose that information to you. We do know that there are companies that are selling face recognition technology to stores and malls, and these companies are claiming that stores can use it to identify shoplifters or even to identify people who are longtime customers who might be willing to pay a lot of money for that next shoplifters. shoe or piece of jewelry. And sorry, go ahead. Go ahead. I'm sorry.

Oh, I was going to say, what we don't know is how do people get into these databases? You know, in order to have face recognition identification, you have to match somebody's image against an existing database of photos. So are stores responsible for putting people in a database? We don't know that. And stores could be basing that, you know, on discriminatory practices. I think that that's even more of a threat from a government database because we don't have access to public records laws

that could let us know whether stores have us in a database. So that's one of your biggest worries if this thing just gets totally flooding, right? Everybody sooner or later is in a database. Yeah, I think we're on the cusp of that right now, and that's why it's so important for communities to have conversations about what they really want to have happen in their communities. We're seeing this happen in California. I mentioned the two cities that have

already banned government use of face recognition, but we also have a bill that just passed the Senate yesterday, our state Senate, that would put a moratorium on face recognition use on mobile cameras for three years. So how concerned should we be about, I'm going to call universal face recognition coming our way? I think we should be very concerned. We can look at what's happening in China right now, there are multiple cameras on every street corner. And those cameras aren't just using face recognition, but they're also using other kinds of technologies like gate recognition to identify people as they're walking away from the camera, object recognition and character recognition to recognize license plates and cars,

all sorts of different technologies like that. And I think that we already have existing networks of cameras in the United States. it wouldn't take much to add face recognition onto the back end of those cameras. What about the possibility of a mismatch? How accurate is facial recognition? Well, it really depends on several factors. So lighting and angle of view are huge, but we also know based on some research studies that face recognition is much less accurate at identifying people of color, women, and children, or young people.

That's a lot of people. That is a lot of people. consider that our criminal justice system is disproportionately made up of people of color, that means that the use of face recognition in the criminal justice system would have an even more disproportionate impact on people of color. We got a comment on this from a listener through Science Friday Voxpop, Robert from Holly Springs, Georgia. Facial recognition is already in use with passports and other security devices.

So why should the criminal justice system not use facial recognition? Yeah, you know, you get your phone opened, right, by facial recognition. Like, you know, you're already being recognized there. Well, there's different ways to use face recognition. So if you're using face recognition on your phone, in general, that biometric is just stored on your phone, and your phone is the only source for that and the only place where there is access to that.

But there are also these vast government databases. Now, the question is, should the government have access to photographs that were taken not for a criminal purpose, but for a purely civil purpose, to be able to drive a car, to be able to travel outside the country? And I think there's a strong argument that we have never allowed the government to have vast access to those databases, unrestricted access, and we shouldn't allow that now. And I guess the other difference would be when you use your phone with a facial recognition, you've given your phone permission to look at your face, right? Yeah. I want to move on a little bit, and believe it or not, there is also emotion detection AI. And this technology that claims to assess facial movements and expressions and make conclusions about whether someone is afraid or nervous and angry, it's just interesting to think about it. And if you follow the money, this is a $20 billion industry.

companies like Amazon and Microsoft and IBM are developing and selling this technology to police departments, among others. So how accurate? How accurate could these emotion detection systems be? Here to fill us in on this is Lisa Feldman Barrett, a professor of psychology at Northeastern University. She joins us via Skype. Dr. Barrett, welcome to the program. Thanks for having me on your show. You're welcome. Let's start with how good are humans at detecting emotions from facial expressions? Well, humans don't detect emotions. Humans infer emotions. So if you and I were in the same room right now, our brains would be processing not only our facial movements, but our vocal sounds, our body postures.

There's a whole broad context that your brain takes advantage of to make a guess about what the raise of an eyebrow means, what the curl of a lip means, and so on. But so can we tell with any surety how if someone is happy or angry or not? Well, I think it depends on how well we know each other and whether or not we come from the same context. So same cultural context. So I think the research shows pretty clearly that humans are guessing. To you, it feels like you're reading someone's face like you would read words on a page, but that's actually not what your brain is doing.

It's making an inference. And if someone comes from the same culture as you and you've known them for a long time, you've learned a lot about the patterns of their facial movements and what they mean, so you can guess pretty well. But if you and I come from a different culture, then we're probably going to have some, you know, mistakes in our guesses. Because the data are really clear that people in Western cultures like ours scowl more often than chance when they're angry, but only about 30% of the time, which means that 70% of the time, which means that 70% of the time, When you scowl, on average, you know, you're feeling something else. And you scowl at a lot of times when you're not angry, like when you're concentrating or when you're confused about something. So we're using not just the feast, but a whole ensemble of signals.

And so face reading is really limited, I would say. Yeah, that's what my question was. If we are not good as humans in knowing what, you know, these expressions means, how do we teach that to an AI to recognize. Yeah, so first of all, I think it's really important to understand what AI can do and what it can't. I have four senior colleagues, and the five of us just published a paper where we reviewed over a thousand scientific studies, some of which are AI studies, where we reviewed

all the AI studies that we could get our hands on. And it's really clear that what AI can do pretty well is it can detect a smile, but not what the smile means. it can detect a frown, not what a frown means. And that's under perfect recording conditions, so when the face isn't occluded and when the light conditions are good and so on. So, you know, AI doesn't make inferences

about what a facial movement means. It just detects the facial movements. Humans are, I would say, you know, reasonably good at guessing under some circumstances and really bad at other circumstances. And scientists study, you know, what makes a good perceiver, when do people make mistakes and so on. So, you know, it's a complicated question.

That's not just about the movement of the face. It's also about what that movement means in a psychological way. Ira Flato, this is Science Friday from WNYC Studios, talking about artificial intelligence and facial recognition. So why is this so appealing then? I mean, why do companies like Amazon want to get in on it, even though it doesn't align with what we know about facial expressions? Well, I think that there's a persistent belief that everybody around the world smiles when they're happy

and frowns when they're sad and scowls when they're angry, and everyone around the world can recognize smiles and frowns and scowls as expressions of emotion. And so companies think, ah, this is a really great way to be able to read someone's emotions in an objective way. and then capitalize on that for selling products or for determining guilt or whatever people want to use it for. But the fact of the matter is that, as I said, facial movements can mean many different things depending on the context, and they're not universal. That's one thing that we know pretty clearly, I think, at this point. Lisa, do you agree? Do you think these companies are aware of the issues with their technology as not being quite ready? I don't think they're necessarily aware that they're making claims, sweeping claims that are incorrect.

I think a couple of companies are becoming aware based on this paper that we published and the press that it's getting. But it's really interesting. Some companies are super interested in trying to figure out how to do what they want to do, which is to, you know, guess at what someone's emotion is in an accurate way. other companies are maybe being a little more defensive and really want to defend what they have because, you know, they've invested a lot of money in it. Jennifer, what do you think about all of this emotion detection technology? Well, I'm really worried about it because I think, as your other guest mentioned, in the best of circumstances, the technology might be able to identify a frown or a smile, but it can't identify

what that means. However, when we see this technology sold to law enforcement agencies or schools, which is where it's being sold quite a lot now, the companies really claim that they can tell if somebody is going to do something. They can predict with this emotion detection technology, and it's just not there. It's just not accurate. And I think it will be used to target people, especially people who are from different cultures, different races or ethnicities, and it will be used to make assumptions about people. Who is going to be the bad kid in the school, who we need to pull out of school, or who is going to be the person who is lying about whether they committed a crime. That's how emotion detection will be used in the near future. Not ready for prime time, but still moving ahead.

I think we have run out of time. I'd like to thank both of you, Lisa Feldman-Barrant, Professor. of psychology at Northeastern University. Jennifer Lynch, surveillance litigation director at the Electronic Frontier Foundation. Thank you both for taking time to be with us today. Pleasure. We're going to take a break, and when we come back,

we're going to continue our AI theme here. We're going to talk about how bias shows up in AI. I'm Iraflato. This is Science Friday from WNYC Studios. This is Science Friday. I'm Iraflato. We're talking about facial recognition. and how it is being used to recognize people and not recognize people so well on artificial intelligence. We've got some interesting tweets coming in. Thomas on Twitter says the fact that the world's tech epicenter of San Francisco has banned the technology should be a major red flag.

And a lot of Twitter, I think it should somebody else on Ryan says, I think it should be similar to when law enforcement wants to enter your home. Law enforcement should need a warrant to use facial recognition. But we're going to move on now and include and being more inclusive about, you know, are machines unbiased and neutral? We think that they might be. They're supposed to be. But machines are made and programmed by people.

AI is trained on whatever data we put into it. So the AI used by police officers and judges may not be as neutral as we think it is. Alex from Madison, Wisconsin, waited on this through the Science Friday Vox Pop app this week. I'm really uncomfortable with our criminal justice system using facial recognition software because that software was created by humans, and we have a lot of implicit bias. We have, some of us are overtly racist, some of us are accidentally racist, and I can't imagine that our technology doesn't reflect that at least somewhat. So what do you think?

Let us know what you think on the Science Friday Vox Pop app or by calling 844-724-8255-8-4-4-Sai Talk, or, of course, you can tweet us at SciFri. My next guest is here to tell us more about how bias shows up in artificial intelligence and what we can do about it. Roja Benjamin is a professor of African American Studies at Princeton. Dr. Benjamin, welcome to Science Friday. Thank you for having me. We just talked about facial recognition, but what other AI is being used in the criminal justice system? So in addition to facial recognition, we have predictive policing, which is identifying particular areas and neighborhoods where more officers should be sent, greater attention. We have risk assessment, a software of all kinds, deciding people's fate at pretrial detention and sentencing and parole.

Basically, every arena where decisions are being made, there are programs underway to automate those decisions to reduce the context in which the criminal justice system operates to a single score that can be used to decide people's fate. So is our listener, Alex, right? Is bias built into AI? Absolutely. I mean, as you said at the top of the show, you know, human beings are creating this. not all human beings, but a very small sliver of humanity are coding values, assumptions, decisions into the software. And the power and the danger of the system is that we assume that it's neutral and objective,

and it has this great power embedded in it. In what ways does the bias show up? How does the bias get into the machine? So a number of different levels. So for starters, you have to train algorithms and software how to make this. decisions. And what do you use to train? Typically, you use historic data. The way that we made decisions in the past, let's say sentencing decisions or decisions about what neighborhoods to patrol, all of that data is then in the input to these systems to train algorithms, how to identify

patterns, and therefore make future decisions. So if the way that decisions were made, what neighborhoods to patrol were based in part on the socioeconomic class of that neighborhood, the racial composition of that neighborhood. All of that data is the input to these systems. And then what spit out is something that on the surface looks objective, but in fact is a reduction of those past decisions into scores, into projections about where police should go or how judges should make decisions. You know, but we are only human.

Right? I mean, we can't escape our own history if we're the ones programming the AI. Absolutely. And so what all of these systems have in common is this. They are trying to identify, project, predict risk, the risk of individuals. And so my interest is not in looking at the risk of individuals, but in the institutions that produce risk. And so, yes, we have this history, but one of the things that we can begin to start to rethink is where is the a locus of risk. And many of the policies, many of the institutional biases are now being put on the shoulders of individuals, and then we're giving individuals a score about their relative riskiness. And I think we have to zoom the lens back on the institutions that create a context

that makes certain kinds of vulnerabilities possible. You have called this new technology part of the new gym code. What do you mean by that? So here I'm trying to get us to really reckon with this history that we're talking about. And so, again, we like to think of technology as a-social, apolitical, a-historic. And by calling it the new Jim Code, which is building on Michelle Alexander's notion of the new Jim Crow, which itself evokes the history of Jim Crow, white supremacist institutions in the United States. And it's saying that that history of segregation, of hierarchy, of oppression, is the input to our new technical systems. And it gets imagined as objective when it's really coded bias in these systems. And so let's talk about it

with this historical context in mind. So how do we fix it then? So there's a number of ways that we can begin to do, and things are underway. So we have legislation, as one of your callers mentioned in terms of San Francisco and other municipalities that are banning certain forms of facial recognition and other automated carceral tools. We have lawyers underway that are thinking about how to litigate algorithms. So when a decision is made about you using an algorithm, should you have the right to take that algorithm to court or the company that produced it if you feel that the decision was wrong?

Another area is organizing, even among tech workers working together to actually challenge the companies that they work for under the hashtag tech won't build it. And so you have tech workers saying, no, even if you enforce. me if you tell me to build something for ice or for the military or for police, I'm going to resist. And then we have a broader campaign of education. That means everyday people need to start questioning these tech fixes and holding our representatives accountable for adopting them. I want to continue with this thread by bringing on someone else. Shared Goyle, who is a professor of management, science and engineering at Stanford University. He joins us via Skype. Welcome to Science Friday.

Hi, thanks for having me. You are doing work with California on improving risk assessment. How are you going to do that? Yeah, it's a complicated question. First, I want to say that I agree with a lot of what's already been said, that science and technology, these are human endeavors, and almost certainly everything that we're doing is affected by this complex history that we're contending with.

And at the same time, my feeling is that if we've done well, that there is a place for having equitable outcomes or more equitable. outcomes by using technology to drive those decisions. So let me give you one example of something that we're doing in San Francisco. So when somebody is arrested, the district attorney's office has to make a decision about whether or not to move forward, whether or not to charge that individual with a crime. And traditionally, how this is done is by reading police narratives with the human experts reading police narratives and then making this decision.

And while I think there's lots of good reasons to do that at the same time, they're in explicit human biases that can taint those decisions. And so what we're doing is we've built a blind, what we're calling a blind charging platform that attempts to reduce the effect of race on those decisions. Removing explicit mentions of race, removing people's names from those police reports, removing hairstyle, location,

other indicators that we don't really think are necessary to make a more objective decision about whom to charge and him not to by using these technicalers. But it's going to be the judges then, though, who have the final decision on these things. Well, in this particular case, it's the district attorney's office, not the judge who's making that decision. But you're absolutely right that in some of these circumstances, when someone is trying to decide, when a judge has to decide whether or not an individual is released, whether or not they're detained on bail,

whether they have other sorts of obligations. And you're right that there is this interplay between the recommendation that an algorithm can provide and then what that human expertise is saying. Can you actually test these new algorithms out that you're talking about to see if they are biased or unbiased? Well, in a way, everything is biased. And so in a sense, we don't need a test. And so that's just the nature of not only algorithms, but human decision-making, that these are going to have consequences that we're not happy with.

And I think in some cases, they're bad, and we have to try to fix those. But so in my mind, I don't try to paint this in this. binary of biased or on bias or good or bad, but is it better, is it moving us in the right direction? And how do you do that? So in some of these cases, we can look, for example, these risk assessment algorithms that are being used now across the country to see whether or not they're resulting in more people being released into the community, which I think many people would agree is a good thing, and at the same time, is there a public safety cost to that increased release?

And you see this as a tool then for improving criminal justice? Yeah, I think it's a tool in many sectors of society, in criminal justice, that's the area that, or one of the areas that I'm particularly interested in. And I do see that this is a tool for bringing equity. And here again, I want to emphasize that my view, I think many people's views who work in this area, is that this isn't a panacea, and we can't treat it that way. that we have to say the criminal justice system is broken in so many ways. There's so many systemic problems that, honestly, algorithms are not going to be the right fix for those. But in some of these narrow instances, I do think that it can act as a tool to improve outcomes. Let me ask both of you, Ruha and Sherrod, what then is a fair algorithm?

What does it mean for an algorithm to be fair in this case? I'm interested in how to employ technology tools, algorithms, less for fairness and more for justice. And so what that would mean would actually be using these tools to expose the work of institutions and of those who wield power through various kinds of carceral techniques. So that would mean looking at the institutions, for example, that create instability, that, you know, thinking about the way that the criminal justice system requires not more tools and investment and resources, but for us to decarcerate and thinking about how to turn the lens onto those who are making decisions about the most vulnerable among us. Is it possible to challenge an algorithm in court for being biased? Do you see that as happening also? There are a group of lawyers that are working to build that capacity, and there are a number of cases that I'd be happy to tweet out later

of individual lawyers and groups that are working to litigate algorithms in order to make them more transparent and just. I'm Ira Flato. This is Science Friday from WNYC. studios talking about algorithms, artificial intelligence and the uses in our legal system. Are you both, well, are you positive? Are you hopeful that this thing, this kind of better fairness or more justice, as you say, is actually going to happen? No, I think we're at that point in history where it's not totally clear. I think with many

emerging technologies, you know, face recognition is one of these examples that we've been talking. about is that I think these can be a force for good. I think they can also be abused. And we're at that tipping point that if we don't have the right regulation to make sure that these things are accountable, that we do have transparency, that we can understand what's going on, that we can take accordingly take action, that these can cause more harm than good. But I'm optimistic that we can have this thoughtful discussion and use, at least set up a system where we can get the benefit. and at least minimize some of the costs. But it's certainly, I don't think it's a done deal.

And I very much worry that these will be disheartedly abused, if not, regulated appropriately. What I'm really positive about is the growing movement of communities in different locales and nationwide, who are creating greater critical consciousness and speaking back to a lot of these automated systems. So we're talking about the Stop LAPD Spying Coalition in Los Angeles, the Algorithmic Justice League, tech won't build it, the Our Data Bodies Project, and a number of other organizations and initiatives that are not just paranoid about technology and what's happening, but are building power in communities to be able to push back and reimagine what a community would look like without these tech fixes.

So do you think this is going to be a grassroots movement? In other words, community by community, state by state, and not a national movement. if on the other hand, maybe other states are looking at what you're doing. It's really both. In almost every locale, you can find some organizations that's really working diligently around tech justice. But it's also a growing national movement and national consciousness thinking about what we as citizens, not as users of technology, because users get used. And so when we begin to think about our obligation, our responsibility, as communities and as people who need to take the power back in our own hands.

How should we talk about? Is our vocabulary the right way? Do we need to change the way we talk about this? You know, I'm a little fed up with the bias and fairness as a more watered-down version of what we're talking about, of systems of oppression, institutionalized forms of racism, and sexism, that the bias talk both individualizes, and it makes it seem like it's a level playing field, like we all have bias, when in fact, power is being monopolized and exercised

in very patterned ways, and we don't all do it to the same extent. Well, we've run out of time. Yes, quickly. Oh, I agree with that. I think this focus on bias and fairness misses the point that these are complex policy decisions that we're trying to make, and they can't be distilled down to these simple binaries. I'm glad I allowed you to say that. It sounds like a very interesting point.

I want to thank both of you for taking time to be with us today. Sherrod Goyle is Professor of Management Science and Engineering at Stanford, and Ruha Benjamin, Professor of African American Studies at Princeton. Again, thank you both for taking time to be with us today. Thank you for having me. You're welcome. Before we go, Science Friday is headed to North Carolina next week on Wednesday, September 18th for an evening highlighting science in the Tar Heel State,

We partner with WUNC to screen three new films shot and produced by Science Friday in North Carolina, followed by a chance to ask the featured scientists more about their work. So don't miss out. Go to ScienceFriiday.com slash Carolina Films to get your tickets at ScienceFriety.com slash Carolina Films. Charles Berkwist is our director, senior producer, Christopher and Taliatta. Our producers are Alexa Lim, Christy Taylor, and Katie Feather. and today we had help from Donya Obdel Hummeid. Sorry, Daniel.

Our intern is Camille Peterson, and she produced that fantastic segment you just heard about AI and the criminal justice system, and we bid her a fun farewell this week. And I know any place she goes, they'll be happy and lucky to have her there. We had technical engineering help today from Rich Kim and Kevin Wolf.

BJ Leatherman composed our theme music. You want to hear your voice on Science Friday, download the Science Friday Vox Pop app this week we're asking you whose responsibility is it to ensure clothing is sustainable clothing brands textile manufacturers or the consumers i'm ira flato in new york