Science Friday - SciFri Extra: A Pragmatic Wishlist For AI Ethics

Episode Date: June 24, 2020

Earlier this month, three major tech companies publicly distanced themselves from the facial recognition tools used by police: IBM said they would stop all such research, while Amazon and Microsoft said they would push pause on any plans to give facial recognition technology to domestic law enforcement. And just this week, the city of Boston banned facial surveillance technology entirely. Why? Facial recognition algorithms built by companies like Amazon have been found to misidentify people of color, especially women of color, at higher rates—meaning when police use facial recognition to identify suspects who are not white, they are more likely to arrest the wrong person.  CEOs are calling for national laws to govern this technology, or programming solutions to remove the racial biases and other inequities from their code. But there are others who want to ban it entirely—and completely re-envision how AI is developed and used in communities. In this SciFri Extra, we continue a conversation between producer Christie Taylor, Deborah Raji from NYU’s AI Now Institute, and Princeton University’s Ruha Benjamin about how to pragmatically move forward to build artificial intelligence technology that takes racial justice into account—whether you’re an AI researcher, a tech company, or a policymaker. Subscribe to this podcast. Plus, to stay updated on all things science, sign up for Science Friday's newsletters.

Hey there, Ira here. If you're a regular listener to Science Friday, you know we've been having conversations about the ethics of new artificial intelligence technology. Just recently, sci-fright producer Christy Taylor spoke with two researchers, Deborah Raji of New York University's AI Now Institute and Princeton University's Dr. Ruha Benjamin about how AI can further racial injustices. Our original radio interview covered the recent moves to limit police access to facial recognition by companies like Amazon and Microsoft, at least temporarily. They talked about where facial recognition is being used and why it can disproportionately harm people of color simply because it's worse at identifying people with darker skin. They also talked about why

even when AI works correctly, it's worth asking if we should use it at all. That conversation should already have come through your local podcast feed. But wait, there's more. Deborah and Ruha also talked about what they thought we should do instead, practical steps for AI researchers, tech companies, and lawmakers. Since we didn't have time to air all of that, we're sharing that part of the conversation now. Have a listen.

So what is an AI researcher to do that is an ethical way to look at creating new things and exploring new technologies? Deb, I'll start with you. My answer is a little bit lame, but the way that I think of, so by the way, this is a conversation that is currently happening in the AI community, right? There's been a lot of recent revelations around the ethics of the work, you know, really reflecting on the impact of some of the things that have come out of the AI community, including facial recognition, but other, you know, other products, other projects that we now

understand to be harmful. Rua mentioned sort of this idea of risk assessment. definitely being one of them. And I think that after reflecting on some of these impacts, there's been a push for further ethical reflection as part of our design process as AI researchers. When you're working on a project or when you're making the decision, like I alluded to earlier, to invest time and effort into a particular research direction, reflecting on some of the consequences, the societal consequences that can exist. And there's a debate happening to what degree the researcher can be involved in that. I personally feel like that debate is interesting and important,

but there's also this other debate, which is reflective of my own experience, being on the engineering side, being on the side of the person building some of this technology. It's really like a free-for-all, as in the AI systems that affect humans today, like that are products that are deployed today, are sort of built in this like completely haphazard, unstructured way. We don't necessarily have like a consistent sort of framework that we all use to think about things like safety, to think about things like even societal impact to reflect on these things, but also to record even the most basic information, like where does your data come from? What are the demographics of your data set? That is like an unknown for so many products that are already

out there, like AI models that are already affecting people's lives today. You know, in situations like immigration, in situations like social welfare, you know, the algorithms, guiding those very important decisions are so poorly documented and so poorly communicated amongst the engineers at a company, externally, even to different public entities that these vendors might be working with. So for me, I see one of the really big issues just being like a lack of accountability and a lack of documentation structure or a lack of even sort of due process that happens in machine learning engineering, like in the stage of the, you know, on the side of the people building the technology, we don't really see a lot of accountability and we don't really

see a lot of proper, you know, established processes of accountability, such as documentation. So that's something that I really see as important. And I think if someone wants to work on a project or if someone wants to build something as part of a company and they understand that it's a AI system that is going to be deployed and it's going to affect people, there needs to be very thoughtful consideration to, you know, recording or at least capturing some of these decisions that are just haphazardly made at the moment. So that for me is a really huge part of this. And how is it possible that, you know, these huge companies, IBM, Microsoft, Face Plus, Plus, Amazon even. How's it possible that they hadn't evaluated it on darker skin faces? Like, who missed that detail?

So I think, I think a lot of exposing sort of these gaps ends up becoming part of the due process or the due diligence that is missing currently in the development process of these systems. So that's something that I'm hoping will get better, or at least more, you know, concrete, more standardized. Yeah, right now we don't, we don't have anything sort of concrete or standardized to point to. Ruha, what's your advice for people who want to develop more ethical AI? I would say, you know, if you're an AI researcher, in some ways you have to go against perhaps your own desire to be a cowboy or a cowgirl and to not have anyone tell you what to do and to try

anything you want and in fact what we're calling for and imagine how powerful it would be for someone who is in the profession people who are in the profession who have potentially the most to lose on one level to be most outspoken about the strong need for public accountability for transparency for a governance structure that exceeds the bounds of any one company and so in some ways you have to call for an ecosystem that would allow us to create technology in the public good. Because right now we don't have that. And part of that ecosystem is to be in partnership

and conversation with organizations that like some of the ones we've talked about, whether it's data for black lives, whether it's a number of digital justice organizations, ACLU and others that have at the forefront of what they're trying to do is to advance justice and equity in our communities rather than simply try to increase the bottom line. And so in some ways it's going against, against the drive to just, you know, be unregulated and be on that frontier and have the next

great thing that's going to sell is to pull back away from that and to realize that's what has gotten to us in this problem in the first place. And so we need to think about what public interest technology will look like and what kind of structures will get us there. You can't, as an AI researcher, trust your own desire to do good as the test that will say, okay, well, I want to do good and therefore what I do will be good. Because the do-gooding ethos is the Kool-Aid is the Kool-Aid of Silicon Valley. And so what we have to do is basically spit up that Kool-Aid and understand that your personal intention to do good has nothing to do with this conversation. What we need is an ecosystem and a structure that will ensure that what you produce will not harm communities,

will be in the best interest of communities, whether or not you want it personally or not. I think that's an excellent point, and it completely relates to what we're seeing with Amazon, IBM, and Microsoft last week. There's been a certain amount of critique coming from the tech community to say, like, oh, you know, did they do enough?

Could they do more? And I'm kind of like, I don't expect anything from them. You know, self-regulation is such a... I will never sort of pin my hopes on these companies acting in a way against their own profit and against their own self-interest to the extent that we need them to for the protection of the people that we're advocating for that we're fighting for. I think it's very difficult for some of these big tech companies to make a move so counter to the benefit of their users. And in this case, for the sake of Amazon, like as an example, their clients, their user is the police department,

not the people that are affected by the technologies used by that department. So it's so easy for them to completely ignore those affected populations. and focus only on that police department. So, you know, really when it comes down to it, it's up to a lot of policymakers, but also, you know, advocates and other community voices, other people that really are representative of the people are meant to be representative of the people

to advocate for that perspective and critique from the outside. I think it's still a good thing for the companies to make that stance. It's a significant thing. It shifts public opinion. It informs policymaker perception of what's possible, but it comes down to sort of Congress and also, you know, some of these advocacy groups really playing that critical role in terms of, you know, shutting this technology down. It's not a decision the companies will arrive at on their own at all.

Tell me more about government regulation. I'm hearing you both, Deb and Ruhas, say that bans and moratoriums are something you would want. But what else should the government be considering when looking at regulating artificial intelligence like facial recognition? So I think the first, like the first sort of point. is around disclosure, right? We don't necessarily understand today where that technology is being used by the government, you know, even the government, but also even more broadly. So even today, we're still fighting, you know, to understand who's using what the ACLU recently sued, Department of Homeland Security and other intelligence agencies to try to understand if they were

using Amazon recognition or Microsoft's product or whoever's sort of facial recognition product. You know, I'm personally sort of curious, I've done a lot of personal digging into the immigration process, and it's been so difficult to understand which vendor that they're using, and the procurement process is a complete black box. So it's too difficult today to understand and to really pick out where facial recognition is being used in multiple processes and multiple government processes. So I think that disclosure is an ideal. Just understand where it's being used is one objective. And then the other one is sort of restriction. And that's where the idea of the moratorium and the bans come in because like I mentioned, you know, facial recognition is this sort of inherently,

I personally call it this like toxic technology or like, you know, you have such an important piece of information to have about an individual, but to have that about millions of individuals, you know, controlled by a central authority figure, a figure that could change over time that might be hard to trust that could easily weaponize that technology, that for me is very scary. So the idea of restriction and the idea of moving towards a ban or a moratorium, one of the reasons why I like a moratorium is that facial recognition actually has so many concerns associated with it. Sometimes it actually requires this very nuanced conversation around, like, what does it mean to, you know, have tools of safety? So some people see facial recognition

as an integral part of security, as an integral part of sort of the safety of a community, We need to be able to track people. Surveillance is tied to concepts of safety. And like untangling that is such a nuanced conversation that I'm glad that there's proposals for a moratorium where we say, hey, we know that facial recognition is this potentially dangerous technology. And because we know that we're not going to sell it

or it's not going to be used while we have this more nuanced conversation, working towards a ban, working towards like ultimate restriction. So I think the concept of a moratorium policy-wise is super appealing to me where it's like we already know that there's a lot of of issues around this technology, why are we still using it? We should at least at minimum, you know, pause its use while we're having this more nuanced conversation around, you know, whether it's appropriate to be used in other ways. And then also the definitions of safety that it is compatible with. Those are sort of two things that make sense to me. And then the sort

of third thing, policy sort of checkpoint that I think about is so the National Institute of standards in the U.S. is really in control of a lot of the facial recognition assessment. They set up a lot of the benchmarks that are used to sort of evaluate the technology that the government considers and really beginning to criticize and understand, critique that assessment itself. And this begins with something like gender shades. You know, this year, 2019, last year, I guess, was sort of the first time that NIST was able to sort of assess the facial recognition models that it usually evaluates for performance on different demographic groups. And they found out that there was up to 100 times more errors on darker skin faces as we've

discovered, but also Asian American faces and other minority populations. And that was because they actually evaluated for that this time. They hadn't done that in the past. So thinking about assessment, like what does it mean for this technology to actually work, revisiting that? And, you know, including in that conversation, you know, required reporting on, you know, privacy measures, required reporting on use and the potential for misuse and how easily manipulatable these technologies are. These are things that I would like to see as part of the policy conversation. So I'm not sure if that was comprehensive enough, but disclosure, all of those things. Yeah. Yeah. And to that, I would add, we need to be thinking about what we do when harms are done.

And so when you think about, for example, in Michigan a few years ago, the state of Michigan adopted an automated decision system called Midas that was flagging people for unemployment fraud and falsely flagged 95% of the people that they flagged over 40,000 residents were falsely flagged and had, you know, all kinds of issues as a result, losing their homes, you know, having a, you know, a record, all kinds of things that created harms in their lives. people committing suicide. So what is the responsibility of those who adopt and implement a system like that? And I'm thinking about reparations of many different forms. How do you repair the damage done? And who is responsible? So we need things in place to actually hold accountable on the back end. And then for me, you know, as a teacher, as an educator, I think we need to think seriously about the training of people who are going into these fields. Like would we allow people graduating medical school, not to have a basic level of understanding of X, Y, and Z before they're considered able to be a practitioner. And there's a lot of movement and advocacy around students there in

terms of the lack of education around things like systemic racism. But I think it's equally as important in STEM fields, in computer science, people going into creating machine learning algorithms and AI that impact people's lives all over the place. And yet, they don't have a basic kind of licensing or knowledge or understanding about the history of racism and technology. And so we need to think even broadly beyond policies governing just the technology, governing the whole ecosystem to actually prevent certain harms from happening in the first place. So I'd love us to think about all aspects of it, not just the nitty-gritty of the development of a single technology. Yeah. And I just want to add a quick comment to that last

point around training and, like, affecting the whole ecosystem, I really do to call back to sort of an earlier point I had made around the lack of documentation and structure that we see in the actual engineering sort of department of when we try to implement these systems in the real world. I think a lot of that is due to training. Like, we're not to, like, articulate what the ethical concerns are, but also reflect on how our design decisions influence sort of the impact of the system overall. These are all things that everyone learns after their first disaster situation rather than before. So it would be amazing to have that integrated as part of our learning experience.

That was sci-fi producer Christy Taylor interviewing Dr. Ruha Benjamin, Professor of African American Studies at Princeton University. And Deborah Raji, a technology fellow with the AI Now Institute at New York University. And for more about the racial inequities in facial recognition algorithms, and how they are used, plus how this is playing out in the city of Detroit. We have an article up on our website, ScienceFriety.com slash community AI. I think you'll find it very interesting.