Short Wave - Are You Overestimating The Algorithm?

Episode Date: September 3, 2024

Humans hallucinate. Algorithms lie. At least, that's one difference that Joy Buolamwini and Kyle Chayka want to make clear. When ChatGPT tells you that a book exists when it doesn't – or professes its undying love – that's often called a "hallucination." Buolamwini, a computer scientist, prefers to call it "spicy autocomplete." But not all algorithmic errors are as innocuous. So today's show, we get into: How do algorithms work? What are their impacts? And how can we speak up about changing them? This is a shortened version of Joy and Kyle's live interview, moderated by Regina G. Barber, at this year's Library of Congress National Book Festival.If you liked this episode, check out our other episodes on facial recognition in Gaza, why AI is not a silver bullet and tech companies limiting police use of facial recognition.Interested in hearing more technology stories? Email us at [e-mail] — we'd love to consider your idea for a future episode!See pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences.NPR Privacy Policy

You're listening to Shortwave from NPR. Hey, Shortwaivers, Regina Barber here, and we have a special episode for you today. One recorded at the National Book Festival, an annual event hosted by the Library of Congress. Hey, how's everyone doing? There I had the honor of moderating a panel on algorithms, specifically how computer algorithms are everywhere. We experience a lot of mediation in culture now when we, watch a TV show through Netflix or listen to a song through Spotify. Like all of this digital technology has become a kind of filter.

That's Kyle Cheka, a staff writer at the New Yorker and the author of the book, Filter World, how algorithms flatten culture. And okay, all of that sounds fancy and new. Algorithm is this kind of buzzy word people like to toss around because it shows how it integrates with technology. But Kyle says that algorithms, they can do things like find new songs on Spotify, yes, but at the end of the day, they're very basic. They've been around since the Babylonians, who made equations to calculate all sorts of things,

like the volume of dirt dug up to make a ditch. Or how to split up, you know, a crop or share things between people. And so these were just basic equations, like step-by-step mathematical processes, to find a result. Dr. Joy Bolam-weeney agrees with Kyle on the basic nature of algorithms.

She's a computer scientist and author of the book, Unmasking AI, My Mission to Protect What is Human, in a world of machines. And she joined Kyle and me on stage to talk about all things algorithms. To me, an algorithm is just a sequence of steps to achieve an outcome.

So for me, I loved video games. I used to love playing Tony Hawk's pro skater too. Mine is like Donkey Kong country. That was a beautiful game in the 90s. We could just do Tony Hawk and Donkey Kong for another half hour. Right. You just talk about those two games were really, really good.

Anyway, can't be right. So you want a character to move. left, you have this input, this action happens. And so that gets you that early introduction to what is our language to communicate with computers so that we can program them or shape them to help us express ourselves, solve problems, and so forth. So algorithms can be fun, obviously, but Dr. Joy's book dives into the much more serious and potentially dangerous side algorithms used in artificial intelligence, like how facial recognition is far from perfect. The work that I do a lot of people know is around facial recognition technology.

So the callback you never get because your resume is screened out by a hiring system. Many of the algorithms that are shaping our lives are the ones you don't see. Or how it can amplify racial bias. Like how it has trouble even detecting people with darker skin. This software has led to false identification and even false arrests of black people. So you had people like Robert Williams falsely arrested in front of his two daughters and his wife for supposedly stealing a watch. So today on the show, the algorithms behind artificial intelligence. At their best, they can expand our worldview.

But what happens when they're at their worst? You're listening to Shortwave, the science podcast from NPR. Let's get right back into the conversation at the National Book Festival, talking about algorithms and their connection to AI. So how do we go from software, like solving problems, calculations like Kyle talked about, to software that, like, thinks for itself? Thanks for itself is interesting. Yeah. Well, we have it in quotes here because my boss was like, put it in quotes. Thanks for itself.

So when I think about AI, I approach it as this ongoing question. to give machines abilities we associate with intelligence or humans, right? So whether it's recognizing a face, a puppy, or making a prediction, are you going to pay back that loan? Are you going to be a good tenant? Are you going to be a good student? These are the types of applications we're seeing with AI. computer scientists like me, we quickly realize you're not going to be able to put out all of the instructions.

It's messy, it's complicated, they're always exceptions, right? And maybe we can imitate the human brain. Maybe we can create pattern recognition systems. So instead of having to code everything line by line, maybe instead we can present a data set of examples. And so now you have these pattern recognition algorithms that are trained on a data set, and then it means the quality of that data set is going to determine the quality of your outputs. Right. Kyle, you start your book Filter World with this machine built in 1769, and it seems to think for itself, right?

It seems to be AI. Yeah, I think we overestimate that idea of artificial intelligence or of the algorithm, That device that opens the book is called The Mechanical Turk. It was a machine that appeared to play chess. It was like a wooden cabinet with this automaton guy in a turban on top of it. And it appeared to move its arm and smoke a pipe and play chess. And it competed against really famous people. Like, I think it beat Ben Franklin in a game of chess.

But the problem was that inside this cabinet was not just a series of gears and levers. but a small man that was maneuvering magnet-controlled pieces under the board. And he had a little, like, lamp in there, and there was, like, a hole in the back for the stuff to go out. And people were, like, why is there smoke coming out of this thing? It was from, like, the lamp. Yeah. So it was this illusion of an artificial intelligence, of a machine that could think for itself,

which, in reality, was a little guy in a box. and I think that kind of became my metaphor for talking about algorithmic recommendations and some of machine learning, because we see it as this automated process, but it's really designed and updated and changed and driven by human engineers who are working at Silicon Valley technology companies. And I really wanted to get back to the source of these decisions that are made about how these things work, because they're not, I mean, in the context of recommendations, Spotify's main goal is not to give you the best music ever. Netflix is, that might be a surprise.

But I wanted to point out that, like, there are human incentives behind how this technology works, and usually they're motivated toward profit, which online now is mostly in the form of advertising. So we can acknowledge the illusion that's happening, while also understanding it can have really real consequences. That's why when you even said thinking, I was really cautious about that

because what we believe about AI systems can be even more powerful than what they actually are. Yeah, and Dr. Joy, I want to get into kind of like unmasking a lot of the algorithms and what's happening into our software. Your work is focusing on this like facial recognition but you kind of go even further that it's not just like they can't see you, but they're also, you have this misgendering whole section

and you also have like people aren't seen by self-driving cars. They're being wrongfully arrested. Like, can you kind of go into those a little bit? Okay, the doom section. The doom section. We're ending on hope. We're ending on hope. I have a list.

I remember when I was working on my master's at the time, I came up with this notion of FML, you all know FML, failed machine learning, failing, freedom, money, love. And so on the freedom side, thinking about AI systems that were being used by law enforcement, right? So these algorithms of exploitation. Now when it comes to love, I was procrastinating my PhD and I went on this social, kind of this dating app. and it required facial recognition to get there. So I upload one photo. It's an artistic photo profile, whatever, and it didn't detect my face.

So I was like, okay, it's because of the type of photo. So then I did another photo looking straight at the camera, right? Still didn't detect my face. So once I realized I was messing with people's love lies, that's when I went straight back. People were going to listen. I was like, okay, we really have something here. but even our connections and our communities who's following you.

Yeah. Your personal relationships are so mediated by these things now, too. I mean, every dating app is driven by an algorithm that sorts people into matching with each other. Like, whose Instagram stories you see most often are a choice that Instagram makes for you based on what you engage with. I mean, which emails you see, who's seeing your posts on Facebook, it's also filtered. that it's not necessarily what you organically want, but it's like this weird technological mirror reflection

of what the platform wants to show you and how the platform interprets your behavior. This is something I learned when I was doing this reporting on AI, hallucinations. And I really, really like this, and I kind of want to go through this before we kind of get to hope. Kyle, do you want to talk about an AI hallucination?

Yeah, I think, I mean, a hallucination is essentially an error of the artificial intelligence. It's when an AI comes up with an answer to something or generates a result that is just completely wrong and irrelevant. In experimenting with chat GPT myself, I was struggling with an article. So I was like, chat GPT, can you find me an example of Russian folklore that has to do with an automaton or some kind of robot? Like, surely there's something. And ChatGPT kept giving me stories and fairy tales and examples of. of these Russian folk tales, but then every time I tried to chase down the source of them,

it turned out they did not exist. And it would give me fake URLs, fake PDFs, like fake book titles. It would say, no, no, no, this is absolutely real. Just go look at this book. And the book was also not real. But that just showed me how, or not manipulative, but just blatantly false these results could be. And even just thinking about hallucinations, it's, fancy way of saying BS.

Yeah. Right? It elevates it to another place, which is good for marketing. Humans hallucinate. Yeah. Right. So it's like, oh, okay. Just lie.

Right. So I push back even on hallucin. It's making things up that are not factual. I think hallucination as a way of framing it doesn't really give the gravity, right? If you're reaching out to try to find information about benefits or health care and you're getting the wrong information, it's not like, oh, it's hallucinating, having this mystical experience. And yet it can be so persuasive in its hallucinations. It's like, it'll tell you exactly where to go. Right. Dr. Joy, you were talking about how after your master's thesis and

like the, you know, the documentary that you're in and all the stuff. Coded bias. Coded bias. You actually put pressure on like big companies like Microsoft, Amazon, and they claim to get better and change things. Well, they actually stopped selling facial recognition to law enforcement, which I thought was huge. Yeah. And part of that experience for me was showing the importance of sharing our stories.

If you have a face, you have a place in the conversation about AI. You don't need all the degrees from MIT or wherever. And so that's why I'm excited about what's to come because when we address our limitations, that's how we actually reach the aspirations of what we have for AI. I want to thank you, Kyle, Dr. Joy, for talking to me, and we're going to take some questions after, but let's give them a round of applause.

This episode, This episode was produced by Hannah Chin, edited by our showrunner Rebecca Ramirez, Hannah, Rebecca, and I check the facts. The audio engineer was Quacey Lee. Beth Donovan is our senior director and Colin Campbell is our senior vice president of podcasting strategy. I'm Regina Barber. Thank you for listening to Shortwave, the science podcast from NPR.