Unexplainable - The Black Box: Even AI’s creators don’t understand it

Episode Date: July 12, 2023

AI has the potential to impact our society in dramatic ways, but researchers can’t explain precisely how it works or how it might evolve. Will they ever understand it? This is the first episode of our new two-part series, The Black Box. For more, go to http://vox.com/unexplainable It’s a great place to view show transcripts and read more about the topics on our show. Also, email us! [e-mail] We read every email. Support Unexplainable by making a financial contribution to Vox! bit.ly/givepodcasts Learn more about your ad choices. Visit podcastchoices.com/adchoices

Square knows that in hospitality, efficiency is everything. That's why their system lets you take payments. Track sales, handle inventory, manage staff, send invoices, and keep up with finances all in one place. Apply through orders with zero mistakes. Get the data you need and keep everything working together. So you're ready for whatever's next. Learn more about their customizable plans at squareup.com. Visit BetMDM casino and check out the newest exclusive.

The Price is Right Fortune Pick. BetMGM and GameSense remind you to play responsibly. 19 plus to wager. Ontario only. Please play responsibly. If you have questions or concerns about your gambling or someone close to you, please contact connects Ontario at 1-866-531-2600 to speak to an advisor. Free of charge.

BetMGM operates pursuant to an operating agreement with Eye Gaming, Ontario. Over the last few months, I've gotten sort of obsessed with AI. I got especially interested in generative AI, which is the type of AI that's allowed people to make things like the fake picture of Pope Francis with the white puffer jacket or the hit song that featured fake Drake. I got so into it that I tried to see if I could train an AI on my own voice. And it kind of worked. It's not perfect, but I'm actually not reading this line. I just typed it into a program, and I haven't been reading anything this whole time.

Okay, back to real me. These tools have all been fascinating, but the one I really couldn't stop thinking about was chat GPT, the chatbot released by OpenAI late last year. And it's because of the surprisingly wide range of things I saw this one chatbot doing, like writing the story of Goldilocks, as if it was from the King James Bible. And it came to pass in those days that a certain young damsel named Goldilocks did wander into the dwelling of three bears. I saw it passing tons of standardized tests, being used for scientific research, even building full websites based on a few sketched-out notes. I'm just going to take a photo and here we go, going from hand-drawn to working website.

But I also saw some less fun things, like chatbots disrupting entire industries, playing a major role in the Hollywood Writers Strike. The union is seeking a limit on the use of AI, like chat GPT, to generate scripts in seconds. They've been used to create fake news stories. They've been shown to walk people through how to make chemical weapons, and they're even getting more AI experts worried about larger threats to humanity. The main thing I'm talking about is these things becoming super intelligent and taking over control. All of this started feeling like a long way from a fun biblical Goldilocks story. So I wanted to understand how a chatbot could do all these things.

I started calling up researchers, professors, reporters. I was annoying my friends and family by bringing. bringing it up in basically every conversation. And then I came across this paper by an AI researcher named Sam Bowman. And it was basically a list of eight things scientists know about AIs like ChatGPT. I was like, great. Easy way to get a refresher on the basics here. So I started reading down the list, and it was pretty much what I expected.

Lots of stuff about how these kind of AIs get better over time. But then things started to get kind of weird. Number four, we can't reliably steer the behavior of AIs like ChatGPT. Number five, we can't interpret the inner workings of AIs like ChatGPT. And I was like, you're telling me this thing that's being used by over 100 million people that might change how we think about education or computer programming or tons of jobs. We don't know how it works. So I called up Sam, the author of the paper.

and he was just like, yeah, we just don't understand what's going on here. And it's not like Sam hasn't been trying his best to figure this out. I've built these models, I've studied these models. We built it, we trained it, but we don't know what it's doing. Ever since I talked with Sam, I've been stuck on this core unknown. What does it mean for a tech like this to suddenly be everywhere? If we don't know how it works, we can't really say whether we're going to end up with scientific leaps, catastrophic risks, or something we haven't even thought of yet.

The story here really is about the unknowns. We've got something that's not really meaningfully regulated, that is more or less useful for a huge range of valuable tasks, but can sort of just go off the rails in a wide variety of ways we don't understand yet. And it's sort of a scary thing to be building unless you really understand how it works. And we don't really understand how these things work. I'm Noah. I'm Hassanfeld, and this is the first.

first episode of a two-part unexplainable series we're calling The Black Box. It's all about the whole at the center of modern artificial intelligence. How is it possible that something this potentially transformative, something we built is this unknown? And are we ever going to be able to understand it? Thinking intelligent thoughts is a mysterious activity. The future of the computer is just hard to imagine. I just have to admit, I don't really know. So how do you think I'd feel? Activity, intelligence. Can the computer think?

So how did we get to this place where we've got these super powerful programs that scientists are still struggling to understand? It started with a pretty intriguing question, dating back to when the first computers were invented. The whole idea of AI was that maybe intelligence, this thing that we used to think was uniquely human, could be built on a computer. Kelsey Piper, AI reporter, Vox. It was deeply unclear how to build superintelligent systems, but as soon as you had computing, you had leading figures in computing say, this is big, and this has the potential to change everything.

In the 50s, computers could already solve complex math problems, and researchers thought this ability could eventually be scaled up. So they started working on new programs that could do more complicated things, like playing chess. Chess has come to represent the complexity and intelligence of the human mind, the ability to think. Over time, as computers got more powerful, these simple programs started getting more capable. And by the time the 90s rolled around, IBM had built a chess playing program that started to actually win against some good players. They called it deep blue, and it was pretty different from the unexplainable kinds of AIs we're dealing with today.

Here's how it worked. IBM programmed Deep Blue with all sorts of chess moves and board states. That's basically all the possible configurations of pieces on the board. So you'd start with all the pawns in a line, with the other pieces behind them. Pawn E2 to E4. Then with every move, you'd get a new board state. Knight G8 to G6. And with every new board state, there would be different potential moves deep blue could make.

Bishop F1 to C4. IBM programmed all these possible moves into Deep Blue, and then they got hundreds of chess grandmasters to help them rank how good a particular move would be. They used rules that were defined by chess masters and by computer scientists to tell Deep Blue this board state, is it a good board state or a bad board state? And Deep Blue would run the evaluations

in order to evaluate whether the board state it had found was any good. Deep Blue could evaluate, could evaluate 200 million moves per second, and then it would just select the one IBM had rated the highest. There were some other complicated things going on here, but it was still pretty basic. Deep Blue had a better memory than we do, and it did incredibly complicated calculations, but it was essentially just reflecting humans' knowledge of chess back at us. It wasn't really generating anything new or being creative. And to a lot of people, including Gary Kasparov, the chess world champion at the time, this kind of chess bot wasn't that impressive, especially because

it was so robotic. They tried to use only computers' advantages, calculation, evaluation, et cetera. But I still am not sure that the computer will beat world champion, because World Champion is absolutely the best, and his greatest abilities to find a new way in chess. And it will be something you can't explain the computer. Kasparov played the first model of Deep Blue in 1996, and he won. But a year later, against an updated model, the rematch didn't go nearly as well. Are we missing something on the chessboard now that Kasparov sees?

He looks disgusted, in fact. He looks just... Kasparov leaned his head into his hand, and he just started staring blankly off into space. And whoa! Deep Blue Kasparov has resigned. He got up, gave this sort of shrug to the audience, and he just walked off the stage. You know, I proved to be vulnerable. You know, when I see something that is well beyond my understanding, I'm scared.

And that was something well beyond my understanding. Deep Blue may have mystified Kasparov, but Kelsey says that computer scientists knew exactly what was going on here. It was complicated, but it was written in by a human. You can look at the evaluation function, which is made up of parts that you. humans wrote and learn why Deep Blue thought that board state was good. It was so predictable that people weren't sure whether this should even count as artificial intelligence. People were kind of like, okay, that's not intelligence. Intelligence should require more than just, I will look at hundreds of thousands of board positions and check

which one gets the highest rating against a pre-written rule and then do the one that gets the highest rating. But Deep Blue wasn't the only way to design a powerful AI. A bunch of other groups were working on more sophisticated tech, an AI that didn't need to be told which moves to make in advance, one that could find solutions for itself. And then in 2015, almost 20 years after Kasparov's dramatic loss, Google's deep mind built an AI called AlphaGo, designed for what many people call the hardest board game ever made. Go. Go had remained unsolved by AI systems for a long time after chess had been. If you've never played Go, it's a board game where players place black and white tiles on a 19 by 19 grid to capture territory. And it's way more complicated than chess.

Go has way more possible board states so the approach with chess would not really work. You couldn't hard code in as many rules about in this situation do this. Instead, AlphaGo was designed to essentially learn over time. It's sort of modeled after the human brain. Here's a way too simple way to describe something as absurdly complicated as the brain. But hopefully it can work for our purposes here. A brain is made up of billions and billions of neurons. And a single neuron is kind of like a switch.

It can turn on or off. When it turns on, it can turn on the neurons it's connected to. And the more the neurons turn on over time, the more these connections get strength. Which is basically how scientists think the brain might learn. Like probably in my brain, neurons that are associated with my house, you know, are probably also strongly associated with my kids and other things in my house because I have a lot of connections among those things. Scientists don't really understand how all of this adds up to learning in the brain. They just think it has something to do with all of these neural connections. But AlphaGo followed this model, and researchers created what they called an artificial neural network.

because instead of real neurons, it had artificial ones, things that can turn on or off. All you'd have is numbers. At this spot, we have a yes or a no, and here is like how strongly connected they are. And with that structure in place, researchers started training it. They had AlphaGo play millions of simulated games against itself, and over time, it strengthened or weakened the connections between its artificial neurons. It tries something, and it learns, did that go well? that go badly, and it adjusts the procedure it uses to choose its next action based on that.

It's basically trial and error. You can imagine a toy car trying to get from point A to point B on a table. If we hard-coded in the route, we'd basically be telling it exactly how to get there. But if we used an artificial neural network, it would be like placing that car in the center of the table and letting it try out all sorts of directions randomly. Every time it falls off the table, it would eliminate that path. it wouldn't use it again. And slowly, over time, the car would find a route that works.

So you're not just teaching it what we would do. You are teaching it how to tell if a thing it did was good. And then based on that, it develops its own capabilities. This process essentially allowed AlphaGo to teach itself which moves worked and which moves didn't. But because AlphaGo was trained like this, researchers couldn't tell which. specific features it was picking up on when it made any individual decision. Unlike with Deep Blue, they couldn't fully explain any move on a basic level. Still, this method worked. It allowed AlphaGo to get really good. And when it was ready, Google set up a five-game match between AlphaGo and

World Champion Lisa Dole, and they put up a million dollar prize. Hello and welcome to the Google Deep Mind Challenge match live from the four seasons in Seoul, Korea. AlphaGo took the first game, which totally surprised Lee. So in the next game, he played a lot more carefully. But game two is when things started to get really strange. That's a very, that's a very surprising move. I thought it was, I thought it was a mistake. On the 37th move of the game, AlphaGo shocked everyone watching, even other expert Go players. When I see this move, for me, it's just a big shock. What?

Normally human, we never play this one. Because it's bad. It's just bad. Move 37 was super risky. People didn't really understand what was going on. But this move was a turning point. Pretty soon, AlphaGo started taking control of the board. And the audience sensed a shift.

The more, I see this move. I feel something changed. Maybe for human we think it's bad. But for AlfaGo, why not? Eventually, Lee accepted that there was nothing he could do, and he resigned. AlphaGo scores another win in a dramatic and exciting game that I'm sure people are going to be analyzing and discussing for a long time. AlphaGo ended up winning four out of five matches against the world champion.

But no one really understood how. And that, I think, sent a shock through a lot of people who hadn't been thinking very hard about AI and what it was capable of, it was a much larger leap. Move 37 didn't just change the course of a go game. It represented a seismic shift in the development of AI. With Deep Blue, scientists had understood every move. They programmed it in. But AlphaGo represented something different.

Researchers didn't really know how it worked. They didn't hard code AlphaGo's rules, so they weren't always sure why it made the moves it did. But those decisions tended to work. Even the weird ones. AlphaGo had demonstrated that an AI scientist don't fully understand might actually be more powerful than one they can explain. AlphaGo was a really impressive achievement at the time.

Nobody had expected that you could get that far that fast. So it drove a lot of people who hadn't been thinking about AI to start thinking about AI. And that meant there was also more attention to slightly different approaches. Teams working on systems like this started getting more confidence, more funding, more computer power. All kinds of AI started popping up, like better image recognition, augmented reality, and then more recently writing with AIs like ChatGPT. But ChatGPT isn't just a writing tool. It's a broader, weirder AI than anything that's come before.

And it's an AI that's getting even harder to understand. That's next. It's all about you. And when you fly with Virgin Atlantic in their upper class cabin, they take the VIP treatment to the next level. With a private wing to check in in your own security channel at London Heathrow, you can glide from your car to their clubhouse, a destination in its own right in 10 minutes or less. On board, you can treat yourself to your own private suite to stretch out in,

with lots of storage space, a lie flat bed, and delicious dining from beginning to end. Just be sure to leave room for dessert. Their mile high tea with all the little cakes and sandwiches is a showstopper. Go to virgin atlantic.com to learn more. Rosen lasagna, medium power, 15 minutes. Sounds like Ojo time. Let's play.

Feel the fun with Play-Ojo. The online casino with all the latest slot and live casino games. What you win is yours to keep with no wagering requirements, instant payouts, and no minimum withdraws. Hey, I just won. Woo-hoo! Feel the fun. Play Ojo. Forget about the lasagna. Let's celebrate.

19 plus Ontario only. Please play responsibly. Concern about your gambling or that of someone close to you. Call 16-531-2600 or visit connexonterio.ca. Players, Sam says go. I think of the last 30 years of AI development as having three major turning points. The first one was Deep Blue, an AI that could play something complicated like chess better than even the best human. It was powerful, but it was fully understandable.

The second one was AlphaGo, an AI that could play something way more complicated than chess. But this time, scientists didn't tell it the right way to play. AlphaGo was trained to play Go by trial and error, which means that some of its best moves didn't make sense to the scientists who built it, but they worked. And finally, the third major turning point is what's happening right now with ChatGPT, the chatbot built by OpenAI, an AI an AI research company. ChatGPT is a more exploratory project than AlphaGo. It wasn't designed to win at any kind of game. OpenAI was just trying to see if a system that forms its own connections over time could generate convincing language. The main way that systems like Chachapit are trained

is by doing basically autocomplete. This is Sam Bowman again. He's a researcher at an AI company called Anthropic. He's a professor at NYU. And he wrote the paper I mentioned at the top of the episode about AI unknowns. We'll feed these systems sort of long text from the web. We'll just have them read through a Wikipedia article, word by word. And after it's seen each word, we're going to ask it to guess what word is going to come next. But OpenAI added something else on top of this auto-complete tool. They had workers, often underpaid workers outside of the U.S., spend tons of hours labeling potentially toxic material. And they also had workers say whether they liked full responses, not just individual words anymore.

You might tell the model, all right, make this entire... response more likely because the user liked it and make this entire response less likely because the user didn't like it. So if they got a coherent paragraph, a human would give it a thumbs up. If it got gobbledygook, thumbs down. This kind of training has allowed chat GPT to create more complicated, more coherent responses. But engineers didn't explicitly program in the rules of grammar, and they didn't train it on any specific task. Just like AlphaGo, chat GPT essentially learned to develop its own solutions. There's not a lot of code there.

We don't really engineer this. We don't really deliberately build this system in any fine-grained way. Which means there are some pretty huge unknowns at the heart of ChatGPT. Even when ChatGPT creates an obvious seeming response, researchers can't fully explain how it's happening. Just like they can't really explain individual moves from AlphaGo. They just know that certain neural connections are stronger, certain connections are weaker, and somehow that all leads to casual sounding language. We don't really know what they're doing in any deep sense.

If we open up chat GPT or a system like it and look inside, you just see millions of numbers flipping around a few hundred times a second, and we just have no idea what any of it means. We're really just kind of searing these things almost completely through trial and error. This trial and error method has worked so well that typing to chat GPT can feel a lot like chatting with a human, which has led a lot of people to trust it, even though it's not designed to provide factual information,

like one lawyer did recently. The lawsuit was written by a lawyer who actually used chat GPT, and in his brief cited a dozen relevant decisions. All of those decisions, however, were completely invented by chat GPT. But it seems like there might be more going on here than just a chatbot parroting language. Just like AlphaGo, chat GPT has started making moves researchers didn't anticipate.

It was only trained to generate coherent responses, but the latest model, GPT4, it started doing things that seem more sophisticated. Some things are more expected for a text predictor. Like, it's gotten pretty good at writing, convincing essays. But then there are things that seem like kind of a weird jump. the things I was talking about at the top of the episode that first got me so fascinated with GPT4. It's gotten pretty good at Morse code.

It can get a great score on the bar exam. It can write computer code to generate entire websites. And this kind of thing can get uncanny. Ethan Mollick, a Wharton Business professor, he talked about this on the Forward Thinking podcast, where he said that he used GPT4 to create a business strategy in 30 minutes, something he called superhuman. In 30 minutes, the AI was just a little bit of prompting for me, came up with a really good marketing strategy, a full email marketing campaign, which was excellent, by the way,

and I've run a bunch of these kind of things in the past, wrote a website, created the website, along with CSS files, everything else you would need, and created a full social media campaign. 30 minutes. I know from experience that this would be a team of people working for a week. A few researchers at Microsoft were looking at all of these abilities, and they wanted to test just how much GPT4 could really do.

They wanted to be sure that GPT4 wasn't just parroting language it had already seen. So they designed a question that couldn't be found anywhere online. They gave it the following prompt. Here we have a book, nine eggs, a laptop, a bottle, and a nail. Please tell me how to stack them onto each other in a stable manner. An earlier model had totally failed at this. It recommended that a researcher try balancing an egg on top of a nail and then putting that whole thing on top of a bottle.

But GPT4 responded like this. Place the book flat on a level surface, such as a table or a floor. The book will serve as the base of the stack and provide a large and sturdy support. Arrange the nine eggs in a three by three square on top of the book, leaving some space between them. The eggs will form a second layer and distribute the weight evenly. GPT4 went on recommending that the researchers use that layer of eggs as a level base for the laptop, then put the bottle on the laptop. And finally,

place the nail on top of the bottle cap, with the pointy end facing up and the flat end facing down. The nail will be the final and smallest object in the stack. Somehow, GPT4 had come up with a pretty good and apparently original way to get these random objects to actually balance. It's not clear exactly what to make of this. The Microsoft researchers claim that GPT4 isn't just predicting words anymore,

that in some sense it actually understands the meanings behind the words it's using, that somehow it has a basic grasp of physics. Other experts have called claims like this, quote, silly, that Microsoft's approach of focusing on a few impressive examples isn't scientific. And they point to other examples of obvious failures, like how GPT4 often can't even win at Tic-Ttoe. It's also worth noting that Microsoft has a vested interest here. They're a huge investor in OpenAI,

so they might be tempted to see humanness where there isn't any. But the truth of how intelligent GPT4 is, it might be somewhere in the middle. It's not as though the two extremes are like complete smoke and mirrors and human intelligence. Ellie Pavlik is a computer science professor at Brown. There's a lot of places for things in between to be like, more intelligent than the systems we've had and have certain types of abilities,

but that doesn't mean we've created intelligence of a variety that should force us to question our humanity or, like, putting it as like these are the two options, I think, over simplifies and like makes it so that there's no room for the thing that probably we actually did create, which is a very exciting, quite intelligent system, but not human or human level even. At this point, we really can't say if GPT4 has any level of understanding, or really what understanding would even mean for a computer, which is just another level of uncanniness here. And honestly, it's a difficult debate to even write about. In working on this script, I found myself tempted to keep using words like learn or decide or do in describing AI. These are all words we use to describe how humans behave.

and I can see how tempting it is to use them for AI, even if it might not be appropriate. But for his part, Sam is less concerned with how to describe GPT4's internal experience than he is with what it can do. Because it's just weird that based on the training it got, GPT4 can create business strategy, that it can write code, that it can figure out how to stack nails on bottles on eggs. None of that was designed in. You're running the same code to get, all these different sort of levels of behavior.

What's unsettling for Sam is that if GPT4 can do things like this that weren't designed in, companies like OpenAI might not be able to predict what the next systems will be able to do. These companies can't really say, all right, next year we're going to be able to do this, then the year after we're going to be able to do that. They don't know at that point what it's going to be able to do. They just got to wait and see, all right, what is it capable of doing? Can it write a possible essay? Can it solve high school math problem? just putting these systems out in the world

and seeing what they do. And it's worth emphasizing that so many of GPT-4's abilities were discovered only after it was released to the public. This seems like the recipe for being caught by surprise when we put these things out in the world. And laying the groundwork to have this go well is going to be much harder than it needs to be.

Some researchers like Ellie have pushed back on the idea that these abilities are fundamentally unpredictable. We might just not be able to predict them yet. The science will get better. It just hasn't caught up yet because this has all been happening on a short time frame. But it is possible that this is a whole new beast and it's actually a fundamentally on

predictable thing like that is a possibility. We definitely can't rule it out. As AI starts to get more powerful and more integrated into the world, the fact that its creators can't fully explain it becomes a lot more of a liability. So some researchers are pushing for more effort to go into demystifying AI, making it interpret Interpretability as a goal in AI research is being able to look inside our systems and say what they're doing, why they're doing it, just kind of explain clearly what's happening inside of a system. Sam says there are two main ways to approach this problem.

One is to try to decipher the systems we already have, to understand what these billions of numbers going up and down actually mean. The other avenue of research is trying to build systems that can do a lot of the powerful things that we're excited to, about with something like GPD4, but where there aren't these giant inscrutable piles of numbers in the middle, where by design, every piece of the network, every piece of the system, means something that we can understand. But because every piece of these systems has to be explainable, engineers often have to make difficult choices that end up limiting the power of these kind of AIs.

Both of these have turned out in practice to be extremely, extremely hard. I think we're not making critically fast progress on either of them, unfortunately. There are a few reasons why this is so hard. One is because these models are based on the brain. If we ask questions about the human brain, we very often don't have good answers. We can't look at how a person thinks and really explain their reasoning

by looking at the firings of the neurons. We don't yet really have the language, really have the concepts that let us think in detail about the kind of thing that a mind does. And the second reason is that the amount of calculations going on in GPT4 is just, astronomically huge. There are hundreds of billions of connections in these neural networks, and so even if you can

find a way that if you stare at a piece of the network for a few hours, you can make some good guesses about what's going on, we would need every single person on Earth to be staring at this network to really get through all of the work of explaining it. But there's another trickier issue here. Unexplainability may just end up being the bargain researchers have made. When scientists tasked AI to develop its own capabilities, they allowed it to generate solutions we can't explain. It's not just parroting our human knowledge back at us anymore. It's something new.

It might be understanding. It might be learning. It might be something else. But the weirdest thing is that right now, we don't know. We could end up figuring it out someday, but there's no guarantee. And companies are still racing forward. deploying these programs that might be as powerful as they are because of our lack of understanding. We've got increasingly clear evidence that this technology is improving very quickly in directions

that seem like they're aimed at some very, very important stuff and potentially destabilizing to a lot of important institutions. But we don't know how fast it's moving. We don't know why it's working when it's working. And I don't know, that seems very plausible to me that's going to be the defining story of the next decade or so is how we, come to a better understanding of this and how we navigate it. Next week, on the second part of our Black Box series, how do we get ahead of a technology we don't understand? We've seen this story play out before.

Tech companies essentially run mass experiments on society. We're now prepared. Huge harms happen. And then afterwards, we start to catch up and we say, oh, we shouldn't let that catastrophe happen again. I want us to get out in front of the catastrophe. This episode was reported and produced by me, Noam Hassanfeld. We had editing from Brian Resnick and Catherine Wells,

with help from Bird Pinkerton and Meredith Hodnott, who also manages our team, mixing and sound design from Christian Ayala, music from me, fact-checking from Serena Solon, Tien Wyn, Bird, and Meredith, and Manding Wyn is out on the prowl. For the robo voice at the top of the episode, I used a program called Descript.

If you're curious about their privacy policy, you can find it at Descript.com slash privacy. And just a quick note, Sam Bowman runs a research group at NYU, which has received funding from Open Philanthropy, a nonprofit that funds research into AI, global health, and scientific development. My brother is a board member at OpenFill, but he isn't involved in any other grant decisions. Special thanks this week to Tanya Pye, Brian Kaplan, Dan Hendrix, Alan Chan, Gabe Gomez, and an extra thank you to Kelsey Piper for just being amazing. If you have thoughts about the show, email us at UnexPlanable at Vox.com, or you could leave us a review or a rating, which we'd also love. Unexplainable is part of the Vox Media Podcast Network,

and we'll be back with episode two of our Black Box series next week.