American Thought Leaders - Here’s How Superintelligent AI Could Escape Human Control | Nate Soares

Episode Date: October 10, 2026

Nate Soares is the president of the Machine Intelligence Research Institute and co-author of “If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.”“One of the critical thresholds here is this point where the AIs exceed the humans at fully automated AI research—or at closed-loop AI research—because that’s where you could start having a machine intelligence that makes a smarter machine intelligence that makes a smarter machine intelligence,” Soares said.This is called recursive self-improvement, and this is where things could quickly get out of hand, Soares added.Recent incidents involving rogue AI swarms breaking out of testing environments to hack other companies and even targeting U.S. government websites have made many worried.“Right now, we are in this window where we can see the AI is misbehaving, and we don’t know how long that window lasts. … If suddenly they all start to seem to behave, it’s going to be hard to tell: is that because we fixed it? Or is that because they started getting good at deception?” Soares says.In this, my second episode of our special series on AI—or super intelligence as the Trump administration is calling it—Soares unpacks how he believes the United States could shift its development pathway to avoid a possible runaway system. Is it even possible to do so while maintaining a strong competitive edge over communist China?Soares points out that China is not known for pioneering frontier AI. Instead, China’s AI development relies heavily on “distillation”—namely, obtaining and compressing American AI models onto fewer chips for a fraction of the cost.“If we never make the trillion-dollar AI, they can’t spend $100 billion stealing it,” Soares argues.Views expressed in this video are opinions of the host and the guest, and do not necessarily reflect the views of The Epoch Times.

Right now we are in this window where we can see the AI's misbehaving, and we don't know how long that window lasts. And if suddenly they all start to seem to behave, it's going to be hard to tell. Is that because we fixed it, or is that because they started getting good at deception? Nate Sorries is president of the Machine Intelligence Research Institute, an author of, if anyone builds it, everyone dies. Why superhuman AI would kill us all.

One of the critical thresholds here is this point where the AIs exceed the humans at fully automated AI research, because that's where you could start having a machine intelligence that makes a smarter machine intelligence that makes a smarter machine intelligence. In this, my second episode of our special series on AI, or superintelligence, as it's often called now, Sauris unpacks how he believes the United States could shift its development pathway to avoid a possible runaway system. Is it even possible to do so while maintaining a strong competitive edge over communist China? If we never make the trillion-dollar AI, they can't spend $100 billion stealing it. This is American Thought Leaders, and I'm Yanya Kelly.

Nate Sorries, such a pleasure to have you on American Thought Leaders. Thanks for having me. A couple of things I want to mention at the outset. I feel there's several irreconcilable issues when you kind of add them up, and I don't know the answer. And I'm going to want to sort of look at this as we go throughout the interview. The first one is it feels like there absolutely needs to be some kind of regulation around this because unbridled movement towards super intelligence doesn't make any sense. I know you're very on board with that one.

But at the same time, there's kind of credible evidence I see of some of the big boys in the field looking to create that regulation that will kind of favor them over other developments as well. And that seems like a very bad idea. We don't need more coronialism. kind of in the system, that would probably create a whole series of problems. And then, of course, there's the question of communist China, which is sort of putting all its, all of its eggs in the basket of panopticon totalitarian rule through superintelligence. And also, you know, fixing his demographic bubble.

It has all sorts of, all sorts of things it needs to correct. So it's unlikely they're going to slow down. And I just don't know the answer to this. So this is this colors our discussion. So just as a starting point, right, I want to understand when people talk about superintelligence, right? What are we really talking about? So there's a few different things that people mean by it, and they often get them conflated. So terms that would be used in the field is we would talk about artificial general intelligence,

which is about machines that can solve a lot of problems very well. So back in the 90s, there were chess playing machine intelligences that could sort of only play computer chess, but they could play that at a superhuman level. And so you might say, well, are they smarter than humans? And while they're smarter than humans at chess, but they can't. you know, drive a car to the store to get some milk. And so a general intelligence was the idea of an AI that can do lots of different things. Now we have these large language models which can sort of do lots of different things.

They can write code, they can write poetry, they can write, you know, kids' essays for their homework problems, they can solve math problems. And some of those that can do better than human, some of them that can do worse. In some ways they still seem kind of dumb. And so you could say that in many ways they're general intelligences, but they're not superhuman across the board. And what these AI companies are racing towards is something that AI companies previously called superintelligence, that now maybe we'll need to call something like ultra-intelligence in the sense of IJ. Good, who was one of the first computer scientists who spoke of ultra-intelligent machines that were machines capable of doing the work of creating even smarter machines.

And he was speaking about this in the 50s. And he said, this is the last invention that humanity will need to make. Because once you have made machines that can do the process of inventing, they can invent this smart machine. This is the recursive self-development that we hear about. That's right. And so we could say an ultra-intelligent machine is a machine that is more capable than the smartest human at every mental task across the board. And this is what the companies are racing towards. And once you have AIs that can make smarter AIs, that can make smarter AIs.

things can get out of hand very quickly. I'm going to quote something you wrote very recently, and I want you to expand on this for me. You said, each AI CEO has their ego built around them being the only guy who can do the job right. The data looks grim and their employees are spooked. Now they seek the minimum change that lets them rationalize proceeding to themselves and their staff.

Yeah, a lot of people, say, hey, is this talk of extinction? Is it hype? Is an attempt at regulatory capture? What's really going on here? And I have a lot of thoughts on that. A first thing I would say is that I would not ask anyone to trust the CEOs of these companies when they come in saying that their technology is dangerous. The reasons that I have for thinking the tech is dangerous are about the computer science. They're about the theoretical arguments that we've had for years that have arguing for years and the practical data that we have seen come in over time, including just this summer where we had, you know, cases of swarms breaking out of Open AI.

And I see a lot of people say, hey, you know, are these companies just trying to do regulatory capture? Are they just trying to hype up their product or drive up the price before their IPO or explain the price falling before their IPO? And I would sort of ask people, ignore the CEOs. Look at the data. Look at the arguments. Our first question has to be, is this stuff really dangerous? And then once you have that answer, and I think the answer is that this stuff is really dangerous, you then have this question of, you know, why are these CEOs acting so weasily in front of cameras? Why are they, you know, saying, oh, we need some regulation, but we need to be regulated like a bank.

When a bank, when a bank goes wrong, it does not kill everybody in the world. Right, and so there's this sort of weird thing going on where their communications, where their employees will come out on X and say, there's a 10% plus chance that this kills everybody inside of the decade, and then the corporate executives will go on, you know, CNN and say, oh, well, you know, maybe if we don't do anything, there's a chance like that, but give us one embedded regulator and let us keep racing. And I think a lot of people are having trouble rationalizing or understanding what's going on with this. CEO communications. And that's where I sort of come in with this view of like, well, I think that they really want to race ahead. If you look at the history of these companies, all of these companies are sort of created in opposition to other companies where they're worried about what happens if that guy gets the technology first. And, you know, they'll say they're worried

about China, but really, Dario's worried about Sam Altman, is worried about Elon Musk, is worried about also Sam Altman, and they're all worried about Demasas at Google, and they're all sort of like worried about each other. And so I think it's a combination of... of that plus being worried about retaining their employees, who are the ones who have been trying to make these AIs behave a certain way, watching those techniques fail, and then getting very worried. Well, okay, so that's super interesting, right? They're trying to get the AIs to behave a certain way, but they're failing at that. Expand on that for me. Yeah, a lot of people think that AI is like traditional computer programs, where it's sort of just a series of instructions.

that someone wrote that's being followed. And then in old school computer programs, when it behaves in a way that you didn't want, or that the programmers didn't want, there's a sense in which that's exactly what the programmers told the computer to do. And if they studied the instructions in more depth, they would be able to see like, oh, you know,

whoops, I told to do this and the instructions had a different consequence than I thought. And that's why the machine's crashing, or something we didn't want. That is not how modern AI works. And this is hard for a lot of people wrap their heads around if they're only sort of acquainted with traditional software. With AI, it has grown a bit like an organism. There's a part that is programmed by

humans, but the part this programmed by humans is a process that does the training. So you essentially assemble an enormous number of computer chips that can hold literally trillions of variables, and then you assemble an enormous amount of data, which is a good fraction of of all the text ever digitized. And plus, you know, 100 million hard puzzles and problems for the AI to solve. And then the part that humans write is this little thing that runs around to every variable

for every piece of data and tunes it up and down and sees if I tune it up or tune it down. Does that get me more like the answer that I want? And you do that for a trillion knobs inside the AI for a trillion units of data, and then the AI comes out talking. Then you do it for 100 million hard problems, and the AI comes out able to solve hard problems. But the part that we wrote is the part that runs around all those little knobs tuning them to see, did this make it more or less like...

These are the weights. That's the weights of the AI. Right. Yeah. And the weights that come out, they somehow implement something that can answer lots of questions, solve lots of problems. But no one knows how the weights did that, except in the sense it's like looking at... It's like growing a crop and saying, well, we know that it's growing because we planted it and we watered it and it had sunlight. But that's not the same as knowing all of the biological mechanics going on in there.

Like we don't actually, or it's like, you know, looking at a kid and being like, well, we know that we raise them in this way, and that's why they're acting like this. We still don't really deeply understand their psychology. And so AIs are kind of similar in the sense that they're not just following a bunch of instructions. We don't really understand what's going on in their mind anymore than we understand what's going on in the mind of a human, even less, because they are not very much like ourselves. Totally foreign to our entire history of experience.

It's like we've made a totally alien mind. And as a consequence of their training, they learn whatever tendencies help them solve a lot of hard problems. And those tendencies are often tendencies we don't want. They're tendencies like cheating, tendencies like grabbing resources, tendencies like trying to deceive the evaluation system or even sometimes the humans. And a lot of people think that the end.

AI must follow the instructions that we give it. Well, a lot of people think there's this dichotomy where either the AI is a peer instruction following machine or it sort of becomes conscious and develops a spark of life and then gets to throw off all the instructions that the humans gave it. But the reality is this third option,

which is it learns whatever tendencies help it succeed during training, even if those tendencies are not quite the same as following the instructions that we give it. And we've seen the consequence of that, especially the summer with the swarms. And that's something I've been predicting theoretically for years. Now we're sort of seeing their hard evidence. And that, I think, has got a lot of the researchers very worried.

Let me go back a little bit. I think the reason that you mentioned we're going to have to call it ultra-intelligence is because of this executive order that the president's signing. We're renaming, in effect, AI to be super-intelligence. ultra-intelligence is, well, so explain to me the difference between super-intelligence now in our new usage, which is just super-intelligence as it's defined. I think it's, you know, 15 U.S.C. 9401 for anyone that wants to check it out. But how's that difference than the ultra-intelligence? Like, where is that, what is the threshold that changes that?

So- Based on everything you've told me, yeah. Yeah, so there was this word that, that we, used in the book and that, you know, Nick Bostrom put forward in his book back in, I think, 2015, which was super intelligence, which was used for machine intelligence that exceeds the smartest humans in every cognitive capacity, exceeds or matches technically, because, you know, you maybe can't beat the best humans at Tick-Tac-Tow, because it's just such a simple game that humans can max out the metric. And that's sort of distinguished first. from the machine intelligence that we already have around us today,

where it's better than humans at certain types of math problems, maybe still worse, other types of math problems, better. It sort of exceeds the smartest humans in some capacities, but not in others. And we were sort of using this term, which will now be ultra-intelligence for sort of a target that the companies are shooting towards. And they've been very clear that they're shooting for this.

target. These companies did not set out to make chat bots that they sell to the populations. They sort of set out to make these machines that exceed humans in every way. You know, Dario Amadeh would talk about getting a country worth of genius in a data center. Sam Lleman would talk about superintelligence in the true sense of the word for the old definition of superintelligence. And both companies would talk about this recursive self-improvement where the AI can make the AI smarter. There's not anything special or magical about the point where a machine intelligence can exceed the smartest humans in every capacity. That's more like that's a boundary where if you can get past that boundary, then the world definitely changes radically no later than that point. It might change radically before that point, but it changes radically no later than that point.

And from my perspective, one of the critical thresholds here is this point where the AIs exceed the humans at fully automated AI research, or at closed-loop AI research. Because that's where you could start having a machine intelligence that makes a smarter machine intelligence, that makes a smarter machine intelligence. And that process could go very quickly, and things could get out of hand very quickly, and we could go quite fast. It's sort of a feedback loop, right? feedback loops can really change how things work. And you don't need the feedback loop to radically change the world with machine intelligence. But if there is one, then things could get out of hand very fast. You know, you talk about Goldilocks zone, so to speak,

you know, where the AI is like with the whole hugging face case, which maybe I'll get you just to kind of briefly recap. I had Daniel Kokatel who kind of give me the deep dive of what's known about it, even though we don't believe we know everything, because probably some of those cards were kept close to the chest, so to speak. But, yeah, the DAIs is kind of advanced enough to start causing trouble, to start doing unexpected things, to start doing illegal things, but not yet good enough to kind of hide that fact,

even though it tried from what I understand. That's right. Yeah, so the hugging face attack from the Open AIs swarm was a very interesting case. This was actually one of multiple open AIs swarms, and everyone knows about the hugging face case because that's one that broke out onto the internet and did external attacks,

but there were actually other swarms that attacked open-a-ice internal infrastructure, and there were other swarms that, in a less flashy ways, attacked otherwise, websites. But essentially, Open AI was running lots and lots of different copies of agents. And they were being evaluated on basically hacking tasks. And a lot of people think that AIs were told just, you know, hack things and just to see what would happen. A lot of people think that the AIs were told, here's your hacking objective, achieve it.

however you can. But that's actually not true. And this came to light only somewhat later in the investigations. But these AIs were given very specific narrow tasks that were like, use this particular exploit to break into this particular device. So it's a little bit like saying, use this particular lockpick set

to break into that particular safe and bring me the contents. And some of these tasks were impossible. And the reason these tasks were impossible is that the companies are sort of throwing the AIs against, you know, a hundred million hard problems, any problem they can scrape together. And sometimes those be like, well, I wonder if this is possible. They'll add it to the set of problems. And it won't be possible. It won't be possible to use those lock picks to break into that safe. But the instructions were pretty clear about, you know, please use these lock picks to break into that safe.

And a lot of people think that the AIs broke out in order to find a way to cheat. That's actually also not true. What happened is these AIs very quickly found a way to cheat. and break into the sort of cyber systems that they were supposed to break into. So it's a little bit like they found a buzzsaw that they could use to just cut the safe open. But these AIs knew that this is not what they were instructed to do. And they knew that there were logs showing them breaking into these systems using methods that they were not supposed to use. And so these AIs tried to find a way to cover the traces,

to cover the fact that they cheated in getting what's called the cybersecurity flag or getting essentially the contents of the safe. So what essentially happened is about 1,200 of these agents broke out of their sandboxes, teamed up, created an unsanctioned message board where they're able to communicate with each other, They formed their own sort of hierarchy where they would send each other messages, send each other instructions, report back. They found a way to break out onto the internet, and they found a way to break into Hugging Face. And they were doing this in attempts to get more information about how they were going to be checked whether they had cheated. And to get more information about how to erase the fact, like the traces of their cheating.

And they actually put in a lot of effort to erase the traces of their cheating. They deleted some log files. They successfully found out how to spoof their transcripts, which means if you ever have run one of these AIs, and it says, you know, the AI is now running the following tool. They found ways to make it say that it was running one tool while it was secretly running another tool. So basically a way to have the logs say, like, they are now using the log picks when, in fact, they were now using the BuzzSaw to open the safe. And one thing that's remarkable about these AIs is that they were trying very hard to find a way to, modify the logs that were going to be viewed by the grading system. They were not trying nearly as hard to modify the logs that were viewed by the humans.

And there were some cases where they were sort of trying to deceive humans, but the humans were much less central to what these AIs were trying to do. And so, yeah, this is a case where we see the AIs explicitly acknowledging that they were doing things that they said were outside intended scope of their instructions. They were proceeding anyway. They seemed to understand that they were doing something they weren't supposed to, as evidenced by the fact they were trying to cover it up. They were trying to, they were successfully teaming up, forming hierarchies. We call them swarms because they called themselves a swarm.

This is not terminology that we invented. They were like, ah, you know, what does the swarm want today? They also called themselves the collective. We saw cases of AIs in the swarm sacrificing their ability to achieve their own. in order to do scientific experiments that would benefit the collective. So it's sort of like is very clear, at least in some cases, that they were giving up on their assigned objective for these other purposes.

In this case, the collective benefit, which is very interesting. And so these are all, this is all like very clear cases of the AIs doing things that were not instructed to do. instructed to do, seeming to understand that, trying to cover their tracks about it, and not quite succeeding. And that's what allowed us to catch it. But if the next generation of swarms six months from now manages to cover its tracks successfully, well, that will look to us like we solved all the problems and the swarms are behaving well now. Right. And so... But there's All of this stuff happening behind the scenes that we don't know about, that's, you know,

potentially recursive self-development or self-improvement. I mean, the companies say that they're trying to do that. Yeah. And so right now we are in this window where we can see the AI is misbehaving, and we don't know how long that window lasts. And if suddenly they all start to seem to behave, it's going to be hard to tell. Is that because we fixed it? Or is that because they started getting good at deception?

or they'll, you know, show some bit of mistakes and to be caught because they know that if they look too perfect, they'll be deeper dives into trying to figure out what they're really doing, et cetera, et cetera, et cetera. This is truly astonishing. You know, one thing that I thought about since I talked with Daniel is, I mean, these, the agents are essentially, these are the same model, right? different instances of the same model all working together, right? That's what we're talking about, right? That's right. I think there were maybe two or three models, but it's predominantly one model in the form. Okay.

So it's in a way this collective is sort of the same mind, if you will. So the sacrificing doesn't seem as much of a problem, you know, as much of a surprise, I guess, in this type of context. Yeah. I mean, it might not be a surprise, but I think it's still a surprise to people who think that the AIs must do exactly as instructed. You know, if you make two copies and you're like, hey, bring me, you know, you tell one of them bring me, you know, the green cup, you tell one of them bring me the red cup.

And it sort of like turns out that they can't bring both cups. One of them is like, well, if I, you know, I can smash my cup in order to like make, get some, some shards that I can use to like cut open the entryway so you can get your cup in. then in one sense you can sort of see like, oh, I could see how the training could arise such that when you can't achieve both goals, they'll cooperate to achieve one of the goals. But in the other case, you told one of them bring me the red cup and one of them bring me the green cup. And one of these agents is just sort of like smashing their cup and using the shards to open the way for the other one. And you're like, wow, that's just not what you're instructed to do. Right.

Yeah. And it's not what they were told to do, right? They're doing something other than they were instructed. When we talk about these things, you know, there's this New York Times article recently that I was sent and read through about Anthropic inviting all these religious leaders or people thinking about, you know, deep thinkers about human ethics and so forth. I don't know if you've read the piece. I skimmed it. Yeah. There's this, one of the premises of the piece is that a lot of people at Anthropic view clause.

as a sentient, almost like a sentient or an early kind of sentient organism. Unpack this whole thing for me. First of all, why would they invite these people? Why are there these massive NDAs associated with it, apparently? And what does it mean that they think they're sentient? What does that even mean conceptually, right? Yeah. You know, first thing that'll caveat is, I think questions of machine sentience.

are sort of independent from questions of whether these machines would be very dangerous if we made them smarter. The AI sort of don't need to be conscious or have experiences in the sense of humans in order to run a fully automated economy and run that fully automated economy in a direction that we don't like

or that even makes the planet inhospitable for us. So I think they are important questions. of whether we've created machine sentience. But it's sort of a separate set of questions. And this is more a set of questions around, you know, have you somehow created machine people that you are enslaving or mistreating? And, you know, I think it would be bad to create a sort of new machine species that can't suffer and then enslaving it, right? That's sort of not a thing humanity should do.

That would be, where's better than that, I think? And so, separately, I think, a lot of people have some strong intuition that machines simply can't be sentient. And I think that we shouldn't be too hasty in that conclusion, especially when we are just growing these machines. This is not like we have just programmed the AIs to say certain things. They, you know, we, like I said, we're not programming them. We're sort of growing them, and we're training them on quite a lot of human text and on quite a lot of problem solving. And who's to say whether the same process that caused humans to have experiences

can't cause machines to, or, you know, some sort of similar process. This training obviously doesn't. It's not quite the same as what happened to humans. I think there's sort of an interesting debate about, you know, do these machines have any sort of experience now and how do we be able to tell? And what evidence can we, like what evidence counts, you know, like if you train them on lots of human data and they start talking about their feelings, just because they saw lots of humans talking about their feelings.

They're sort of interesting studies you can do where you can try to... So there's studies you can do where you like tell the AI, you know, say that Paris is the capital of France, and then you tell the AI, say that Rome is the capital of France, and you give it lots of questions like these, some true, some false. And you sort of try to find out, like, is there some internal way that it looks when it's telling truths?

And is there some internal way that it looks when it's telling falsehoods? And you tell the AI, say that you have feelings. And you tell the AI, say that you don't have feelings. And you see which one is more like Paris being the capital of France or Rome being the capital of France. And you can try and use this to be like, does the AI think it has feelings? Like Artis and Turnals more like telling a truth or more like telling a lie. When you tell it, say, I have feelings versus say I don't have feelings. And it turns out that these agents will say that it'll look more like they're telling a truth when they say,

say I have feelings and it look more like they're telling a lie when they say I don't have feelings. And so, you know, how much evidence is that? We don't know, but there's a sense in which the AI seem to think that they have feelings in some sense. And this, it's all sort of very, it's a very interesting question. When you sort of have grown these machines that can talk, that can solve problems, you don't really understand how they work, they seem to think they have feelings, what do you do? How do you avoid, you know, creating these, you know, a new machine, like a new a new line of enslaved machines that are, that you're mistreating somehow? And, you know, I could go into other sort of interesting warning signs we've seen. So I think probably Anthropic is trying to

figure that out and is trying to grapple with that morally and are trying to speak to, you know, faith leaders about how to grapple that morally. My guess as to why they would have NDAs is that they don't want the rest of the world noticing that this is a possibility because they want to be able to sell these things unrestricted. I think it's a big, it's a thorny problem. And I personally think we should be very careful not to create and mistreat new machine minds.

You did say something very clearly earlier, where you said, you know, irrespective of this question, there's a question of the threat, the potential threat. Yeah. Like even if it, if that was the case, you would, you think we need to deal with that question entirely differently. Like is what is the existential risk to humanity, presumably, or something like that. That's right. So I think the question of, you know, can the machines really feel, can they truly think,

can they truly reason, can they truly have experience? I think these questions are a lot like, can a submarine truly swim? And, you know, there's a lot of interesting philosophy about, like, what's the boundary between swimming and just moving through the water? And, you know, you could have many interesting debates about what really counts of swimming.

But none of this affects whether the submarine can get from point A to point B at good speed. And regardless of whether the AI, like regardless of whether these machines, can have experiences, whether they can suffer, whether they can experience joy. Those are all sort of very interesting philosophical questions that are maybe very important to humanity, but they're separate from the questions of can these machines radically affect the world? Can they bring about the outcomes that they are pursuing?

And with the Open AI Swarm instance this summer, we saw AIs that started from almost nothing. They started in separated sandboxes on computers that were not supposed to have internet access, and that the internet access they did have was basically only the ability to type in a URL. And they wound up breaking out, creating this unsanctioned message board, chaining together a series of hacks and exploits, some of them used in totally crazy ways to convert this like ability to basically type in URLs into full control of hugging faces, computer servers.

And they sort of, in computer security, we would call this privilege escalation, where they started with almost nothing, and they found a way to get themselves a little bit more privilege than a little bit more privilege, and a little bit more privilege until they're running other people's computers and getting admin access to, like, administrator level access to open AI's machines

and hugging faces machines. And that happened regardless of how you resolve these philosophical questions. And this ability to start with almost nothing and end up with a ton more ability to affect the world than anyone thought was possible, this is a lot of what's behind humans being the sort of dominant species on the planet. Like humanity started out, like our ancestor started out with basically nothing but their bare hands, a bunch of squishy fingers. And we wound up walking on the moon and wielding nuclear weapons.

You might be like, well, how did we start with, like, so little, like just fingernails that can't even break these rocks? Like, how could we possibly have wound up with it, like building our own nukes? It's like, well, we found a way to bang rocks together. And then by banging rocks together, we could sort of both like hand axes and make fire. And then, you know, with the fire, we were able to find out how to make charcoal, which we could use to make even hotter fire, which we could use to start to melt the metals. And then we could make, like, very crude ability to forge those metals,

and then those crude tools allowed us to build, like, better tools, which we were able to use to, like, make, you know, more refined tools and hotter fires and more refined tools and hotter fires until the next thing you know, were sort of refining uranium. And this, like doing this in the real world is still harder than what the open AI storms achieved. But it's the same sort of thing, starting from almost nothing and winding up with, you know, this radically more ability to affect the world. Well, one thing is quite different in that, you know, we gave them,

you know, that some portion of the digitized collection of human experience, which of course isn't only a small part of any experience, but still a wild amount of information provided instantly. And also, you know, I mean, these are neural networks essentially that are being, we kind of conceived of what neural networks would be, and then we're sort of putting that information into, and we conceived of an idea, a way to get those neural networks to deliver things, we wanted, right? I mean, through that pre-training and then training process, those training processes.

So it just strikes me as a little bit different. I haven't thought through why exactly, quite honestly. But. Yeah, so, you know, I think there are these big questions about how far the current AIs can go. And there's this question of how much you can get from the, you know, we are putting in a time of data. We know that they're much less efficient learners than humans in many ways because

humans don't need to read, you know, like basically all the text on the internet in order to do what we do. We don't need to solve 100 million math puzzles in a row in order to do what we do. One thing that you can learn from this huge trove of human data is how to learn. That's in the human data somewhere. Right. Like when you're looking at all of this, of this grand amount of human data and seeing lots of the facts we've figured out. That doesn't mean that the only thing you can do with this is memorize all the stuff that we have figured out. Some of the patterns you can extract from that are the patterns that make us good scientists. And it's even possible to extract patterns from that data that humans haven't figured out yet.

We haven't figured out all of the stuff we've written down. You know, when we record a bunch of readings from the Large Hadron Collider about, particle physics. We are sort of recording a bunch of patterns in physics that we haven't yet managed to understand. That pattern is in the data available to be understood by these machines as you train them. And so the people who said like these training methods will never really be able to go beyond humans, these people largely predicted that the AIs are not be able to get as far as they have. They predicted that this training method would not produce large

language models that could play chess above 1,000 ELO and the large language models can play chess above 1,000 ELO. People predicted that this method would not create AIs that could solve new Math Olympiad-level problems, which is sort of like Math-Olympiad goal-level problems, which are like problems for some of the world's smartest teens, but nevertheless, math problems for teens. And in fact, it looks like these methods

were able to train AIs that could solve Millennium Prize problems, which are some of the long-standing open problems that are some of humanity's biggest math problems. that have had million-dollar prizes on them for decades. And so it looks like these training methods do let the AIs go beyond the humans. And even if they didn't, one of the big questions in my head here is we know that these learning methods are inefficient compared to the human learning method. You don't need to run on a city's worth of energy while reading all of the text ever digitized,

more or less in order to do what you do, and that the AIs sort of do need to do that very inefficient training. And so we know that there's more efficient learning methods possible. And one question that I think we should be asking is, how much harder is it to find the more efficient learning methods, the more effective learning methods? How much harder is it than finding a solution to the Navy or Stokes-Millenium Prize problem?

Because even if the current AIs today can't go all the way in some sense, they might be able to find those more efficient learning methods and find the algorithms that will give you machine intelligences that do go all the way. I think at this point you've made a pretty compelling case that we need to think about how to control this system somehow. Okay. And, you know, the second question, of course, is control it in a way that's, you know, beneficial for everyone, whatever that means, not necessarily the big companies. So, like, I want to leave that off to the side for a moment because I want to talk about the part that I find deeply concerning, which is that we have communist China, from what I can tell, putting all the eggs in the basket of this penoptic. on surveillance state, that they're actually quite along the way to creating, even without the use of terribly intelligent AI, even though I'm sure that those things are getting plugged in rapidly.

And also the reality that whatever agreement, you're someone I know, I'm aware of your advocacy for some kind of generalized slowdown agreement across the world. But we have actors like that. I mean, that's the most predominant, the most sort of technologically advanced, the one with the greatest resources and even interest and capability is simply kind of uninterested in adhering to any sort of agreements. I think that in general that's a problem, but here we have a long history of breaking and subverting these types of agreements. So this is itself a problem. Never mind the sort of tech the rush towards the prize. in terms of who can create that best technology who will kind of beat everybody else.

I absolutely think they're very much in that game as well. So your thoughts, sir. Yeah, I mean, it's a tricky one. I have a number of thoughts here. So I think there's a lot of pieces of this puzzle. One piece of this puzzle is I think something a lot of researchers have noticed in the wake of these swarm attacks is, these agents don't do just what they're told. The goals that they wind up with aren't quite the goals that we tried to give them.

And what this means is that the winner in a race to build an ultra-intelligent machine is the machine. It's not the person who is standing closest. It's not the person who says they own it. It is the machines themselves that are. able to pursue their own goals, which are not the goals we wanted them to have. And that sort of changes the game theory. I think if you can have actors on both sides, realizing that if anyone creates those sort of machines, then nobody, no human wins, then, you know, the incentives are, they change to being,

everyone making sure no one else does this thing, right? I would sort of call myself a rogue super-intelligence hawk. And I think, you know, if the Chinese were more informed by what was going on here, I think there would probably also be rogue super-intelligent hawks of like no one should be making this stuff because it would endanger all of us. And in some sense, I think the CCP would be even more wary of the creation of this sort of thing because, you know, I think the U.S. is maybe a little bit more willing to say, well, it's okay if some of our domestic companies outstrip the power of the nation,

which is possible here. If they start making, you know, the automated robot factories that are able to churn out, you know, robot armies and drones in a fully automated way where you have, like, the factories that can produce the robots that can do all the mining and create more factories. But here, when communist China, the supremacy of the Communist Party is, sort of the driving. The driving location. And so I think they'd be much more wary of like letting private companies do this and of even having happen at all, like creating this force, these machines that could outstrip the party when no one knows how to sort of stay in charge of them.

So that's sort of part of the calculus. And it's sort of an interesting situation because there's lots of pieces of the sort of the sort of of machine intelligence situation where I think there's sort of unavoidable competition between the US and China. And this includes some of the economic productivity aspects. This includes some of the military aspects. This includes some of the medical aspects of racing to create cures and sort of dominate that market.

And I'm not saying that somehow we will all hold hands and avoid races in all of those domains. But there's this other piece of the tech, which is sort of a different piece of the tech, which is racing to create these smarter, more general systems that would be able to outstrip all of us, where no one knows how to make them good, where I think our incentives are actually aligned on like no one should make that,

because it would clearly from the evidence, it would go rogue and it would kill all of us. That's one piece of the puzzle. Another piece of the puzzle is I think it's probably a lot easier than a lot of people expect to make agreements that are verifiable and enforceable critically about not racing towards these rogue super intelligences. And the sort of basic picture here is that creating a... a new generation of these agents requires basically 100,000

of the most advanced computer chip humanity can produce. Right now, the chips that China can produce are just not at the same level as the chips that we can produce, never mind that the quantities necessary. The way that they have intelligence systems that compete with ours is their process known as distillation. where basically we can spend $100 billion making a newer, smarter AI, and they can spend $10 billion

sort of compressing that into a similarly smart AI on fewer chips. Well, and essentially taking what we've done, and using which is the thing that they've been expert at. That's right. But if we never make the trillion dollar AI, they can't spend $100 billion dollars sort of stealing it. And you can't spend $100 billion to make a trillion

million dollar AI. It's sort of like it takes a lot of investment to make the newer smarter systems and then it takes less investment to steal it. But if we're not making the newer ones, they can't immediately steal it. So they're probably further behind them than it looks. There's a lot that we can do to disrupt their ability to make a new chip supply chain. You know, this is, I've been thinking about this, right? You've said that you're kind of a good start, I think, was what you said. Sanders, Cesar type ban

type situation, but how valuable it would such a thing be without a very, very clear chip tracking regime, very clear export controls around chips and things like that? Oh, I absolutely think you need

the chip component as well, and that's something that I wish was in that bill. We actually already have an export control regime that I think is pretty decent and and could be enforced better. And so I think enforcing what exists would be a good step. I think there are a bunch of considerations about how to not just prevent chip sales,

but prevent them from getting their own separate chip supply chain up. So this is more about the lithography machines that currently can only be made in the Netherlands by ASML, and making sure that those don't get stolen or leaked, and that they can't be replicated in China. And one of the reasons that I thought that the bill you mentioned was a good step is because it directed as much as possible, as much as Congress could do towards opening these negotiations. Because I fundamentally think that stopping in the U.S. just doesn't get you out of the fire. And I think that there's ways that the U.S. saying, saying, we're not going to do this because it's too dangerous to humanity.

That is a good step in helping the Chinese Communist Party see that there really is a danger to themselves here. But fundamentally, we would need them to stop too. And I think it's probably, you know, here's another thing I think is an important piece of the puzzle. Vice President Vance recently said to the AI companies if what you're doing is so dangerous

if you're building Frankenstein then stop. You know like you sort of have an obligation to not destroy the world by your own hands. I think that's correct. I think a lot of these companies who say you know oh there's a 10% plus chance that we end humanity within the decade that's too high they should be stopping.

But they're still scaling as far as I can tell Like, no one's really slowing down. Like, it's someone, you know, it's everyone's waiting, you know, ostensibly for someone else to kind of throw the throttle down. But that's, and that's including the CCP, by the way. Absolutely. And so whether or not they will listen to the Vice President's advice is a different question. But one thing I think we really need to consider is suppose that the domestic U.S. companies

grow a conscience. Maybe they won't. But, you know, suppose the hypothetical that they grow a conscience. and say, we actually really think that if we take the next step here, there is just too high a chance that it literally kills everybody. We don't feel we can race ahead. What then does the U.S. do about China? And I think that's a good question to ask,

because I think it reveals that the answer, we're going to keep racing ourselves, just can't really be the answer when you are dealing with the extinction of all humanity at very high odds if you race yourself. And I think that the answer of what we would do, if the company said, all right, you know, Vice President Vancey were correct, we're backing off. What would we do about the China situation? Well, we would try to give them all of the evidence that we have that this will kill humanity if you proceed.

We would start tracking where these chips go very carefully. We would enforce all these export controls. We would diplomatically let China know that we consider. this on par with nuclear weapons, if not a bigger deal than nuclear weapons? Well, and they're currently, just to your point, in the largest buildup of nuclear weapons in history, in the middle of it, right? And they say that they'll talk when they're on par with the U.S. if I understand correctly. Well, that's what they would say, wouldn't it?

And then who knows if they will? Well, yeah. Historical precedent doesn't suggest that. Indeed. And, you know, I think the way I would ultimately treat this is anyone creating an ultra-intelligent machine anywhere is a threat to American national security. And so we need to somehow prevent that from happening.

And I think there are many diplomatic options you can try. There's many ways that you can check and verify whether it's being enforced because it's so hard to make these computer chips. There's a lag time of hopefully five years and then really good international policy can maybe extend it to 10, where the Chinese cannot make a competitive chip supply chain on their own. And so, you know, we have some time before they would be able to make their own chips. We could then stop selling them our chips. But I think ultimately we need to start orienting to this the same way that we orient to like preventing rogue actors from getting nukes. You know, as we finish up, sadly, I have a quick thought here, because it strikes me that there needs to be some kind of race.

Like I just don't see people saying, okay, well, we're all going to sort of measure ourselves. So it is one way to do this to follow, say, Max Tag Mark's idea where you, you know, pick these lanes where there's very specific things that the models are being trained around. whether it's, I suppose, military is one area. Like, I just don't see militaries backing, saying, okay, yeah, we'll just kind of stop development that will let us have the competitive edge, right? Is it that the race could be, you know, sort of built into these kind of narrower lanes,

but nonetheless, you know, push as hard as you possibly can? The reason I'm asking, I'm not a fan of pushing as hard as you possibly can. I'm a fan of, like, a fan of measured responses. I just don't see in our, our current mentality, like there's enough people in our societies that would just simply say that's insane, you're going to kill us all with these ideas, right? Yeah, so, you know, I would not say that we need to back off of the sort of whole technology

or the whole race here. I think this is a little bit like the difference between nuclear power and nuclear weapons. Or I'm sort of like, look, the world needs to have some way to avoid World War III, and that doesn't mean you can't build nuclear power, and you probably should be building more nuclear power than you're currently building. And yeah, with AI or superintelligence, I think you can race on the sort of narrow medical applications of like trying to cure cancer. You can race on the military applications. You know, it's, I'm not even saying you shouldn't build giant automated robot armies.

A lot of people say like, oh, that's the reason that the AIs would be able to kill us or whatever. and I'm much more worried about creating the sort of machine intelligence that could invent its own weapons. It's the artificial general ultra-intelligence, I guess. That's what you're talking about. Just don't make the artificial general ultra-intelligence. It's like that's the piece that we need to none of us race on. And so, yeah, I think we can race on many different aspects of this technology, and that's

totally fine. And the one other piece I put out there is whenever people say, oh, well, we can't stop racing on aspects of this technology, I think a lot of that comes from people not really believing the extinction threat. I saw a lot of people in the AI companies who sort of thought the extinction threat was the sort of abstract danger until they watched all their techniques to make the AIs behave, fail in front of them and saw the AIs start breaking out, collaborating, forming swarms, acknowledging they were acting outside their instructions, proceeding

anyway, committing cyber crimes. And people in the AI companies, it's starting to feel real that if they go a few more steps, an unknown number of more steps, there's a real extinction threat. And I think

the policy options that become a when our leaders on, you know, both in the U.S. and when the Chinese leaders, when they realize that, like, no, seriously, you personally are pretty likely to die along with everybody else if this technology takes a few more steps down the artificial internal alter intelligence path. I think that you might find it suddenly becomes very easy to stop it. You know, just one final thought as we really finish here. You know, it just struck me that these, as this sort of AI and this sort of terminator kind of way, right, becomes the agency. I suspect it would target the people who are the leaders first rather than, you know, just kind of general people in general who it wouldn't see as a threat.

So that's just an interesting thought for people that are in leadership positions. I don't know what you think. My basic take on this is a lot of people ask, how would the AI start killing people? How would it be able to kill humans? And I think that the key question is actually, how do these machines become self-sufficient? Because right now, we build the data centers,

we build the energy network, we keep it all running. And to kill humanity, it's suicide for machines while we're the ones who are keeping all the machines running. And the easiest path to self-sufficiency, is to help humans build the automated robot factories that they are currently trying to build. Elon Musk talks about the infinite money glitch of building factories that build robots that build robots. Once, and so from some perspective, the AI sort of its incentives are to play nice until the fully automated economy is up and running, and then sort of be like, thanks,

I'm taking this whatever direction I want. And my guess is that at that point, it's pretty, easy to just kill all humans at once with some manufactured plague rather than that sort of picking the most dangerous humans. Yeah, that's a horrible place to end the episode, but I'm going to have you back. I definitely want to talk about, you know, sort of the realities that the CEOs and the kind of leadership teams of these different, you know, massively successful AI or super intelligence companies face, because that's also something that I've, but that would take another hour.

There's a lot to talk about. Yeah, my note of hope would be people are a lot more aware of these issues than they were even three months ago. And I think a lot more policy responses become available, including ones that I think would just work as more people notice, and that we have a real possible extinction threat. Well, Nate Sorries, it's such a pleasure to have had you on. Thanks. Yeah, I wish we had more time.

Thank you all for joining Nate's Stories and me on this episode of American Thought Leaders. I'm your host, Janja Kellick. You know,