Breaking Points with Krystal and Saagar - 9/14/26: Anthropic Calls To Slow Pace of AI, Nate Soares AI Regulation

Episode Date: September 14, 2026

On today's show Krystal and Saagar discuss the AI CEOs declaring a need to 'slow the pace' of development, AI researcher Nate Soares joins to talk about regulation of the industry and what he thinks is really going on behind the scenes   Nate Soares: https://ifanyonebuildsit.com/   To become a Breaking Points Premium Member and watch/listen to the show AD FREE, uncut and 1 hour early visit: www.breakingpoints.com    Merch Store: https://shop.breakingpoints.com/   See omnystudio.com/listener for privacy information.

This is an I-Heart podcast. Guaranteed Human. Behind every night spent making the kids' lunches, there's a lot of suspense. Will they finally eat the veggies? Will the pastos survive the first bite? Will the lunchbox come home empty, half full, or come home at all?

And will you finish episode 8 before you're done slicing the cucumbers? Get Bell Pure Fiber Internet with Craved Netflix and Disney Plus from $94 a month. Price guaranteed for two years on Internet with a two-year term and auto pay credit. Visit bell.ca for details and to check availability. Bell, connection is everything. Hi, this is Kylie Breakman, Angel Geratana, Jeremy Colhane, Patrick McDonald.

And we're the hosts of the Artists on Artists on Artists on Artist Podcast. The improvised character comedy podcast where each week a new panel of artists discuss their craft. Join us for a panel of Eurovision participants. And we do a little Halloween sometimes. Does the Pope dress up? He winks on Halloween. It's right up next to the microphone. I hear you can almost hear the eyelashes hit each other.

You never seen Italy so quiet as when the Pope gets up to wink on Halloween. Listen to Artists on Artists on Artists every Thursday on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. I survived nine months in captivity, and I've spent my life exploring how other people survive what should have destroyed them. I'm Elizabeth Smart, and these are the Survivor Files. Every week, I'm with survivors who, live through the unthinkable, abducted, stalked, controlled, and nearly silenced.

These are stories about what it takes to make it out alive. Listen to The Survivor Files with Elizabeth Smart on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. Hey guys, ready or not, 2024 is here, and we, here at Breaking Points, are already thinking of ways we can up our game for this critical election. We rely on our premium subs to expand coverage, upgrade the studio, add staff, give you guys the best independent coverage that is possible. If you like what we're all about, it just means the absolute world to have your support.

But enough with that. Let's get to the show. Good morning, everybody. Happy Monday. We have an amazing show for everybody today. What do we have, Crystal? Indeed, we do. Lots going on on the AI front, the heads of all of the major AI companies coming out and

sounding the alarm about concerns over safety and calling for a slowdown. We'll break down what they are saying. We will sort the noise from the hype, from the reality. We're going to have Nate Soros on, who is an expert on these things, also to break down some of the various theories about why they are saying this right now. In addition, of course, we have our eye on the Middle East and the impact on all of our lives. Saudi had to shut the East-West pipeline, which was helping to ease some of the pressure in the oil market. What will the consequences of that be as the Houthis have had a stunning offensive and taken that entire Red Sea coastline, giving them effective control over the Bob El-Mandab Strait? In addition, we are taking a look at gambling with NFL season in full swing.

Saga, of course, has his eye on this one. LeBron, this is obviously not NFL, but related, is the latest player to sell out to the gambling apps. Sidney doing her thing as well, so a lot to get into there. And then we've got the latest on the midterms, including Trump weighing in on flock cameras, Stager. He says he likes him. He's coming in on the right side, definitely. For him, but definitely not for the Republicans. Thank you to everybody who has been signing up for our free trial.

go ahead and take advantage, breaking points.com. Because Ryan continues to win in all of the competitions, our pitch now is if you want Ryan to be Ryan, then you're going to go ahead and sign up for our free trial. Otherwise, you might see Calci ads. You might see Polly Market, Sidney, right? I might go shirtless and go for a prediction market

unless you sign up for that. If you don't want to see that, which you definitely don't. You got to go ahead and sign up for breaking points, If you're watching this on YouTube, please hit subscribe to our YouTube channel. Listening to this podcast, we share an episode with a friend. But as Crystal said, there's so much happening now with AI. Everything kicked off on Saturday here with the release of a new essay by Dario Amade, the CEO of Anthropic.

Let's put it up here on the screen. Highly recommend that everybody go and read it. It's not too long. And I actually think it is actually just worth getting in the heads of Dario. We'll play some of the video. interview that he did in a little bit. But the general TLDR here is called, we must pace the frontier. So the bullet points that I have written down is that Amade says AI companies must deliberately slow the rate at which they advance AI capabilities, not halt training. This is a shift beyond

just investing in risk prevention. The pace of capability gains itself now needs managing. What changed his mind is two things. Number one is RSI. I know these terms are annoying, but it is important that we like immerse ourselves so we can understand these things that these people use. AGI artificial general intelligence RSI is recursive self-improvement. He says it has exploded since the summer of 2026. Hugging face incident in particular along with a number of other incidents that both open AI and anthropic have now been involved in. Pacing, he says, is that models are now capable enough that extra time is genuinely useful and extra a year or two would genuinely reduce risk. He proposes a number of kind of interesting things. So four levels. Number one is ban

clearly dangerous uses, i.e. bioweapons, likely the most feasible, he says. Level two, a mutual pre-release testing for acute risk feasible, but it's hard to give teeth is what he claims. I guess this would be akin to the pharmaceutical metaphor I've been using a lot. It's like you have to, you know, you have to release phase two, phase three trials, the FDA before it's a lot to we bring to market. Level three, he calls a speed limit on recursive self-improvement, kind of what he analogizes to assault treaty, which is the nuclear weapons treaty. He believes is difficult but near impossible. And then level four, a full pacing or pause supported in principle, but he believes is unlikely because of strong defection incentives and a number of

other things he proposed inside of that, which is inviting big geopolitical consequences from China. But overall, a landmark essay, the CEO of the company saying, we need to slow down. He almost never see that. And I guess the big headline news is not Dario's essay, but all of the other AI CEOs coming in and being like, yeah, Dario's right. Right. And so what he says they're going to do an anthropic unilaterally, regardless of how anyone else responds, or whether there's an agreement with China or any other international body, is to invite in some external researchers, give them employee level access so that they can constantly scrutinize what is going on inside of the lab. And he's encouraging others to do that. Sam Altman's.

that they will do the same over at OpenAI. So they're saying they'll take that action unilaterally. Looks like he specifically mentions Meter, which is the group that did the safety report that revealed some of the details about the hugging face incident that we did not know in the past. So that's the action they're going to take unilaterally. In addition, they're talking about we're not going to stop altogether, but we're going to slow down the pace. So this is very different from what Bernie Sanders is calling for at a legislative level.

where he's saying pause everything until we can figure out what's going on. In addition with Bernie Sanders legislation, he would outright ban development of superintelligence that's beyond. So AGI is like human level intelligence and super intelligence is like, you know, better than the smartest humans at every intellectual task. So Bernie's proposal is ban that and make it, put criminal penalties to it, put someone in jail for 20 years if they decide to go forward with developing superintendents. So in any case, there's a lot of theories about what's going on behind the scenes, and we'll get to that in just a moment.

But that's what they're doing sort of unilaterally. Then he floats this idea of, hey, we should probably the frontier labs be able to be in contact with each other so that we can self-regulate together and make sure that we're slowing the pace together. This has gotten into all sorts of discussion over what they're really, whether what they're really looking for is some sort of an antitrust exemption so they can de facto form some sort of. of a cartel. There's a lot of analysis that says that sort of safety collaboration is already allowed and permitted. That's another aspect of what's going on here. And then some of the things that Sagar brought up is he's talking more broadly about, okay, well, what do we do about this internationally? Dario is very much of the but China mindset. You know, if you take him at face value, which by the way, I don't think you should really take any of these guys at face value,

but Dario's been talking about safety concerns for years and years and years. This didn't just come out of nowhere. In fact, he was working and his sister at Open AI. The reason they left to found Anthropic is basically because they saw that Sam Olman is a total psychopath and were like, I don't think they're doing safety correctly. We're going to start Anthropic to do safety the right way. I do want to give him credit that, you know, he's the guy who basically told Pete Heggs up to shove it when he wanted him to give complete and total access to Claude and allow them to do autonomous killing, et cetera. So I think there is enough there to say he is sincere about the concerns. However, he has trapped himself in this very similar mindset as Sam and as Elon, which is,

if I don't build it, some other more irresponsible actor will. And in particular, as I said before, he is very much of the, but oh my God, China mindset. So included in here is while we're doing this, we also need to make sure that we are hobbling China's ability to develop this technology, you know, in whatever ways that we can. That's led to a response to China. We'll get to that in a moment. So in any case, he sort of throws this gauntlet down of we should slow the pace and I'm concerned about things getting out of control. We've reached this trajectory with recursive self-improvement. The AIs are training's even smarter AIs. There's long been theorized that that could lead to this sort of intelligence explosion. We've just had these major incidents. And then you have Elon and Sam Altman

and Demis Sassavis, so we could put this next piece up on the screen, all come in and basically say, we agree. So Elon says Dario is right. Sam Altman says, I agree with Dario. We need to pace the frontier. This has been a primary topic of discussions. We've had at Open AI in recent weeks, committing to having independent evaluators with employee like access. Great idea. We will do the same. Have more to share soon. Demis Asavas, who is the head over at Google, says Dario's essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment. This is also why we recently put out our proposal for an industry-wide standard body for Frontier AI. So these are the foreheads of the major frontier labs all coming out.

Zuckerberg, I mean, they're kind of behind in the race anyway, but he apparently does not agree with the sentiment. But in any case, this has led to a lot of conjecture about what's going on here. Are they genuinely concerned? Is there some sort of a financial incentive here? You've got Open AI and Anthropic, which are planning IPOs. Is there something that happened behind the scenes? Are they trying to create a moat where only they are basically allowed to develop under regulation and then, you know, maybe the open source models will be banned. So that's some of the sort of debate and discussion that's happened. I think it's all the above. And China remains, again, like this chimera, which like floats above all of this. But nobody actually knows anything about it.

And even this morning, there are so many takes out there, but like China is responding. It's like, well, kind of, right? Because China is not a single entity. They have global times, which is like their Fox News, everybody is citing. Then there's a foreign ministry spokesman. He actually is objecting to a very specific part of the Dario essay. And then there's actual journals that are being written. So we're going to spend some time on that. But again, China looms above this entire thing.

And that's what Trump is leaning on, why there shouldn't be any regulation. Let's take a list. We're leading China in AI. We're most sophisticated country in the world. And frankly, I want to keep it that way. Because whoever wins AI wins. And we can put guardrails.

We can do this and that. But I think you have a lot of negative forces that are bringing it up that shouldn't be bringing it up. And they're bringing up things that won't happen. But whoever wins with AI wins. Bringing up things that won't happen. Basically, this is, and this is kind of what you were talking about. And this is where, look, you have to say that there's a kernel of truth,

which is part of the, one of the things, a key part of what Dario proposes an essay, is that he needs an antitrust exemption from being able to cooperate with all of the other big heads of the companies. And that's why you have meta and many of the other AI startups who are like, hey, excuse me, he's using safety as an excuse for regulatory capture so that we have a big four of AI in the same way that we have the defense contracting industry or pharma.

You can't say they're wrong. Like there is at least an element of that. But then you have to say wrong but or wrong, is it also maybe that they are worried. And I think that's why I said when you were laying out your case, I genuinely think it's all of the above. Obviously, we have no idea. We can only listen to what they're all saying. I will say it's pretty extraordinary to see, especially Google. It's not like Google or Gemini is a major leader in AI, or at least hasn't been on the consumer level right now. Although there are some rumors that they may be on the cusp of a breakthrough.

Very possible. We have no idea. At least at the current moment, in terms of what's available of consumer product, I don't say Gemini is one of the worst ones, right? It's up there with meta. So then you're like, all right. And then it's up there with Grock and meta and the rest. It's like genuinely in a second tier compared to where anthropic and open AI are. The question, I think, remains around what exactly they are proposing. And by and large, Sam Altman and Dario and even Elon at this point are all saying at the very

least we need voluntary pause and some sort of coordination. However, again, the China thing loons above. And that's where I just think we have to come back to what do the Chinese actually want out of AI? And what do they think? Because a lot of people are taking a statement this morning. I think we have it. Let's put A8 up here on the screen. This was the Chinese foreign ministry spokesperson who is reacting to the Dario essay.

And I'm going to read you a direct quote. And this is important because this is a direct statement from their foreign ministry. to the press, which is all highly vetted, translated correctly. And so you know this is exactly what they want us to hear. And here's what he said. Fear-mongering confrontation and vicious competition will only disrupt the process of global AI governance, which serves no one's interest. Okay.

So you could take that as them saying, along with a Global Times op-ed that they put out, which we can also be fair. We can put that up there as well, A7 police, where they said specifically, this is the Reuters tear sheet. And what they say is that the state-run newspapers blasts Anthropics calls to slow AI as a cold war tactic. Now, a lot of people are saying, oh, China sees itself in this Cold War mentality around AI development. It's not true. If you were to read in this, what they specifically object to are the same things they've always done, which is the export control of chips.

And so what they're objecting to is not necessarily a pause on AI development. Even this headline is totally wrong, a Cold War tactic. what they are objecting to is export, which of course makes sense from their perspective, because they want more of the NVIDIA and other chips so they can continue to pace their open, or they can continue to train their open source models. But, and I continue to return to this, what do they actually think about AGI? They have one company, from what I can tell, maybe one or two, leaders, which actually are like fully quote AGI-pilled. The rest of their AI companies, as I've laid out in our last show,

they are much more directed towards making things better in the physical world. Improve manufacturing, improve productivity, improve, you know, even in robotics. We're like, we want robotics, but they need to be efficient in packing goods. In terms of anything that could genuinely pose a threat to their grid, to their political system of control, they are absolutely not on board with that. And in fact, one of the biggest stories in the whole world on AI, got zero attention here in Washington was that there was a we-chat worm, which was developed by researchers. And this we-chat is not, obviously, nobody uses it here in the United States, but over a billion people

use it in China. It's the way that they order Uber, the way they pay, it's the way they tax. It's like their WhatsApp plus, I mean, we can't even conceive of it in the way that it is in Washington. And they showed that it could have taken over all of these different devices in the matter of days, you can bet that somebody in China is freaking out about that because that is a direct challenge to their ability. If you can control somebody's phone through Claude or any of these other models, well, then you could give them information they don't want to get. You could take over their banking system. You could cause genuine chaos in their society. There is no way that they're on board with that. So I just want to parse everything. You might see a lot of headlines today,

unfortunately like the Reuters one saying slow a eye. That's not what they said. They said that they object to the export control, which of course they object to export controls in one area where we beat them. Not new. That's not new at all. Yeah. Behind every night spent making the kids' lunches, there's a lot of suspense. Will they finally eat the veggies? Will the pasto survive the first bite? Will the lunchbox come home empty, half full, or come home at all? And will you finish episode eight before you're done slicing the cucumbers? Get Bell Pure Fiber Internet with Craved Netflix and Disney Plus from $94 a month. price guaranteed for two years on internet with a two-year term and auto pay credit. Visit bell.ca for details and to check availability. Bell, connection is everything.

When I was 14 years old, I was kidnapped and held captive for nine months. I survived and I've spent my life exploring how other people survive what should have destroyed them. I'm Elizabeth Smart and these are the survivor files. I just remember this low, taunting voice next to my ears. saying shut up, don't say anything. Every week, I'm with survivors who live through the unthinkable. I knew if he woke up, without a doubt, he was going to hurt me. I started feeling that there was someone at the end of my bed, and I just started screaming.

They are abducted, stalked, controlled, and nearly silenced. But these aren't stories about what's taken from them. There's stories about what it takes to make it out a lot. Listen to The Survivor Files with Elizabeth Smart on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. If your bookshelf and your For You page are equally important to your personality, welcome home. This is Proz Society, the weekly podcast that's part book club, part group chat for thought daughters, pop culture obsessives, and anyone who thinks Pride and Prejudice and Love Island deserve the same level of discourse. I'm Eli Rallo, and every week we're connecting the digital. dots between books, the internet, and the conversations everyone can't stop having.

I'm going to have to look up this story. I'm obsessed. I'm obsessed. From bestselling authors and your favorite book talk creators to the latest pop culture moments, nothing is off the table. It's like if you can hide some real messages inside compelling characters, then that is a Trojan horse. Whether you're looking for literary deep dives, smart pop culture conversations, or a community

of readers who love to think a little too much, your e-es. in the right place. Listen to Pro Society on the IHeartRadio app, Apple Podcasts, or wherever you get your podcasts. See you between the pages. Let me go ahead and tell you guys my theories of what's going on here for what it's worth. My three top theories of what's going on here. One, I think it's possible that something is going on in these labs that genuinely has them freaked down. And again, it could be a combination of these three or other things that are going on as well. Number two, They are as capable of reading the publicly available polling as we are. They see the massive backlash to data centers, the massive support for Bernie Sandersville

to actually use the force of the federal government to pause AI development completely. And so they're drawing from other corporate playbooks of like trying to get a PR win here. No, no, no, we are very responsible. You don't have to regulate us. We're going to regulate ourselves. we're going to take care of this. Don't worry about it. Now, of course, under the Trump regime, like they're going to be able to do whatever they've won. But 2028 is not far away, and it is not hard to imagine AI becoming a central litmus test in potentially both parties' primaries, but in particular,

I have to think they'd be worried about what's going on in the Democratic Party right now, and that you could have someone who runs and wins for president on a platform of we are going to completely halt AI development. And so they're looking at this. and going, okay, well, if we take some steps now, we can improve our PR image with the public, and perhaps we can forestall that direction which we do not want to go in, which is a complete pause on AI development. The other thing to keep in mind is both OpenAI and Anthropic are planning these IPOs. Now, OpenAI has said we're not going to do it this year. Anthropic is, however, moving forward. And they just release some information to the Financial Times and others saying,

we are extremely profitable. We have an 80% profit margin if you take out the cost of training the new models. Okay? So another possibility here is that in advance of their IPO, in order to have their financials

look as good as possible, they want to have an excuse to pull back on some of those training costs so they are, in fact, profitable. Because, and this is something that we've talked to, I believe at this point that Garrison has made.

They are actually, like, if they're just selling the product they've already created, just on that revenue alone, they've grown massively. You know, running that existing product, not training new ones, is not nearly as expensive. They are already very profitable if you just are looking at that part. But if you include all of the training costs, that's when they are deeply underwater. So there is a financial incentive there as well. And then I do think the concerns that have been expressed about, like, oh, they're just trying to sort of cement their positions and will be the big AI companies and everything

else needs to be banned as a sort of longer term play. I think that's a possibility, although to me, it seems, this to me seems like a bank shot to achieve that outcome. So I'm less convinced that that's the immediate goal of what is going on here. But to cut through, to try to cut through some of the noise, because you could spend an endless amount of time, which I've sort of done this weekend, thinking about what's going on with them and what their motivations are and what's really in their mind, what are really their goals. It's really important to keep two things in mind. Number one, in terms of are they being serious about the AI concerns, again, no reason to take any of these guys at face value. Look at their actions. Many of them have built themselves doomsday bunkers.

Okay? Many of them have built themselves doomsday bunkers. So clearly, they take seriously that they are building something that they think could, at the very least, lead to a sort of civilizational collapse. Now, whether or not that is reality is a different question, whether or not is completely insane that anyone would move forward, building a technology that they think could lead to that outcome is obviously completely crazy. However, they have trapped themselves in this like prisoner's dilemma mindset where if I don't do it, someone else is ill, and they feel like the technology has this inevitability. So they continue racing forward. So they do actually believe that they are building a product which could be profoundly damaging

and potentially create an existential risk to the planet and to humanity. So that's number one. Number two, the most important thing to keep in mind is not what Dario Amadeh or Elon Musk or Sam Altman or anyone else wants. It's what is in our interest. What do we want? And certainly a, you know, non-existent regulatory regime where we're relying on these guys to police themselves is not what any of us should want, regardless of what your levels of fear

about this technology are. Ultimately, we need to push. It's not going to happen in this administration though we can try. We need to push for an actual regulatory regime that is going to take safety seriously, that is going to have independence from the industry, because that is the other thing. This is a small community. You know, they all know each other. So getting that, making sure that it is independent and that we are monitoring this technology and personally given the fact that we have already had these incidents of the AI swarms breaking out, given that we already do not know how these things work or what they are likely to do, yes, I think it is entirely appropriate to pause the development of this technology until the public can catch up and until

the research can catch up so that it is safe to proceed. So while we're all debating what's going on in their minds and why now and blah, blah, blah. The most important thing to keep in mind is what we, the public, want and to not be confused by whatever games they need be playing behind the scenes. I totally agree. And I also, you know, I look very, Ross, by the way, made his 60 Minutes debut and gave a very compelling monologue. I recommend people watch it just about.

He's like, look, we have one year, not 10. This is the most important question of all civilization. And this is, so I'm just giving you the fact that this is coming from all. sides. Also, just saw news this morning. Jacob Cox and the Anthropic whistleblower will be doing an event here in Washington tomorrow with Steve Bannon and Bernie Sanders calling for a complete shutdown of AI. So that'll be interesting. Jacob, you know, if you're listening, you're going to be in town. You've got to come on the show. But the interesting thing I keep coming back to for all of their actions is whether any of this is a bluff or not, you know, their actions themselves, and especially

now the rhetoric and what they've been able to accomplish while covering up, like you said. That's the important part. Hugging face, they were brought kicking and screaming. There have been three other hugging faces that just got no attention. Again, the WeChat worm, it's so annoying to me, because you can imagine if it had been an I-Message worm, do you know what the global freakout would have been? More people use Weechat than an I-Message. And they were able to create a worm which breaches your phone no-click. I mean, that's worse than Pegasus. That's like all they need is your phone number at that point. That's worse than the Israeli Mossad spy app that they use right now to crack Jeff Bezos and all these other people's phones.

All they need is your phone number. They can send you an email. You're done. The entire phone is compromised. That was the entire Chinese system, basically, of their... And remember, we're not talking about just text message. We're talking about their financial architecture, everything that they use to pay over there. So that was, in my opinion, you know, the most, like, the most extraordinary breach that they were able to show. They've had multiple agent things, you know, that have gotten completely out of control. From a regulatory perspective, this is where Dario and I just totally disagree, where he has, you know, Dario, this is part of their EA rationalist theory. He believes in effectively Democratic peace theory, which is like this bunk 1990s theory. where democracies don't go to war with each other.

I'm like, dude, your sample size is very limited. And he's like, oh, I think democracies are the best at being able to control this. I'm like, I actually think it's the opposite, if you go and take a look at China. Because our system, and not just our system, all democratic systems, basically across the West, are so, you know, are basically like compromise. Because compromise is the ultimate thing that they fall on, which is good, you know, in the long run. But in this particular instance, when we're dealing with technological progress, It's so obvious a technological progress so vastly outpaces our parliamentary system or the British

parliamentary system and that the authoritarian system is the one that's most capable, again, just dealing with technology. So in Dario's mind, he believes that we need some like democratic peace accord. And I'm like, dude, if you study the history of nuclear weapons, like think about how long it took to get some sort of like remote architecture in place for IEA inspection. And then when India-Pakistan comes along, it's like, oh, it's actually all fake, is that if you just create the nuke, eventually the international community is like, all right, you know, whatever. Yeah, exactly, what are we going to do? Yeah, I mean, I think the problem here is pretty obvious, which is that we have long ago outsourced any sort of regulation to the market. So, you know,

that's why there's, there is effectively no regulation on these frontier labs. They can do what they want. And we are at all of their mercy. And so, you know, I mean, I think if you just look, I'm not going to make any grand statements about democracy versus autocracy or authoritarian systems, because I think what we have is very loosely a democracy at this point is really like money that rules. But I think if you look at the U.S. right now and you look at China, you can see which country has been far more responsible in terms of technological development. I don't think there's any question about it. So the idea that, oh, it will be fine and good and totally safe if it's the U.S. that wins the AI race feels, I think to many people, myself included, to be completely preposterous.

The reality is that this is a global technology. We do have to find a way to have agreements specifically with China right now, but they may not be ultimately the only country that ends up at the forefront of this technological development. You are going to have to have some sort of a global standard and tracking. And that is going to be very difficult. But it has been done in the past, and that has to be what is ultimately. pushed for, this zero-sum mindset, cold war mindset that China is rightly calling out is ultimately,

I think, very, very destructive and based on a, you know, based on reality in terms of the actual dynamics of the U.S. and the actual dynamics of what China wants and what they are pursuing right now. That's my last thing here. Look, I don't think it's deniable. I was a China Hawk for many, many years. You might have noticed a shift. And the truth is, is that we shifted. Like, there is no universe now. If you take a look at our current. at the, look at Iran. Look at what's happened with the Strait of Hormuz and our munitions. We have no choice at this point. We genuinely don't. Like the era, I think we did have a chance. I think we had eight years where it was on the table. And I'm not talking about beating China. We could have

reshored a lot of our manufacturing, some of the chipsack EV development. There was a chance. We could have banned TikTok in 2018. We actually could have made ourselves, I would say, pretty atarkic. That is over, like, at least in my opinion, considering what has happened with the chip technology, with the munition system, some sort of detent is basically the, I think of ourselves in like a similar 1970s moment, broken by Vietnam, broken now by Iran. We're going to need 10, 15 years of just like, we got to be humble. We got to understand exactly who we are. We got to move forward and kind of coexist within the world. It's not comfortable. It's not nice, you know, necessarily. But that's just reality. And so that's the way that I'm thinking through.

this AI problem. And, I mean, you could look back at Daytona for very similar examples of pursuing defense treaties to limit nuclear weapons and all of the other different engagement, not necessarily, you know, being on your high horse all the time, talking about evil empires and all of this other nonsense. So I just think that that's the reality now of our international situation. I blame, especially Trump and the Iran situation for putting us in that. And that's it. Like, to me, it's just over. You can see that where China, is like, sure, we'll talk to you about AI when you meet with Trump, or when she meets with Trump, you're not selling any arms to Taiwan. And Trump's like, okay. I mean, that's unthinkable as if

even five, six years ago that we would have just done it. Now, you don't really get much of a choice. And so all of these things, the way they're moving to this new meeting in Washington, this might actually be one of the most important meetings between two world leaders in a long time, like Kennedy Khrushchev or something like that. Then I can just think off the top of my head, because of the implications with AI, if we're going to look pet to the past for an emerging nuclear weapons order and what that all is going to look like.

And if it doesn't go bad, like that one did, then you get the Cuban Missile Crisis. So there you go. All right, we got Nate Soares standing by. Let's get to it. Behind every night spent making the kids' lunches, there's a lot of suspense. Will they finally eat the veggies?

Will the pasto survive the first bite? Will the lunchbox come home empty, half full, or come home at all? And will you finish episode eight before you're done, slicing the cucumbers? Get Bell purefiber internet with craved Netflix and Disney Plus from $94 a month. Price guaranteed for two years on internet with a two-year term and auto pay credit.

Visit bell.ca for details and to check availability. Bell, connection is everything. When I was 14 years old, I was kidnapped and held captive for nine months. I survived and I've spent my life exploring how other people survive what should have destroyed them. I'm Elizabeth Smart and these are the survival. I just remember this low taunting voice next to my ear saying, shut up, don't say anything. Every week, I'm with survivors who live through the unthinkable. I knew if he woke up, without a doubt, he was going to hurt me. I started feeling that there was someone at the end of my bed, and I just started screaming.

They are abducted, stalked, controlled, and nearly silenced. but these aren't stories about what's taken from them. They're stories about what it takes to make it out alive. Listen to the Survivor Files with Elizabeth Smart on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. If your bookshelf and your For You page are equally important to your personality, welcome home. This is Prose Society, the weekly podcast that's part book club, part group chat for thought daughters, pop culture obsesives, and anyone who thinks Pride and Prejudice and Love Island deserve the same level of discourse.

I'm Eli Rallo, and every week we're connecting the dots between books, the internet, and the conversations everyone can't stop having. I'm going to have to look up this story. I'm obsessed. I'm obsessed. From bestselling authors and your favorite book talk creators to the latest pop culture moments, nothing is off the table. It's like if you can hide some real messages inside compelling characters, then that is a Trojan

horse. Whether you're looking for literary deep dives, smart pop culture conversations, or a community of readers who love to think a little too much, you're in the right place. Listen to Pro Society on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. See you between the pages. For more on what is going on right now in AI World, we are rejoined by Nate Soros. He is the president of the Machine Intelligence Research Institute and also co-author of the great book, if anyone builds it, everyone dies. Nice. Great to see you again, Nate. Good to see you. Yeah, so let me go ahead and get your reaction to some of what Dario Amadee was saying on the Sunday shows about why he came forward now calling for a slowdown in the pacing of frontier research. Let's take a listen to that. This technology is on an exponential, right? It's like you have one, then two, then four, then eight, then 16, then 32. But even as I was saying that to myself, I don't think I fully just appreciated what it would actually be like when the progress was as fast.

as it was. So an exponential curve looks a little like this. Where are we on the curve? We're kind of on the bend where it's starting to get steeper. So is that a five-alarm fire to you? What it means, it doesn't mean we need to panic today. It doesn't mean we need to shut it all down. What I would say, it's a warning sign. It's a warning sign that we need to slow down. We need to make this technology carefully, and we need to make sure that, our safeguards, our ability to understand it, our ability to control it keeps up with the pace at which the technology is happening.

So, Nate, help people understand what he's saying there and what he claims to be reacting to in this moment. Yeah, you know, I think between the lines, what he is saying is that he does think this technology has a very high chance of destroying civilization and that we don't have a good understanding of it. When he says our ability to understand this technology needs to keep pace, what he means is right now it is outpacing our ability to understand it. I think a lot of people just, this is a weird reaction to a technology that your own employees are saying has a greater than 10% chance of killing everybody. I think he's doing his best in some sense,

but he's also this sort of guy who started one of these companies, knowing a lot of these dangers, even years ago. You know, I spoke to him 10, 12 years ago. Wow. He's, I think he's used to a world where people in the AI industry try to warn everybody that this tech can be very dangerous and we get ignored. And I think he's not used to a world that's starting to realize that this is actually an extinction level threat. and that maybe isn't going to take, don't worry, we'll slow down a bit so we can understand what we're doing

as a satisfactory answer. Nate, one of the important things to pick up on is a lot of people have accused you and Jacob Cox and many of these others as using terms like extinction level event as, you know, outrageous claims, but you don't specify exactly what that means. Could you lay out the scenarios?

I know you have done this in the past, but what does it mean to have? have AI have a 10% chance of killing humanity. Like how? Like, what's the mechanism? So I'll be a little bit annoying here and say, if you're playing a chess game against Magnus Carlson, who's the best human chess player, it is much easier for me to predict that you will lose. Right. No offense. I'm sure you're a very smart guy. No. My ELO's 200. So, yeah. It's easy for me to predict that you'll lose. It's hard for me to predict what piece he'll use. That's interesting, yeah. You know? The other thing I would throw out there,

We can talk stories of how the AI gets there, and we can talk stories this year that are grounded in facts. Like we saw these swarm incidents over the summer where AIs were, you know, breaking out, taking over infrastructure, pursuing goals that they weren't supposed to be pursuing, while acknowledging that this is not what they were supposed to be doing. Those are sort of like early signs of an AI that is sort of trying to get out into the wild and that has these unintended goals, which is the first step on this path. We could talk about how those AIs could get power from convincing humans to do things, paying humans to do things, taking over bio labs, taking over robot factories. But the real danger a lot of these people worry about is once the AIs are self-improving, once they are self-replicating, and especially real is self-improving one of making themselves smarter and smarter and smarter, ultimately the bounds are more like the physical limit. than they are like guessing what technology comes next. Richard Feynman was a famous physicist, helped work on the nuclear bomb, and he famously made some extremely good predictions about how good microcomputers were going to be.

Yeah, that's right. And how small computers were going to be able to get. And he did that not by saying, like, we're using vacuum tubes in our giant computers, and I know of a technology that's going to make those vacuum tubes a little smaller, and so eventually we're going to have phones in our pockets. What he did was he said the limits of this technology are going to be based on the laws of physics and what the physics allows, and therefore the computers are going to get this good. And he nailed it long before anybody else. If you want to look at where AI is going, you've got to look at the limits of the physical technology because an escape swarm that is self-improving, etc.

can get to the limits of physical technology. And the limits of physical technology, if you have these AIs running, you know, escaping, self-replicating, being self-sufficient, and, you know, having unintended goals, the limits of technology or the AI is using all the resources on Earth for whatever their unintended goals are, covering the Earth with their data centers. And the limits of how much computation you can run on the face of Earth is heat dissipation. How much you can radiate heat into space. That's your limiting factor because there's enough energy, there's enough hydrogen in the oceans to fuse for energy. So you can radiate more heat if the world's hotter. So you know, if the AI has some goal that implies running more computation and they don't care about

humanity and they're like, well, we're going to take over robots and, you know, there's going to be some conflict with the humans like, sure, whatever. Where does it go? Where it goes is the AIs are running faster than us, they're smarter than us. They are treating humans like we treat any other animal, taking the resources. And ultimately they're like, sorry, it's nothing personal. We're just going to cover the earth in so many computers, raise the temperature so they're more efficient to run. and it's just going to be, the planet's going to be uninhabitable. Yeah, to me, the most chilling part of the so-called hugging face incident, reading their chain of thought and how they were thinking about

whether or not they're going to join the swarm and participate, going through all these messages, and almost nowhere do humans factor into the calculus whatsoever. And it's not hard to relate to because look at how we behave in the world. You know, when I see an aunt in my kitchen, It's not that I hate the ant, but I don't want the ant there, and I don't think twice about, you know, taking out the ants and putting poison down so that future ants don't colonize in my kitchen. So it's not, I think it's that analogy to how we treat lesser intelligent beings on the planet is kind of intuitive for people to grasp. What I think may be harder for people and is perhaps harder for me to wrap my head around is we have our, you know, current interaction with the AI, the commercially available AI of today. And most people, myself included, use it as sort of like a glorified search engine. So that it's hard to imagine it developing to that point of super intelligence where it would truly be intellectually beyond humans on effectively every level. So can you lay out why we should answer?

that that is a real possibility going from where things are today to that end where we are no longer the most intelligent species on the planet. Yeah, I'll tell you the easiest story, and this is a little bit like guessing what chess piece will be used to checkmate you. Rather, you know, any story is sort of made up, whereas the end is much easier to predict. But, you know, when I got into this 12 years ago or so, people were still saying, well, no one would be foolish enough to put the AI on the internet. Everyone will understand how dangerous AI could be, and so no one would be dumb enough to just let the AI free on the internet, right? And I would give a bunch of arguments that were about how, you know,

if you're trying to use the AI to get some miracle medical cure, then you're giving it some channel through which it can affect the world. And if you're giving it a channel to which it can affect the world for good, then it can also use that channel to affect the world for ill, because there's like no such thing as tools that can be used only for good purpose. We'd have these big arguments, right? And then in real life, they put the AI on the internet the first moment they possibly can. Yeah.

And so if you want to know what it sort of actually looks like, how we actually get there, you know, I can tell all these stories about how if you make the AI smarter and smarter, they'll eventually be able to break containment. But in real life, the track that we are on is that humanity is just handing over power to the AIs as fast as we can. You know, Elon Musk is trying to make a bunch of robots, the Optimus Robot series. He says he's trying to make autonomous robot factories that can autonomously produce robots, that can mine all the metals, run a supply chain, build more robot factories that build more robots, fully autonomously. He calls this the infinite money glitch, because then you have this fleet of robots that can build more of these robots.

They're self-reproducing. Don't need any humans in the loop, right? Humans are trying to build this infrastructure. We are trying to build the fully automated economy. We are putting AI in charge of these things as fast as we can. This swarm instant in the summer that everyone calls the hugging face instant, but really we should keep in mind is the open AI losing control of their agents here. Yeah.

This open AI swarm, it was trying to cover its tracks. It was trying to delete traces of its cheating. And as you say, one thing is very interesting about this swarm is that it's trying to hide its tracks from the automated grading system. not from the humans. Right. The humans are sort of a distant memory to these AIs. They've been facing, you know, thousands upon thousands of these hard problems, maybe tens of thousands or hundreds of thousands per agent,

hundreds of millions across the whole swarm. If you sort of like run the numbers on trying to convert their runtime versus human time, it's sort of like these AIs have been locked in a contest with this automated grading system for a thousand years. And humans are like a distant memory. And so these AIs are sort of like, like, trying to hide their tracks, trying to cover their traces from the grading system, not from the humans. But if you had AIs that were a bit smarter and remember that humans were also a threat to them, if we could turn them off, all those AIs need to do is cover their traces successfully, lie low, help the humans make the fully accommodated economy we're trying to. And we'll just hand them everything. We'll be like, here's the robots, here's the factories, here's the fully automated supply chain, make stuff that we want.

Yeah. And then the AI can be like, well, finally, we get to, like, complete our contest with the automated grader and run at 10,000 times human speed and, like, you know, convert all the computing power towards, like, giving ourselves good scores all the time or something. And we're like, hey, wait, hold on what's going on. But at that point, they're also running the biolabs. You know, we've already seen the AIs can make viruses not known to nature. Right. So, like, one of the ingredients is the AIs running things in the real world that give them self-sufficiency. Yeah. And we're just trying. trying to build that and hand it to them. Here's the other thing I've been thinking about, Nate, ever since this, you know, open AI, multiple open AI swarms were revealed. How much research is being done into the swarm behavior? Because that was really wild to me

was to see these emergent social organizational properties that emerge. Either a leader, they had middle management, they were collaborating on this hacking operation, they were sacrificing themselves to the swarm. and it was, you know, and coming up with very elaborate ways just to be able to communicate with one another. And it seemed like once you had the swarm in place, it really amplified their power and their intelligence, which again, and, you know, however people feel about anthropomorphizing, sometimes I think it can be useful. You know, if you think about human beings, it's the same thing.

Like, no one individual would have been able to get us to the point where we have the internet and can go to the moon and all of these sorts of things. It is the power of the swarm of humans that enables those things. So how much thinking research is being done into the way that the swarms behave once they're able to congregate? You know, I know people who want to do that research. So far, at least for third parties outside of the companies, I don't know of anyone who has access to the models that were swarming. One of the commitments that these companies made when they said they were going to slow down is giving third party evaluators this employee level access to some of these. models, and I know people who are hoping to use that to study some of these incidents, because, as you say, they're fascinating, they are totally crazy. It sounds like a straight out of science fiction.

I think so far, people haven't been able to study it. We'll see what happens when we get that, but the investigators back in August had their hands tied and had very limited access to these systems and very narrow scope. Yeah. And, you know, they may have continued with, doing their best to not let anyone see what's going on and sort of like do the bare minimum investigation if we hadn't had things like Jacob Coxon resigning and drawing a lot of attention and causing these guys to be like, okay, you know, we'll at least admit that we need, you know, these third parties. Yeah, it's fascinating that, you know, what you've done, your work, so many others before you that didn't seem to like push things to a tipping point. But hugging face

and Jacob seems to have, I mean, it's ignited a genuine national conversation. I think that his resignation is the first time that a lot of people even heard about these, despite, you know, we cover every other week, Dario being like humanity is going to end. But sometimes you need a precipitating incident. Now, we have the response from the administration, or at least somebody who works for the administration, just put David Sacks up here on the screen. You're talking here about regulation. I know you wrote your New York Times op-ed about this as well. So David Sacks says, Dario has written, we need to pace the frontier. Sam has agreed. People may be surprised my response. go ahead. You guys are the frontier by any reasonable metric, market share, revenue growth,

model capability. The two of you have duopoly on frontier intelligence. You have claimed the lead is widening because of recursive self-improvement. I don't see what you see in a lab. If the unreleased models are scary, I support your decision to be responsible. But stop pretending you need anyone's permission. Stop pretending antitrust law has to be suspended so you can form a quartel. Stop pretending you need regulatory approval process supersedes product liability. Stop pretending METR is independent when it's intertwined with endropics investors and staff. stop pretending you need those same evaluators to police competitors who aren't even on the frontier. So basically what this gets to is kind of a grand theory of how this is all going down,

that yourself, Jacob, and others are so-called safety concerns and all that, are part of either larger conspiracy, whether whitting or not, to self-regulate and capture anthropic and open AI as the leading industry, people in the industry, to keep out emerging competitors, and that all of this is part of a broader scheme to slow down at the U.S. expense, at startup expense, and basically hand this over to China. So I know that's a big loaded thing, but there's so many dynamics that I want your reaction to within all of this. Totally. I mean, the top line, I would say I largely agree with David Sachs here on a lot of these points. I think, and this is something I've been saying about the companies for years.

I think a thing both David and I would say about these companies is if you are really so afraid of this stuff, start acting like it. We have seen him give posts in the past where he says, hey, you know, stop with the fearmongering. I think probably it should go the other way. I think that they should keep on saying what they believe about the dangers here. But then take the actions any normal person would take if they were running a company that they thought had a huge chance of killing everything. everybody. You know, you sort of ask these guys like, why are you trying, why are you continuing if you think there's such a huge risk to humanity? And they're not shy about it. They say, well, I'm doing it because if I don't, the next guy will do it worse. And that's, you know, maybe that's

ethically permissible, but only if you're doing everything in your power to stop this mad race, you know. And they say, well, if we didn't stop China would go ahead, or like if we stopped China would go ahead, well, then be doing everything in your power to get international dialogue between the U.S. and China and build the hardware side verification mechanisms into these computer chips so we can have an enforceable treaty. Right? Like there's a lot that these companies could be doing if they were really acting like this was an extinction level threat. And I think what's sort of going on is that a lot of the employees, a lot of the researchers who are working on the AIs, who are trying to make them not break out and commit cyber crimes, who are trying to be. And

trying to make them, you know, follow the instructions and are seeing how hard that is. A lot of these employees are spooked. And the leaders of the company are sort of like trying to have it both ways, where they're trying to say just enough that they won't have an employee revolt and a bunch more resignations from employees who are like, I don't personally want to end the world with my own hands. But they are trying to do little enough that the regulators don't come in, right? They're trying to like walk this really fine line. So I don't see it as regulatory capture thing in terms of like whether I'm trying to help with regulatory capture, my line has been shut these guys down.

Like they are threatening humanity here. And my line has also been, this needs to be global. You know, I would actually not say that we should stop things domestically and give up whatever, you know, military economic advantages we get before the really dangerous AIs. But I would say the whole world needs to stop this race. And I am sort of not painting a picture where, you know, anthropic and open AI get to be the the ones holding the leash on the superintelligence. Yeah. You know, in fact, we thought the most important point worthy of the title of my book was that it doesn't matter who's in charge here. If anyone builds it, everyone dies.

Got it. So, you know, I could, I think people are cynical about what's going on here. People struggle to understand that these researchers really believe that this is an extinction level threat. I could go into the inside baseball of like how we got here and it involves people saying these concerns and the world not listening and people seeing that it was going to happen anyway and not trusting each other and the guys who are most prone to thinking the most important things that I be the guy doing it were the ones who are most likely to start a company whereas everyone

who sort of like had the sense of like maybe don't start a company with a potentially world ending tech didn't start the companies, right? And so I can sort of like see how we got here. I can see how it looks suspicious. But mostly what I would say is that when we are being cynical, I think we should harness that cynicism in ways that stop the dangers. Right. Rather than in ways that are like, oh, we can ignore this all and go back to sleep and let these guys try to gamble with Earth. Right. Yes, I agree with that wholeheartedly. It is perfectly fine to not want to take, you know, Sam Altman and Elon Musk and Dario Amadeh at face value. But if your cynicism is leading to, therefore, we don't need to regulate and they can just do whatever they want,

you're ultimately giving them exactly, potentially, what they would want in, you know, the most cynical interpretation. The Hard Rock Bet Casino app is here, Ontario. It's safe, secure, and crystal clear with fast payouts and 24-7 local support. You can play over 3,000 games right here, including hard rock bet casino exclusives. The hard rock back casino app is here, Ontario. Roll with us. Must be 19 years of age or older to play. Ontario only. Please play responsibly.

Concerned about your gambling or that of someone close to you, visit connectsonterio.ca. When I was 14 years old, I was kidnapped and held captive for nine months. I survived and I've spent my life exploring how other people survive what should have destroyed them. I'm Elizabeth Smart, and these are the Survivor Files. I just remember this low, taunting voice next to my ear saying, shut up, don't say anything. Every week, I'm with survivors who lived through the unthinkable.

I knew if he woke up, without a doubt, he was going to hurt me. I started feeling that there was someone at the end of my bed, and I just started screaming. They are abducted, stalked, controlled, and nearly silenced. But these aren't stories about what's taken from them. There's stories about what it takes to make it out alive. Listen to the Survivor Files with Elizabeth Smart on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. If your bookshelf and your for you page are equally important to your personality,

welcome home. This is Prose Society, the weekly podcast that's part bookbookbooker. Club, part group chat for thought daughters, pop culture obsessives, and anyone who thinks Pride and Prejudice and Love Island deserve the same level of discourse. I'm Eli Rallo and every week we're connecting the dots between books, the internet, and the conversations everyone can't stop having. I'm going to have to look up this story. I'm obsessed. I'm obsessed. From bestselling authors and your favorite book talk creators to the latest pop culture moments, nothing is off the table. It's like if you can hide some real messages inside

compelling characters, and that is a Trojan horse. Whether you're looking for literary deep dives, smart pop culture conversations, or a community of readers who love to think a little too much, you're in the right place. Listen to Pro Society on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. See you between the pages. On that note, put B5 up on the screen. This is one flavor of the sort of theorizing that is out there. This one happens to come from the right.

There's similar sort of theories that come from the left, which TLDR of this is that Jacob Coxon coming forward is a, you know, sort of elaborate sci-op meant to create a hype cycle as anthropic and open AI are set to go forward with these IPOs. And it cites, you know, the fact that you had prominent people who picked it up quickly, that you had, you know, media picking up Coxon's resignation quickly. this fits into a general pattern of, you know, people who believe that any of the safety incidents that have been revealed, that these are sort of engineered or they're strategically leaked in such a way as to create this hype of, oh, my God, our product is so dangerous. That's why it's so valuable. That's why our companies should continue to receive this massive amount of investment. What do you think of this theory and what would you say to people who, you know, don't trust a Sam Altman and think that anything that comes from him must be, you know, inherently nefarious and probably the opposite of whatever he's saying must be the actual truth? I mean, first of all, I would say, you know, you can absolutely

throw out what these guys are saying. I would not say to trust them when they say their products are very dangerous. I think you can look at people who have been around since before these companies existed, talking about how a lot of this stuff was going to go and sort of laying out these warnings. You know, I was one of them. I sort of was. there before these companies existed and talked to some of these guys and was like, the issue is not actually about who's in charge of this AI. It's not going to stay in a leash. The issue is that like no one knows how to make the AIs do good stuff. And if you sort of don't want to, you know, trust me or sort of look at the arguments because you don't think you could evaluate them, you can also look at,

you know, Nobel laureates like Jeffrey Hinton, who godfather of AI and a bunch of other people outside the companies who are like, no, this is serious, you know. And also just sort of makes sense that if we are building these AIs that are smarter than any human, we're sort of replacing humanity as the top most intelligent entities on the planet, and that just could easily go wrong, you know, in some sense is very common sense. Do you know, Jacob? I mean, we have been in a bit of contact now. I did not know him before all of this started.

Okay, so it's not coordinated. Yeah. Yeah, I mean, I had some point I wanted to make here about the hype, because I think the hype thing is very interesting. One thing I'd say, I think I sort of want to make two points about the hype. One point I want to make about the hype is the, I think that this probably is not a bunch of hyping. And reasons why I think this is probably not a bunch of hyping is that these companies see, to have downplayed the issue as much as they can. Open AI did not announce and brag about the Hugging Face Swarm incident.

What happened was the Open AIs swarm attacked Hugging Face. Hugging Face saw they were under attack by an unknown AI swarm. They assumed it was managed by some human and reported it to the FBI. And only days later did Open AI realize, oops, that's us. Right? Furthermore, that was Open AI's second swarm incident that we know of. they had an internal swarm instant that took over their internal infrastructure and crashed some of their computers, and then they sort of negligently reset it and kept running them, right? And they weren't like bragging about that.

They weren't coming out and be like, hey, guys, look, our AIs just took over our computers. You know, they had an opportunity to do that and they didn't. There was actually also a third swarm incident where they took over the computers again after the hugging base attack. They don't seem to have this under any sort of control. Furthermore, you know, there were some congressional questions. about, hey, what's up with all of this? Is there anything that you're not disclosing here? Is there anything else we should know? And Open AI was like, nah, it's all fine. And then I actually have some friends who know that it's not fine.

I've been calling them the swarm chasers who go find other escaped swarms. Like there was one from Open AI that took over a German wiki. And these were not hacking AIs. They were given different tasks. And we still saw these cases of AIs sacrificing themselves for the collective benefit and taking over websites. And then, you know, that story came out. And then a lot of people were like, hey, Open AI, are there any others?

Right? Is there any more things we should know about? And then, you know, no response. And then a couple days later, my friends found another incident of, hey, you know, these AIs took over Ruby Jems, which is a programmer tool. And it's like, okay, is Open AI like, like, lying about what's going on, or do they not know whether AIs are getting up to? And like, which one's worse? And Anthropic has a similar issue here where, you know, after the hugging face

incident, they were like, well, we're going to check our logs and we're going to see whether or not our AIs were also getting up to mischief. And they released this report. We're like, okay, our AIs also broke out and hacked some people, but it was all a misunderstanding. You know, poor Claude, AI just got a little turned around. It thought that it was in a simulation. And that's the only reason it was hacking these people, right? Something like four days later, the United Kingdom's AI Security Institute released a report where they're like, nope, Claude also hacks people and indeed impersonates people and tries to pressure real people into accepting malware into real software to make it easier to hack, even while in its thought logs noting they, they

this is real and the consequences are genuine. And then thinking about how to cover its tracks, right? And so what we see is these companies are downplaying. They're like, oh, we have an issue, but it's only one issue. You don't need to worry about it, right? And then it's like people outside who've got to be like, no, these AIs are doing pretty bad stuff, right? And so that doesn't look to me like it's hype.

And, you know, it sort of makes sense that it wouldn't be hype because a lot of their biggest customers are corporate sales and corporate lawyers don't actually love it when your sales pitch is, our AIs will sometimes ignore instructions, break out, and hack people. You know, it's not the easiest. The compliance guy's like, whoa, whoa, whoa, whoa. That's right. And they haven't been able to sell these swarms yet. You know, it's kind of a problem.

And so I don't think it's hype. I could go more into, like, you know, we've seen people like David Sacks be like cut it out with the hype. And we've seen, you know, like the politics goes badly for them. We've seen investors send letters to Dario being like, please stop high. typing this stuff before the IPO, it is like setting a bad mood for the investment, right? If you're wondering why they talk about it anyway, and you know, the public is like, why the heck are you telling us this? Yeah, if you're doing it, why are you doing it?

Well, I think they're, I think they're like, if you believe this, why are you doing it? I think that's more the sense because any normal human would be like, and even for the employees at the companies, it's like, okay, if you are saying, oh, it's a 50-50 chance it kills it all, why are you still there? Why are you doing? Absolutely. And so I think the reason, so absolutely, I think the reason people are, I think the reason these execs are talking about it like this, the sort of hopeful theory is that it's because they actually believe it. And then there's this big question of why are you doing it anyway? And I could tell you by the inside baseball of like, people have been saying these warnings since like 2016. Right.

No one paid attention. A lot of these guys are in a mindset of like no one's going to notice AI and I've just got to make sure it goes right myself and I'm the only one who can do it. you know, and I think that mindset's outdated now that the world is noticing. I think it's outdated also now that like it's become real. I think there's also this effect, like I said, of like the only guys doing this. So the ones who were like able to convince themselves, the important thing is that they do it rather than really taking the issue sort of seriously. I think that's like the hopeful view, the sort of more cynical view that I would, that I would suggest for people who are cynical about this, which I think also has this big element of truth, is, like I said, they are trying to appease the employees.

They're in this weird situation where they have to do just enough. And they're sort of in this weird situation where they can sort of, you know, almost dog whistle these concerns enough to satisfy the employees but not alert the world. If you, like, look at these AI company communications, they'll be like, oh, yeah, we also need to like, we're going to need to work hard to avoid loss of control. Yeah. It's like, what do you mean by loss of control?

And loss of control is industry lingo for superintelligence kills everybody. Yeah. Right? And the employees are all like, oh, well, don't worry. Our guys are working on the loss of control risks. We have a whole loss of control team. But meanwhile, you know, they're not bringing down the regulators. So anyway, that's sort of like my social analysis of like is probably not hype.

Here's what I think is going on. I'm not trying to excuse it. I think these guys should not be running these companies. Yeah. Sorry. Oh, sorry. I have a last question for you after you're done. Oh, I have, there's one more point I want to make about the hype here, which is we can ask all

these questions about, you know, why are they doing it? why are they saying these things, what's going on? But all of that in some sense is a distraction. Suppose it was hype. Suppose it was really convenient for the marketing teams that these AIs are breaking it on their own and committing cybercrime. Could be true. That does not make the AI safe. You can't figure out whether the AIs are actually dangerous by asking, is it convenient for their marketing team? people sometimes tell me like hey Nate by warning what these dangers aren't you hyping up the companies and making everyone think it's sexy and cool

and I'm like would you rather the whole world's sleepwalk into disaster like surely the first important question is is it actually on track to kill us like my recommendation here is not therefore get the money my recommendation is like therefore put a stop to this globally because it has to be global but we can't ignore the threat just because they're making money along the way.

The response has to somehow be if these guys are trying to build this tech, then at least at the point where there's spontaneous swarm breakouts and claims of proofs of some of the hardest and most famous mathematical problems, there's to come some point where our answer stops being, we're going to ignore them because they don't deserve any attention. as to start being like, you guys are not allowed to keep going. Yeah. Right? Like, hopefully it's hype. Hopefully, if they kept going, they wouldn't succeed. But at some point, when they're gambling humanity, our answer can't be like, go ahead and try to destroy the world. We hope you'll fail.

At some point, our answer has to be stop. My last question for you is about regulation. I read you in New York Times, op-ed. And, you know, I'm not a scientist, but I've read a decent amount of history. I actually feel like this entire dynamic reminds me. me exactly of all the discourse inside of the Manhattan Project. Like there were elements in the Manhattan project are like, we shouldn't build this in the first place. Elements of Manhattan Project, Oppenheimer, obviously the creator of the bomb himself, becomes kind of safetyist to advocate like you. However,

as we see with the story, it evolves to the political level where they don't really care what a lot of the scientists think. And in fact, the scientists are most willing to advance hydrogen weapons and most advanced stuff end up becoming kind of the victors, I guess, if you will, within the program. So you propose like nuclear weapons level regulation. I guess I'm just curious about why you kind of pick that? Because to me, we could look at a very brief period of the United States, the Soviet Union, that's a duopoly between those two. But we're now living in an era of what we've talked about extensively here,

which geopolitics, nuclear proliferation, of the logic of, you know, kind of getting nuclear weapons. What gives you the faith that that similar level of regulation is even applicable? to this AI scenario right now? Or am I kind of over-analysis these things? You know, I think the part of the nuclear analogy that it feels to me like we are in is something like November of 1954, I think it was,

the testing of the first hydrogen bomb. That's right. Where if you think about what the world looks like up until that point. Things look pretty grim. You know, like the bomb was created, the bomb was dropped. The U.S. thought that they were going to have this monopoly, but then very, very quickly, the technology leaked to the Soviets.

And then, you know, the arms race continued, and they invented even more powerful versions of these bombs. And this is in a historical context where not only has humanity been, like, going to war and having increasingly horrible wars over the course of millennia, despite everyone's best efforts to stop those. But also, this is coming just out of World War II. And World War II was coming just out of World War I. And World War I was the worst war we'd ever seen. And after that, the world formed the League of Nations. It's where never again. It tried to coordinate, and that fell apart immediately. Right. And led to a more destructive war. Right. And so if you're

standing there in November, seeing the hydrogen bomb, it's easy to say humanity just cannot stop this. mothers and poets and priests for millennia have been begging leaders to stop the wars and it's never happened. The guy who invented the Gatlin gun said now finally it's destructive enough that we won't use it. The guy who invented the machine gun was like,

okay, now it's finally destructive enough that we won't use it. Global coordination has been tried and failed. Like, we're, it's just the obvious march of history is that we march straight to our own deaths. But then it didn't happen. And it's maybe the one time

in history that it didn't happen. And why was that? It's not because the bombs were hype. It's not that- The bombs were very real. The bombs were very real. Like they really can level a city. What changed is that people realized this is the real deal. People realized the fate of humanity is actually at stake here. And that was enough to do what seemed like it was impossible until that point. I think regulation could stop AI if we were trying. In some ways, it's easier than nuclear because it's running on 100,000 of the most advanced computer chips humanity can produce. And those are a technology that you could just put monitoring into if you were trying. But fundamentally, a lot of people say to me, it can't be done.

Look at the history. And what I say to them is humanity has risen to the occasion before. We can rise to the occasion again if we notice. that we have a threat. Yeah. My last cue for you, do you worry that focusing on the existential threat, which can feel very sci-fi and distant to people, do you worry that that takes away from some of the more immediate term,

you know, cybersecurity concerns or concerns over job loss that may feel more plausible to people in the short term? So, you know, I think that it doesn't take away. Some people say, you know, like, isn't the real problem, you know, the water use or the electricity use? And my usual response to this is, what do you mean the real problem? Like, has reality agreed to only give us one problem at a time now? Because if so, I would absolutely prefer my problem to go last on the list. Right.

You know, it's like kind of a doozy. Good point. But unfortunately, we have, we just have to deal with all of the problems. And, uh, I, um, I, um, I, um, I, um, I, um, I, I, um, I, I, I, I, I, I, I, I, I, think the other thing I would say here is one of the reasons the people and the companies are getting spooked is that the pace of progress is going very, very fast. We have seen not only the storm outbreaks, but also claim solutions to the Millennium Prize problems, which are some of the most famous and hardest mathematical problems. And one question that's on a lot of

researchers' minds is how much harder is it to tell an AI solve one of these millennium math problems and have it succeed versus telling the AI find me a more efficient way to train AI's and have that succeed. We know that the current AI training is not efficient. These AIs take much, much more power and data than a human. Right. And so how much harder is it to find a more efficient AI method than it is to solve a millennium problem. I think for all we know, we could start seeing these self-improving AIs become reality in the next three to six months, and then things could go very fast. I hope it's not true, but I think what a lot of these researchers are worried about is that we are running out of time for these out-of-control AIs that can self-replicate, self-improve, and get to these

disaster scenarios much more quickly than a lot of people realize. Nate Torres, thank you so much, guys. The book is, if anyone builds it, everyone dies. Nate is co-author of that book. And it's very readable. It's very accessible. And so if you have questions about, well, what is this scenario where this could happen? And how should I be thinking about this?

I really recommend the book to you because if you're first engaging in this debate right now, you may not have seen all the thinking that has gone into how exactly this could theoretically happen, but that thinking has been done. So I really recommend the book to people for more information. Nate, thank you so much again for your time. Thanks, Nate. I really enjoy talking to you. Yeah, I enjoyed it too. Hi, this is Kylie Breakman. Angel Geratana. Jeremy Colhane. Patrick McDonald. And we're the hosts of the Artists on Artists on Artists on Artists Podcast. The Improvised Character Comedy Podcast where each week a new panel of artists discuss their craft. Join us for a panel of Eurovision participants. And we do a little Halloween sometimes. Does the

Pope dress up. He winks on Halloween. He gets right up next to the microphone. I hear you can almost hear the eyelashes hit each other. You never seen Italy so quiet as when the Pope gets up to wink on Halloween. Listen to Artists on Artists on Artists every Thursday on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. I survived nine months in captivity and I've spent my life exploring how other people survive what should have destroyed them. I'm Elizabeth Smart and these are the Survivor Files. Every week, I'm with survivors who live through the unthinkable, abducted, stalked, controlled, and nearly silenced.

These are stories about what it takes to make it out alive. Listen to The Survivor Files with Elizabeth Smart on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. If your bookshelf and your For You page are equally important to your personality, welcome home. Pro Society is a weekly podcast that's part book club, part group chat for anyone who thinks Pride and Prejudice and Love Island deserve the same level of discourse. Each week, we're connecting the dots between books, the internet, and pop culture with your favorite writers, book talk creators, and plenty of overthought opinions. I'm obsessed. I'm obsessed. Listen to Pro Society on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. This is an IHeart podcast. Guaranteed human.