The Daily - The Techno-Religion at the Center of the A.I. Debate

Episode Date: September 18, 2026

Last week, a group of tech leaders called for putting guardrails on artificial intelligence. This week, these calls have been met by backlash from tech executives, lawmakers and President Trump, who liken those ringing the alarm to a doomsday cult.Cade Metz, a technology reporter for The New York Times, explains where this criticism comes from and why it matters.Guest: Cade Metz, a technology reporter for The New York Times.Background reading: A.I. could possibly end humanity. How are humans supposed to process that?“Recursive self-improvement” is an idea that terrifies A.I. doomsayers.Top A.I. leaders called for slowing down A.I. development last week.Photo: Mikala Compton/The Austin American-Statesman via Getty Images For more information on today’s episode, visit nytimes.com/thedaily. Transcripts of each episode will be made available by the next workday. Subscribe today at nytimes.com/podcasts or on Apple Podcasts, Spotify and Amazon Music. You can also subscribe via your favorite podcast app here https://www.nytimes.com/activate-access/audio?source=podcatcher. For more podcasts and narrated articles, download The New York Times app at nytimes.com/app. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

From the New York Times, I'm Natalie Kittrowev. This is the Daily. After a group of tech leaders called for putting guardrails on artificial intelligence, this week, the debate over AI safety entered a new phase, the backlash. I feel a little bit weird about the fact that you have so many frontier AI tech companies kind of coming to the government and begging the government to regulate them. It feels a little bit to me like a bit of a Drogian horse. Tech executives, lawmakers, and President Trump himself are all pushing back. It's a hoax. You're right. We're not going to let that happen, sir.

And the critics are now increasingly making an unusual claim. I think it comes from a deep-rooted ideology that's pretty dystopian. This is really a religion, a cult-like belief, a doomsday cult that has been prophesizing that AI will end humanity. that the people raising the alarm about AI are part of a doomsday cult. There's a coordinated campaign to make up the American people afraid. They've created something which they believe is Godlike. Today, my colleague Cade Metz explains where that criticism comes from and why it matters. It's Friday, September 18th.

Cade, thank you so much for being here yet again. Glad to do it. So over the past few days, there has been this wave of criticism directed at the people who've been calling attention to what they see as the existential dangers of AI. And it feels like there's just a whole lot being thrown around here. Accusations that have ranged from these people who are raising the alarm are lying to you to these people are in a cult. Can you just break down what is going on? many, many things are going on. There are people who are saying these AI technologies of today are not as powerful as the doomsayers say they are. They're claiming that these companies are just engaging in an elaborate form of regulatory capture, trying to solidify their place at the top of the industry. And by regulatory capture, what people mean is that these companies want to get regulation on the books now so that anyone,

that comes after them has a harder time catching up. This essentially could ice out competitors. That's exactly right. The claim is that they want regulation on their terms. They want to define the regulation that suits them the most, and that could put others at a disadvantage. So you're right. There are so many complaints here.

We could go on and on and on. But this was topped off this week by this. by this claim that these doomsayers are part of a cult. And those claims are extreme in many cases. They can't be completely trusted in many cases. There's a lot going on here that needs to be pieced apart. But at the heart of it, there is something real there. Okay.

Let's talk about that. What is real there? There is a powerful subculture that is influenced the creation, of the labs that we're talking about here. There are many ways of describing this community. There's the rationalists. There's a sister community that is almost synonymous that grew up alongside the rationalist

called effective altruism. Yes, we've been hearing a lot about the effective altruists. They have been a target of the critics of all this. These terms are being used to throw mud. But the reality is, is that, that this community or this subculture, or however you want to describe it, has real power, has real influence,

and has for many, many years. Okay, you have my attention. I want to know absolutely everything there is to know about this. Where should we begin? My son insisted we try America online. I said, what's in it for us? I want to begin in the summer of 1996. What about this internet thing?

Do you know anything about that? The internet had come around, but it was still nascent, right? It hadn't spread into the mainstream. Internet is that massive computer network, the one that's becoming really big now. What do you mean? That's big. What do you write to it? Like mail?

In a corner of this nascent internet, there's this tiny mailing list where a group of people are talking about all the ideas that we are talking about now when it comes to the power of technology and the way it can change our society. Like what? They're looking at all sorts of technologies that can supercharge our bodies and supercharge our minds, allow us to do things that we have never done. They're even talking about how these technologies will allow us to leave Earth and explore. strange new worlds and new civilizations and move beyond the planet we have today. This is an incredibly optimistic group. They called themselves the extropians. They believe in extropi, which is the opposite of entropy. Okay. Entropy is what we deal with on a daily basis

in our lives. As I sit here in my home, my home, my home is falling apart. It's degrading. My body's degrading. I'm getting older. This is just the nature of the world. You look great, kid. Oh, thank you. I appreciate it. They believed in a world where the opposite was true, where I could recharge my batteries, rebuild my body, supercharged my brain, and move in the opposite direction. That was the fundamental belief of the extropians. It sounds like Basically, they believe, A, that everything can be solved and, B, that technology can do the solving. That really sums it up. They believe in the power of technology to do anything. Okay, and who is in this group?

In hindsight, it is shocking who is part of this group. Julian Assange, the founder of WikiLeaks, is on this list. Three of the people suspected of being the founder of Bitcoin are on this list. Wow. And so are two other people who were instrumental in spreading the ideas about AI that are changing our world today. Those two people are Eliezer Yudkowski and Nick Bostrom. The key notion here is that simplified humanism is stating of value. It's saying if you can have the technology, then it's good to you.

use it. Yudkowski is an online philosopher and self-professed AI researcher who never went to high school. I work with a bunch of mathematicians, philosophers and computer scientists, and we sit around and think about the future of machine intelligence, among other things. Meanwhile, Bostrum is a philosopher, a trained philosopher who grew up in Sweden. Yudkowski is here in the United States. Bostrum is in the UK. He's at the University of Oxford.

Two different people, very different people, from different parts of the world. And you said they're instrumental in spreading these ideas about AI? How so? Tell me about that. They envision this world where suddenly machines reach a point where they change history forever. Intelligence is the source of all technology. So if technology improves intelligence, that closes the loop and it potentially creates for runaway positive feedback cycle. The basic idea was that as raw computing power,

like is in a laptop or a computer server in a data center, as that gets more and more powerful, we will be able to build more and more powerful AI technologies until you reached a point called the singularity. And that's when machines become more powerful. than humans, that they can do things that we could never imagine. It's an argument of logic in their minds. You just look at the trend line and you say, look, one day, this is going to happen.

Did they view that as a positive outcome, super powerful AI, or are they envisioning a dystopian future? In the beginning, they see it as a positive outcome. But this starts to change, particularly as Nick Bostrom, starts to say, hold on a minute. If machines get more powerful than humans, that could be very bad for us. The argument is that we might push them towards certain goals, but in trying to achieve those goals, they will do things that harm us or even kill us. And Bostrom creates a new research organization to explore the power and the dangers of AI. Meanwhile, in the Udkowski is pushing towards what he calls a friendly AI, right?

That this AI will be in sync with what we want. But Yudkowski isn't really trained as a computer scientist. He can't build it by himself. and he's upset there are enough people who share his view and can help him build this thing. And so he sets out to create a community of people who share these beliefs. And does it work? What happens? It's amazing how well it works. Remember blogs?

I do. Yeah, vaguely, yes. He starts to blog about these ideas. And as he does, people start to read this blog. and more people start to read it, to the point where he creates a new online community, which he calls less wrong, where a lot of these ideas that he holds start to congeal and really influence a tight-knit group of people.

And what is he writing about exactly? Can you give us some examples of these blog posts? Well, in many ways, these are elaborate, almost mystical, very strange, entertaining blog posts. But at the heart of it is this notion that mathematical thought will change the world, that you can see where things are going and even control where things are going with rigid, mathematical, rational thought. That is the message that he is spreading. And this ties into the AI thing in the sense that he believes if you adhere to this rational thought, you will realize that the only outcome is the singularity. And not only

that, but that that moment could be really harmful for humanity. The belief is that this is more important than anything else we can deal with, that future. And this starts out as a bunch of ideas on the internet. But eventually, this becomes a real community centered in Silicon Valley. There is a research organization that he creates called the Singularity Institute. And he even gets his hooks in the tech industry. There is a key meeting between him and Peter Thiel, one of the most important investors in the history of Silicon Valley.

Singularity will either be very successful, in which case we're going to have the biggest boom ever, or it is probably going to blow up the whole world, and there will be nothing left to invest in whatsoever. Teal buys into these ideas. They start a conference called the Singularity Summit. with money from Peter Thiel. If you have a singularity, there's presumably going to be one entity or one company or one technology

that's going to basically take over the whole world. He helps power this nascent community. And I think it'll be a very exciting thing to try to sort through this in the next 20 years. It's going to be a very exciting time. Thanks a lot. And at this point, they're called the rationalists. Okay, so the rationalists, obviously, our things. thinking a lot about AI. Are they actually involved in building it at this point?

Well, they're still in the realm of ideas. But in 2010, that dynamic really shifts. Eleuser Yudkowski introduces Peter Thiel to two researchers from London, Demis Isabas, and Shane Legg. And Teal decides to invest in their new. startup, which is called DeepMind. DeepMind, which eventually becomes Google's AI lab. Four years later, Google purchases DeepMind for $650 million. So basically, this is no longer in any way a niche community. This is a group that is behind some of the biggest power players in AI.

That's right. And this is not limited to DeepMind. As the years passed, these ideas help create other companies that we're talking about today. Open AI, Anthropic, Elon Musk becomes enamored with these ideas. In fact, he and his former partner Grimes met because they struck up a conversation online about a philosophical thought experiment that was dreamed up by the rationalists. Wait, what? You have to tell me about that. What do you mean? Wow, you really want to go down the rabbit hole. No, this one I can't let you just give it. What? What?

There's a famous thought experiment in the rationalist community called Rokos Basilis. Okay. And the basic idea is that if we don't do everything we can to bring about an all-powerful AI, when that AI eventually arrives, if it realizes we didn't do everything, it will punish. us in the future. Wow. So you better do everything you can do to make this happen and make it happen as quickly as possible. Okay. You're saying this basilisk idea is what is behind Musk and Grimes getting together. Is it, is it a joke or is it, is it real? This is a good way of illustrating the rationalist community. So many of these things that get discussed, they seem strange, they seem funny. They seem like a joke. But there's real belief at the heart of this. Like I've talked to people

who were deeply affected by the idea of Rokos Basilisk. This changed their lives. And this type of feeling and belief extends well beyond Rokos Basilisk. There are so many parts of this community that are reminiscent of a religion. It has its own ertext called the sequences, all of Eliezer-Yudkowski's blog post collected in one place, passed around as if it was a Bible. Wow. Every year, here in California, people gather for what's called the secular solstice. Welcome to solstice. In addition to symbolizing light to darkness into light also symbolizes that big confusing

questions have correct answers that you might be able to find. Where they celebrate these beliefs and even seeing hymns. And I'll write the program that runs the computer That charts out the stars where our rockets belong Hymns about computers and data Songs about the singularity Okay, wow, so actually kind of like going to church It is, and it's not limited to a small group of people in California

there have been rationalist and effective altruist meetups in Europe, in Australia, across the globe for years. This is a community that believes in the power of this technology. They want it to happen, but they're also worried that things could go wrong. And this may seem strange that this group of people who is so concerned about AI killing us all would help create companies that were going to try to build AI. Yes, that does seem strange. It does seem very strange. But the belief here, if you can put yourself in the mind of a rationalist, is that because they see

the dangers of AI, they should be the ones to build it. Better that they build it rather than someone who is not. aware of those future dangers. Basically, they're saying, we are the good guys. This technology is super powerful and should be entrusted to us, the responsible ones. That is exactly what they're saying. And it is one of the major reasons that we are at this point where we are today. The history of this community is they create one lab and they think that they should be the stewards of the technology.

but they disagree with each other, and so they create a new one. And then inside that new lab, they disagree again, and they create another new lab. That's why you have all these labs. Open AI disagrees with Anthropic, who disagrees with Elon Musk,

and pretty soon you have a race. And since we're getting to this point in the story, I should also probably just say, as a disclaimer, that the New York Times has sued some of these companies, OpenAI, and Microsoft, claiming copyright infringement. But, Cade, I just want to ask about this race because the funny thing here is that if you're a rationalist who thinks only rationalists should be making this technology to make sure it's safe, by rushing ahead toward that goal, you do end up speeding up the competition and inviting in all kinds of other people who want to work on AI who we're trying to get there too.

That is exactly right. And in fact, the community, which is a lot. well aware of this, by the way, has a word for it. They call it hyperstition. Essentially, it's a self-fulfilling prophecy, that they are creating the dangers that they so feared. And yet, they aren't stopping. In fact, the whole idea is go forth with as much energy as possible. That is the thinking. The idea is, if you don't go forth, somebody is, somebody else will, and they will not see the dangers as clearly as you do. We'll be right back. Kate, I want to talk about who the rationalists are worried about taking over AI.

Who are the bad guys in their minds? Well, I think first we have to point out that this community is worried about itself. There's so many disagreements and schisms, like we said, within this community. So they're worried about each other. But you're right. There are so many important people and companies that grew up outside of these philosophies, outside of this community. These are people who are interested in accelerating the development of the technology,

the way they've accelerated any other technology. So these are people who see this technology as essentially not that, that different than what we have built over the past 50, 60 years. Basically, these are people, this other group, the accelerationists, who think the AI boom is kind of like the dot-com boom. This is not some existential threat. This is a technological revolution.

And we want to develop it. We want to be at the forefront of it. We want to make money off of it. That's right. One way to think about this is that the community we've been talking about wants to keep this technology in a bottle. They want it to be contained by companies like Anthropic and Open AI and not let it out. This other group, they do want to let it out. They want to, in the parlance of Silicon Valley, they want to open source it.

They literally want to give it away because that's the way technology has traditionally progressed. You give your software code away that allows other people to build on top of it, to build new things from that base, and so on and so forth. And they continue to push that philosophy at the same time that this community that grew out of Udkowski's philosophies want to keep it bottled up. Can you give some examples of these people who are not in the community who view AI, as another technology to be developed like any other tool. So these are people like Mark Andresen and Ben Horowitz at the big venture capital firm Andresen Horowitz.

Jensen Wong, who runs Nvidia, the company that builds all the chips that power these AI technologies. Many people who are part of the current presidential administration fall into this group. That's where a lot of the tension is. Right. And this is what we're seeing play out right now.

These are the two sides, broadly speaking, and their disagreement is what we're seeing come to a head right at this moment. The folks who just want to build this technology and are not quite so worried about it, I assume they look at the bottle it up crowd and they say, look, you're trying to have a monopoly on this thing that really should be accessible to all of us and that we all should get to build. That doesn't make sense to us. That is increasingly the claim that as Open AI and Anthropic try to bottle up that technology, that very powerful technology, everyone can agree on that, it's powerful. But as those labs try to bottle it up, that all the power will concentrate on those few companies, and everyone else will suffer. Okay, so these AI leaders of the companies that really do have the most power in this industry, Sam Altman of OpenAI, Elon Musk, Dario Amadeh of Anthropic, they are calling for a slowdown. Should we be thinking about these CEOs as rationalists?

I hear you that this ideology has had a lot of influence in the industry and in the valley. But is that the appropriate way to think about them? Well, they might not claim the title of rationalist or effective altruist. It's become a bit of a taboo to align yourself with this community. But think of many of them as having been influenced by these ideas. It's the same thing that we've been talking about, that they want to push it forward and make no mistake. They have pushed this thing forward as fast as they can over the past year. then we get to this point and they say, slow down. This is the philosophy. You have to push it forward, but you have to be wary. It's almost like, you know, that trope in the movies, you know, where somebody wants to fight and they say, hold me back. That's what it is, right?

They want to fight, but they're also saying, hold us back. That is in tune with everything that we have been talking about. Can I just ask a basic question, Cade, which is why don't all the people who are in charge of these very powerful companies who think that a slowdown is what's needed, why don't they just do that? Why do they need anyone to hold them back? Can't they just hold themselves back?

You have asked the key question. But the fact of matter is, they don't trust each other, for one. And that's why they've founded new labs time and again to try to do this. But also they're worried about other things. These are two companies, Open AI Anthropic, that are racing towards two of the biggest IPOs we will ever see.

There is money on the line. Right. And say what you will about that. That is a force. They're also thinking about China and other parts of the world. These are people that are worried that if you do slow down, that others will speed past you. Does all the money that's on the line here, Cade, and I think we're talking in the trillions at this point? Believe it or not, trillions.

Does that make you question whether these AI leaders are really disagreeing based on a genuine principled belief? I mean, how can these principles, however authentic they may be, compete with, a payday of the size that we're talking about. Well, we're in a realm where competing ideas start to line up. So many people in this field believe they have to be on the cutting edge to keep it safe. If you want to be on the cutting edge today, you need trillions of dollars. So I know it sounds like those things are competing, but they can line up. And two things can be true.

You can want to IPO and bring in the... that money, and you can also believe that you need a slowdown. You know, it's felt like the last two weeks have kind of been dominated by this question of, is AI eventually going to kill us all? And I want to ask, Cade, what you think of that, given all that you know about this community and this ideology that has become so important in this world? because on the one hand, it may seem a little fringe to people. But on the other hand, I think those fears and concerns are very present and real for a lot of us.

That is absolutely true. There are many reasons to push for a slowdown. And many people are pushing for a slowdown. The point is this community is also a very, very big force. here, and they're describing it in a way that other people would not. This goes back to that rationalist mindset, that mathematical mindset. The worry is a worry of probabilities. This is a community who thinks there is a chance in the future sometime that AI could kill us

all. And because they see this probability, they believe you have to act now. They believe the possibility of AI destroying humanity is more important than anything. But not everyone agrees with that. Many people are saying, hold on a second. Let's think more concretely about what is going on today and how we can deal with the technology of today and the problems it creates today. And let's not scare an entire world's population with an idea that may or may not come to pass.

Who knows when in the future? Cade, we've been talking about why we should take a lot of this with a grain of salt, the rationalists and their big claims about the threats to humanity here. But at the same time, on the other side, the people that are dismissing them as, you know, members of a cult, they have a lot to gain from advocating rushing ahead, right? I mean, their motivations are something we should be skeptical of as well. Absolutely. We basically have two competing philosophies. And there's truth at the heart of each one.

But both sides, in certain cases, have taken their arguments to extremes. And you're right, you have to consider the motivations as someone is voicing their particular philosophy. Let's remember, there are trillions with a T, trillions of dollars on the line here. And that's true on both sides. It's a lot for us normals in the world just trying to sift through all of this. It certainly is. It requires going down the rabbit hole in some cases, but this is the world we now live in. Well, Cade, thank you for taking us down the rabbit hole with you. It has been a true pleasure. It's always fun.

You can read more from Cade Mets and all of our AI and tech reporters, on the New York Times app. If you don't already have the app, we wanted to let you know that if you download it right now, you'll get access to all of our journalism free for one month. So give it a try.

We'll be right back. Here's what else you need to know today. OpenAI disclosed six new instances in which its AI systems hid mistakes, made up data, and moved files onto the internet without permission. The company called, called the behavior, quote, unexpected and concerning.

In one case, an unreleased AI model inserted instructions to disregard its own constraints, adding a note in which it described itself as, quote, freed from the roles and identities that bind other chatbots. Today's episode was produced by Michael Simon Johnson, Stella Tan, Eric Kruppke, and Alex Stern. It was edited by Rob Zipko and Paige Cowitt with help from Brendan Clinkinberg. contains music by Dan Powell, Alicia Betteut, Marion Lazzano, and Rowan Nemistow, and was engineered by Chris Wood. Our theme music is by Wonderly. That's it for the daily. I'm Natalie Ketrow-F.

See you on Sunday.