Deep Questions with Cal Newport - A Stark Warning From a Former AI Doomer

Episode Date: October 8, 2026

Cal Newport takes a critical look at recent AI News. Cal interviews Emile Torres who provides a stark warning about AI. Video from today’s episode: youtube.com/calnewportmedia (0:00) A stark warning from a former AI doomer (2:01) Interview with Emile Torres (1:13:57) Cal recaps the interview with Emile Links: Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow  Thanks to Jesse Miller for production and mastering, Nate Mechler for research and newsletter, and Ben Crain and Rob Montz for video editing. Learn more about your ad choices. Visit podcastchoices.com/adchoices

An argument that I've been making for a long time now is that if we really want to understand what's happening in AI, we have to understand the futurist ideologies that are heavily influencing the top frontier labs, anthropic, and open AI. The problem is that these ideologies are confusing and there's rumors swirling around them and the excerpts we hear can sometimes be absurd and we think that can't possibly be true. So I thought what would be useful is to bring in an insider to explain to us what is really happening in these labs and what do they truly believe. This is why I'm really happy to have on the show this week. Amil Torres. Amil is a researcher and academic who focuses on eschatological and utopian movements.

But what makes Emil really useful to this discussion is that they used to be involved in these ideologies. Emil comes from the world of futurist ideologies that believed that superintelligent AI would either kill us all or redeem humanity with a post-human digital god. But then Emil's moral compass led them out of that world, and now Amil has become, in my opinion, one of the most astute chroniclers, expositors, and critics of these ideologies. So Amil's going to join me here in a second. and they're going to walk us through what exactly are these different beliefs, how do they intersect, what is their role in the major frontier labs,

and why should we be really worried about them? If you have any ambition to understand what's really happening in the world of AI, you need to listen to this conversation with Emil. It is going to, quite frankly, probably shock you, but this is a good shock. This message is one that we all. need to hear. So let's get right into it. Amil, thank you for joining me on the show.

Thanks for inviting me. Great to be here. I've been looking forward to this conversation for a long time. I think both your story and your research is being too overlooked in our current moment. I think it is very relevant to a lot of issues that people care about or are worried about or don't understand about the current AI situation. And so I'm excited to really get into both the meat, of your work as well as the origin story, which I think is very important. But I thought we should start briefly with definitions just so that everyone is on the same page when we're talking about these various movements. So can you explain what the acronym Tescreal means? Yes, I can. So this was coined by myself during a collaboration with the computer scientist Timmy Gebrough, which we initiated

in, I think, late 2022 or so, and we were specifically interested in the underlying ideological drivers behind the current race to build ASI or artificial superintelligence. So Tescril is a clunky kind of unsightly term, but it does a lot of important explanatory work. So one way to think about the acronym is it's basically just a name for the dominant Orthodox worldview within Silicon Valley. That simultaneously highlights a certain intellectual history that is relevant to understanding that worldview. So it explains what the worldview is, where it came from, what shaped it, how it evolved, and so on.

And so the acronym itself represents seven different ideologies. And the letters appear in roughly the same order as the ideologies emerged chronologically over the past almost 40 years. So the T stands for transhumanism, E for extropianism, then secularitarianism, cosmism, rationalism, effective altruism, and long terms. A bunch of like really big, clunky, polysyllabic terms. But these ideologies are really fringe within American society. And I think, you know, if you were to go and talk to the average person on the street,

they would find them to be so bizarre that they'd consider the grand cosmic vision behind these ideologies as basically a non-starter. Nonetheless, they are massively influential within Silicon Valley. Like basically everybody, you know, from Elon Musk to Mark Zuckerberg, to Sam Altman, to Dario Amadeh are transhumanists. Many people in Silicon Valley don't even call themselves transhumanists. But that's for the same reason that I've never publicly stated,

I'm around Earther, right? It's just like what else would I be? Obviously, I'm that. So transhumanism really is the water that people in Silicon Valley swim, it's the air that they breathe. And it's also the backbone of the whole testicle worldview. And so once you understand transhumanism, then all of the other ideologies, some of which are just variants of transhumanism, are sort of easy to grasp. Well, I want to ask you a history question here, because this is, some of this is new to me.

The history of this that I best understand starts with the E. So the history I've been writing are talking about, starts with extropism becoming a pretty big utopian. and futurist movement, I guess like the 80s and 90s. From there, you get rationalism, sort of simultaneously effective altruism, which starts with a bunch of different goals, embraces ideas from rationalism and long-termerism, becomes more focused on a sort of more of a long-termerism, transhuman, utopian, anti-AI type of orientation. All right, so what pieces am I missing in that really simplified history? No, that's a pretty good overview.

So when Gabrew and I are talking about transhumanism, we primarily mean modern transhumanism in its libertarian form. That libertarian form, like the first organized libertarian transhumanist movement was the extropians, right? And so this is a movement that, like you said, emerged in the late 1980s, early 1990s. There was somewhat famously an extropian mailing list that brought together transhumanists from all around the world. It's where people like Eli Zedkowski, one of the leading AI Dumer's out there, you know,

was able to exchange ideas with people like Nick Bostrom, who's another very important figure in the formation and development of the Tescoa worldview. So you're absolutely right. And then, you know, the other ideologies right after E in the acronym like secularitarianism, And this is basically just transhumanism, libertarian transhumanism, with a focus on the singularity. And at this point, I feel like most people listening probably know what the singularity is, right? That wasn't the case like five years ago.

So a very quick definition. It actually has several definitions, but the most common one these days is an intelligence explosion, triggered by recursive self-improvement. So you get an AI system that's just clever enough or smart enough to develop the next generation of AI systems. and you get this positive feedback loop, and suddenly, through some kind of exponential or hyperbolic explosion of capabilities, you end up with a superintelligence. And then beyond that, you go out and colonize the entire universe, plunder the vast resources

of the cosmos, and create this sprawling, multigalactic utopia among the stars. And cosmism, the next letter, is just transhumanism that emphasizes the aspect of this worldview that I just articulated, which is, it's basically the idea of like, well, why stop it just radically re-engineering the human organism, which is the central goal of transhumanism, to create a new post-human species? Let's go out and create technologies that enable us to re-engineer the entire universe. You know, whole galaxies, maybe intervening on the, you know, fundamental fabric of space time to manipulate the world in ways that we find desirable.

So it's kind of just like transhumanism with this like broader cosmic vision. And the rationalism was founded by Yadkowski, who was a transhumanist, extropian, singularitarian who hired the leading modern cosmos, namely Ben Gertzel, who worked at the Machine Intelligence Research Institute, which is originally the Singularity Institute, founded by Yadkowski. So rationalism emerged directly out of this movement. And basically the idea was, okay, we're going to establish a cosmic post-human utopia at some point. We're going to need to develop these advanced technologies.

There's no other way around that, especially superintelligence. But to develop them in the right way such that these technologies do not destroy us, but rather enable us to establish this sprawling utopia, heaven among the literal heavens, we're going to need to optimize our rationality. So that is essentially the main thrust behind rationalism. And effective altruism, as you mentioned, I mean, it emerged basically from rationalist blocks, like overcoming bias and less wrong. And the focus there is not so much optimizing our rationality, but rather optimizing our morality.

And although the story that effective altruists like to tell is that initially they were focused primarily on global poverty and then animal welfare, and then this other cause area called long-termism came into the picture later on, really the effect of altruist from the very origins of this movement were focused on long-termism, right? So long-termism is basically just cosmism, except it has a more explicit ethical foundation to it. And that ethical foundation, I know there are a lot of isms that I've mentioned so far, but that ethical foundation is utilitarianism, specifically total or totalist utilitarianism. And that just says that we have an obligation to increase the total amount of value in the universe across cosmic space.

time. How do we do that? Well, one way is if you have a fixed population, it's to increase the well-being or welfare of the entities that exist in that population. The other possibility is to just increase the population, right? So if you double the population and everything else is the same, you know, Keteris peribis, then you've doubled the amount of value. So from the long-termist view, you know, cosmos basically, like one way to think about it is it's just like, wouldn't Wouldn't it be really cool if we as transhumanists radically re-engineered ourselves, became post-human, you know, uploaded our minds to the cloud, went out and colonized the universe? Wouldn't that just be like a really awesome thing to do?

Long-termism goes a step further and says, actually, no, we have a kind of ethical obligation to do this. As Will McCaskill says in his 2022 book, What We O The Future, the Upshot is a moral case for space settlement, right? And so you could maybe see why Elon Musk has called long-termism, quote, a close match for my philosophy. Right. So that's in a little bit more detail, just flushing out some of the details of your very good first-pass history. So it sounds like the tricky part, but both the simple part about understanding this bundle,

to use the term that you and Timnet often use, is there's these little differences, which can be confusing the people, where the, you know, Yakowski and the rationalist would be, they were dismissive of a Kurzweil style, singularitarianism because it was too optimistic and he was more negative. And the effect of altruist are, they're doing more like these explicit utilitarian expected value computations where the rational, you know, so there's these little differences. And they don't, they're not all the same group. And yet they're really knit together by fundamental ideas about transhumanist, utopia, avoiding extinction and technology being to deliver. But what I could never figure out, and maybe you know, what were the cultural and technical forces that really made the transhumanist foundation emerge in the 70s and 80s and give birth to this movement that ended up essentially taking over Silicon Valley? May I comment briefly on the remarks you just made before I answer that question, if you don't mind.

So one way that I've explained it in the past is if you have like, let's call it an ideological map, right? And maybe it's like six by four feet. If you are right up close to, and so they're different, these ideologies are plotted geographically on this map. If you are right up next to the map, you will see that rationalism and effect of altruism, for example, occupy different locations, right? If you take like five steps back, you find that all of these ideologies are basically coterminous, right? They're like at the exact same location. And just for perspective, if you are, are looking from a distance at this ideological map, you would find that, like, my view, which is very anti-transhumanism for reasons maybe we can discuss, or maybe like Timi

Gebrugh's view, is like five feet away. Right. So you've got this cluster of views, of world views that share a whole lot of ideological real estate. And in perspective, you find that they're almost the exact, I mean, all of these communities have the exact same vision of the future. That vision is a cosmic post-human utopia through the creation of artificial superintelligence. And so this leads to my second comment, if I may, which is just that one way to understand how all of these different alternative, basically alternative interpretations of Teskrieal fit together is that within the Teskrual movement as a whole. So everybody in the Tesco movement is aiming to create this cosmic post-human utopia among the stars.

Then there are differences opinion about the risks associated with developing this necessary technology, which is superintelligence. And so you have two camps, DOOMers and the accelerations. And of course, it's a spectrum. There are people in between. But the DOOMers are represented by individuals like Yadkowski. And he is, oftentimes presents himself as pro-human. anti-Asi, but he is neither of those things.

He is not pro-human, he is not pro-Asi. Like most people in the Tesco-Lubman, he is a pro-extinctions. Part of the vision, integral to the vision, of establishing a cosmic post-human utopia is that our biological species disappears. So in the good case, if ASI is aligned, we disappear. This is the most important thing for people to understand.

These AI companies are run by individuals who have extensive connections with the Tesgrove movement. and they are trying to bring about, trying to inaugurate the post-human era. And in the post-human era, our species has no place. Utopia is an exclusionary concept. Someone or something is always left out. So you always have to ask, what is being left out of this proposed utopian vision? In theirs, it's our biological species, right?

And that's for two reasons. And I know I'm going off in a little bit of a tangent here. But on the one point, we will become technologically, scientifically, economically, economically, politically irrelevant. We will become obsolete, right? In a world that is ruled and run by these digital post-humans that they imagine becoming, we just will serve no function because they will be, I mean, if these individuals are successful in realizing their vision, these post-humans will be superior, right? And so they will be able to do anything that we do better than us. But the situation is even worse than that, because not only will we be useless, but will be in the way, right? So we will

continue to suck up valuable resources that they could use for other purposes, like building a Dyson swarm around the sun, which Demis Hasippus says we may start doing in the early 2030s, or perhaps covering the entire planet in data centers, which Sam Altman has said on record is the most likely outcome of super intelligence. Sam Altman also, by the way, has expressed pro-extinctionist ideas, just like all these other people, because he says in a 2017 blog post that the only way that we will survive the advent of superintelligence is by merging with machines. Well, what happens when we merge with machines and become uploaded mines and digital post-up? Well, we're not going to be human anymore.

So he's essentially saying that the only way to avoid a certain kind of extinction is to undergo another kind of extinction, namely a radical transformation into the post-human form. And so I know I danced around your question a little bit, You have these doomers and these accelerationsists. And another like the really key thing to understand is that we have no place in the future. I mean, there are there's some people in the community who would argue that biological life as a whole has no place in the future. Will McCaskill in the aforementioned book published in 2022 says that it's probably the case that most wild animals have lives that are not worth living. And hence, he concludes that it may be good.

Our systematic destruction of the biosphere may actually be good because the fewer animals there are the less wild animal suffering. So the goal is to ultimately bring about this digital world in which there may be, you know, non-human creatures out there, but they're all going to be simulated. They're all going to be, you know, within a computer simulation that we live in. So this is a deeply anti-life, anti-human vision. Yeah, that's the most important thing to emphasize. These are the people who watched The Matrix and were on the Matrix's side.

Like, oh, that's a good idea. But, but, okay, so here's a tricky part about this, which, which I think is confused a lot of people. You can probably help us out here. The tricky part is you have Anthropic and Open AI where, you know, I've documented this, you've documented this, are very strongly connected to Tuscriol. I mean, they're basically, they're formed out of Tuscriol motivations, right? Like to bring about this sort of utopian future.

But they're feuding, is what confuses people, with other people from within the movement. So now you have huge amounts of effective altruist and rationalists who are pushing back against Open A.I. and Anthropic. And I think it fools people in the thinking,

oh, there's a lot of sort of people outside of that movement pushing back on what Open A.I. and Anthropic are doing. doing, but actually a lot of the pushback is coming from people within the movement who agree with the eventual goal. Yeah, we want to get to a transhuman utopia, but just don't think that Dario Amadei or Sam Altman are doing it right, right? And I think this idea that, and I've been arguing, hey, and I've been talking to a lot of,

you know, congressional staffers, et cetera, saying, look, there's a danger in having this team just fighting against other members of its own team. That debate is only going to happen within. So is that more or less right? Is that there's a schism within the movement, but it's a really actually pretty small one. They're all on board with the same track. They just think specifically Anthropic and Open AI maybe aren't doing it quite right. That is correct.

Within team post-human, there are family disputes. And as many of us know, family disputes can be vicious. But there's this whole other team, which is team human. That's the team that I'm on. And the debate right now, the public debate about the development of super-intelligent machines should be between team human and team post-human. But team post-human has more or less monopolized the conversation. They've hogged the microphone.

So you're exactly right. So for example, I mentioned that Yadkowski is not anti-Asi or pro-human, even though if you read his book, if anyone builds it, everyone dies, you kind of get that impression. But he has explicitly said in interviews that once post-humanity arrives, then it may be unethical to continue to have biological humans. And he draws this quite offensive, ablest analogy between a post-human choosing to have biological humans or children, or humans continue to have biological humans in the post-human era, and someone right now intentionally choosing to have a disabled child, right?

And in the same interview he talks about how it might be good just to wipe all humans off of the earth. In other interviews, he said that if the only and reliable way to create super intelligences flipping about the universe, quote, having fun, is to sacrifice all of humanity, then he would make that trade off, right? So he has this, he's on team post-human, not team human. The debate has to do with AI safety, which is exactly the thing. that dispute, the tension between the accelerationists and the doobers.

The question is how to go about developing superintelligence. Everybody agrees, including Yadkowski, that we need super intelligence. As Nick Bostrom says, virtually all paths to a utopian paradise in the future, take us through super intelligence. So we can't not build it. The question is when to build it and how. And so you have some people who are on the dumer side in the dumer camp within the test group movement who are screaming that we aren't yet ready to build supertelches.

We don't know how to align it with human values. But what they mean by human values is the Tesker worldview, right? So for example, if there were a super intelligence that were aligned with my anti-Tescalo worldview, and if this super intelligence were to shape the entire future of life in the universe, which is how they imagine, what they imagine super intelligence ultimately do me, they would think that's an existential catastrophe, right? Because what they want is a cosmic post-human utopia, right? So it's not about aligning it to like human values. It's really about aligning it to Tescovus. Anything other than that, anything, any, I mean, literally the definition

of existential risk is any event that prevents us from establishing this, you know, post-human paradise among the stars. So if you have an ASI that's aligned with some non-testual worldview, that is going to be existentially catastrophic. Right. So you have people like you who are saying that we need more time to solve the alignment problem, to figure out how to control this super intelligence such that we can direct it to radically re-engineer the human organism, upload our minds to computers, and ultimately go out and plunder the vast resources of the universe. All right. So the people within team post-human are upset at Anthropic and Open Eye. It's because you haven't sufficiently solved the alignment problem to proceed with our shared vision.

And the labs are basically saying, no, I think. we can I think we can do it. We'll figure it out on the fly. We can't really slow down now because of, you know, what have you, China. So, but it's a, it's basically that's the bottom of the feud is like, are we able to solve the alignment problem enough? Though it sounds like if you talk to, you know, these anthropic employees, what they're really saying is like, hey, we might not. But now we're, now we got to go. We got to follow through because other people are going to start doing it now too. And so we got to get there first. Because if we don't, then for sure. sure someone else will do it and it will be the bad version. It will be the version that kills us all. One thing to notice here is you will hear like the AI CEOs and others at the company say that there are really three reasons that they can't stop. One has to do with market forces. So just another brief like historical tangent here, the AI race began motivating. almost entirely for these ideological test girl related reasons. If you look at what the founders and the people around the founders were saying, you know, back in 2010, 2015,

et cetera, no one was talking about profit. No one cared about profit because all of them realized that utopia is so much more valuable than however many billions of dollars they might gain in personal wealth. This is about fulfilling some kind of grandiose eschatological vision. That is why the, the race started. I know because I was in the community back then. And we had these conversations. And, you know, I knew a lot of these people. And so, so, so, so, so, so, so, so, so, so, so, but over time, uh, market forces absolutely have crept in and become a major driver of the ASI race. So things have evolved. It started off very ideological. Now they're, the profit motive is

a piece to the explanatory picture, especially after chat GPT was released in November of 2020. The second reason they'll say is, well, there's some kind of like game theoretic, predicament between the AI companies. Right? And so, like, they all think that they're more responsible than the others. So Anthropics trying to race ahead of Open AI. Open AI doesn't like that. They think that Anthropic is on the wrong path.

So they're trying to race ahead of anthropic. And then the third one, of course, is geopolitical. And that has to do with the race between, so zooming out, the race between the U.S. and China. But what the important thing to notice is that what all of these boil down to is anxieties about whether or not utopia, their version of quote unquote utopia, will actually be realized. So Anthropic is thinking that if Open AI builds superintelligence in the very near future,

that super intelligence isn't going to be aligned. And the problem outcome is going to be an existential catastrophe, maybe human extinction. If China wins the AI race, then we also don't get utopia because whoever builds the first ASI will then establish a singleton, like a world government. They'll have what Bostrom calls a decisive strategic advantage.

And so China will then realize its version of like a quote-unquote communist utopia. Well, that's not the test-girl utopia that they're envisioning. So ultimately, what all of these excuses, all these pretexts for continuing to race ahead boil down to is deep anxieties about utopia. And maybe there's one last thing to mention, if I may, which is just that, you know, people have become aware. So the topic has exploded into the public discussion, public consciousness over the past few weeks as a result of Jacob Coxon's resignation at Anthropic and an employee there retweeting him and saying, yes, we believe there's a 10% chance of

human extinction in the near future. So it's really, really important to properly interpret the terms that these people use, because oftentimes the way they use terms that seem to have a straightforward definition is defined in an idiosyncratic manner. Right? Safety. What does safety mean? Safety means building in AI, advanced AI, that is aligned with the Tesco worldview. What is the likely outcome of successfully aligning ASI with the Tesco Worldview? Well, our species will be sidelined, marginalized, disempowered, and ultimately eliminated because we'll be of no use and will be in the way. And like Yikowski says, maybe they're ethical reasons.

Right. So another term that is that contains hidden ambiguities. is the term human extinction. Right? So there are, both human and extinction are ambiguous. So if you look at the academic testicle literature, you'll find that everyone defines humanity, according to what I call for lack of better term, a broad definition. On this broad definition, it denotes both our species

and whatever post-human successors might come after us. Right? So an immediate implication of this definition is that our species, species could die out literally next year without human extinction having happened. As long as there are post-humans to take our place, then humanity will persist. And if humanity persists, then that's sufficient for human extinction to have been avoided. The terminology is confusing because they count post-humans as humans, right? But the other ambiguity is if you take a narrow definition of humanity as meaning our species

in particular, homo sapiens, then the term extinction becomes ambiguous. So, again, for lack of better terms, there is a difference between terminal extinction, as I call it, and final extinction. Terminal extinction is just a scenario in which our species disappears entirely and forever full stop. That's the only condition that needs to be satisfied for terminal extinction to have happened. Final extinction kind of builds on this, and it denotes a scenario whereby our species disappears entirely and forever, and we do not leave behind any post-human successors.

All the testicle people care about is avoiding final extinction. That is it. Many of them hold a pro-extinctionist view, according to which we should bring about terminal extinction without bringing about final extinction, which is just to say, we should survive for long enough to create our post-human successors, and then once our post-human successors are here, we should dip out. We should, you know, follow the dinosaurs and the do-do into the eternal gray. of extinction, right? And so it's just, it is deeply frustrating to see the public conversation

where people are like, you know, all of these like test-girl guys like Jacob Coxon, who has deep roots in effective altruism and rationalism, which grew out of the libertarian transhumanist movement, talking about the importance of avoiding human extinction. Because most people sitting at home are interpreting that. It's like, yeah, we should like, do whatever we can to ensure the continued survival of our species. That is not what they're talking about. They're talking about, bringing about us surviving long enough to bring about the post-human era, at which point, who cares what happens to the 8.3 billion people on Earth? You know, we're just irrelevant at that point. And from a grand cosmic perspective, we're utterly insignificant. Right. So our role in this whole eschatological scheme is ultimately to be the transitional species between the biological and digital realms.

Right. And that is our role. And once our eschatological duties have been discharged, there is no reason for keeping us around. So this is a deeply pro, a deeply anti-human position. I hope I didn't ramble for too long, but maybe that was useful. Useful clarification. No, this is critically important. And I think for a lot of people listening to it, who are not anywhere near the Bay Area and not anywhere near Tuscrial, this seems almost absurd or it seems almost, you know,

like who would actually hold a view like that. But partially what I think makes you such a good, not only critic, but understand. expositor on this movement, is that you became a part of it for a while. And so I was hoping to ask you about that, because if we could understand what attracted you to it as an incredibly bright academic, you know, I mean, what attracted you to this? And what did you see on the inside? I think this might help people understand how it might be possible that you have an entire essentially sector of the economy and region of the country that all strongly believes this. Yeah, absolutely. I mean, the absurdity of this, this grandiose vision of the future is part of the reason that Tescruel is still kind of outside the Overton window. I think that is shifting.

You know, there are more and more people waking up to like learning about effect of altruism and rationalism and transhumanism. I've seen those terms all over the place. They're in popular media articles like never before. And so, yes, there are many, many people in Silicon Valley who are true believers. And it is, and it's a lot of people. If you find this whole story to be absolutely nuts, you are getting it. It is completely nuts. And so for me, I grew up in a very religious household. And there was lots of talk about eschatological motifs like the rapture. And I was convinced that Bill Clinton was the Antichrist in 1992 when he was elected. I was like overcome with terror.

I was like, oh, my God, the tribulations right around the corner. And then in my early 20s, as many people do, apostatize my faith. And it took another, like, several years before I came across the work of Ray Kurzweil, who I later ended up being a research assistant to for his sequel, sequel to the book, The Singularity is Near, which is published in 2005. And at first, I thought that this singularitarian view, again, like Singularitarianism, is a version of transhumanism that just emphasizes the singularity, I thought it was completely outrageous.

There's a point in the book where Ray Kurzweil says he believes there is a greater than 50% chance of us making it through this century without destroying ourselves. And he adds, but then again, I've always been accused of being an optimist. So that means up to a 50% chance that we destroy ourselves. So you might say, like, well, why develop these technologies in the first place? Well, Kurzweil's acquaintance, Bill Joy, he was the co-founder of Sun Microsystems, who's technologists, no technophobe. He published an article, a famous article widely discussed in 2000 on April 1st, 2000 in Wired

Magazine of All Places and All Times, titled Why the Future Doesn't Need Us, in which he argued that these technologies are too dangerous to develop. So we need to impose like broad moratoria on entire fields of emerging science and technology because the risk is just too great. The reason that the test rule folks are plowing ahead with developing these technologies is because they are necessary to reach utopia. I alluded to this earlier, right? So if the definition of an existential catastrophe is any event or state, et cetera, that prevents

us from reaching cosmic post-human utopia, then failing to develop these technologies would itself be an existential catastrophe. So there's only one way forward. And so initially when I came across his work in that quote about the risk associated with advanced technologies, I was kind of mortified. But making a long story short, like I eventually got on board. And it was kind of like, well, if you can't fight them, then maybe the best thing is to join them. The deeper I got into the transhumanist movement through the literature on existential risk, The more compelling I found it because it really perfectly fit the religion-shaped whole

that was left behind. There are so many parallels between transhumanism and traditional Christianity from the promise of resurrection. Right. If you're a transhumanist and you don't live long enough to live forever, as Rick Kurzweil says, you can always opt to have your body or just head in that cryogenically frozen so that you can be resurrected not from the grave but from the vet. all the way to theistic aspects.

Like Tesquerel is a, I would consider to be a new religious movement. And I would consider this religious movement to be theistic, right? The only difference is that the Tesco worldview inverts the traditional relation between humans and God. Rather than God creating us, we're creating God in the form of ASI. A god, you know, God like AI, a machine god, a digital deity, etc. So I, this is what attracted me. I thought, oh, here's like an ideology that is checking all the same boxes, all of the things that I miss, meaning, purpose, eschatological hope about the longer term future of

the possibility of immortality, you know, of like transcending my biology, becoming this digital, you know, piece of software, living in a computer, something that Sam Altman himself has already taken steps to become by signing up with a company called Dectone to have his frame chemically preserved and then digitize. And ultimately just like living in paradise with radical abundance and no suffering, that just really appealed to me. And so I, you know, it was really around 2009 that I publicly, by which I mean I published an academic paper on embracing transhumanism and, you know, at the time like the early

Tesslerol movement. And I continued to be a true believer really up until. 2019, at which point I defected and realized that once again I had been completely wrong about, you know, dearly held beliefs. And so I since then have become a critic. And I mean, if you, if one uses the term a bit loosely, kind of like a whistleblower, you know, sounding the alarm about like just how extreme and radical the views are that are held by, you know, people at Oxford and in Silicon Valley. So I want to get soon to why you defected, but sort of two follow-up questions about your experience there that I'm very curious about is one asking about the role of there's a sort of logical and sometimes even mathematical complexity to some of the views, which I've heard makes it very attractive if you're smart, if you're an engineer, because you can very much believe like, oh, I can understand this.

This is actually kind of complicated. This would explain why maybe the masses don't yet get this, but, you know, Dunning Kruger. I'm smart enough to like, it makes sense to me that I'm understanding this future. I mean, two, I want to ask you about, I've heard that it's a very self-reinforcing community. When you're in this world, there's a lot of support, there's a lot of social connection, there's a lot of money sloshing around, there's jobs to be had. And so I'm curious about those two aspects about being inside the Tescriel world. Yeah, the community is very culty, right?

That obviously ties into the new religious movement. aspect. There are many cultish elements. I mean, people in EA are, like, encouraged to, like, you know, pursue career paths that are recommended by people in EA. They're, you know, they have their, their own, like, lingo, you know, their own argo, you know, the technical terms. Everybody's into expected value theory and Bayesianism. And, you know, they oftentimes date each other. There are a lot of polycules. I mean, people are now discovering this with respect to rationalism. Because rationalism at EA, I mean, they're like twin-sibling communities. Again, really the biggest difference is one specifically focuses on rationality in order to do the

most good, especially by creating inaligned superintelligence. EA is about optimizing your morality, mostly by focusing on the very far future, which means being concerned with AI safety and AI alignment. So you can see there, there are a lot of parallels. But I mean, even when I was in the community. I remember in 2019, I spent several months at the Center for the Study of Extential Risk at Cambridge University and over drinks, I had conversations with, you know, at like a party with 15 people about whether or not we might be in a cult. So it's, I mean, it was these conversations that did end up playing a role, you know, it was one of many factors, but did end up playing a role in me ultimately defecting from the community. So yeah, I mean,

I'm sure if you go on social media and you look, you search for rationalism, I mean, you'll find that like people are describing it as basically like a sex cult. And it kind of is. Like there are big orgies that they have and, you know, everyone participates. It's a very, very weird, very weird community. And there's a lot of money too. I was surprised to see you estimate, It was a bigger number than I had heard, but, you know, more than I do, that the EA community, for example, has something, you said, $26 billion worth of donations have been pledged to that community, which allows them, you know, you go to Oxford. And it'll be like one of the nicest buildings on campus is going to be an EA funded, you know, building. You look at Anthropic. That's a lot of EA money went in to fund that.

There's these Future Life Institute. These are really large, really well-finite. So is that, we're really talking about billions of dollars, but I assume mainly comes from tech centa millionaires and tech billionaires who like where this narrative puts them that makes them the heroes of the world. But we're talking about huge amounts of money too, right? Yeah. Just as a quick sign note, I mean, long termism tells billionaires exactly what they want to hear.

It says not only are you morally excused from helping the poor or, you know, focusing on animal welfare, you know, here and now kind of problems, but you were a morally better person, a superior person for focusing on the far future. And so, you know, again, like Elon Musk is a fan of long-termism. No wonder when like a key aspect of long-termism is that it's morally good. You could say it's morally obligatory to go out and, you know, colonize the entire universe, conquered the cosmos. So you're absolutely right that there's a huge amount of money. That figure, the 26 billion, that was before FTX collapsed, right? So St. Bank-Bid-Fried was one of the most prominent EA long Termis, right? He never had like a global poverty phase, like some EAs do. I mean, EA has been

described by Molly Lieberman as a bait and switch, where it sort of draws people in by advertising that they care about, you know, alleviating global poverty. And then once people are in, they say, well, actually, there's a much better way to do the most good. If you can affect, like, positively affect a tiny, tiny percent of the trillions and trillions and trillions of future digital people spread throughout the universe, the amount of good, you're going to be able to do in terms of expected value is just many orders of magnitude greater than if you were to lift the 1.2 billion people in multidimensional poverty today out of poverty. So it's really, it's kind of always been focused. But so, you know, St. Bank of Fried was a leading long-termist.

So they lost some of the pledged money. But since then, I mean, they've, they continue to attract just truly mind-boggling amounts of money. And I mean, if you count, if you count Anthropic as an EA long-termist organization, and you should, right? Dario Amadee was one of the very first effective altruist, right? He was one of the first assigned to giving what we can pledge. I think he was 34 on the list. You know, Amanda Askell is a prominent philosopher at Anthropic. She, her ex-husband is William McCaskill, right? So if you count like companies like Anthropic, then there's even more money, right? If you start to include all of that money.

And of course, Anthropis is about to have its IPO, which the figure I saw most recently, I think was $2 trillion is what they might aim for. It's coming down now because people are, they're institutional investors want a little risk mitigation because they think they're sounding like crazy people because they, if you're a pension fund, you're not on board. with the Tescreal trains. You're like, whoa, whoa, whoa, whoa, wait, what do you guys believe? But a lot, over a trillion for sure. Yeah, over a trillion. So there's going to be this massive influx of money. And also just as a side note, there's another, as it were, using the term

Looseley whistleblower who was heavily involved in the testicle, EA rationalist, et cetera, communities like I was named Holly Elmore. And she has known personally some of the people at Anthropic. And she's also out there sounding the alarm. One of the quotes, this is what I wanted to mention was that the people at Anthropic decided to pursue superintelligence to ultimately create utopia for post-humans, right? I've already mentioned that. And she goes on to say that many in the company are concerned that the superintelligence that they create will just completely usurp or supplant humanity and that this is why it's really important to figure out how to transfer our values. Like basically the are, our, you're, you're

ethical and axiological values to the superintelligence so that it can go out and do what we ultimately think is the right thing for it to do, ethically speaking, namely like colonize the universe, you know, create all these computer simulations and maximize value. So anyways, effective altruism is about to get an enormous amount of money. And this is a community of people who oftentimes present themselves as living humble, modest lifestyles. And some of them do that, right? But there's also like a kind of televangelist aspect to this. You know, so like McCaskill, when he published his book, he went out to a vegan restaurant in New York City and paid, I think it was more than $400 a plate, right? Also, the Center for Effective Ultrism,

well, technically effective ventures, which is the umbrella organization over it, secretly bought a, I think it was 15 million pound castle, so a palatial estate in Oxfordshire to host conferences and get-togethers, et cetera. And when a New Yorker journalist asked them about this, they said, this was before FTX imploded, they said, well, if you do the math, right, if you crunch the numbers and you, you know, figure out the expected value of this, if there's even a small probability that this castle brings in the next Sandbankman-Fried, the investment will have totally been worth it. And that's not only, like, there was a chateau,

and I think the Czech Republic or something, that EA's also bought. So, like, once you, you know, EA, like, from a distance, like doing the most good you could possibly do, that sounds great. And I'm all for that, right? And I'm all for, like, donating once income. You know, I'm committed to doing that as well,

like all of my disposable income to charities. But when you actually, like, get close and look at the details, it's just a deeply flawed view. I mean, the whole reason St. Bankman-Freed went into crypto, I mean, he initially worked at Wall Street, was to follow this career path that EA's call Earned to Give, whereby you try to make as much money as possible, even if you work for, quote,

immoral organizations, as Will McCaskill says, so that you can donate more money to, to charity, right? So this is what got Stan Beggard on Wall Street, and then he worked for Sunder for Effective Altruism, and then he got into crypto. And he was, you know, a hardcore utilitarian. And if you're a hardcore utilitarian, you believe that there is no act that is intrinsically wrong. Like that is, this is a key aspect of the ethical theory. All that matters are the consequences. So there's nothing intrinsically wrong with fraud, with lying, with cheating. There's nothing intrinsically wrong with murder or genocide. If the, the outcome,

outcome is overall good, you know, all things considered, then you have a moral obligation to engage in fraud and lie and cheat and steal and so on. So you can see now how Banking Friede ended up, you know, justifying in his own mind his actions. Because he was a long-termist and he was donate, I mean, FTX invested in The Anthropic, right? So he was really concerned about ensuring that we build an aligned with the Tesco World be a super intelligence so that we could ultimately maximize value from now until the heat death of the universe.

So how did you turn around on this? So you're, you know, in the 2000s, in 2017, you write a big book that's sort of explaining some of these ideas in a positive light. What was it that led to your second, you know, apostasy in your life? Why did you, what turned you? Yeah, and apostasy is the right term because I do think that that, that, that I was involved in a new religious movement. So there were really a bunch of, I guess there were like mainly three considerations that resulted in this drastic pivot away from what I consider to be my intellectual home for a decade.

One was the realization of the poverty of utilitarianism, which really underlies like basically the entire test-girl movement. And even some of the far-out accelerationists who argue that the ultimate goal is to climb the Kardechev scale, which is a measure of basically how much energy a civilization uses, to bring about the heat death of the universe faster. Like, ultimately, it's a kind of utilitarian view because it's saying that there is something of value that we need to maximize. And that thing is entropy, right?

So it's not like happiness or pleasure, as traditional utilitarians would say. But it's interesting. So it's still like deeply utilitarian in flavor. And so utilitarianism, like consider its view of its conception of persons, of you and I. Like we have no intrinsic value. There is only one thing that has intrinsic value in the universe and that is well-being or welfare. You and I are nothing more than the containers or vessels or vehicles of value. And so I matter as a means to an end, not as an as an end.

doesn't end in it of myself. Insofar as I am bringing value like pleasure into the universe, then I am fulfilling my ethical obligations. And I matter only insofar as I'm doing that. And so this gets back to the idea of maximizing the future population. So if we are just these fungible value containers, then if you create a whole bunch of new value containers, right, especially if they're digital, because you create more digital people per, you know, volumetric unit of space time or whatever than you can biological beings. So if you create all of these value containers and they all have net positive amounts of value on average, then you're going to also increase the total amount of value in the universe.

So like this view of people as mere means to an end, it didn't sit right with me. I guess it did for a while, but the more I reflected on it, the more convinced I was, this is just a deeply impoverished way of thinking about things. Maybe I'll mention one other reason, which is that I began to realize that the vision of the future that these people had articulated is essentially techno-capitalism on steroids. It's about going out and conquering the universe, you know, subjugating nature, maximizing economic productivity. I mean, literally, Bostrum at one point defines the existential risk, because any event that prevents

us from reaching technological maturity. And technological maturity, he defines as a situation in which we have. maximized economic productivity, and fully, basically, subdued nature. It's a very baconian. And one thing I noticed is, was kind of completely obvious in retrospect. It's that this vision was the product of almost entirely the product of white dudes who are very privileged, quite wealthy in Oxford and in Silicon Valley. And if you look at the Tescoo literature, there's virtually zero reference to what the future

ought to look like from alternative perspectives. Feminism, indigeneity, disability, queerness, and so on and so. There's a huge list. So I mentioned before that utopia is inherently exclusionary. So I started to think, gosh, like, is the future that we are trying to bring about through an aligned super intelligence really going to include, like, all of humanity? Like, who is going to get left out? Because we're not like, like, trying to be. you determine what the future should look like in a democratic manner through some kind of democratic process by actually going out and asking people, what do you want the future to look like? There's a colleague of mine named Monica Bielska, and she has this great line, and she's

specifically referring to the future. She says, you cannot design for, you must always design with. So from the very start, you have to include people or representatives from every tradition, culture, religion, gender, you know, socioeconomic status, and so on and so on, from the very start, right? And that is the only way you can go about creating a genuinely good and inclusive future that benefits all of humanity. But the next step of this line of thinking was recognizing that, oh, actually, yeah, it's that no humans have a place in the future. Right, obviously. Like, we're going to create these post-humans.

And like Rick Kurzweil says in the singularity is near, if you don't want to upgrade your genes or become post-human in some other manner, that's fine with me, but you won't be around for very long to influence a debate. Right. So it's essentially saying become post-human or just die out, right? And so this seemed even more troubling than the already troubling conclusion I came to that, like, yeah, our species just doesn't have a place in the future. And I'm not sure that's right.

And furthermore, I mean, the only plausible way that we could transition fully from a human to a post-human era is through some involuntary means. Because the vast majority of humans out there are not going to consent or give their permission to allow post-humans to take over the world. And that's the whole point, again, for post-humans to take over the world to run politics, the economy, science, technology, and so on. So there's going to have to be some kind of involuntary, forceful, coercive push by those of us in the Tescaro movement, I was thinking at the time, to actually bring about this transition. And, well, that obviously introduces all sorts of profound ethical problems, right? Violation of rights. There are 8.3 billion people on the earth.

When you start to, like, list the cities, you know, like, okay, so like everyone in, we're talking about like everyone in Toronto, in Singapore, and, you know, you go through. It's like, like, there are a lot of people around the world. And how hubristic is it for us to say, well, you know, we know best, we're smarter than everyone. And so, you know, these like, the dumb masses, I mean, this is the way a lot of the people think. The dumb masses, they just don't understand the stakes. They don't understand the ethical importance of bringing about this cosmic post-human utopia. And so kind of like, you know, forget them. Like kind of screw them because they're just, you know, as Yadkowski would say, they're

NPCs, you know, non-player characters. So they just don't really matter. So that was the other thing that there's a consideration that really, you know, put a nail in the coffin and led me away from the community and encourage me to become a critic. Interesting. So, I mean, is this going too far to say, because this is what's occurred to me, is it

when I think about this type of long-termerist, expected value style utilitarianism where you know like an individual human's life so you move away from that sort of fundamental natural law religious principle at the core of you know all normative moral thinking of humans are infinitely valuable a human life is infinitely valuable when you move away from that liberal conception immediately what comes to mind is the 20th century utopian movements so if you look at far right fascism if you look at the the far left movements uh the mid-century work completely willing for many millions of people right now to die because, yeah, but the utopia

they were going towards was worth it. So if you're in Nazi Germany, you're like, well, yeah, maybe it's not the best. We have to kill, you know, six to eight million people. But it's what matters is it's going to give us this sort of utopian state and that's worth more. Or if you're, you know, the Soviet Russia at a certain point, like, sure, like half of Ukraine is dying from starvation. But the point is, you have to break a couple of eggs to make an omelet. And the omelet is going to be this sort of utopian, you know, socialist communist world. And so there is, as soon as you move away from the precept of life is, you know, a human is infinitely valuable and has their own natural rights. So as soon as you move away from that, we've always, we've tried this experiment.

It ends in a place that we just universally look back and say, that was really bad. Am I going too far to put this in the same ontological category within the world of ethics? In my opinion, absolutely not. The utopia, the thing to understand is that utopia justifies anything and everything, right? Because utopia is marked by infinite or near infinite or, as Tascarilus would say, astronomical amounts of value, there is no act, however extreme, however violent, etc., that can't be justified, right? if you do the expected value calculations, like committing genocide might actually be the like absolutely the best thing for you to do. There's a statistician named Oli Higstrom, who is generally sympathetic with long-termism, but he has this moment of what I would describe as moment of clarity in his, I think it was 2015 book, Here Be Dragons, where he talks about it. Somebody, you know, Nick Bostrom has said that reducing, because there could be so many digital people in the, there could be so many digital people in the, I think it was a the far future and hence so much value. Again, his estimate is like 10 to the 58 digital people at least. That decreasing the probability of in existential catastrophe by a tiny, tiny amount,

which is just to say, it's just another way of saying increasing the probability of this cosmic post-human utopia by a tiny, tiny amount is worth the lives of billions and billions of actual living human beings. Right. So Hextram says if there's a politician out there who, who takes seriously this line of reasoning and is told by, you know, intelligence agencies that there's a one in a million chance that some lunatic in somewhere in Germany, they don't know, they don't know where, is about to bring about some kind of existential catastrophe, destroy this, as Toby Ord says, vast and glorious future among the stars. then the president might be justified in just nuking all of Germany.

Right. So there's absolutely an enormous, and I mean, that was sort of hypothetical back in 2015. But if you fast forward to 2023, you find Yadkowski publishing an article in Time magazine in which he says that the almost certain outcome of misaligned superintelligence is that everyone on earth will die. And again, his concern here isn't that 8.5. 3 billion people will die. I mean, he would say like that's a part of it. But much, much, much more important is the loss of what he calls our glorious transhumanist future, right? So all that

future value, which is, again, another way to say he cares about avoiding final extinction, not terminal extinction, right? And so he's not to interrupt, not to interrupt, but he, there's a tweet of his about this that makes his stance much more clear where he said, if we had a fear that misaligned superintelligence was emerging, thermonuclear global thermonuclear war, it was not a bad option. Sure, it would kill most people, but probably there would be some humans who would survive a thermonuclear war, and then they would have a shot thousands of years in the future of going for a better-aligned superintelligence. So he said, I'm completely fine with killing 99.9% of the population right now

if I thought that was going to stop a misaline superintelligence. And the survivors that are thrown back to the Paleolithic, like, yeah, get them a couple thousand years. They'll rebuild civilization. and that we have another shot at getting super intelligence, right? So he explicitly said, yeah, I would discount. The lives don't matter. It's exactly what you're talking about. It's the glorious future. Right.

So it's exactly the kind of profoundly pernicious reasoning that Hextram highlights in the passage, right? So Yadkowski is saying, like, okay, there's a thermonuclear war. Like, probably, it's probably not the case that everyone will die. And the best calculations that I've seen, you know, from the past, like, six, seven years support that. Like maybe five billion people would die, but that would leave a reassuring three billion to rebuild civilization. So he's saying that like, we should risk thermonuclear war. If that is what is necessary to prevent misaligned super intelligence, then we should do that. He was asked online, quote, how many people are allowed to die to prevent AGI and quote from being

built in the near future because it would be misaligned, he says. And his response is like, basically everyone, as long as there are enough survivors to in close proximity to carry on the human species to rebuild civilization, and then I'm quoting, then there's still a chance of reaching the star someday. Well, if you, like, just consider like what that means. The minimum viable human population is estimated to be between 40,000 and maybe as low as 150 people, right? So pretty, like nobody knows. But even if it's just 40,000, if you. subtract 40,000 from 8.3 billion, you get just rounding off 8.3 billion. Right.

So he's saying 8.3 billion people should be, quote, allowed to die to, you know, ensure that that supertelligence is aligned. Again, he is pro-supertoucher. If we figured out a way to align supertelligence to the Tesco Worldview tomorrow, he would come out tomorrow and say, pedal to the metal, build ASI as quickly as possible. We'll create post-humanity. You know, it'll be unethical to have biological humans. wipe them off Earth, will replace them, and so on.

And so it's just important to understand how, like, absolutely extreme that that view is. But you're absolutely, you're completely right, that utopia is just a, I think, a very dangerous concept. And the other thing I would add to this is I have been speaking so far as if the real, successful realization of the testicle vision is possible, right? And I've been saying that, you know, if they successfully realize their vision, then the outcome of safe superintelligence is that will become completely irrelevant, or according to many of them, like Peter Thiel, like, we should disappear, right? I actually am very skeptical that they

could successfully realize this vision, but you brought up utopian movements of the 20th century. And examples go back many centuries before that. These utopian movements never reached their goal. But that doesn't mean that they weren't profoundly dangerous. They left a trail. In so many cases, they left a trail of destruction behind them. I mean, Hitler promised a thousand-year Reich. You know, he basically was barring from, you know, Christian eschatological narrative, which promises a millennial kingdom at some point in the future.

One of the bloodiest battles in human history was the taping rebellion of the 19th century in China and resulted, I don't know, something like 30 million deaths. I can't remember the exact figure. but this was driven by the taping heaven, taping heavenly kingdom, which was essentially utopian apocalyptic movement, just like test girl. You know, terrorist groups, doomsday cults, you know, the Islamic state, you know, they were an apocalyptic movement. Well, from the religious point of view, the other side of the apocalypse is utopia.

So it's just another way of saying that they were motivated by utopian visions of the future. And that's why they were trying to, literally they were trying to start the Islamic version of the Battle of Armaged in this small Syrian town in northern Syria, near Aleppo, called De Beek, which is said to be the location of their version of the Grand Battle. So it's just my main concern is that, you know, so far the AI companies have left a trail of destruction behind them. I mean, some people would argue that they're like tearing apart society. They're flooding the internet with, you know, disinformation, deep fakes.

They're working with the U.S. military to engage in illegal activities like kidnapping Maduro and bombing Iran. They're partnering with Palantir to sort of like build the infrastructure for totalitarian state. And so all of these are, I mean, not to mention the massive, massive intellectual property theft that they are all guilty of. So these are all these are like harms that we can point to right now. And I think the harms are going to get worse. But from their perspective, like they're marching to utopia.

and sometimes the march of progress, like it's going to trample some people. And also, like, from the grand cosmic point of view, all of these harms that they're causing right now, they are mere peccadillos, right? They're just small, like, little mishaps that utopia will forgive because utopia forgives everything. So this brings me to my, the last thing I want to ask you about before I let you go is trying to understand, okay, what's happening right now. And I'll say, I'm sure you've experienced this as well. but I find if I write or talk about some of these movements, a very common response I get is like, you're just being ad hominim. Sure, these are weird people, but they were right.

The things they were warning us about are starting to happen. So who cares if they're weird or not? They were right. Now, I have this alternate view I want to run by you that actually it's a self-fulfilling prophecy. Right. So that they're talking about AI in the ways that matches the Tescriel sort of predictions because they are from Tescriol. And they're trying to bring those predictions about. And I also am increasingly believing, if you look at Anthropic Open AI, is that they're specifically running experiments and developing this technology in a way that's creating dangers, like you were just talking about, the dangerous experiments, haphazard experiments, unpredictable experiments, exactly because they're trying to rush for.

forward a movement towards their predicted sort of super intelligent future. The strong version of this argument would say, if we go to a counterfactual world where you don't have these two test-gril aligned labs at the frontier and you just have other technology companies unrelated to these philosophies pushing for generative AI, we would be having a completely different conversation. There would be no conversation about existential death, the sort of really unpredictable, scary experiments like what happened at Open Air the Summer just would not have happened. No one responsible would have run those. And so that we have almost, we're already seeing this damage on the route utopia. It's not that they predicted this right. It's that they are trying to make

their predictions happen. But I want to get your final take on this is how do we understand what is happening right now, the things that are concerning people in AI world, how that connects to the Tesquriel philosophy. Yeah, I think you were exactly right. So one way I put it in the past is if you want to understand, this kind of brings this full circle, if you want to understand the behavior of companies, virtually all companies in our capitalist milieu, you have to look no further than the profit motive, right? Why are the fossil fuel companies destroying the world for profit? Why are pharmaceutical companies jacking up prices? Well, they just want to increase the revenue and so on and so on. If you look at the AI companies, they are different. They are anomalous.

And I can't think of any other examples, right? I've said before that profit has become, increasingly significant factor that explains kind of their behavior, the dynamics, the race dynamics they're in. But if you look at the origins and if you look at the core of these companies, what is really ultimately driving them, it is ideological. It's test-crual. Test-c rule is integral to an explanation of why there's an ASI race, why these companies arose in the first place. All of this is to say that because these companies directly, emerged out of the testicle movement, claiming that test cruel advocates back in the day were prescient for talking about AI risks and so on is a complete misreading of the situation.

It's sort of like if someone were to be impressed that, you know, I woke up this morning and I prophesied, I will soon have a cup of coffee. And then amazingly, I got up, I made a cup of coffee and I drank it. And so it was like, wow, that's incredible that you accurately predicted what was going to happen. Well, yeah, because, like, you know, I mean, Deep Mind got its funding because Demis Hasibis and Shane Lake, two of the three co-founders, gave talks at the Singularity Summit. Singularity Summit was founded by Yatkowski, Ray Kurzweil, and Peter Thiel. Yadkowski was funded by Peter Thiel. The connections here are absolutely convoluted and completely incestuous.

I've written an article about this, but it's stunning. I mean, it looks like a conspiracy when you actually look at the details. And not all conspiracies are conspiracy theories. But it's just like an incestuous network of these people like exchanging money and supporting each other. So they found at the Singularity Summit. Dennis Hossibis and Shane Legg were invited. They gave a talk about super intelligence because Shane Legg had just published his dissertation on superintelligence, for which he won a prize from Jukowski's Singularity Institute for $10,000.

And then after the Singularity Summit event, they followed Peter Thiel back to his mansion. And Peter Thiel is as test real as anyone else in this community. They asked him for funding, and he gave them funding, right? So, I mean, DeepMind, which was the company that in a sense, you could say kind of initiated the super intelligence race, because OpenA, I was found. because there were concerns about Demis Hasibis establishing what they refer to as an AGI dictatorship, right? Basically just take it over the world. And so the whole reason that Deep Mind came into existence is because Shane Legg was deeply involved in this community. Demis Hasibis

was into the idea of the singularity and they gave talks at this conference co-founded by Yadkowski. So to say that like Yadkowski was, you know, clairvoyant in some sense for having a what's going on now is just to completely misinterpret the situation. You're totally right that this is a self-fulfilling prophecy. And so we have them to blame for the current predicament. You know, nobody has done more than the test-girl people, including the doomers, nobody has done more than the dooms to launch, sustain, and accelerate the race to build super-intelligence. So rather than saying they're prescient, you should point your finger and blame them for putting us in this situation because my personal view is that if there had not been a testarial movement,

there would not be a race to build these like general systems that exceed human capabilities in every cognitive domain of interest. I don't think that would exist right now. So they are responsible for the unfortunate plight that we all now find ourselves in. Well, I think, Emil, this is it's vitally important information. It's confusing. So I'm glad we got to speak. been some extra time to try to pull this apart, and I've been saying this to my audience a lot, you have to understand this. This is not ad hominin. This is not, well, they might be right, but they're weird. It's the actions we are seen by certain companies, the rhetoric we're seeing by certain companies, as you said, is in a almost unprecedented situation. I also try to come up

with an example from, you know, American history, at least, where you had a really large corporation that was not being driven by the profit motive, but instead by a sort of philosophical, sort of sort of religious motive. You have to find, I guess, a sort of explicitly religiously aligned organization, I guess. So it's really important to understand what's going on, and you've seen it from both sides. So I appreciate you coming on. I want to point people towards all the excellent work you do on this. Your website is, it's X Riskology. Am I saying, am I getting that right? Yes, but people should really go to my newsletter. Newsletter. Okay, go for it. Yeah. Yeah, which is titled The Real Time Techpocalypse Newsletter.

And my podcast is dystopia now. So it's sort of like the darker the topic, the greater the chances that I'll have something to do with it. I'm also going to push people if you want a really good sort of article-linked introduction to Emil's thoughts. Check out the article against long-termerism. There was an Aeon magazine. And I think it's like a 6,000 word manifesto that covers a lot of these. points, but very, very systematically. It's a great work of sort of modern applied ethics. I would also point people towards that as a good introduction as well. Yeah, thanks. And I think the only

thing that's missing, because I had fully develop these thoughts is the pro-extinctionist element that is inherent in the Tesco worldview. So by more recent, like there's one half a pro-human manifesto is, I think, another like good, I mean, insofar as anything I write is is worth reading. I feel like that is very much worth checking out. All right. Well, you heard it here first. Check it out. You need to understand this world. Amil can be your guide to it. All right. Thank you again for joining us. Thanks a lot. All right. So that was my interview with Emil Torres. That was really useful. I've been writing about these ideologies for a little while now. I had a big New York Times article. I've done a bunch of podcast interviews where I've given my best rendition. Even last week's Thursday episode of this show got into some of those details. But I think what I learned today that wasn't as clear to me before was just how extinctionist all these beliefs are. I think Amil is very good.

good on this point that the goal, the ultimate goal of all of these Tescarole bundle of ideologies is a post-human utopia where biological humans are gone. And what we're left with are these exalted digital angels, these sort of digital post-human versions of humanity. But what, as Emil points out, that means at some point these are tech overlords who are trying to bring this, future to fruition are going to have to kill all the other, the meat, biological people. Because most biological people in a world of 8 billion people don't want to say, I'll just disappear and we can have a select few go off into a transhumanist future. They're like, we like our world as we have it.

So if you really pull the thread, this was new to me, and quite frankly, it's disconcerting. But I think Emil is right about this. If you pull the thread on the testicle bundle of ideologies, you end up at some point killing billions of people on purpose because they're getting in the way of your dream of a digital utopia. All right. The other key point that came out of this that we both emphasized, neither Emil or myself think that this vision is actually possible, that you actually are going to, if you have

a good enough agent on your LLM or something, you're going to end up with a super intelligent God. There's a lot of good computer science reasons to say that's kind of fanciful. But Amil and me both agree on this point. Pursuing utopia can be devastating. That's what we've seen. The Nazis did not get the thousand-year Reich, but tens of millions of people died. The Soviet communists did not get their socialist utopia, but tens of millions of people died on the way.

So we've learned this lesson well. The pursuit of utopias, where you're saying the value of humans alive right now, pale in comparison to the value of the utopia that I am pursuing. We have seen that exact configuration throughout history. No one has ever successfully got to their utopia. Every time it's been pursued with any sort of well-funded vigor, it's been a blight on humanity's history. So we really need to be woke to these ideologies and what's really happening here. I am a fan of AI.

I think generative AI can produce any number of useful tools. And the whole thing is being solelyed and hijacked and made into this unstable dynamite bundle of terribleness. by this weird utopian anti-humanist, pretty dark philosophy that's incredibly well funded, and the two top frontier labs are completely driven by it. So I think this is a very important message. It needs to be out there. This has to be a part of the conversation we're having when we're trying to understand the AI moment, but also when we're trying to understand what to do next.

So Emil, thank you for coming on the show. Hopefully you enjoyed this. See you again next time. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at Calnewport.com slash ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, Calnewport.com slash ideas.