Better Offline - Stopping The AI Safety Cult ft. Adam Becker & Cal Newport

Episode Date: September 16, 2026

Ed Zitron is joined by computer science professor and writer Cal Newport and journalist and astrophysicist Adam Becker to get the word out that the narrative about existential risk from AI a narrative driven by a religious cult of effective altruists and rationalists.AI Is Already In Dangerous Hands newsletter: https://www.wheresyoured.at/ai-is-already-in-dangerous-hands/ Adamhttps://bsky.app/profile/adambecker.bsky.social https://www.youtube.com/@DreamingAgainstTheMachine https://www.hachettebookgroup.com/titles/adam-becker/more-everything-forever/9781541619593/CalPodcast & Videos: https://www.youtube.com/@CalNewportMedia/Newsletter: https://calnewport.com Save $10 off a year of my premium newsletter: https://edzitronswheresyouredatghostio.outpost.pub/public/promo-subscription/gzqwkv54e1 YOU CAN NOW BUY BETTER OFFLINE MERCH! Go to https://cottonbureau.com/people/better-offline and use code FREE99 for free shipping on orders of $99 or more. --- LINKS: https://www.tinyurl.com/betterofflinelinks Newsletter: https://www.wheresyoured.at/ Reddit: https://www.reddit.com/r/BetterOffline/  Discord: chat.wheresyoured.at Ed's Socials: https://twitter.com/edzitron https://www.instagram.com/edzitron https://bsky.app/profile/edzitron.com https://www.threads.net/@edzitron Email Me: [e-mail] omnystudio.com/listener for privacy information.

This is an I-Heart podcast. Guaranteed Human. Didn't catch the latest Roland Martin unfiltered podcast? Here's what you missed. There should not be a single law enforcement agency. It does not have body cameras. It's real. Black farmers have been under attack.

This is just the latest example of them just slapping DEI on anything. It's raw. And I'm sitting here going, they are playing y'all for fools. Catch Roland Martin's daily commentary on the Black Information Network. And download Roland Martin Unfiltered on. On the IHart Radio app, Apple Podcasts. Or wherever you get your podcasts. 30 years ago, an island off Manhattan almost brought down New York City.

Who will you trust? Your friends and neighbors? Or will you trust the people five miles overseas? This is a story about neighbors turning on each other. And what happens when a forgotten place decides it's had enough. But we're not stopping, are we? Listen to revisionist history, the Staten Island problem, on the IHeart Radio app. Apple Podcasts, or wherever you get your podcasts.

Also Media. My name's Ed Zitron, and this is Better Offline. Better Offline. I am back and I'm announcing that we will be speeding up the process of researching powerful and dangerous podcasting technology. My researchers are spending over $200 million a day on GPU compute, trying to create and generate a recursive self-improving Zitron, and no amount of pressure for world governments will stop me.

But before my greatest work is complete, I must continue to record. And today we've got an absolute barn burner. I'm joined by the clavikulars of computer science, by which I mean Professor Cal Newport, an astrophysicist and author of More Everything Forever. Adam Becker, fellas, welcome back to the show. Thanks for having us. Yeah, I'm terrified of the idea of recursively self-improved.

That means your newsletters will be 200,000 words long now. Is that what we'll expect? Yeah, they'll either be 200,000 words or 100 words long. Oh, maybe it's not the worst thing then. All right, keep going. The perfect matrix of Zitron is complete. Well, today, though, this was inspired by your wonderful piece, Cal in New York Times, about the rationalists. And we are going to start very simple.

Who are these people and why are they suddenly notable? Well, I'm probably going to actually throw the baton to Adam here in a second to give the history of this group because he and his book was one of my primary sources in trying to understand them. But let me just set the stage for why I was writing about them. I don't think people realize just how weird some of the lead figures are who are at, in particular, Open AI, Anthropic and Deep Mine. And I don't think people realize that until last week when you had a whole group of 20 somethings casually debating in their lunch hour, whether it's 10% or 50% chance that we're all going to die within the next couple of years. And that, I think, caught a lot of people's attention of like, wait a second. what is going on over there in those labs? Why is this type of rhetoric only coming out of those particular labs?

Why is Jensen Wong think that this is ridiculous? Why is China laughing at this narrative? There's something going on here that's not right. I think people are starting to wake up to this. And I think it really is this family of futurist philosophies that come out of the Bay Area that have mutated and split in various ways that are really defining our current moment and no one really realizes it. but Adam spent years, God help him, immersed in this world, has read everything they've read, he knows the history well.

So actually, Adam, I'll throw the baton to you. Maybe you can give us a story of rationalism and effective altruism and where they came from and how they've split. To me, it's a fascinating odd story. Oh, God. I mean, my standard response when people ask me this is, you know, well, I could write a whole book about it and I did. And, you know, I'm not going to get into the same level of detail I do in the book. But, like, basically, this is something that was spawned out of transhumanist message boards and email lists in the 1990s. And what is transhumanist?

Yeah, that's a good question. Transhumanists, I guess the shortest good definition I could give you is transhumanists are people who don't want to be human anymore. They want to use machines and especially computers to basically escape human bodies and solve the problem of death. Okay. Yeah. So those guys. Yeah. Normal shit.

Normal shit. The normal way to do things. Yeah. Exactly. Yeah. And so, you know, these are people who are really excited about the prospect of AI. the prospect of, you know, a nanotechnology that could, you know, basically serve as a kind of magic to do whatever they wanted. This was the 90s.

Nanotech was like the hot thing. And it was a bunch of ideas that basically come from science fiction. Right. You know, they read too much sci-fi and confused it with reality and thought, you know, oh, well, because all of the sci-fi I read. takes place in the future and all of the sci-fi I read has super intelligent machines in it and nanotech and you know people uploading their brains into computers that must be what's going to happen in the future uh if we just you know work hard and wish hard enough for it and so just so i'm clear this is this is a certainty for them they're 100% sure this will happen they're 100% sure

that they could make it happen right so it it's in their hands. Yeah, yeah, yeah, yeah. Like, you know, they, they think, oh, yeah, this is going to happen unless something, you know, ends civilization first, or, you know, nobody listens to us or something like that. They think that they're working on the most important stuff in history of humanity, basically. They think that they are going to, uh, you know, make themselves immortal. And when you say working on, that seems like a fairly broad thing, though, because a lot of these rationalist types don't actually work on the stuff. They work next to it.

Yeah, it's like philosophy people. Yeah, I mean, it's not even so much that they're working next to it. It's like they talk about it. They like to talk about the prospect of nanotechnology. They like to talk about, you know, super intelligent AI. They like to talk about these things. But, but yeah, so these people spent a lot of time doing that on message boards in the 90s. And there was this one guy who showed up in the mid to late 90s on one of the most sort of influential of these message boards, the extropians, which, you know, don't get me started on that name.

It comes out of a misunderstanding of physics. That's an annoying name. No, that's how you know. This is a serious group of scientists. That's right. The extropians. Yeah. That's correct.

Yeah. That's exactly right. And they're like, yeah, we're going to fight the second law of thermodynamics to which my response. as a physicist is, you know, yeah, good luck with that. But basically, uh, this guy, this kid shows up, uh, named Aliezer Yudkowski, who is a teenager at that point. And dropped out, dropped out of middle, after middle school dropped out of schooling. This is like that is, that is wild. Yep, dropped out after eighth grade. That's correct. That's crazy, man. Yep. Um, and, uh, and yeah, Yudkowski then goes on.

to write an enormous amount of, you know, his own deep thoughts on what he thinks the future of AI is going to be. And relatively quickly comes to the conclusion that AI is going to, you know, he takes an article of faith among these people, which is that AI is going to become self-improving and lead to a super intelligent AI at some point in the relatively near future and lead to a singularity where everything changes and all of the stuff that the transhumanists believe is inevitable is going to happen. He comes to the Yadkowski then comes to the conclusion that such an AI might not be friendly to humanity and probably wouldn't be unless it's done properly. And the singularity is when the

computer is smarter than us, just so that I understand. It's when, it's really ill-defined. It's when the computer is smarter than us, or maybe it's when the computer can upload us into more computers so we can all be computer and everything is computer. The singularity is when, uh, is when a lot of tech, but all the time and everything, like, it's not a well-defined thing. Um, yeah. And this is the thing they obsess over the thing that's ill-defined. Oh, yeah, yeah, yeah, yeah. They think that that is pretty much inevitable unless, you know, we have like a nuclear apocalypse first or something like that. This is worth, let's just pause on this for a second, right? I think it's really important because these ideas that I was talking about, so I'm very familiar to what people are talking about today. They were talking about this. This is about the early 2000s. Yep.

It's roughly when Yacalski kind of had this change of heart of, um, super intelligence, of course, is an inevitable, but now we was worried about it instead of being excited about. it. This was all thought experiment. Okay. I mean, this is, there's no LLMs. There was, this was during an AI winter from the computer science perspective. It was entirely inductive thinking directed by ideas from sci-fi. So they just sat there and thought about this, right? So these are not, this is not an engineer looking at a particular technical trajectory in saying, uh-oh, I'm worried this is going to go. This is people just thinking through the thought experiments of, well, if AI could recursively self-improved, it would get super-intelligent, what would it happen if you had a super-intelligent AI and then just sort of exhaustively working

out in their own minds various scenarios about, well, maybe it would be friendly, but maybe it would trick us and what if it did this or that? And so I think that's important. So you have all these ideas are established, well before there's any particular technology that people are worried about, established entirely based off of at this point thought experiments. Yeah, I think that's So this isn't through proximity to actual scientists. Nope. So nothing about witnessing the actual technology following other than an arm's reach. Yeah.

No, I mean, it is exactly as Cal just said. And another way of phrasing that is it's essentially a kind of extremely dry fan fiction that they're engaging in. They are writing, you know, anxiety-fewishing. scenarios based on these ideas that they got from science fiction thinking that that's what the future is going to be. Do they believe it? Oh, yeah. So they're sincere.

They're just odd. Yes. Yeah. And at the time, you can dismiss them as sincere oddballs. What happens is Yudkowski just keeps writing and writing and gets invited to contribute to some group blogs and then spins off his own sort of. blog called Les Rung, which becomes

a community hub for people. Yeah, that's him. But the other thing that happens is it's around this time, like the late 2000s, early 2010, somewhere in there, that Yudkowski meets Peter Teal. Oh. And Peter Teal gives Yudkowski a bunch of money.

How old is he at the time? Oh, we did this math. We did this math, Adam, with the fact checkers. Yeah. This is like 2000, the first decade of the 2000s, he was like mid-upper 20s. Yeah, something like that. Yeah. Yeah, that's right. Huh. It's like a decade of him, a decade after teenage. He'll just give in a child money.

I mean, that's not out of character for Peter Thiel. But he literally pays people not to go to college, right? Exactly. The thing is, though, there's the Teal Fellowship where it's like, I'll pay you not to go to college to build something. This is just like, I'll like your blog, kiddo. Pretty much. You think the world's, going to explode. You better blog about that some more. They met at, as I'll tell me get this right on, but they were at an event called the Singularity Summit. And that's when, I believe that's where they met. Yeah. Or maybe that's where it gets complicated. They definitely get complicated. Yeah,

it does. Yeah. But a lot of money enters the scene, right? Like, this is where like a lot of money starts entering this world. Yes. Yeah. Yeah. And, uh, and Yadkowski, along with some other you know, transhumanist futurist types like Ray Kurzweil starts this set of conferences, the Singularity Summit, and, you know, just gets people together every year to talk about

this stuff. And the community of rationalists on less wrong just, you know, starts, you know, gets bigger and bigger. And they start meeting up in the Bay Area. They start living together in group houses. What does rational

mean? Rationalist basically means you spend a lot of your spare time reading Yadkowski's writing. Yadkowski thinks at this point that if you don't believe that the singularity is coming and that AI is an existential threat to humanity unless you figure out how to do it right, he thinks if you don't believe those things, you're not thinking rationally. And so he decides to post a lot about how to think rationally. That's it. Yes, that's it. That's it. That's it. That's the entire rationality part. Yeah, that's why it writes. It's not about living rationally. It's just about the rational, their so-called rational belief that the computer might wake up. Are you fucking kidding? Well, they apply it to their whole lives. Yeah, exactly. That's why the two big blogs are called

overcoming bias and less wrong. It's all about this sort of like really sort of tech obsessive way of I'm going to be super accurate in the way I, think, right? And this is, I'm going to, I'm, I am not going to be overcome by cognitive biases. I'm not going to be tricked. And if we can just be like hyper Sherlock Holmes is, like, that's how we're going to unlock the potential of the world. But Yukowski thinks like the ultimate application of this is preventing the extinction of humanity by preventing super intelligent AI. And it like, that is why we have to raise an army of like hyper rationalist people. It's our only chance to survive the, the coming AI. apocalypse. So that kind of becomes his obsession is that like this idea of like hyper rationalism that he's the leader of is going to be, that's the John Connor of this Skynet scenario. Right, but they don't seem to be doing anything other than talking. Yeah, now at this point they're not. They're doing some things. I mean, they're in the polyamory. I mean, there's some other things going on. But like, to be very, very clear, Yudkowski is anointing himself as an expert on

on, you know, the art of rationality and cognitive biases and how to think. And, you know, this is an entirely self-trained, self-taught guy whose experience of the world is pretty limited. And he is posting hundreds of thousands of words about this on Last Wrong and then starts writing as, you know, a combination of artistic outlet and recruitment technique. he starts writing literal fan fiction, a Harry Potter fan fiction called Harry Potter and the Methods of Rationality. Oh.

Yeah. There's like 600,000 words long or something. Yeah, I'm sorry, even I would never write something that long. Even I would, 200,000 words on anything is terrifying. I mean, this is 600,000 words. But what the fuck was it even about? Was it like... Oh, it was about Harry, who is, you know, I think fairly clearly an author self-insert of Yarkowski.

Harry being the specialist boy and finding ways to, you know, change and save the wizarding world of Harry Potter using his special logic and science powers. Through logic and reason. Yeah, exactly. Yes. Right. Well, why? Great question. I guess it's just creative. Yeah, I mean, I interviewed him for my book and that was what he claimed. And I believe that it started out as a creative thing. But, you know, it was also very clearly him playing with these ideas that he has about, you know, this is the only way to think clearly. And it did end up becoming, you know, quite a recruitment tool. for the rationalists. And just to put a look, just make sure we have like the timeline beats clear

and then Adam fact check me here. There's sort of the shift that happens where in the early 2000s, actually I think his first money was, it was Peter Thiel, but it was also that other internet entrepreneurship couple. They fund something called like the Institute for AI Singularity or the,

what is the exactly? The Singularity Institute for AI. Originally, yeah, originally the funding comes from this couple, the name of which I don't remember, but eventually, like a few years after that in like the mid to late 2000s, he meets Teal, and Teal gives them like real money. Yeah.

And this is where, so as the 2000s move on, so that Singularity Institute was superintelligence is going to be a beneficial godhead that's going to redeem history. Well, he shifts pretty, my understanding is like, once that's a full-time job and he's thinking about it a lot, he shifts pretty quickly. He does that first big report. He shifts pretty quickly, like, actually, wait, I don't think we'll control. AI, we should not build it at all, is going to kill us. Like there's this shift that happened, like kind of quickly in that area, and he shifted Valence, and then eventually they renamed it

to Mira, Miri, the intelligent research institute. Yeah, that's correct. Yeah. So, so just to get the timeline straight here, um, Yudkowski makes that shift, uh, from, you know, superintelligence is going to solve everything and we should be aiming for it as fast as possible to, oh my God, it's going to kill us all. He makes that shift around the same time Singularity Institute for AI is founded in the early 2000s. Right. Is there an antecedent or does he just one day wake up? I mean, he's just been on these message boards for a long time and he also saw that like, you know, the nanotech guys who wanted a lot of the same things and made a lot of the same promises back in the 80s and 90s, people like Eric Drexler. They had their own.

sort of institute and Yadkowski, you know, at this point, he's, you know, unemployed and and about 20 and I think he's, he's like, I want to work on this full time. Can I turn this into a job and finds a way of doing that by, you know, co-founding this institute? But I also, I want to, I want to just push back on one thing that you said, Cal. Okay. It's not quite that, you know, Yudkowski wants to stop AI at all costs. He, he wants to stop AI at all costs. until he and others figure out how to do it safely, at which point he wants to go full speed ahead. He still believes in the, you know, the AI utopia

if you can get a super intelligent AI to not kill everybody. So he just thinks that that problem is harder than he originally imagined. So he's like, this might take longer. Right. There's a, there's an evidence towards him being the thinking king. Yeah. Like the thought king that can tell the computer how to be best that loves us. That's basically, yeah.

He will be the right-hand man to the digital god. But Adam, is this right? There was no technical antecedent for this shift in his thinking from actually this might be harder. The best I could tell from your book and looking at other sources is that the shift in thinking just came from as soon as he got the funding for the Singularity Institute that he was helping to run. He had this big writing project he took on where he was like, okay, I'm going to produce this big report on superintelligence. and it was in just the internal, you know, cognition of thinking through this report that he just convinced himself, oh, wait, I don't think we're going to be able to control this as easily as we think. Because I think it's really important to emphasize, again, this is not someone who has seen something in a technical lab that is scaring him.

All of these developments and is thinking at this stage, it's really just, it's Yukowski in his own mind. Didn't catch the latest Roland Martin unfiltered podcast? Here's what you missed. And the argument is, do you want to win? If you don't win, you have no power. It's real. Those are the very people who I always say vote against their own economic interest, and they complain about the very thing to receive.

It's raw. I knew that the issue of transgender athletes was not going well with a lot of black men. Roland Martin unfiltered. I went off, I said, this is exactly what I'm talking about. Y'all going to scream at you. Chanel, but you don't want to see a black luxury company in the same vein. Catch Roland Martin's daily commentary on the Black Information Network. Twelve seats, those are moving south.

So if you Democrats, you don't wait to 2030, you better be hitting the ground going hard in the south now. And download Roland Martin unfiltered on the Iheart radio app, Apple Podcasts. Wherever you get your podcasts. The NFL never stops, and neither does it. newest podcast. The NFL report hosted by me, Andrew Siciliano. It is your home for everything football. Williams drifting. Sprint it for space. In all kinds of trouble put it up for next. The end of stuff. We're tapping the biggest news stories, giving you expert analysis, fantasy previews, game picks, and more. We fly around every single play. We dominate every single play.

Want to hear from the league's top stars ahead of the biggest matchups? We've got you covered. They're a really good football team. We've got to have all hands. on deck. Join me at an all-star cast of analysts for everything you need to know from around the league. Get new episodes of the NFL report Monday through Friday all season long. Listen to the NFL report podcast on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. Hey there, everybody. Amy Robach and T.J. Holmes here from the Amy and T.J. podcast with midterm elections just around the corner. We're breaking down the biggest political stories leading up to November.

those big political stories aren't just what's happening on Capitol Hill. Yes, we just saw Lindsey Graham the death of a long time senator. Also, we've got a couple of missing members of Congress. Also, we've got a lot of members of Congress behaving badly. We've got accusations, allegations flying all day long on Capitol Hill. It's just like a reality show. We're going to cover everything you need to know during this heated political season. And it starts with our morning run. Yes, go for a run with us. Maybe you can take us along when you're on your actual run outside, or if you don't feel like going out, still, you get what we're saying.

It's our morning run. It's our daily news rundown of the top stories that will get you going before you even finish that first cup of coffee. Listen to Amy and TJ on the IHeart Radio app, Apple Podcasts, or wherever you listen to your podcasts. I'm Mimi Brown, and every day on front page podcasts, I help you make sense of the stories shaping our world. Each day, I break down the biggest headlines,

provide important contexts and explain how these. stories could impact your life, your family, and your community. My goal is to deliver clear, trustworthy information in a format that fits your schedule. And when a story deserves a deeper conversation, I'll sit down with experts, newsmakers, and thought leaders to help you better understand the headlines. I spoke with Gary Mayor, Eddie Melton, about what's happening on the ground, whether race is playing a role, and what help is available for families who simply can't afford the added cost. Joining us now is Tennessee State Representative. Justin Jay Pearson.

Thank you so much for having me. I appreciate it. Absolutely. Whether it's a major national story or an issue impacting your community, join me on Front Page. Listen to Front Page from the Black Effect Podcast Network

on the Iheart Radio app, Apple Podcasts, or wherever you get your podcast. Every case has a story. Every victim has a voice, but will it be heard? I mean, it's He Grace. On crime stories, we go beyond headlines to,

Examine evidence, challenge the story we're fed, and pursue justice for victims and their families. Please join us at Crime Stories with Nancy Grace on the IHeartRadio app, Apple Podcasts, or wherever you find your podcasts. Hope you join us. I'll see you there, friend. I'm Brett Coleman. And I'm Trevor Sycamma. Have you ever watched a game and wondered what really made the difference beyond the final score? That's exactly why we created NFL Prox. He's throat got a man wide open. Jayhawks!

Every week, we use exclusive next-gen stats insights and NFL content to break down the biggest stories, matchups, and moments from around the league. We go beyond the box score to uncover the numbers, the tendencies, and details that help explain what actually happened on the field. Fires it out in the flat to Barclay, who beats one man, spins past another,

leaps over at third. And don't worry, this isn't just a show for stats experts, we use advanced analytics to help tell the story behind the game, turning the numbers into insights that every fan can understand and enjoy. So whether you're a casual fan learning more about football or a hardcore fan who can't get enough analysis, our goal is to help you see the game in a whole new way. If you've ever wanted to know why a playwork, how a matchup was won, or what most fans missed on Sunday, NFL Pro delivers insights you simply can't get anywhere else.

Listen to NFL Pro on the IHeartRadio app, Apple Podcasts, or wherever you get your podcast. What would change if you stop letting money stress run your life, sis? I'm Mandy Money, host of Brown Ambition, The Podcast. And right now, I'm taking you through my six-part Money Reboot series. This is your chance to hit reset, break the debt cycle, get honest about where your money's going, build a savings plan that actually sticks, and start making financial choices that support the life you want. not just the bills you have. Because wealth building is not about being perfect. It's about having a plan, knowing your numbers, and making your next move with confidence with a community that cares

like we do on Brown Ambition. Each episode gives you practical steps, real talk, and the motivation to stop putting your financial goals on hold. So if you've been saying, I'll get serious about money next month. This is your sign. Start now. Listen to Brown Ambition. That's Brown Ambition and begin your money reboot today. Listen to Brown Ambition on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. I think that that's a really important point. Is that consistent through time, though? Yeah, I mean, he's, from that point on, he's convinced of this and has been saying- But he's not seen anything and these people have not seen anything. No, I mean, you know, he, he, he, later on he starts pointing to various advances that are happening in machine learning as

evidence that the amount of time that we have to solve this problem that he came up with is, you know, running out. But no, you know, that original idea and his, his faith in it is based purely on thought experiments and conversations he has with other people in this community. Because the way these fucking people frame themselves when they talk about this, they talk, I'm speaking from, I don't have a podcast episode that just did while them. They don't talk about, and maybe this is how they varnish here, they don't talk about that they would be the authority in some cases. They just like, we should have one, which sound, that's how they always get you these people,

people like this. It's always, they sound reasonable, and then behind closed doors, trousers come off. Okay, that's not true, but, well, maybe. But it's about, it's about how, oh, actually, we are, we are, like Boris Bolkin in the Ninth Gate. I alone am worthy to deal with this. That's terrifying. I don't like that one beard. I mean, oh, okay. Let me, yeah, because we're going to make this worse now. Yeah, right. All right. So this, this then becomes the key, this is one of the key arguments in my New York Times op-ed. Like, why is this relevant to right now? Well, the issue is, so this world is going

on. Yukowski has his followers. He's convinced of this. It's mainly thought experiments, right? There's not a particular AI technology of looking at. That world severely overlaps with open AI anthropic and deep mind. And let me give you like the very condensed version that Adam can add the pieces that I'm missing here, right?

But let's go back to the founding of deep mine, right? There's three founders of co-founders of deep mine. Two of them, Hissabies and Shane Legg, they're at a singularity summit. They are talking with Yikowski. Yikowski says, let me introduce you to Peter Thiel.

And Peter Thiel gives the initial funding for DeepMind. Shane Legg has been deeply involved in sort of the Yukowski rationalist world for a long time. He was motivated by a talk that Yukowski came to give about superintelligence at the place where Legg worked. He went on to write a doctoral dissertation on Superintelligence, a frequent poster on Less Wrong. So DeepMind has these deep connections to that community. When they get bought by Google, that's when Sam Altman and Elon Musk are like, well, that's bad, right? Because this is, in a true rationalist sense, this is the technology going to either redeem humanity or kill us all.

We can't have a company like Google control it. That's when Musk puts up the money for OpenAI. You have Sam Altman, you know, tweeting favorably about Yikowski at this point, saying he's done more for AGI than anyone else. Yukowski should get the Nobel Prize. The current president of Open AI, Greg Brockman, was running a less wrong reading group at the company where he worked before. Wait, even Brockman? Brockman was the normal one. No, no, no.

All right. So you have these deep connections. the board of OpenAI, half of that board comes strongly out of the sort of rationalist EA communities. When they try to oust Sam Altman in 2023, as Cade Metz reported at the time, it was the faction of the board with rationalist EA connections that felt like he was being insufficiently aligned

with rationalist type ideas. The temporary CEO that the board put in place to try to replace Sam Altman is someone who is so entrenched in rationalist circles that there's a character named after him in the Harry Potter fan fiction that Yukowski wrote, Dario Amadeh and his sister and the other employees who left Open AI to start Anthropic did so because they felt like, again,

they were being insufficiently aligned in part to rationalist ideals about how do we usher in superintelligence safely? You have Amade a couple years ago parroting common rationalist talking points. Like, yeah, we have about 25% chance that AI is going to kill all of humanity. So these worlds, those three companies, they come out of this world of rationalism,

which had a really big foothold in the Bay Area of the time, which means our leading AI labs, the way they think and talk about AI is heavily influenced by everything we just talked about. Yep. No, that's all correct. So when they say alignment, because alignment as an idea is not the worst idea. It's making sure the models give good outputs and the outputs aren't inherently evil. That's not really what they mean when they say alignment. No, but that's exactly what I'm getting at, though, because there is the term alignment,

which means the thing I just said to most people I talk to, and then what they mean by alignment, which appears to be something else. Yeah, no, what they mean by alignment, and I mean, this is basically word for word. What Yodkowski says is an AI that when it becomes. super intelligent will not kill 100% of humanity. Wow. Yeah. That's fucking annoying. Yes. I know I'm meant to sound more eloquent than that, but that's, that makes me very frustrated. Well, to make you more frustrated and to make it even more annoying and infuriating, because these guys were so deeply entrenched and involved with the AI industry. And

And because they'd been writing about it for so long, it means that when the sort of AI moment arrived for the public with, you know, the release of chat GPT in late 2022, the rationalists and effective altruists were able to convincingly portray themselves as experts who had been working on AI for decades. And this is how you end up with things like, you know, Bernie having, you know, a sit down with Elias or Yudkowski. Right. This makes more sense now because the way they frame themselves and they'll say like, oh, AI 2027 has been accurate in some ways even. And as you'll know, in a podcast coming up, I push them on that and they refuse to actually open the blog in front of me.

But it's it's how they have scary stuff framed in the term. of academia without actually being academic, right? Yeah. Yeah. Cool. Now, there's two important sidebars here. So sidebar one, we've been kind of casually referencing effective altruism. Yeah.

This is important, right? Because so effective altruism, this is where it gets complicated. So this is another movement that has like similar sensibilities that became intertwined and heavily influenced. They sort of mutually influenced each other, right? But you have effective altruism. you know, it comes out of a bunch of utilitarian philosophers at Oxford. Then they have this idea of, oh, we can apply.

This should sound familiar. But it's like if we're very like rational and empirical about altruism, we can do altruism better, which I think is like a, it's like modern utilitarianism. That's a fine idea. This is where they're going around and doing research. Like what are the actual best charities to give money to? How do we measure that? They have these ideas like actually if you are, if you're at Oxford, like an elite university. the best thing you could do for charity is to go get a high-paying banking job and then give up half your salary of charity.

So they're trying to think through rationally how do we do altruism as effectively as possible. Well, they overlap and are influenced by rationalism. And pretty quickly, you have a large segment of effective altruists. You say, actually, the most altruistic thing we can do is prevent an AI apocalypse. Yep. Because why bother, why care about, you know, saving 20 million people now from malaria? when we could instead be solving, you know, saving a hundred trillion future lives that will never be born if AI kills us all.

So effective altruism, not all of it, it's a complicated field, but big segments of effective altruism, and this is where Nick Bostrom comes out of at Oxford and talking about existential risk and superintelligence. A big segment of effective altruism really begins to care about superintelligent AI is like one of the core things to care about. And a huge amount of money pours into effective altruism. we're talking hundreds of millions of dollars, starts going to these institutes and groups. And nothing's been done yet, though.

Just want to be clear. Like, no actual actions have been taken to stop superintelligence, no safety standards. No actual science has taken place either. Well, they wrote the Harry Potter fan fan. Okay, so, no, I'm not even being sarcastic. Like, they wrote some posts. They wrote some posts and kept talking about how it was really hard because, you know, you know, what does it mean to align something with human values?

And like, well, maybe, maybe your question is ill-posed. Maybe go work on a fucking large language model. Go and work on AI instead of this weird thing. I mean, some of them did, right? And that's how we got anthropic. Yeah, yeah, yeah, yeah, and deep mind and an open AI. Open AI, yeah. But, but like, just to everything Cal said is right, except one thing, and I think it's important,

And Nick Bostrum is definitely like part of the whole effective altruism community. But Nick Bostrom, he wrote, he's probably best known for a book that he wrote called Super Intelligence that came out, I don't know, a little over 10 years ago that made a big splash, basically, you know, expounding on all of these ideas and saying, yeah, you know, super intelligence is really scary. And we should, we should figure out, you know, how to control it or stop it before it kills us all. And again, this is before, you know, transformer models and the current AI moment. And it's not from any particular thing he's looking at. But the thing is Bostrom, who at that time is a philosophy professor at Oxford, came out of the same, you know, place, basically, as Yudkowski. He was on those same 1990s list serves.

And he's the one who sort of is most responsible for the connection between rationalism. and effective altruism because it was when his colleagues at Oxford were sort of working on effective altruism that he went over to them and said, oh, if you're worried about that, you really need to be worried about the AI apollos. But these are two different worlds.

Yeah, but at this point, they're kind of conjoined. Yeah. At this point, they're like, they're really like, you can't disentangle the two communities. Yeah, there's a talk you can find. So William McCaskill is one of the two big utilitarians at Oxford that helped found effect of altruism.

And I'd cross paths with him sort of earlier on in the EA world when they were really caring about, especially like the idea of donate most of your salary to charity. There's a talk you can find. And it's on YouTube. And I forget exactly when this happens. But there's a famous talk you can find in McCaskill where he basically comes out and says, this is the new thing we should care about is existential, the existential threat of super intelligent AI.

this is the sort of long-termerism. So they used this, there's another idea that comes out of Silicon Valley, but this long-term view of what's going to happen over the next thousand years is actually, if we're just going to do the math, 100 trillion future lives matters more than, you know, 50 million lives now.

So he kind of has this moment where he gives this big speech that where he's sort of announcing this is the new direction for EA. And so now they're like roughly interchangeable. This is why like if you read like Kade Metz's reporting on like the Open AI board or reporting about Amadeh and his philosophical connections. They'll sort of use the terms rationalism and EA interchangeably. They'll use one or the other, say both. And that's why these circles have become very, very simple. And they seem to have had different goals that have now kind of conjoined. Yeah, I think that's correct. Yeah. I should also point out, you know,

the most famous example of someone who took McCaskill's advice and, you know, you know, went off to make a bunch of money in the finance industry and then tried to donate large amounts of it. Uh, is Sam Bankman-Fried? Oh. Yeah. Yeah. Biggie A.

Yep. Yep. And actually, uh, early investor in Anthropic. Oh, yeah. Yeah. Yeah. Yeah. Yeah.

Yeah. Yeah. Yeah. rationalist and effective altruist community organizations. And yeah, no, I mean, McCaskill met SBF when he was a college student at MIT. Yeah, I get around with a lot of kids. Yeah, well, the Charlie Kirk style thing where they go to college campuses almost. Yeah, well, I mean.

For thinking online dudes. That's the thing. They do do this. There's a, God, there's a piece. I want to say it was in the Washington Post a few years. ago by Natasha Tiku about campus recruitment strategies that the effective altruists were into because they have

just enormous piles of tech money that they were using to get them while they're young. And we're talking like the effective altruists will come in and someone will just sign a check for like 60 million dollars to start an institute. Like we're talking huge, huge money coming from. It's tech billionaires.

Yeah. Because think about it. it. Who is the hero? Who are the heroes in the the rationalist? You are Neo from the Matrix. You're John Connor from the Terminator. You are the people who are going to save the world. And if you're a tech, you know, a hundred million, a centa millionaire, a billionaire, you're like, yeah, that tracks. That kind of matches my understanding of myself. And I assume the EA rationalist types don't think, hey, wait, are we just playing into the hands

with the richest people alive? I mean, they kind of either don't think about it or don't really care. They don't think, they think that if you just think really, really rationally, then things like conflict of interests and corrupting your motives. Well, you know, those are just cognitive biases and you can overcome those by, you know, the power of pure logic. That reminds me of this guy Alex Heath who is ostensibly a journalist who has also become a venture capitalist this week. I saw someone comment on his post saying, now you've got skin in the game. You can be truly objective and they were serious. Oh, Oh my God.

But that is that it sounds like the rationalist people. It's like now you know the stakes. You'll be bereft of any sort of human biases you may have and work towards this vague goal we all have that none of us can describe. I mean, they, they believe that, you know, making bets on things is a way of thinking clearly about stuff. When you say making bets, what do you mean? Money.

I mean, I mean like, you know, like what they betting on? Well, I mean, this is, this is all their like, you know, decision theory stuff. But they have, there's this forecast gambling website called manifest, sorry, manifold, where they, you know, like bet on, you know, it's a prediction market. and they love prediction markets. Because economic theory, right? You're going to be the rational consumer.

When your money is on the line, that's going to help you overcome biases and get towards, you know, all this theory behind it. Is there going to get a clear, rational solution? Yep. So this reminds me of just like the most annoying early people online. Oh, yeah. They were doing brain, like mind mastership being like, yeah, we're so smart. Oh, well, do you want to make a gentleman's bet about who is?

is the smartest. And then they set very strange, vague goals that they can move the goalposts on. Yeah, yeah, I've seen this before. Except they control so much money now. Well, okay. But now we've got to connect us to the current moment because it gives you, this is the argument I was trying to make my times piece. It gives you a different filter to understand what's going on because it's a

confusing moment because we have two things happening simultaneously at the, these small number of frontier labs that are heavily connected to this world. A, you're developing a complicated, powerful technology, spending giant amount of CAPEX on this, right? So transformer-based LLMs are a legitimate breakthrough in AI. They can, if you push them, put enough money into them, they're very jagged, but there's all sorts of interesting things that happen. It's a live wire technology.

And so you have a real technology that's developing. But then it's being run by people who have these sort of futurist, almost apocalyptic sort of eschatological views where basically I feel like that community split where the hardcore original rationalists like Yukowski or, you know, Nate Sores, etc. would argue like, you know, we need the bomb the data centers because we don't know how to control this yet.

We're all going to die. And then you have like the Altmans and the Amadez who are like, I think what's more important is that we control the digital God we get there first. Right. So it's either going to redeem history or destroy us all. So you have people running these labs and they're hiring people who think way. Andrew Ho Online was talking about there's probably a lot of pressure within these places to think that way if you're hired there. They're thinking in these almost like apocalyptic terms.

Like we need to build the digital God before someone less trustworthy does, right? Like the end of history is coming and we need to be the ones to get there. And you have this actual live technology that's powerful and a lot of money is going into. And so these two things are now mixed together. And I don't think our sort of public or our media landscape was ready for that. Like this mix. of weirdness with a real company and real technology. And it's like these two things together is blowing everyone's mind. They're like, well, I don't understand.

Like there's these people saying we're all going to die. Like, why would someone say that? Why are they so calm about it? And I think it's almost like the Rosetta Stone to understand what's really happening this moment is that we have a mixture of a lot of money and a powerful technology with a weird group of people. And that mixture is blowing up. What does that manifest as?

What is the specific thing that people need to know? my concern, I mean, there's a lot of different things are going on here, but my concern, if you look at what specifically is causing trouble, what are we worried about coming out of the frontier labs? Like, in particular, this sort of rogue AI narrative is what seems to have caught people's attention. The experiments that led to something like the hugging face attack were extremely negligent, right? Yeah.

It boggles the mind, you know, what we, I've talked about on your show before, Ed, and people point to that episode, so I won't go through the whole thing, but, you know, they're taking an agent is basically a prompt loop. It's a, it's a program that tells an LLM with a prompt, hey, I'm an agent that has access to these tools, here's my goal, what should I do next? And then it gets a response from the LLM, and it executes that, updates its state, and it says, okay, here's my new state, here's my tools, what should I do next? So it just blindly executes what an LLM says, which could be okay. Millions of programmers every day use agents, there's no problems with them. But they ran these, over the summer, Open AI was running these negligent experiments where they gave them very powerful tools that you could use for computer hacking.

They hooked them up to an LLM that had no guardrails and what information they would give them. They made the programs persistent. So they said, we want you to just keep demanding an answer. Don't accept an answer of like, I don't know what to do next. And then they let them run for days without supervision. And they read thousands of them at the same time. And why did they run thousands of them at the same time is because what we now, know is what they wanted to do is see if any of these versions happen to like stumble onto a

particularly good hacking strategy, they would then use that trace to go back and reinforcement trained a model to push towards that type of solution in the future. So it's like putting a thousand self-driving cars onto the interstate and say, yeah, most are going to crash and maybe hurt people, but those that actually make it to the destination, we can use their data to help update our system. So it's this very negligent experiment they were running. And I think, my claim, And they had no idea that their computer was running as well. They just was like letting these little bastards run. They, I don't think they cared.

And the reason why I don't think they cared is because they have this bigger eschatological belief of we got to get to the digital God before anyone else. And you think that that's what's driving open AI, though? Well, this is what I'm worried about is that when that's your mindset, instead of focusing on useful products, things that can give you return on investment, common sense safety standards, you're going to take big risks because when you're racing, when you're racing, towards super intelligence, you know, there's going to be some casualties along the way that's acceptable. So that's my concern. I don't know, Adam, if you agree with that. But that's my concern is because

they are so convinced this is going to happen. And it's so important. And it trumps, this is what their writing says. It trumps any measly concerns we have about like the current moment that they will be reckless. It's also why now Amade and Altman are talking about, well, we need a global slowdown, whatever that means before we can stop doing dangerous things. Because if you believe that, that That's the only way to get digital God. You're like, we can't slow down unless everyone else does, because if someone else gets the digital God, it's going to enslave us.

Didn't catch the latest Roland Martin unfiltered podcast? Here's what you missed. And the argument is, do you want to win? If you don't win, you have no power. It's real. Those are the very people who I always say vote against their own economic interests, and they complain about the very thing to receive.

It's raw. I knew that the issue of transgender athletes, Athletes was not going well with a lot of black men. Roland Martin unfiltered. I went off, I said, this is exactly what I'm talking about. Y'all going to scream at Chanel, but you don't want to see a black luxury company in the same vein. Catch Roland Martin's daily commentary on the Black Information Network. Twelve seats, those are moving south.

So if you Democrats, you don't wait to 2030, you better be hitting the ground going to are in the South now. And download Roland Martin Unfiltered on the IHart Radio app, Apple Podcasts. Wherever you get your podcasts. The NFL never stops and neither does its newest podcast. The NFL Report hosted by me, Andrew Siciliano. It is your home for everything football. Williams, drifting, sprinting for space.

In all times of trouble, put it up for next to the end of us. We're tapping the biggest news stories, giving you expert analysis, fantasy previews, game picks, and more. We fly around every single play. We dominate every single play. Want to hear from the league's top stars ahead of the biggest matchups? We've got you covered.

They're a really good football team. We've got to have all hands on deck. Join me at an all-star cast of analysts for everything you need to know from around the league. Get new episodes of the NFL Report Monday through Friday all season long. Listen to the NFL Report podcast on the IHeart Radio app,

Apple Podcasts, or wherever you get your podcasts. Hey there, everybody. Robach and T.J. Holmes here from the Amy and T.J. podcast with midterm elections just around the corner. We're breaking down the biggest political stories leading up to November. And some of those big political stories aren't just what's happening on Capitol Hill. Yes, we just saw Lindsey Graham the death of a long time senator. Also, we've got a couple of missing members of Congress. Also, we got a lot of members of Congress behaving badly. We got accusations, allegations flying all day long on Capitol Hill. It's just like a reality.

show. We're going to cover everything you need to know during this heated political season. And it starts with our morning run. Yes, go for a run with us. Maybe you can take us along when you're on your actual run outside or if you don't feel like going out still. You get what we're saying. It's our morning run. It's our daily news rundown of the top stories that will get you going before you even finish that first cup of coffee. Listen to Amy and TJ on the IHeart Radio app, Apple Podcasts, or wherever you listen to your podcasts. I'm Mimi Brown. and every day on Front Page podcast, I help you make sense

of the stories shaping our world. Each day, I break down the biggest headlines, provide important context, and explain how these stories could impact your life, your family, and your community. My goal is to deliver clear, trustworthy information

in a format that fits your schedule. And when a story deserves a deeper conversation, I'll sit down with experts, newsmakers, and thought leaders to help you better understand the headlines. I spoke with Gary Mayer, Eddie Melton, about what's happening on the ground,

whether race is playing a role and what help is available for families who simply can't afford the added cost. Joining us now is Tennessee State Representative Justin J. Pearson. Thank you so much for having me. I appreciate it. Absolutely.

Whether it's a major national story or an issue impacting your community, join me on Front Page. Listen to Front Page from the Black Effect Podcast Network on the Iheart Radio app, Apple Podcasts, or wherever you get your podcast. Every case has a story. Every victim has a voice, but will it be heard?

I'm Nancy Grace on crime stories. We go beyond headlines to examine evidence, challenge the story we're fed, and pursue justice for victims and their families. Please join us at Crime Stories with Nancy Grace on the IHeart Radio app, Apple Podcasts, or wherever you find your podcasts. Hope you join us. see there, friend. I'm Brett Coleman. And I'm Trevor Sikima. Have you ever watched a game and wondered what really made the difference beyond the final

score? That's exactly why we created NFL Prox. Got a man wide open. Touchdown! Seahawks! Every week, we use exclusive next-gen stats insights and NFL content to break down the biggest stories, matchups, and moments from around the league. We go beyond the box score to uncover the numbers, the tendencies, and details that help

explain what actually happened on the field. Fires it out in the flat to Barclay, who beats one man, spins past another, leaps over at third. And don't worry, this isn't just a show for stats experts. We use advanced analytics to help tell the story behind the game, turning the numbers into insights that every fan can understand and enjoy. So whether you're a casual fan learning more about football or a hardcore fan who can't get enough analysis, our goal is to help you see the game in a whole new way. If you've ever wanted to know why a play worked, how a matchup was won,

or what most fans missed on Sunday, NFL Pro delivers insights you simply can't get anywhere else. Listen to NFL Pro on the IHeart Radio app, Apple Podcasts, or wherever you get your podcast. What would change if you stopped letting money stress run your life, sis? I'm Mandy Money, host of Brown Ambition, The Podcast. And right now, I'm taking you. you through my six-part money reboot series. This is your chance to hit reset, break the debt cycle, get honest about where your money's going, build a savings plan that actually sticks, and start making financial choices that support the life you want, not just the bills you have.

Because wealth building is not about being perfect. It's about having a plan, knowing your numbers, and making your next move with confidence with a community that cares like we do on Brown Ambition. Each episode gives you practical steps, real talk, and the motivation to stop putting your financial goals on hold. So if you've been saying, I'll get serious about money next month. This is your sign. Start now. Listen to Brown Ambition. That's Brown Ambition and begin your money reboot today. Listen to Brown Ambition on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts.

Yeah, no, I agree basically with all of that. I mean, this is why they're doing what they're doing. And it's just awfully convenient, isn't it? Right? When you have this belief that this is inevitably what's coming, that this is what you are building, and you know that you're right about it, well, it makes it really easy to ignore any other concerns. And especially any concerns about, you know, something concrete that's definitely happening as opposed to this. I mean, the nicest way that I can describe their belief in this is that it's highly speculative. I think that that's too kind and that it's a more accurate way to describe it is it's a religious delusion.

So, but could there not be? Because I don't know if I buy that this is everyone working at these companies, but I think what you're seeing right now with this slowdown, talk might be a result of the schism between there are a lot of rationalists and then there are other people who are working at computer company. And so that's why you've got weirdos like Jacob Coxon going out being like, I'm so scared at a computer without really saying what they're scared of. And also people being like, no, no, I love AI, of course, but this is cloud software.

Because if it was just rationalists, I don't think they'd be so apologetic. But it kind of explains why Amaday is like, I think it's time we finally tell the truth about this while also not telling the truth or saying what they're telling the truth about. I don't think you need that, Ed. I mean, I think if you are, you know, in this sort of rationalist philosophy, the slowdown is like, no, they're they're afraid of the digital God. But they're like, we can't stop if other people are going. Like, I mean, I've, the more I understand about the belief system here, the more I discount, for example, the arguments about. marketing or the arguments about this is a good way to make your technology seem powerful. I increasingly believe like, no, this is just what they believe.

Like, they started these companies for this purpose. And I think it's the controlling atmosphere in those companies. Not everyone at the, but so how do we know that, right? I think it's a controlling atmosphere. Yeah, not everyone else feels that way. But what do we notice with Open AI in the last four or five months? They've had a steady stream of high profile hires from other segments of the tech industry that within like a week or a month leave, right?

So we've been seeing this revolving door. I have to imagine, you know, they're hiring you because they're at a size now where you have to hire executives that aren't from your own, you know, world. They're coming in and perhaps what's happening is they're hearing the rational rhetoric and are like, uh-oh, you know, danger Will Robinson and they're out the door again. Like, I don't want to be involved. Oh, so you think that Jacob Coxon might have, might be genuine? Oh, I think Jacob Coxon is definitely genuine.

Yeah. Look, he gave an interview to Wired a few days ago, okay? And in that interview, this is what they used as the headline at Wired. They said, you know, that he was calling this crunch time for humanity. What they didn't say, what Coxon didn't say in the interview, or at least wasn't published in the interview is, that is a direct quote from Eliezer Yudkowski from like, you know, 15 years ago. What was the exact quote? That this is crunch time for humanity. So the reason the only pushback I'll give you there is he uses the phrase crunch time and he says, and I'm going to pull the quote now, these are actually just literal quotes from my colleagues at Anthropic. The reason I push back is because all of this started with him doing an exclusive with the Wall Street Journal, which is not traditionally how that works. I used to run a PR firm. You don't get an exclusive with the Wall Street Journal because you left to

company and you're scared without the assistance of a PR firm or at least an organization like EA. So I'm like, it's, it feels like this is a co-opted thing. Well, almost. I mean, that might be true. I don't, I don't get the sense it's like he can't even explain what he's worried about. Yeah. Well, no, I mean, I think it's true that, right, you have this schism in the EA rationalist world where you have the EA rationalists who are at the actual labs themselves who are like we have to be the ones to create the digital god. And then you have the giant infrastructure of EA and rationalist institutes that are incredibly well funded that are saying don't try to build the god at all because it's going to be really bad. So I think that's

completely plausible that one of these, you know, obviously has connections to some of these EA institutes from before he took the job at Anthropics. So sure, that's, but I think that just matches what we're saying, right? So he is more from the camp of don't. Yeah, but I think that there's a difference between a young man taking an opportunity, which is genuinely what I think. I really think that that's what he's doing. We can disagree. That's fine. And the EA cause taking advantage of that, because like Daniel Kokelachee, whatever his name is, Kukaracha from AI 2027, quote posted it within like seven minutes. It was very clearly coordinated on some level. But what end, I mean, like, I've seen that, and it might be true.

Maybe he's the Rosa Parks of, like, the EAs or something. No, I don't think he's, I don't think, I don't think, I think he knew that if he did this, it would get a bunch of, a bunch of attention. Because my big thing with this is, I think, like, I think this is a burn after reading situation. I think that the vast majority of, the vast majority of this, the problem's caused by the situation are there are people who genuinely believe this stuff. There are people who cynically believe it, and then there are the rest of them who don't know who to trust, who are just going like, yeah, right? Like, sure, I agree. Like, this is really bad. Like, Sam Altman, I don't think he's a rationalist. I think he's a guy. I think it's actually quite funny watching Sam Altman adjust to this, because he's just like, yeah, slow down safety council, I guess. What do you think? I don't really care. Dario Amadee, I don't.

now that you've said this, I will buy that he actually believes this stuff, which only makes the whole thing weirder. But I think that the mishmash of different cynical people, honest people, everyone's kind of weird, is just, it's causing chaos in the AI industry right now. Like, this is not, people are saying, oh, it's an op, oh, this is all a plan. No, I don't think they have a plan at all. Yeah. No, I mean, I think that's a fair way of putting it. I mean, like, to me, what was more important than Jacob's tweet was the multiple responses from current anthropic employees are like, yeah, he's right. Yeah, probably going to kill him. So strange. And it was so calm. And I think that blew a lot of people's minds, but I'll tell you who was not confused by that,

was probably Adam, right? Yeah, no, that's exactly what I expected. Yeah. Yeah. Yeah, exactly. No, because, you know, there are a lot of people who believe in this stuff who work at these AI labs because they think that that's the best way to keep a handle on this is to be the ones who control coming Godhead. And, you know, I understand, Ed, the impulse to think that, you know, that this is something that people are using cynically. And there probably are some people in the industry who are using it cynically. But there are an awful lot of true believers. There are so, so many of them. And these beliefs about what's coming have been so, so influential in the industry. It is the conventional wisdom about AI in the tech industry at this point is that AGI is coming,

superintelligence is coming, and it is going to radically and permanently transform the world and it's going to come out of these AI companies working with these kinds of AI systems. And there's just no evidence for any of that and a lot of evidence against it. And the thing, just to be clear about what I'm saying, Yeah. Actually, this is a point. I actually need to make this point very clearly. I think that there are a ton of EA rationalist types in this industry. I think Anthropic is full of them. I also think the overall industry led by Sam Altman played into their fear tactics and said, oh, we're scared of what this might do insincerely. Some of them were sincere. Some of them were not. The outcome is they've created a very real panic about nothing. Like they won't talk about the real harms of Ella. They'll talk about the big scary computer that might wake up one day. Well, and I would push back on Sam Altman being insincere about it. That's just the way people talk in the world he came out of.

That's just how they talk. That's how Elon talked. That's how they all, Elon was running all these safety conferences. That's how they all talk about this. I'm serious. Here's the thing, though. Using Elon Musk as an example of someone with a sincere belief, I've read those emails. These are just like, these are just regular rich.

guys being rich. Like, I will buy that Yudkowski believes this or what have you. The rationalist types believe this, what have you. But it's like the idea that Sam Altman, it's just laughable. I just, I don't, that one I don't really, I can't get my head around it. Same with Musk. Well, in your defense, right, the defense of your, your claim there would be the fact that the, the EA rationalist faction of the OpenA. board tried to get rid of Altman, which was at the time widely reported as, this must just be because he's kind of a slimy guy. But, you know, it's also true that specifically the people who voted to do this are strongly aligned with Yukowski's world and thought that he wasn't serious enough about it. So that would actually probably back up the E. Tona, who was a big EA person, she got kicked out by Altman.

Yeah, they cleaned it out. All the EA people on the board got kicked out, including what's, what's her name? God, I can't remember her name. Joseph Gordon Levitt's wife got kicked off. Yeah, that's the person whose name I can't remember. Yeah. Jesus. But, okay, so putting aside that sort of nuance, what's, how else can we empirically test, you know, the effect of this philosophy on these particular companies? Well, I laid out a test in my piece. I said, here's two things we can do. Look at other companies in the space that aren't as strongly connected to rationalism, but are just as cutting edge and really understand technology and see how they talk about it. And I pointed the Navidia in particular because, you know, Jensen Wong is, he started that company, well, Yucast he was still a teenager. He doesn't have a lot of strong connections to that, you know, Silicon Valley futurism.

And I said, also you have to keep an eye on China. They don't, they have no connection to that sort of Silicon Valley philosophy. And for the most part, they really don't talk about AI with that same sort of fear narratives that we do. Well, what happened this week? Jacob Croxon came out. They became a big discussion. What had Jensen Wong immediately said, he said, you know, that's dangerous and outlandish. So immediately, he's like, that claims makes no sense, right?

today China came out what are you talking about a global slowdown? What is this like sci-ops cold war nonsense? Why would we slow down our industry? Because they don't see it that way. They don't see their industry. They don't see AI as like a nuclear weapon or a godhead

they're building. They see it as like an economic tool that they're trying to get trade advantage. So I feel like this week validated the claim that this particular philosophy is playing a big role. It's like that Sherlock Holmes story, the dog that didn't bark, right? The thing The thing that matters here is not the things being said by the two labs that we know are deeply connected to these philosophies.

It's all the other people who are doing cutting edge research on this who know it well who are not coming out and saying, yeah, we also think we're about to kill all of humanity. It's Navidia, which is, you know, they supply all the chips to all these companies. They're intimately involved in these models and their evaluation. And, I mean, they customize the API interfaces to their chips to specifically match with these companies. They know what's going on. And he's like, this is outlandish. What the hell are you talking about? But what's funny is, Jensen Huang is barely a week divorced from declaring AGI is here.

Yeah, but this was a bridge too far. This was a bridge too far this week. So I'm kind of yes ending, because what I'm saying is he has, as I think some have cynically, engaged in this, oh, AGI superintelligence wank, this conversation about it. and now suddenly it's like, oh wait, maybe not that much. Maybe we don't superintelligence that hard. Maybe we actually it's not super intelligence. It's cloud software.

Stop freaking out. And I think that this is the thing. It's regular greedy assholes locking horns with a cult, a religious cult of posters. Yeah. I think that's definitely part of what's going on here. What am I messing? Well, no, I mean, I think, I mean, I just think that it's totally, totally possible that even some of the rich assholes genuinely believe this stuff. It's just that genuine belief for these billionaires, you know, manifests in a different way because they believe that they have, you know, like the power to warp reality with their money, which, you know, to a point is true.

crew. Like, you know, Mark Andreessen is someone who I would never ascribe sincere beliefs to. You know, like I don't, I don't, I'm sure he has some sincere beliefs and I just don't think of any evidence of what they are because I think he just says the stuff that he says because he thinks it'll make him money. But, you know, he put out a manifesto a few years ago where he said that slowing down AI means that you're guilty of murder. murder because cool yeah cool yeah because uh you know that means you're slowing down the advent of a super intelligent AI that will save lives this is EA in reverse exactly yes and you know that's a great excuse to say well but no I have to make the investments that I make and you have to let me do

it without any regulation or slowdown of any kind because otherwise you're a murderer I, okay. As we wrap up, though, I was about to rant about these people. They sicken me. What should we do about this? As in, okay, maybe not the first thought in your head, but what is the way that people should look at this differently? How can they pierce the kind of rationalist veil and at least have a more sensible way of looking at how everything's happening? I mean, I have a few suggestions right off the bat because these are the calls I've been taking all day, every day, if you're like for the last week. First of all is clarity and terminology, right? Just as I wrote that op-ed, Michelle Goldberg has a similar op-ed that came out the next weekend. I think it's really important that people actually understand there is a particular philosophy that's heavily influencing these

companies. At the very least, we should be factoring that in to trying to understand what's going on. There's a good quote in Adam's book from Timnette Garibrew, who basically said, and I'm paraphrasing a little bit, but it was referring to this issue. If it turned out that the major AI companies were run by Scientologists, we'd want to know what's the influence of Scientology on the decisions they're making or how they run their companies. That's what's happening. It's just rationalism EA instead of Scientology. But it's a very eccentric sort of almost cultish internally looking self-police type of belief system that's playing a big role. We should at least know that's the case.

So we can factor that in. Two, we cannot allow the people in these circles to get away with. co-opting the general term AI to refer to specific things they're doing that are causing problems. I see this terminology everywhere in the media this week where it's all about AI is getting out of our control. We need to slow down AI. AI is getting smarter and what's going to happen. We have to be very specific, right? And what they're really- And I made this point in my newsletter as well as it's the terminology of pacing the frontier is in their terms. It's taking their philosophy about this. It's we need to

slow down. Everything we're doing is fine. We're just doing it too fast. Well, and in fact, they want it all connected. This is, this is a Max Tegmark mental model from his book, Intelligence 3.0. They have this idea of AI. Is that the future of life guy? The future life guy. Yeah, which is, which is also deeply, deeply connected to this movie. He's nothing special. But he had this idea, he had this idea of thinking about AI capability like a water level that rises, right? So that, um, if anything is getting better. It leads you to think all of AI capabilities are getting better, which is partially why

OpenAI is spending so much money on obscure math problems. There's not a financial, why aren't you building tools that like businesses will buy? It's because of the mindset of, well, if we can solve a math problem this hard, we can do everything comparably as hard. So they want to connect it all together. But if we get more specific,

the issue is a very specific thing. First of all, it's in the world of LLMs. Most AI is not LLMs. Within LLMs, it's specifically in the world of agents. But not all agents. We use, you know, millions of people use coding agents every day. They never go rogue. There's no issue with them.

We're not worried about that technology. So it's a particular type of agent, which I call over-equipped under-constrained. Or they give them very powerful tools. They run it with no constraints. It's a very specific type of system that caused a hacking attacks and all of their concerns about recursive self-improvement is because they want to next take these over-equipped under-constrained LL empowered agents and allow them to update their own code. So now they're completely unconstrained and unmonertable.

It's that that we're worried about. So I think it's very important to we talk about specifically the thing we're worried about because it's very narrow. And then third, if I'm Congress, don't pass a bill about we need to, in some vague sense, slow down AI, bring these people in to testify. And say, tell me about these over-equipped under-constrained L-L-empowered agents. why are you running those experiments? Was there no way to run those safer?

Is there other things you could be? Why are you focused on that? Is this what you mean when you're saying you're worried? Like, okay, what do you believe? Do you believe? And going through like Yikowski, like we really need transparency, I think, on this situation. That's what I would do, because I think it would really change the dynamic. If we could pick apart what's true about the technology,

what is being hyped and what is being driven by these beliefs and is causing more danger than we. Because you could shut down those over-equipped, under-constrained, ill-empowered agent experiments tomorrow. It would have almost no impact on your ability. All the products they're working on that are actually money-making. It should have no effect on their ARR and their revenue or anything, but they're not because they think that's the way to get to super-intelligence.

So that's what I would do. It's like I would be super clear about what's really happening here. I'd be super clear about what rationalism, EAS philosophy is. And I would start pouring transparency on specifically what they're doing and why, what they believe, what experiments are dangerous, why they're running them so dangerously. That's the hearings we need right now. Adam, any additions? Yeah, yeah.

I think that those are all good ideas, but I think the thing that I would add is we also really need to try to shift the public conversation on this stuff, right? The response to, you know, Cox in the whistleblower at, I don't even want to use the word whistleblower. the guy who left Anthropic and made a big splash recently, the response to that should have been, yeah, of course you believe that because, you know, you're part of this group that has these weird beliefs. Like it's a nothing burger. Like Cal said, I wasn't surprised by this. You know, I wasn't surprised by what he said and what his colleagues and former colleagues said in response. I think that we need to get the message out there that this narrative about existence, risk caused by super intelligent AI going out of control and that that is the most important and most likely thing that we need to worry about with AI. We have to get the word out that that is just a belief that comes out of a almost religious cult-like object, the effect of altruists and rationalists. Like this is not something that is based. in reality. And the real harms imposed by this technology are very concrete and very well known.

Awesome. That is great. I'm so glad you guys were here for this because I feel like at this particular time in history, as both of you have said, there is no clarity here. There are a lot of assumptions being made. And I'm really, thank you so much for joining me because I think this will really help people clear this up. Well, thanks, I think it's the more we get this message out, the better. Oh, by the way, I would add one little extra caveat. Don't look at X. Yes. Yes. It's completely caught up in this engagement farming off of this, this rationalist EA belief system. If you're a reporter out there who's listening to this, do not look at the X conversations.

Call real sources, talk to them in person, because X is off the rails right now. Indeed. All right. I'm going to end it there. This has been better off line. I'm Ed Zichron, joined by Adam Becker, and of course, Cal Newport. I'll be back with the monologue next. Well, I think it's going to be Friday. Anyway, this Friday. Cheers, everyone. Thank you for listening to Better Offline.

The editor and composer of the Better Offline theme song is Mattersowski. You can check out more of his music and audio projects at Mattersowski.com. M-A-T-T-O-S-K-I.com. You can email me at E-Z at Better Offline.com or visit Better Offline.com to find more podcast links, my newsletter. I also really recommend you go to chat. Where's YourEd.com to visit the Discord and go to our slash Better Offline to check out our Reddit. Thank you so much for listening. Better Offline is a production of Cool Zone Media. For more from Cool Zone Media,

visit our website, coolzonemedia.com or check us out on the IHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Can't catch the latest Roland Martin unfiltered podcast? Here's what you missed. There should not be a single law enforcement. Spend agency. It does not have body cameras. It's real. Black farmers have been under attack.

This is just the latest example of them just slapping DEI on anything. It's raw. And I'm sitting here going, they are playing y'all for fools. Catch Roland Martin's daily commentary on the Black Information Network. And download Roland Martin unfiltered on the IHart radio app, Apple Podcasts. Or wherever you get your podcasts. 30 years ago, an island off Manhattan almost brought down New York City. Who will you trust?

your friends and neighbors, or will you trust the people five miles overseas? This is a story about neighbors turning on each other, and what happens when a forgotten place decides it's had enough. But we're not stopping, are we? Listen to revisionist history, the Staten Island Problem, on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. This is an IHeart podcast. Guaranteed human.