Taylor Lorenz’s Power User - Why AI Doom Content Is Everywhere

Episode Date: May 27, 2026

Have you noticed your social media feeds suddenly flooding with terrifying content about AI existential risk? From TikTok clips warning about "AI psychosis" to viral YouTube shorts claiming chatbots are pushing people to the brink, the anti-AI movement has taken over the internet... But what if these "anti-AI" influencers aren't actually what they seem?SUPPORT MY WORK: Buy a paid subscription to my newsletter at usermag.co   Support my work on Patreon for bonus episodes, monthly Q&A livestreams, and more: http://patreon.com/taylorlorenz    In this week's episode of Power User, Washington Post journalist Natasha Tiku join me to pull back the curtain on the massive, billionaire-funded "anti-AI" astroturf campaign. We track the money and expose how the exact same millionaires and billionaires building the world's largest AI companies (like OpenAI and Anthropic) are quietly paying influencers thousands of dollars to promote AI doomsday scenarios.We dive deep into the 25-year history of the AI safety movement, from Eliezer Yudkowski’s rationalist blogs and Harry Potter fan fiction to Effective Altruism and the "paperclip maximizer" theory. We talk about how Silicon Valley elites are co-opting VERY genuine public anxieties about jobs and real-world harms related to AI technology in order to push their own pro-AI, anti-regulatory agendas.We also talk about how to spot the hidden corporate pipelines, dark money nonprofits, and staged research papers designed to control the narrative around AI. If you're seeing AI doomer videos all over your feed, please watch this one!✨ SUPPORT THE CHANNEL: 👉 Substack Newsletter: http://usermag.co 👉 Patreon: patreon.com/taylorlorenz Nitasha's WaPo story on all this: https://www.washingtonpost.com/technology/2026/04/18/ai-doom-influencers-safety/

They just started a YouTube channel and like one of the first videos got 10 million views. You probably would not know that it's like two degrees removed from Anthropic or Open AI or some of these billionaires if you found it on YouTube. Have you seen content creators on Instagram, TikTok, YouTube and podcast clips talking about the existential risk of AI? These creators often appear critical of AI. They seem like they're part of the broader anti-AI movement. And they're talking about ways that AI could go dangerously off the rails and kill us all. But these anti-AI influencers aren't exactly what they seem. Today, Natasha Tiku, a journalist at The Washington Post, is joining me to talk about the AI

safety movement and their quest to take over the internet through influencers. AI safety organizations are paying influencers thousands of dollars to seemingly promote anti-AI content. But the people funding a lot of these AI safety groups are the exact same millionaires and billionaires that are building some of the biggest AI companies on Earth. So we're going to talk about the origins of the AI safety movement, who the main main players are and how this whole thing is basically just a massive astro-turf campaign by billionaires to push their own pro-AI agenda. Hi, Natasha, welcome to power user. Hey, Taylor, our second pod today. I'm so excited to have you here because you wrote a story for The Washington Post that frankly

I was so jealous of because I feel like it's something that I'm just noticing so much online. It's titled Inside a Growing Movement to warn AI could turn on humanity. And you talk about the rise of kind of like, I don't know if you, what you call it, like, AI safety content that seems to be taking over the internet. So, first of all, describe the type of content that you're talking about that people might have seen online in their feeds. Sure. So AI safety is, it's an evolving term. Like, this is sort of the one that the community has landed on. This is almost like a 25 year old movement. So if you've heard about like the singularity, effective altruism, rationalist, transhumanists, like, it's kind of at the, you know,

this intersection. So you may have seen something about, like, have you heard the one about the AI that blackmails you or the AI that might have resorted to murder? It's basically stories about humans losing control of the technology and the technology taking over. Super Intelligence, like it's doomsday scenarios in a way, right? The art will all be very similar. It's like clip art of different computer networks and like maybe spliced with the Terminator. It's maybe not very visually stimulating, but it's very dramatic. It's supposed to make you feel like this is a risk that is not being properly listened to, and you need to pay attention because it is coming now. It's happening soon. The titles are usually very grabby in terms of how to protect

yourself from this or why we have to listen or what these researchers have found. Yeah, to me, I see a lot of the, like, this researcher is blowing the whistle on this, this, this original person, you know, there's all these kind of figures that you start to see once you watch enough of of this content that kind of come up and we're going to talk about some of them today. But it feels very like urgent and scary. And I feel like it's especially doing well on short form video. Yes, there's a lot of long form content that feeds into this as well. But I notice a lot of it on TikTok and Instagram reels where it's like this child used chatupte 56 days later they were dead. That's not as much existential risk. But it's this, this like very scary thing where it makes AI seem like this like

terrifying new technology that like we should be stopping to develop immediately. Yeah. Often if you go to the end, if you look at the long form version of the YouTube short, it is about why the development of this technology needs much more controls. Many of the organizations behind this content would like to see a pause or a slowdown of AI development. I mean, it's videos about the dangers of chatbots, AI psychosis. I mean, these, I would say like the article versions of these videos also perform

really well. I mean, I've written some stuff like this as well where it's like, oh, here's this terrifying story, right, about kind of what happened. But the existential risk seems to really be taking off. I mean, Bernie Sanders posted a very viral video that was him itself on CNN earlier this week talking about how AI is an existential threat to humanity. And you sort of start to hear a lot more of this doomsday scenario content creeping up. I think to a lot of people that encounter this content, it seems good. It seems like they want to share it, especially if they're on the left, because it seems like, whoa, this is a scary technology. Like, we need to raise the alarm.

But today I want to dig into some of the groups that are actually pushing this, and especially the groups that you mentioned in your article. So how did this AI safety movement start? And when did it begin? So it began, I would say, around 2000. So it's actually been around for a while. And it started with people who are not AI researchers. A lot of them are former philosophers from Oxford,

self-taught people who were prescient, right, about the general director. that AI technology was moving toward. People now give them a lot of props for anticipating some of the concerns that we've seen around like chat GPT and very human-like AI. And what this can do to human interaction,

how quickly the technology is developing. But what used to be called, like, concerns about the singularity, sometimes you hear about long-termism or effective altruism. It's basically a group of people who feel a human human a huge emotional, psychological weight to warn the world about the possibility that this technology could very quickly become a threat to all of human civilization. Oftentimes it jumps from just the technology being self-improving to ways that it could replicate itself and various reasons why the AI, once it reaches like a superhuman level

of intelligence, might want to destroy humanity. Not necessarily because it's sentient or has an emotion, but just because, like, we are an inefficiency. There's something called the paperclip maximizer. Have you heard about this? I've heard of this. Correct me if I'm wrong. It's the idea of, like, we could program a super intelligent AI to maximize the number of paper clips that are made. And in process of doing that, I guess they destroy all of humanity to maximize the amount of paper clips that could be made.

This sounds very silly to me, at least when I've talked to kind of researchers, they say, like, well, any sort of of super intelligence system would also understand context. Like part of intelligence is also understanding context of a task and motivation of a task. And so like the idea that an AI would go off the rails in that way seems a little bit silly. But for sure, the people developing these technologies are not always responsible. I think like a lot of people hearing this though, like might have a lot of sympathies for these people, right? Like, I mean, when I hear these arguments, I'm like, yeah, that is scary. I don't trust Elon Musk to not destroy humanity.

This stuff is really scary. So speaking of this like paperclip theory. Who are some of the main proponents pushing this in the 2000? Yeah, I think the one person that you really should probably know about is Eliezer Yiddkowski. He is the founder of this blog called Less Wrong. He's also the author of this fan fiction series called Harry Potter and the Methods of Rationality. And these names might sound goofy, but this has been an entry point for a lot of the young people working in AI today.

You would be surprised how many of them kind of came up through Harry Potter and the methods of rationality through less wrong towards effective altruism. And Eliezer is the one who came up with this paperclip maximizer experiment. And I mean, part of the reason I think it's so interesting that they're only now kind of leaning into influencer marketing is because they really understood the power of memes. Like the whole point is to have this super simplistic analogy that you can share with the world. and that's very memorable. But it is wild to having heard this like a decade or 15 years ago to see like politicians bring it up or like world leaders, deeply serious thinkers with a lot of geopolitical power. I think it's hilarious that Harry Potter fan fiction ended up being this on ramp to this like stop AI movement.

You mentioned effective altruism and it seems like that's what a lot of it, you know, evolved into. So you had Eliasor Yudkowski, you had a couple other kind of figures in his orbit in the early 2000s, screaming about AI. I think everybody thought they were kind of probably pretty sillier that it seemed quite a while off at that point, given where we were with technological development. Can you explain what the effective altruist movement is and when it began? So I would say the effective altruism movement became popular maybe like 15 or so years ago. And when you first started hearing about it, the term that got popularized was earned to give. So this was also a movement targeting young people, ambitious young people, intellectual young people.

And it was arguing that rather than go and work at an NGO, you go work on Wall Street, you go take like that kind of corporate job, earn as much as you can, and then give away those money to efficient charities. At the time, the movement was really focused on global poverty. The idea is like the utilitarian argument about like doing the maximum good means helping the most people. But when this subculture kind of merged together with the rationalists and the transhuman they started to argue that actually the way to help the most people was to warn people

about existential risk from AI, both because it could harm all of humanity, human civilization, but also because in some people's minds not bringing about this good version of superintelligence would mean we wouldn't be able to like colonize the stars and like merge with digital beings. I mean, part of the reason they don't open with some of these arguments is because it really stretches the imagination. It's very hard to talk about in a concrete way. A lot of it is based on this Oxford philosopher named Nick Bostrom. He has this book called Super Intelligence. And as much as they were fringe characters, what's been really interesting is seeing how pivotal they were in the founding of a number of these, like, extremely powerful, close to a trillion-dollar

AI companies. It was Eliezer Yadkowski who introduced Peter Thiel to some of these ideas. And Elon Musk. He's like responsible for some of the like funding going into these companies in early stage because the founders of these companies were also talking about this idea of artificial general intelligence or AGI. I mean, you really don't need to know the terms. You just need to know this concept of an AI that is smarter than a human at anything a human can do. I mean, it really sounds like science fiction. Okay. So there's this group of people, Yudkowski, et cetera, the singularity people, they are pretty anti-AI. Yudkowski for years, even as a teenager was anti-AI. He's writing the Harry Potter fan picture and everybody's getting involved. Then they, as you mentioned, kind of like

loop up with the effective altruists. It sounds like get a little bit more mainstream credibility, start to work the way into these tech companies, amass a little bit more power. But you also kind of alluded to something, which is that like it doesn't seem like they're as anti-AI as you think. Like a lot of them, it seems like are interested in this idea of an aligned AI or at least this idea that like it's not even necessarily that AI is bad, maybe, but it's just that like we don't have an AI that's working in the interest of humanity. Is that right? Yeah, that's such a good way of putting it. And this is one of the concepts that is, I think, hardest for people outside to grasp. Because if you think that a technology is going to destroy the planet, destroy every human on Earth,

cause this kind of existential risk, you would think that you would be doing everything in your power to stop it. But what you quickly realize is that there are not that many groups of people that are also so enamored with the potential for utopia from this technology. Aligned AI is one term, and that means kind of aligning AI with human values. Again, a very, very abstract concept, but basically it's like, you know, you do what the developers want. You have morality system, like ethics and you approach problems in a way humans might, even though you don't have any real world experiences or emotions of your own. Sometimes they used to call it friendly AI. It's both like a utopia and a dystopia. And these are the people who get very excited about like some of

the latest AI developments. They keep very, very close tabs on these so-called frontier lab, so anthropic, open AI, deep mind. And there is a huge amount of overlap between the people funding these technologies and also funding the nonprofits and the other institutions. Okay. So basically like these people that are the anti-AI, stop AI, existential risk people, it's not even that they don't necessarily want AI or they want to pause AI. They want to sort of pause it or alert people to the dangers so that we can develop actually a better AI that works in the interest of humanity theoretically and brings us all utopia. Yeah, I would say there's definitely split within the community about like should we just stop.

Stop it, stop here. You know, this is the safest route. But a lot of these people had been exploring ways that they could bring about the good AI for a long time. Like here, let's look for technical solutions. Let's try to research different ways that we can imbue the AI with a better sense of morality. And you have some of those people now saying, it's not working, we have to stop right now. And others of them are like at the top level of anthropic and open AI.

And they believe that they can still bring about AI in a way that is safe. You mentioned that these people are working in some cases at the top level of these AI companies, but they care a lot about AI safety. They care a lot about where sort of like the development of AI goes. And so they're also giving money to these nonprofits. And these AI safety nonprofits are the ones that are going out and pushing these influence campaigns. Yeah. So some of these nonprofits include the Future of Life Institute, control AI.

Center for AI Safety. You might remember some of their work from very early in the Chachybt era. These were the same people who put out those letters, those open letters signed by all of these top scientists and AI executives talking about existential risk. That's what they were leading with a couple years ago. Now, a lot of them are focused more on these influencer campaigns as a way to, yeah, bring their ideas to the masses.

If you're watching this video and you like like my work, please support me on Patreon via the link below or buy a paid subscription to my tech and online culture newsletter at usermag.co. That's usermag.com. I don't have any long-term brand partnerships and a lot of my content is effectively demonetized. I've lost major brand deals for speaking out on certain issues and for challenging power. As you can imagine, advertisers are not exactly eager to work with somebody who covers a lot of the topics that I cover and talks about the things that I talk about. These videos I make are entirely funded by you and I can can't continue to make them without your support. So if you get any value out of the videos that I

create and you want me to be able to create more, please support me on Patreon or Substack via the links below. On Patreon, I do bonus episodes, monthly Q&A live streams, and post frequent updates about my work. My Substack newsletter gives you a bi-weekly roundup of everything that I'm seeing and reading and paying attention to online. You can also get my newsletter on Patreon. Once again, the links to everything are below in the description. Every dollar of your support makes such a difference. Can you describe what these campaigns look like? Like, who are they reaching out to? And what's the goal of these campaigns?

Why are they working with influencers to push this message of existential risk? One of the nonprofits that I talked to, I think, articulated it best, like the control AI. So they were saying that they already have all of these top AI executives, all of these top scientists talking about existential risk, but it is not proliferating to the general public. Now, all of a sudden, because AI is this central political issue, you have. people who have very strong feelings, deeply care about AI, potentially voters who might actually

change their vote based on what a politician does, and people do not care about existential risk. That sounds so false to me. That seems like all people care about. I mean, this content is so viral. And we have people like Bernie Sanders and the whole media and so many people talking about existential risk all the time. I feel like it's actually become like a pretty major issue. Well, I think that the polling shows that it's actually pretty low down as an issue. If you look at the ranking of AI concerns, oftentimes it is jobs or human AI relationships or things that impact your real life immediately. An existential risk is just hard to get voters really animated about.

That makes a lot of sense because I guess if you're thinking of like what is scariest about AI right now, obviously you're more concerned about. your job before you're concerned about the end of the entire world. Yeah, if you look at like the reaction to the deaths by suicide that got so much attention last year from ChatchipT from Character AI and how much it animated parents and that therefore animated senators, then you have hearings and you have bills, you are not seeing that same attention, that same kind of energy that feels authentic, that feels like a real people are agitating for this change around existential risk.

I mean, I was listening to this like kind of a back and forth debate. Like should we try to join the rest of the people that are upset about AI? And they were like, they have the energy. They have the bodies. We have to move towards this direction. You cannot tell somebody who's worried about their job going away and about their kids' college being totally irrelevant to prioritize existential risk. You have people who care.

Let us join them. Like, let's build a coalition. And what do they want to do when they get that power? Like what are there, what is their sort of policy agenda? Because I think a lot of like progressive see this content and think that it's aligned with their values or maybe think like you said, they might be looking at someone like more perfect union, which is having a lot of success on YouTube by fighting against data centers and, you know, seeing like, oh, wow, maybe this existential risk is kind of a version of that.

But when I look at this movement, it seems a lot more sort of like reactionary than people might imagine. Why should people maybe like look at this a little bit with a side eye? I mean, I would say that asking these organizations straight up, okay, well, what are your political priorities? What's the bill that you're supporting when you're going and talking to these senators about AI blackmailing you or loss of human control around AI? And I would say it hasn't coalesed. It's not, these are not like very politically seasoned, savvy policy wonks. A lot of time, what they told me their goal was was just to make sure that politicians feel,

a sense of urgency around this. They want to be able to have this be part of the mainstream discussion. Future of Life Institute, for example, has this program where they've been paying like $100,000 a month for the past year for creators. You can apply to it. You have to talk about some of the concerns that overlap with them. But you can put your pitch in on their website. They recently convened all of these different groups, including folks from from the far right, labor unions, the AFL-CIO, SAG AFRA, family values groups, and also like some of the more labor activists or like people you might think of as on the left. And they came together and they wrote this like pro-human declaration. I think that within the movement, it's very split about whether this will actually help existential risk in the long run.

Yeah. It seems like there's just so much money in this like AI safety movement. And I'm inherently suspicious of it because this is billions and billions of dollars collectively being poured into something. And I'm sure you're like this as a reporter too. Like when you see that amount of money going to something, like it's usually not something that's ultimately going to benefit like working people or like the average person. And I even think of like the relationship that some of these like safety doomers have with the billionaires that they claim that they want to. regulate. There was a big event in LA a little while ago that Yadkowski and Nate Soros, who co-authored the book, if anyone builds it, everyone dies, had in L.A. and first of all, they banned any

journalist that was critical of AI, which is hilarious because you guys are supposed to be this AI critical movement. You banned anybody that's been critical of AI from your big party, and the party was with Grimes and a bunch of billionaires. So it just seems like this movement is not quite what it seems. And I'm curious if you can kind of talk about that. Yeah, I mean, obviously, I think, follow the money, follow the money, follow the money. And if you look at where the funding is coming from, it is coming from the same people who are benefiting from the technology. I mean, you can just highlight a handful of people, which also goes to show you that it's not necessarily a grassroots movement, right? So you have Dustin Moskowitz, the Facebook co-founder. You have Elon Musk,

Jan Tallinn, who is co-founder of Skype, Vitalik Buterin, who is a co-founder of Ethereum. And then you have a number of very wealthy AI researchers who are also putting, you know, millions or hundreds of thousands of dollars into the AI safety movement. And one thing I think that people just do not appreciate is that these same people are also funding Anthropic and Open AI, literally. The vehicle for Dustin Moskowitz, it used to be called Open Philanthropy, They have since changed the name to coefficient giving. But the former head of the organization, his wife is a co-founder at Anthropic.

He now works at Anthropic. While he was at Open Philanthropy, they gave a $30 million grant to Open AI in order for him to have a board seat. Again, this is like part of the utopian bent. Maybe if we're inside the organizations, we can change from the inside. We can emphasize safety. but you also have the same founders putting a million dollars here or there into various AI startups. If it seems from the outside like this is like a countervailing force against the concentration of power in AI, it's not. It is being funded by people who are at the center of the concentration of power in AI.

It's something that is so hard to explain to people from the outside because, again, the natural responses, if you really think that this is so dangerous and is going to kill everybody, why would you back it? but I think the fact that many of these people are technologists themselves and are so enamored with the idea of being able to recreate human intelligence. Yeah, I think like a lot of people might look at this movement and think that it looks really organic or they see these science YouTubers or, you know, TikTok or that they really respect talking about AI risk and how existential and bad it is. There's a big group chat of progressive leaning influencers that have been talking about getting these deals.

they also were offered a deal from a pro-AI super PAC. A lot of them were against that AI super PAC deal saying, you know, oh, we can't be on the side of billionaires. We can't be seen to promoting AI. That's really bad. But then they're taking these other deals that I would argue are just as much on the side of billionaires. It feels like this sort of astro-turfed billionaire war that's happening on the internet. And I think the pro-AI groups kind of acknowledge that. They say like, yes, I mean, we got a statement from the co-leader of leading the future. a big pro-AI super PAC being like, yes, it's an all-out internet war right now. And like, we need to get our side out. Like, we need better PR for AI that's pro-AI. But I think a lot of what people don't realize is like the quote unquote AI safety or the existential risk or all this sort

of seeming anti-AI content is often also funded by billionaires or is also in service of like a billionaire's mission. Yes. I mean, this is why people who cover this particular intersection or faction of AI have this kind of spreadsheet in their head about how the money is moving around. They really do not make it easy to find or easy to track. Those same handful of billionaires that I mentioned, they have so many different vehicles. There are like crowdfunding sites where you can move some of this money around. If you've heard of anything called like the Survival and Flourishing Fund. I mean, I can't even, let me just go back.

on how hard it is to track. It is so hard to track because not only do they have all these weird like C4 non-profits, like dark money groups, like there's so many dark money groups in this area. But it's also just like for instance, Ayela, the pro-billionaire only fans girl, the girl that literally hosted a pro-billionaire march outside San Francisco when certain politicians were talking about a billionaire tax. This girl is now running an AI-dumer existential risk content creator program where content creators can get thousands of dollars a month and mentorship from girls.

rhymes to post about AI existential risk. So I guess I like kind of really want people to like look at all of this content with more of a critical eye because when you see this stuff, I think a lot of people think like, wow, amazing, progressive content, like you're speaking truth to power. You're calling out the AI industry in this really intense way. You know, you're talking about this threat to humanity. But that's not what these videos are doing. Like these videos are often also funded by billionaires to push a specific message that serves their ultimate. agenda and it's not so clean cut as people might think. Like you said, it's not so like easy to parse if you don't have all that stuff in like this mental map in your head of like where

everyone is with AI. Like it's not so easy to identify. No, not at all. And oftentimes you find a nonprofit. You scroll to the bottom. It says, oh, I'm a sister organization of this organization. You go to that one. You scroll to the bottom. You have to like Google around and find it. I mean, I have pitched to the Washington Post many times like a map of this and they're like, Natasha, I can't put your murder board on the cover of the Washington Post, but I'm like, but people need my murder board, you know, because it's really, really hard to keep track of it. I mean, it's not, I had a similar one for like antitrust policy groups because it was like, oh, Google has funded everybody on the right and the left. And this is why you don't hear about antitrust. Tech organizations just have so much money.

It's like if you put it all together, it doesn't even cost them that much to have this much. influence, right? But they certainly don't want to make it, you know, very readily visible. Partly, I think, like, I wonder why open philanthropy just change its name to coefficient giving. One thing I will say is they used to be pretty transparent, used to be able to go to the site and at least use the search function. And now, like, that's not, that's not working either. So, yeah, I mean, I really have so many Excel sheets that just try to track this behavior. And, like, one thing I found that was like super interesting to me is that a lot of these young people who ended up saying yes to some of these like fellowships or influencer campaigns, they had already been touched

by existing organizations that are part of the AI safety movement like Blue Dot Impact. It's this group that talks about AI safety that encourages people to work to go towards these careers or if you've ever heard of 80,000 hours. Again, another career advice podcast, organization and, you know, they have gotten $31 million over the years from Open Philanthropy, just Open Philanthropy. They've gotten millions of dollars just for their marketing budget in recent years from Open Philanthropy Coefficient Giving. And they are actually one of the inspirations behind focusing on influencers because they just started a YouTube channel and like one of the first videos got 10 million views. And everyone I talked to mentioned that is called AI

in context. You probably would not know that it's like two degrees removed from Anthropic or Open AI or some of these billionaires if you found it on YouTube. Well, that's the thing. I think so much of this content, in fact, and maybe it's just like my personal frustration as well because like a lot of people that watch my content consume. They end up consuming this content or they send it to me and they're like, oh, look at this like brave content. You know, like, oh, they're really calling out like the dangers of AI. And I look at it and I look at the person. And as you said, it's like you kind of go through this like chain of events. You're like, well, what's that and what's that? And is this person getting funded by that person?

Or like they seem like they're getting followed and boosted, you know, inorganically by one of these strange groups. And ultimately it's all sort of comes back to these billionaires. I think like your comparison with Google is so correct. I think of this with meta as well. I've talked so much on my channel about like meta age verification laws and how they literally fund both sides. They fund like the It's sort of like social media, you know, the child safety groups, Open AI as well has been doing this. They've been funding these like child safety groups to push these online child safety laws. And then they're also funding the politicians fighting those laws. And so it's just like either way they have influence.

And when I see a lot of AI content on the internet, aside from the like pro AI accelerationist, like tech maxing people that are like just get on board. AI is coming. We love it. A lot of the content that is performing well is this like doom content. And I don't know. I think that like it's hard to know how much of that is organic and how much of it is shaped by this stuff. Because I think even when it's not being overtly paid for, it's being created by people that are in those networks or influenced by these billionaire campaigns or in this sort of broader EA community.

And so it's hard to know. I don't know. It makes me feel like depressed because it's like anyone that's trying to have like a nuanced opinion on AI. You're being like warped in every direction right by this like ecosystem of of content. Yeah. I mean, I find it increasingly challenging to accept. explain these power dynamics and what's happening behind the scenes for a mainstream audience at the post. Because like I said, you kind of have to have this internal map of who everyone is funded by, what's their ideology, where are they coming from? Part of what really compelled me to focus on the story is looking at the fact that often they were advising these influencers to pull content ideas from research papers.

And a lot of times it's like research papers from anthropic. Anthropic. Like that blackmail example actually is from an anthropic paper. And like people who work at Anthropic have said they put that example in there as a way to alert politicians to what might happen. They don't hide it. Like if you look a little bit further down in the paper, it says this might have been a little bit of a checkoff's gun because we gave them a loophole for the ability to blackmail. We put the information very close together. So this is a really contrived situation. But here you have the fastest growing company. this year, right, beating the other AI companies. Exactly. You have basically like the biggest and most powerful AI company right now putting stuff in their research papers that then influencer groups and AI safety groups will flag to influencers to then make this content that seems like it's challenging the AI industry and seems like it's holding like power to account almost, but they're getting it from anthropic and it's like all funded by billionaires. Like it just feels like it's all part of this same big ecosystem and these people are so cozy with each other as well. I just want to like shake a lot of

people too, like, that I see on consuming this content on the internet and thinking that it's organic and sort of falling for it because I'm like, you know that all these people are together in the same parties in San Francisco. You know that they all hang out together. I mean, I think that that Yadkowski event in L.A. was such a perfect encapsulation because you had like the richest of the rich people. You have people that they're ostensibly against Ayala, the pro billionaire girl, is there, you know, and then they're claiming to be like holding power to account. Like, it's just very ridiculous. Yeah.

Even a couple of years ago, Sam Altman, the CEO of Open AI, was saying that Yudkowski should maybe deserve the Nobel Prize for highlighting these issues early. One of the people that I talked to for my article, Daniel Kokotajlo, who was an Open AI researcher and, you know, came out and spoke out against the company's safety practices, which he said were irresponsible. He's also behind this violence. essay called AI 2027, which is one of the things that many of the influencers use as a source material for their viral posts.

But I was asking him about this. Like when people understand that it is the same people who are funding Open AI and Anthropic that are behind the AI safety movement, like, how do you talk about this? And he was addressing this criticism of the movement that's existed for a few years, which is like, is it just inorganic hype to make these companies appear more powerful, their technology more inevitable, their ability to shift our entire economy and the way we are oriented with each other in the world. Like, is it just making them appear not like a normal technology where you have tradeoffs,

but like this event horizon thing beyond which we can't even see? And he said that is completely false. That's not why the people who raise these concerns are doing it. But he did say, like very plainly, but I do think they're too. too cozy with the AI companies, and that leads to them pushing for solutions that won't really hurt the AI companies. Recently there was a march in San Francisco about stopping AI development and pausing, and one of the people who was not part of the march, but she identified, oh, that's an anthropic employee,

and immediately got a lot of flack from it online. It just goes to show you that this is the milieu in which this environment is operating. You are very close to the same people who are building the technology that you believe will lead to this potential extinction event. And the closeness I don't think you're right is probably not apparent at all to somebody who's consuming the content via YouTube, Instagram, TikTok. No. And also, I mean, I know that that person says it's completely false and they might truly believe that. That it's not all marketing. And I'm sympathetic to that. I think a lot of people say, oh, this is just a marketing sci-op to make it seem more powerful.

But that is ultimately the effect that it has as well. And I do think that there is this like fervor around AI where it is treated as this like radically different technology. I was talking to a tech policy person a little while ago that was like, you know, the internet was radically transforming technology that had like insane bad effects and good effects and stuff. Right. But we didn't have this sort of like hyperbolic like existential like crisis over it, although we could have. But here like with this technology we do and it's I think part of the reason we do have that existential. crisis is because we have all these billionaires involved and people at these companies involved. And I do think it's ultimately, it's not even that it's sort of in their interest to push these

narratives, these hyperbolic narratives, but I think they also buy into a lot of these hyperbolic narratives. They really believe that they could create a doomsday scenario or something with their technology. And maybe they can. But I don't know. I just think like at the end of the day, this is new technology. And we can regulate technology. And we can understand technology. And this is not like they've invented like a portal to a new dimension. You know what I mean? Like these technologies are invented through incremental developments that thousands of people have hands in. And that is something that we can control. Yes, yes. And I would say from long time AI policy academics and experts that I talk to, I think one thing that they really caution against is

this worldview requires a kind of, requires you to anthropomorphicry. morphies the technology in a way, right? Like even the paperclip thing, you're saying that it's, it doesn't have human motivations. But I mean, it is pulled from sci-fi, right? And if you think about it as what you're trying to stop is a superhuman intelligence, rather than a product that so many knobs and dials are being pulled behind the scenes, in order for this technology to get better at any different domain, they have to go out, find PhDs in biochemistry, have them write it down, generate tens of thousands of synthetic examples. So these are the people that are really, really twisting the knobs and dials to get the most out of this technology.

I've just never understood and I've asked so many people, why is it the people that are doing the most manipulation, most human manipulation, the most product design, the most, you know, the most optimization to the technology that are so convinced that it will be this kind of runaway scenario. I would just say though that like one thing I thought a lot about while doing this piece is this, great paper that came out a few years ago that talks about this community and like the field building and epistemics. And one thing they highlighted is that it is the money, but also the AI safety movement is very involved in knowledge production, cooperative knowledge production. And this is a work of a lot of the AI safety nonprofits. They are writing papers, developing technical solutions that reference each other. If you start going into the footnotes of the papers, it might even reference a less wrong blog post or this is not peer review. The whole industry has moved away from

peer reviewed. But it talked about like how it's it's like social cohesion. One of the animating beliefs is that people are not listening. This is an urgent issue. You have to show them how important this is, many of the ways that it can go wrong. And it talked about like right now, this This information is just being shared with elites and it's influencing academia, it's influencing policy, but what happens when these ideas kind of break containment and go to the masses? That's partly why I was really interested in in looking at this because it's AI 2027. If anyone builds it, everyone dies. It's anthropic papers.

That's what's going viral in this AI safety content. It's content created in the belly of the beast. Yeah. Given the fact that so much of this content is sort of so manipulated, it's so hard to tell, like, what's organic, what's not organic. Some of this AI, anti-AI content is not actually, I would argue, very anti-AI. It's coming from the same AI companies. Like, how can somebody know what to trust? And how can somebody know what content to take seriously when we talk about AI?

Because I think, I mean, you mentioned this earlier, but like positioning it as this existential risk can be really scary and people can feel overwhelmed. And I think it's bad to buy into their. ideology. I think that's a really bad ideology to have, like this existential risk nonsense. Like, it ignores, as you said, like all the real humans turning the knobs to like make actual changes in these products. So like, how do you kind of navigate and know what information to really trust and know who to actually listen to, you know, about AI? I mean, I wish I had a simple answer for people. This in the entire, I don't know how long I've been covering tech, like 15 years longer. this is the most challenging time because you could technically call up 10 reputable experts who would show up on the cover of the New York Times and get like very different answers.

So I think that it's integral to know where people are coming from, to know where their funding is coming from, to know who they studied under, to know where they got their ideas, to look at what forums they participate in. And unfortunately, though, you, it's, again, it's like you almost need to have an understanding of the ecosystem to be able to find it. You could go to their LinkedIn, scroll down. You could look at the last five papers. You could look at who they work with and not necessarily know that they might be coming from a strong ideological bent when it seems like it's just in the pursuit of science or in the pursuit of truth. But I would definitely say, don't stop at the YouTube video. Look at whether there is a disclosed sponsor. Look at the sponsor.

Go to their website. I'm not saying, do your own research. I'm actually just saying that be extremely skeptical of everything that you hear from AI. And maybe hopefully they'll let me put my murder board on the cover of the Washington Post. I know. I hope so. I just want them to be skeptical of this like AI safety content and like look at it twice. Because I think a lot, there's just all these people that I see grifting online right now that are getting huge audiences by ostensibly challenging.

challenging power, speaking out against AI dangers. And these people are extremely well funded. These people are well connected. And they're not doing that. Like they're not, this is my frustration with this whole movement. It's like they're not challenging AI.

They're not really anti-AI. They're not really, like they're grifting off this like very organic anti-AI movement. And I think like stoking the flames among the public in a really corrosive way. We're like, as you said, like we're not able to get smart regulation because now we have a bunch of people that believe a bunch of hyperbolic nonsense about.

AI, when we could actually like make a lot of changes to AI products that would be good and engage with these products in like, I think, more nuanced ways than than probably a lot of the AI safety people are putting forward. I mean, I will be really curious to see whether some of the research we, we saw that shows like 70%, 85% of people believe that AI should be regulated, but that ranks existential risk of like among the populace very low. I would be very curious to see if it changes. because like that's the thing about coalition building

and trying to kind of like crest off the wave of this very organic pushback against the concentration of power in AI companies or your inability to say no. But I don't think they need it to rank high, right? Like some guy from one of these AI safety groups wrote an article in New York Times pushing what I would argue are horrifying laws,

restrictions around speech and information, quoting a very well-known anti-social media group that has partnered with a slew of anti-LGBQ hate groups. They weren't even sent. entering AI risk in it necessarily or not existential risk, but they're putting forward really dangerous laws. And yeah, maybe they're not getting people on board with existential risk as much, although I do think they are, you know, even if they don't get people on board with the existential risk stuff

as much, they can hop on the anti-data center stuff. They can hop on all the other stuff. They can hop on all the idea that that AI is drinking up all of our water, which is debatably untrue. Like, you know, I think they can just like, there's so much anti-AI sentiment for good reason that these extremely well-funded groups are seeking to co-op. And I just want people to, like, think twice before you let yourself be co-opt by this really deranged movement of people that just hang out with a bunch of billionaires and, like, also work in the AI industry, I guess.

Like, I don't think that they necessarily have our best interests at heart. I mean, maybe I'm even more cynical than that, because actually I would say they have already so much power in D.C. because Anthropic and at the time, opening, was like during the Biden administration was pushing a very different type of message. I would say the whole way that we think about AI safety, the way you think about like testing beforehand, red teaming, looking at bio, chemical, nuclear, very particular kinds of risks, down to the way that they test it. Part of the reason why we didn't fully anticipate or understand or some people didn't anticipate

or understand how it might affect people mentally when you have eight hours. long conversations with the chatbot is because they were in these red teaming, in these safety exercises, they were just doing like one back and forth or a few back and forth because they have a particular idea of the ways that this technology could go wrong. So I would argue we are already living in a world where they have had massive influence on how we think about testing this technology, the direction it could possibly go. So this to me is more like a resurgence of it. Well, Natasha, thank you so much for joining me and talking about all of this.

It's been so great. Thanks for having me talk about one of my favorite subjects that I'm obsessed with, so really appreciate it. All right, that's it for this week's Power User. If you like my work, please, please support me on Patreon or Substack via the link below. You can buy a paid subscription to my Substack newsletter at UserMag.co. I send a weekly roundup of everything that I'm reading and following online. You can also support me on Patreon where I do bonus episodes, a monthly Q&A live stream, and more.

I'll be back next week with a brand new episode of Power user, see you then.