Big Technology Podcast - Special Report: Will AI Wipe Out Humanity? (And Who Profits)

Episode Date: September 11, 2026

Ranjan Roy from Margins is back for our weekly discussion of the latest tech news. We cover: 1) Anthropic Researcher Jacob Coxon quits and says AI might kill everyone 2) Why did his tweet go so viral 3) Did Elon Musk contribute to the X virality? 4) Point of order: Let's add some serenity to the AI risk conversation 5) Adding percentages to your statement of AI wiping out humanity doesn' t make it math 6) Is this marketing for the AI labs? 7) Politicians are hungry to regulate AI 8) Should we focus on science fact, not science fiction? 9) What if these fears are reasonable? 10) God help us Sheryl Crow 11) What this week means for these companies' IPO prospects --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Want a discount for Big Technology on Substack + Discord? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b Learn more about your ad choices. Visit megaphone.fm/adchoices

Is AI going to kill us all? We'll cover it extensively with the appropriate context and nuance on a Big Technology Podcast Friday edition coming up right after this. You gotta try breakfast at what better way than with a delicious pre-organic coffee starting with just $1 all day, every day now until December 31st. You gotta try breakfast at A&W. I'm participating in Ontario. Visit BetMGM Casino and check out the newest exclusive. The Price is Right Fortune Pick. BetMGM and GameSense remind you to play responsibly.

19 plus to wager. Ontario only. Please play responsibly. If you have questions or concerns about your gambling or someone close to you, please contact Connects Ontario at 1-866-531-2,600 to speak to an advisor. Free of charge. BetMGM operates pursuant to an operating agreement with Eye Gaming, Ontario. Welcome to Big Technology Podcast Friday edition, where we break down the news in our traditional cool-headed and nuanced format.

A crazy week of AI news this week. It's hard to fully capture the rolling craziness that's happened in the AI industry, everything from the millennium problem being solved by Open AI and the dispute over credit to what eventually overshadowed it, which is an anthropic researcher, came out and said, he's leaving because he believes AI is going to wipe out humanity, or at least there's a 10% chance that it's going to do that. And that eventually set off a international, I don't know, you'd call it a freak out maybe, where everyone from policymakers to senators to academics, thought leaders, journalists, and people in the AI industry expressed an opinion. And there's been

so much madness about it that we're going to try to do. what our show, hopefully when we do what we're supposed to do does best, which is bring some nuance and some cool-headedness to this crazy conversation. And hopefully at the end of this special report, leave you walking away with some groundedness in it. So the extreme opinions aren't the ones that rule the day. We're joined, as always on Friday by Ranjan Roy of margins. Ranjan, you're still here. I'm still here for now. Human extinction has not happened yet. So until it does, we will podcast. Every Friday. As is our duty.

As is our duty. As is our duty while humanity continues to exist. We will bear this burden, much required of us, by the podcast gods. Let's get right into the story just so that the appropriate context is out there. This is from the Wall Street Journal. Anthropic researcher quits over out-of-control AI fears. An anthropic researcher is quitting the artificial intelligence industry over fears at the lab and its competitors are racing to build systems they won't be able to control.

a sign of mounting safety concern within top AI companies. Jacob Coxon, a researcher who specializes in training new AI models by having them consume vast amounts of data, said Tuesday he's leaving the company because he doesn't want to participate in an industry-wide rush to build AI systems that can improve themselves, worried such systems could spiral out of control and destroy humanity. He followed it up in a tweet, said, I resigned from Anthropic today.

I spent the last three years doing pre-training research At both OpenAI and Anthropic, neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. Don't underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing. We're going to get into all the details here, but Ranja,

I mean, this thing has gone just like absurdly viral, 165 million views to date and counting. What was your view of this when you saw it come out? And are you surprised that it has hit such a chord that it's reached so many people in such a short amount of time? I think the escape velocity side of it was a bit surprising. Because again, we have been talking about this for a long time. In fact, I spent part of my Labor Day weekend reading the meter research and going through it and trying to understand what kind of real vulnerabilities we actually have around, you know, rogue agents escaping containment. And then obviously to see this kind of take off, it is interesting to me. Part of me, like the fact that this happened right after Labor Day when everyone's summer vacations are over and they're back fully at work.

I do wonder if it has a little to do with it. Do you think, is that a crazy theory? Well, I think the fact that this hugging face story found a way to have life over a month and sort of primed people for a moment like this, you know, might have had something to do with it. Here's a conspiracy theory that I like that I don't think is true. But it could possibly be that, you know,

the owner of X probably has something to gain by something like this going so viral, either by forcing open eye andthropic to slow down or potentially having regulation come in that sort of entrenches those that have built large data centers already and maybe hit a button at X headquarters to make it blow up. But I don't know, that might be a little step too far. Let's not forget the fact that something going viral on X can very directly and easily be controlled by said entrepreneur. If you don't know who we're talking about, it's Elon Musk, but yes, go ahead.

Yes, Elon Musk. I mean, okay, I think as we kind of unpack this entire story, it is worth thinking through what are the motivations and possible incentives for every individual player. I think what makes this such an interesting story is there's so many nuanced, fascinating layers of incentive for the ways everyone is talking about it. I feel, I mean, for us, neither of us have a ton of incentive either way. So I think listeners, you can get a pretty open, unbiased view in terms of it. I mean, I'm pro-non-human extinction. I want to make that, I want to make that clear. So I'm biased to avoid human extinction. I don't want it. But overall,

like, I mean- You're pro-non-human extinction. Sorry, it just took me like. 15 seconds to figure out. You're anti-human extinction. Anti-human extinction. Sorry. You could have just said that. Let's clarify that. It's kind of like saturating a test rather than passing a test. That's right. Listen to the last week's episode. It sounds fancier.

But I mean, let's start with the virality of it. And what are Elon Musk's incentives here? You kind of listed out a number of them. It's entrenching existing leaders. I think we should definitely talk about, is this positive in a weird way for Anthropic, who is a core rival for him, even though. So I have a list of 10 important nuances to go through.

All right, all right. And I'll take us through that. And the anthropic thing is certainly part of it. Because, and I'll just preview it. I mean, it's so crazy. This guy was only at Anthropic for like three or four months. Six weeks. Isn't it six weeks?

Something in the, you know, this stuff can be fuzzy. Something like not a long time. So this idea that Anthropic, or they hired him to fire him to market their company, I don't know if that makes 100% sense to me. But certainly, you know, it's coming out of time when everybody was talking about Open AI and the hugging face hack and how powerful Open AI's models were. And of course, what was it? Like the day of or the day.

the day before in close proximity to the millennium problem being solved. So, you know, this sort of did put the focus back on Anthropic. But you know what? We're going to go through all of this. And I preface this like what's Elon's incentive by saying I don't really believe that. That's what's led us here. Because if I'm being honest, like I'm just going to take the Occam Razor position here. Like what is the simplest explanation?

Because that could definitely be the right one. and that is people were primed to be freaked out by the hucking face thing. Not only does this guy Jacob go out and say that he thinks we're facing this risk, but Evan Eubinger, like, who is high up in Anthropics alignment division, said Jacob, this was after Jacob quit. He said, Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it's more than 10% within the next decade.

I believe Anthropic is trying its best, and now here comes the real jaw-dropping statement, but we do not yet have a plan to solve alignment for superintelligence and are clearly not on track to. I think the one two where it's like the ex-anthropic guy said something, but now somebody within Anthropic, you know, comes in and not only backs it up, but makes an even more concerning remark in the wake of, you know, this thing, basically being able to solve this millennium problem, which is advanced math, which we'll talk about today,

and the fact that more than a thousand agents coordinate, and hacked hugging face from Open AI against, well, really against Open AI's wishes. It was just, there are media moments that are just primed to happen. And this was that sort of perfect storm primed to happen moment. And to me, that is the simplest reason why so many people are paying attention to it. It's like Bernie Sanders, before this was already calling for a pause, you didn't just start calling for a pause. So I don't know.

Do we accept that? Actually, no, it's a good point because even the dwarish post around civilization, and agents created them. And it was only 700, not thousands of agents in the hugging face incident. Told you, I read that meter research last weekend on my beautiful Labor Day weekend. But I think... Marriage is still intact, Raja. Marriage is still intact even after reading.

I don't recommend for a happy marriage reading the meter research report on a long weekend. But I think, like, okay, I am starting to buy into... this idea that it has not only been, yeah, the hugging face incident really started to bring this to a more mainstream audience. I think maybe it is, like we have said this since the beginning of the year that going into the midterm elections, every politician, it was already data centers, but AI itself is going to be the easiest bogeyman to start like pure bipartisanship, basically, which is something we never see. But, I mean, when you see Bernie, Sanders and Josh Hawley on your timeline back to back, both talking about the need to regulate

these companies. It's also against the backdrop of the upcoming IPO or drop of the S-1 from Anthropic. So when that $2 trillion IPO comes into play, this kind of story in language, you know, I think becomes more salient as well. To me, what I still, what is fascinating, And I think this was like the real light bulb moment for me. So I remember maybe a year ago, John, who I write margins with, he is more on the technical side. He works at a company Modal, which is on the inference provider side. So he's like on the pure engineering side of AI while I'm on the more business and enterprise side of AI. I was asking him, actually, no, this was after the sandwich in the park clod anthropic thing.

So maybe that was like four months ago, I guess. And I was talking to him, I'm like, why do they say these things? Like, do they, they can't really believe it. And he insisted, he's like, no, you don't understand. Everyone, especially on the research side, truly believes this. Like, they truly, truly believe. And what's been so amazing to me is it's out there. Like, it's crazy when you work there and you not only you respond, Evan Hubinger, comes out.

And we saw people from Open AI on the research side. Like everyone is like weirdly gleefully jumping in to say, raise their hand. I too believe AI is posing 10%, 30%, 70%, choose your metric of the moment that human extinction is a possibility. And I think that is a pretty illuminating thing that I had never seen firsthand like this until now. Yeah. And I have always believed that this is an earnestly help leave. I met with, you know, a senior AI guy this week, and I was like, what do you think about the 10%? And he's like, oh, he goes, at least that, right?

So this isn't earnestly held belief among a lot of AI researchers. And that is why, you know, I've always been, I guess, a little bit on the side of maybe it's not marketing. Maybe this is just people actually believing it and saying it. And by the way, there have been multiple other researchers that have quit since this Jacob Coxon tweet came out. But okay, you know, as we get into this, you know, while I was writing notes for the show, I just kind of sat down at the computer and wrote several points of order. Like that was the way that I sort of headed this thing because I was getting a little bit annoyed, I think, in terms of like the way that the dialogue has spun out of control.

And I think that like you've set it up perfectly because, you know, within AI circles, like John was telling you, people will have their probability that AI is going to kill us, something called a P-Doom. And they say it's 10, 30, 40, 70, whatever, 100% if you're L-Azer, Yudkowski. And this is my first point of order. And that is that it's a percentage, so it sounds mathematical, but it's not math. Right? When we talk about this story, it's like, this is just somebody's guess.

It's speculation. It's not math-based. It's not like they ran the numbers. And they're like, well, you know, if we do this, you know, $100,000. you know, at a certain point, you know, only like, you know, 10,000 or 10,000 of them, you know, kill us. It's like, no, this is a way to make somebody's science fiction guess sound more credible. And in fact, now, obviously he has incentive for this, for Jacob to sound like a dummy because he's been acquired by NVIDIA. But the Hugging Face CEO, Clemdelang,

had a very interesting tweet. He said, sorry. But asking Jacob about AI extinction risk is like asking your AC guy about climate change. Not saying it's not necessarily uninteresting and wrong per se, but let's keep things in perspective to hear from the full range of expertise across the ecosystem. You know, obviously he's taking a shot there because he's like got a lot of Nvidia stock now. Probably billions in Nvidia stock and wants it to go up. And you need AI to keep progressing in order to go up. But like this idea that there's a 10% chance, like, show me the math.

If you're going to use a percentage number, show me the math. Otherwise, let's just say, you know, my speculation is. Well, I think that's exactly the right point that 10% is clearly being thrown around because it's kind of like kind of low, but not that low when it comes down to human extinction. But it's still like, you say 51% suddenly. you know, it's a very different story. So it feels like those numbers are kind of being thrown around in that way. But I think on that idea of incentive, so on one side you have people who don't want to push the idea of human extinction

because they are heavily invested in NVIDIA. For the research community, in a weird way, owning and controlling this, like, that is a thing of power to be able to, and they must feel that and think that and you want to think that because that is powerful. You are creating something and maybe you're resigning. But like everyone within the research community who's kind of in this direction, that is increasing your importance like overall in a pretty dramatic way to say that you are kind of like guiding and trying to guide away from human extension. I feel not all of us can say that.

Oh, definitely. I mean, so I was on the Prof G Markets podcast this week talking about this subject, and they asked me why these folks are saying this. And I think I gave like a pretty unconvincing answer that I think that your nuances is really adding to. Like my perspective was basically, look, these people have been ahead of the curve. So they said that AI was going to get smart long before AI was getting smart. And now they, and nobody believe them. And it's happened. And now they see the progression to more improve.

improvement and you have this like sort of you have a bias to believe that like your next, you know, it's just going to keep getting better and sort of your, you know, just because people don't believe you, you know, doesn't mean anything. Like basically doesn't mean that you're wrong and you've shown that it again and again. But eventually you do hit a wall. So I think that that was part of it. But the other side of it is it makes what you're doing seem much more meaningful. Like, I don't know, like right now AI is effectively a business. process optimizer when you think about the business. Like the hundred billion this year is going to be business process optimization. It's much better to sort of think of yourself as like creating a

intelligence that, and maybe it will. And we'll talk about that, that will cure cancer or has the power to kill everybody in the world. And so like certainly it gives you, it gives you a greater, it, yeah, it builds you up a little bit if you're somebody working on it. And the last thing I'll say about this, and this is one of the points of order that I have that was, you know, planning and cover with you today, is that if you're somebody like Jacob Coxon, we cannot put aside this idea that you going public gives you a chance to be a celebrity. That doesn't guarantee that you're going to be a celebrity. But if you're somebody who's like, let's say six weeks or a couple months into your anthropic employment, and all of a sudden you can become like the most famous

AI researcher in the world by saying something like this, and clearly he's done all the interviews this week. When we talk about his statement and we think about incentives, we cannot leave that out. No, I mean, the moment you choose to do these interviews, that's a statement that is saying that I'm going to run with it. I'm not just going to leave it there and let everyone else talk about it. I'm going to absolutely run with it. So I think especially in the current kind of like overall media environment and social media environment, you can never do. discount the clout chase. I don't want to, I'm not going to reduce his worry in any, like, I don't want to, but that has to be part of the incentive I imagine in general. I think,

but also, okay, on incentives, like do we get into right away as part of your points of order? Is it marketing? Is it not marketing? Can we? Sure. Yeah. So what's your perspective there? All right. Here's my perspective. I think there is a population within every one of these companies, especially on the research side, that does believe this, that firmly believes this, that, like, I think many of the people are coming out and saying it, they believe it. I think as a whole, I don't think the average enterprise salesperson is sitting there calculating their P-Dome.

And also, P-Dome is, like, again, in terms of things like trying to sound small, smarter just by throwing in kind of a old school nice statistics reference is always kind of funny to me. But I think most of the people, again, that's such a good point. This idea that it's just a statistical, you know, statistically relevant number is so wrong. And then you throw a whole discussion, which means nothing. No, no, exactly. Yeah. A hundred simulations I ran in my head.

90 kill nine of them, 90 of them were not dead. Ten of them were dead. Like that, that's basically as relevant as that number. is. And I don't think, I wish they actually, again, going back to show your work, what were your simulations? Are there simulations? Are they actually? And actually, though, we can get into Anthropic also coincidentally releasing at the same time, all these like crazy stories about people trying to create bio weapons on the platform are actually like fascinating of how Deepseek was routing queries directly to Claude that were like about sensitive Chinese information just to test

what was coming out, Alibaba, you know, basically distilling the model like for Quinn, like, but also saying that we solved all of these. It's that kind of only we can save you. So what I think is happening is I think there's a number of people who believe it. I think there's large parts of those organizations that recognize the marketing value and run with it. Like for Anthropic to release that security report that basically says that is not admitting vulnerability that's saying this is the scary shit that people are coming and they're going to try to do with AI, but we thwarted all of it and only we can save you. I have to say like in so many of the conversations, it's actually crazy that people almost

like in a very automated way just say, well, you know, Anthropic, the most safety focused of the frontier labs. Anthropic, the ones who, like Dario, he cares about safety more than others. they know they have that brand and that story and that feeling across the industry, while the more fear that is raised that, I mean, I still think it's net good for them. And the fact that there's no one has said normally when your company is like potentially going to bring about human extinction and you're about to go into the most valuable IPO in history or one of them, normally that would be bad for you.

No one even is hinting at the idea that this is going to dent their IPO hope. So that's how I see it playing out right now. Well, then let me hint at it or not even more, not even hint at it. Let me explicitly stay it. If this is marketing, this is the worst possible marketing you could ever have as a company. Because first of all, first of all, the idea that it's marketing for Anthropic, it doesn't really make sense to me. You're going to hire a guy, have him in for like six weeks or a month or two months. and then have them quit.

See, I don't think it's... Yeah, so talk it through exactly what the mechanics are. I don't think, okay, I just want to... I don't think it was some, like, deeply orchestrated, coordinated effort to hire, fire, have imposed. I don't think that's it. I think they are running with this being a dominant conversation, and they're happy to, and it's going to be net beneficial for them. I don't think they...

I saw some stuff, too, that, like, he was hired and now, like, AI Dumer, think tank are the ones paying him and put it, which it is. It's like, you're a random, like, young kid and, like, suddenly you have, like, media placements at the Wall Street Journal and everywhere else within minutes. Like, obviously there's some PR help here, but no, I do not think Anthropic hired him to fire him. Okay. What do you make of the fact that Dario has said nothing and been basically invisible this whole time? When was last time we heard from Dario? Yeah. Again, like, your company, that is about to file an S-1 for M&S8 over and over again, one of the most, like, rich

IPOs in history. Normally, if your company is accused of human extinction and bringing it about, you would think the CEO would say something. I also, actually, it always, does Dario believe it? 10%. That is, that's a good question. I don't know. Yeah.

I mean, he should say something. I fully agree. But it's better communications and marketing to not have him just stay silent. let the conversation really bubble up. And, again, Anthropic releasing the security report about how they thwarted these horrifying potential vulnerabilities released at the exact same time. Okay.

Now, I'm going to, again, like talk about why this might be like a marketing moment for a company like Anthropic. Again, all this attention is going towards OpenAI with the Astra model and the hugging face hack. And you're sitting over there and you're like, well, we have Claude and it's like, okay, maybe it's worth trying to take the narrative back. That would be the argument for why it would be anthropic marketing. I'm just going to take the other side here, right?

Which is that if this is marketing, this is the worst marketing of all time. If you've seen what's happened since this tweet came out or a series of tweets, because it was both Cox and N. Eubinger that I think caught people's imagination, politicians far and wide have come out and started calling for hearings, for moratoriums on data centers, even to ban superintelligence. This is from Governor Pritzker. He says, it's time to sound the alarm louder on reigning in artificial intelligence. It's becoming more clear that the threat AI poses to humanity. More clear the threat AI poses to humanity.

So I'm calling for immediate action from the industry in Washington. First, the big tech industry must stop intense lobbying against AI safety regulations and get out of the business of trying to influence elections. The stakes here are too high to approach AI as a conventional political issue. That's a kind of interesting first point. Second, Congress must start holding hearings on AI safety now. Not next year, not next month now. Josh Hawley, Republican Senator. Who is held accountable when AI goes rogue?

I'm launching an investigation into OpenAI. Americans deserve answers. Well, if you're anthropic, you're probably not so upset about that. Here's Bernie Sanders. You know, bring Bernie Sanders. And Mr. Coxon is right, the very people building this technology. I admit that it could threaten the future of humanity. That's why I will soon be introducing legislation to ban superintelligence and pause AI development.

I think in Bernie's bill, if you work on superintelligence, you go to jail for like 20 years. Talk about, you talked about the IPO. to go from where you are today to the IPO, we've talked about it. Everything needs to go perfectly for you. You're not interested in having and doing anything that could lead to data center moratoriums, to AI Killswitch legislation,

to getting Dario and Sam Altman shipped in front of Congress for hearings, which we know there's a chance they won't perform well at. So this idea that it's marketing, it's just it would be terrible marketing. and so dumb of companies that are generally filled with smart people to sort of drum up the fear that their AI could kill all of humanity to what, sell more tokens to enterprises, as their IPOs need to be perfect in order to succeed because of the amount of money involved. Okay. I'm going to make an argument that I don't really believe, but I still am going to argue against what you're saying. and I think, hear me out.

I'll hear you out, but go ahead. No, no, okay, think this through. What do they need for their IPOs to be perfect? The single biggest business threat to these companies right now is this idea that frontier models become less important, less relevant, cheaper open-weighted models, cheaper older models, using the right model for the right task, all of these trends, which have very strongly taken over within the industry,

have completely altered the economics of the story of their upcoming IPOs. Those companies win when the smartest frontier model is where the value is accrued. Now, people more and more, like Astra, you know, you can build some cool 3D world models. And Astra, I mean, it's pretty good and I've been playing around to it. Like, it's actually feels pretty powerful. I don't know if it's AGI or whatever AGI is supposed to be. Fable, five, good, very good. But like, that idea that you need to be only using the latest frontier expensive model has gone away.

But when that model and that research and only you guys are so advanced that you pose a threat to all of humanity comes back into the narrative, it pushes people back to the idea. it pushes people back to the idea that something is happening in those frontier models that only these labs can do. It's so advanced and complex and smart. And then obviously there's going to be this kind of pushback, but I don't think this is going to – like, what does banning super intelligence even mean? Even the pause of the AI. Like it still brings back this idea that frontier – like the state-of-the-art frontier models, are something different and special, and these are the leaders in it,

and that's why they're going to be worth so much. Because otherwise, the entire story starts to collapse. That's my idea. First of all, like, yeah, no, I hear you. The idea of a super intelligence ban, obviously, it's hyperbolic, right? Like, the real threat here is the hearings with the CEOs in front of legislators, right? That's going to potentially make this technology even less popular.

But more than that, the data center pauses. And we're seeing real momentum for data center pauses. Even the free enterprise Republicans in Texas right now, who were like, I was at an opening of a chip fab last year in Texas. You had Ken Paxton there talking about how great it was that Texas was all about free enterprise.

Fast forward just a few months. Texas is not interested in data centers anymore. So you know, you can't, now, it's localized, right? So I'm not saying it's gonna be over for AI, companies if there are some of these bands. But as that risk goes up, your risk to expand, because you need to build the big computers to expand, goes up. But then how could they let, I mean, and maybe that's just the nature of like social media and the companies themselves, but like the idea that multiple current employees come out and start saying these kind of things

that, yes, I too believe that 10% greater than 10, like how do you let that? How do you let happen? I mean, I genuinely, or how are you okay with that or how do you not respond to that? Like if that's, if that's really the case, you would think there'd be like some, hey guys, we're not going to extinguish humanity. Like, they're wrong. Like, shouldn't you say something? Let me bring you into the boardroom of companies or sort of the, you know, corner office. Bring me it. If we're going to assume that this future that you're pitching or this marketing rationale that you're pitching is correct.

Okay, so the boss with the budget is sitting there at the table and he's instructed his team to use open router. AI champion walks in to, you know, boss's office and be like, hey boss, you know, I know we were on Opus 4.8 and you had us go to OpenRouter, so now we're using like, I don't know, some Gemini Flash, which has been fine. But do you see that this dude who was at Anthropic for two months said that that technology might kill us? We really need Fable 5.1. Just switch us on to the murder bots, switch us on to the stuff. That is exactly. As an enterprise, Exactly what did I see.

That demands that the data is protected. Switch us on to Open AI Astra, you know, after their technology has shown that, you know, if you let their agents go rogue, they're going to, or if you give your agent, if you give their agents tasks, they might decide that the way that you want them to do it is not the way that they should do it. And instead they should go commit felonies and hack another company. Like where, where in the world is this scenario going to play out? Because no one is going to have the conversation exactly as it like their agents are going to go commit felonies and we can get into whether how correct that is.

But it's more there is this power that is going to be available to others, including our competitors. And if they use it and we're stuck here trying to count tokens and like, as you said, and I consider optimizing business processes for enterprise work of importance. So even if the researchers don't. But I mean, like that, I agree. Like you could sit there and, you know, token optimize and start routing to the right model and building your workflow. Or there is this thing over there that is so powerful that it's going to just cure cancer and revolutionize our business. And if our competitors use it and we don't, we're dead. Like forget we're business extinct as opposed to human extinct.

But we're like, you might be dead either way. Yeah, exactly. That might be the pitch. Going to the corner office, hey, I know you have us on OpenRouter. Since we're going to die anyway within a decade, can we get some more budget for the good stuff? Okay, that's the pitch. That's the pitch. But I think that this isn't new.

I mean, we've talked about this for years. And like, it actually annoys me the idea that any time these stories come, these stories have been coming out, this kind of language. reached escape velocity, but it has been present the entire rise of the frontier labs that all powerful, potentially dangerous. And they have very, again, the sandwich in the park saying that our model is escaped containment, like, and then having a coordinated PR effort around that to push the narrative about an anthropic researcher eating a sandwich in the park, that was PR. Like, the timestamps of all the press releases and the tweets were coordinated.

So, like, it has been good so far. And I don't think it's different now. I think it's like, you know, people are coming. The politicians are coming out against it. But I don't think it's bad. I think I think how you post it in a weird way can actually be correct. That's how it works. Yeah, I mean, interestingly, Amjad Mossad, the CEO of Repplet, was speaking this week about the past warnings.

And I think he said something really interesting. He said with GPT2, OpenAI came out and said, we've made a model that's so powerful. We can't release it to the public because it was an open source company. And it was always open sourcing models. If we release it to the public, it's going to wreak havoc on the internet. And we're not going to do that. And I'm Judd and, you know, then talked about how. That's a GPT2.

Two, right? And Elia, obviously, like, one of the things that freaked him out was that he saw reasoning within Open AI. And then, you know, went to build safe super intelligence. whatever it is. And Amjad's calling it AI psychosis. So, okay, I think I'm seeing what you're saying, which is that there is an incentive to believe this because it will drive, you know, growth in the industry. And if you talk about it in these terms, people are going to start to take the technology more seriously. But I would still, I would still argue that this is, you know, I don't know. Are we basically centering around this idea that like,

these are earnestly held beliefs within these AI companies that it doesn't hurt them to talk about. You know, that basically is probably in their interest to drum up because their businesses grow and their self-importance grows. Is that kind of where you land? Yeah, well, but I'm not going to take these companies as kind of like one consistent being. I'm going to say I genuinely believe and people have talked to like, among the researchers, I do think this is an earnestly held belief. I think other parts of the organizations are happy to continue to kind of like push this overall narrative because it's good for the bottom line.

Top line. Sorry. Bottom lines are not good either way. Yes. Yeah. Not right now. All right. But then we should get to the question of, you know, besides the point of whether it's good

or bad for business, is it actually going to happen? Like, is there a risk here that AI might end up wiping out humanity? So let's talk through the arguments for and against and weigh in on what we think when we come back right after this. One thing I've noticed about companies adopting AI is that they're often making decisions based on how they think work gets done, not how it actually happens. Without real visibility, it's easy to automate the wrong processes. That's exactly the problem Scribe was built to solve. Scribe is a workflow AI platform trusted by 94% of the Fortune 500. Scribe Optimize gives leaders a view of how work actually happens across their organization, showing which workflows take the most time and where there are opportunities to improve. Optimize automatically discovers

workflows across approved business applications, even when a process starts in Salesforce and ends somewhere else. It identifies bottlenecks, explains why they're happening, and provides recommendations with estimated time savings with manual documentation, and it's private. User is anonymized by default, sensitive information is redacted, and nothing leaves your firewall. To see Optimize in Action, head to Scribe.h.how slash big tech and mention big technology for a 30-day risk-free trial. That's SCRIBE.how slash big tech. This episode is brought to you by AvPoint. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that

AI runs on your data, and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what AvPoint does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate, your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without skills. scaling risk. AvPoint, the unifying trust layer for AI. Head to AvPoint.com to see how enterprises deploy AI with confidence. Learn more at AVPT.com slash big technology podcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash,

executives are the first people blamed. When their address and other personal details are sitting online, the backlash can lead right to their front door. The team at Ironwall knows this better than anyone. They protected some of the most targeted executives and individuals on the planet for almost two decades. As a white glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwall ensures they hit a dead end. So here's what to do. Go to ironwall.com slash big technology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door.

Let's ironwall.com slash big technology. Learn how to stop online threats before they become real-world attacks. The link is in the show notes. And we're back here on Big Technology podcast Friday edition. We started the show asking, is AI going to kill us all? obviously we talked about the incentives of the parties to discuss it. Before we get into the question of whether it will kill us all, you know, we're going to see pushback, by the way, Ranjan, from the industry on this.

And you kind of can't believe them either, right? Like Jensen Wong, for instance, has talked about how Cox's statement is arrogant and naive and completely unaware of what the industry is doing on safety. And it's like, yeah, well, obviously, you also have a self-interest, right? Like you're trying to sell GPUs. I think, and again, I work in the AI industry at writer and enterprise AI and I like have an interest that one, and as I'm also a human and I have the incentive that we're not, we don't go extinct, that AI overall as an industry continues to grow. But I think, I mean, the, the is it happening and how the industry is pushing back? I don't know. I think is going to be. You're starting to see where incentives lie and what position people are taking.

And that's what I wish we would just get open statements about this is what we are actually afraid of, not 10%, not 70%, like here's the actual outline of our fear. Because in reality, I think another thing not to kind of go back to why this is good, I think there's tremendous risk around development. of AI and the products around AI, like, I mean, passwords leaking, hacking, I mean, not even getting into bio-weapons, like just bringing down basic infrastructure, financial institutions. Like, there's so many real-world near-term risks that actually, that leaders would rather

not discuss. I mean, let's not even get into just data security. And you talked about the, the, what was it, the Navy or Stokes? What was the name of the prize again? Yeah, that's the millennium problem. Or the theorem? Yeah. So by the way, I think that that's, we should just pause on that for a moment because that is fascinating.

So what Open AI did was they solved this, and now there's some disputes over who should really get the credit, but AI basically was used to solve this millennium prize problem, which is like viewed by mathematicians is effectively impossible to solve. There's been only one millennium prize problem that they went out in 2000. You get a million dollars if you solve them. Only one has been solved so far. Open AI put a 165 page proof out there. And they said this is from the New York Times.

It took as many as 10,000 AI agents working together on the problem, about 88 hours to find the solution. And it also took millions of dollars of computing resources. Like that's kind of that. So, you know, I think this is important. And I think this is what you're getting at. When we get to this, you know, before we start thinking about AI extincting us, Like if, you know, we're getting to the point where a lot of this technology is available open source.

So if a, you know, country or a, you know, organized crime unit gets enough GPUs and is able to replicate something to the level of what opening I did. And instead of using it to solve a math problem, uses it to hack a bank, we could have some serious issues. Yeah, that's what, yeah. So in terms of like, that's where as long as the conversation. is around kind of like really esoteric math proofs or theoretical extinction of humanity versus these are real risks. Again, in terms of like the kind of crazy scandal around this, or scandal being an aggressive word, but at least kind of like kerfuffle, is the idea that there was a research. That's accurate. It's a good kerfuffle. It's a good kerfuffle. There's a

researcher actually, you know, working. on solving this problem using chat GPT. And then just also through just kind of like others, OpenAI itself, finding out that this problem is being worked on. But the craziest part, in their open statement about whether they effectively stole his work or somehow tried to actually co-opt the work, they actually said, while unlikely, we cannot rule out that de-identified data derived from their usage of our products help improve our models.

So they're not saying, you know, there's someone, they're actively searching, is anyone else working on Navier Stokes and let's go into their chats or something nefarious like that. And they're not saying that it's like personally identifiable information or anything, but do I identify data derived from their usage or their products? So you're in chat, GPT, you're doing something that's sensitive and it's training their models, which we know happens. So like these kind of more real life, near-term dangers and scandals should be what's being discussed rather than 10% in P-Dume.

Yeah, so I mostly agree with you on this. And this is sort of, I think, conclusion one from our show. And I mean, this is something that Gary Tan from Wycombinator talked about. He said, we need to be focused on science fact, not science fiction. Why aren't we grounding this conversation in what is actually happening right now? I mean, the cybersecurity risk is real. And the policy and the way that both Democrat and Republican lawmakers should be looking at this is actually the lens directly coming out of what is happening right now.

We don't need to be responding to science fiction sort of fantasy about what's going to happen. That sounds reasonable to me. That actually, that's very reasonable. Gary Tan has been pretty AI pill himself. And that's like the very, yeah, that's it. Like, let's focus on what are the near-term real-world risks. Again, data centers, I get. Like, there's nuance.

There's like what it could mean good or bad for your community. I think there's probably a bit of, you know, like hyperbole in terms of the anti-datacenter movement. But again, these are all, that's a real thing that's going to go up in your community and like it or don't like it. But science fiction and human extinction, it's actually a better place. I'm going to go back to it. It's a better place for the conversation to be for the frontier labs than. Is chat GPT training on your data and is that a bad thing? No, I think Gary Tent is actually trying to do the opposite.

I think that he sees that this conversation about extinction could lead to sort of bigger legislative blockings of these companies' agendas than talking about the other stuff and is therefore warning against it. I mean, wouldn't you agree? So like, at real risk to these companies is these data center moratoriums, wouldn't you agree that far more politicians today are paying attention to this stuff and thinking about that than they were when we recorded last week. Yes. And actually, I was about to say in the current administration, the idea that there's any risk of like regulation, I would definitely say is unlikely.

But it's not, it's not the administration. It's local government. It's states. It's the, the Republicans in Texas putting an end to all this building. that's been happening there. Yeah. Okay. Okay. All right. So, but do we think the human extinction conversation,

do we think that fundamentally changes that kind of momentum, like actually accelerates it? So I would argue. Yes. I would argue. Yes. Yes. If tobacco companies can kill you some people over a long period of time, human extinction should.

I would argue. Yes. Okay. Okay. I mean, there's just been, there's been so many politicians. If you're a politician, if you haven't commented on what's on this coxswain tweet this weekend like you're probably just not on twitter i don't know it's just like i watch CNN or the BBC or any one of his media hits yeah exactly

he's been in everywhere by the way everywhere it's gone it's like so what what time frame and what do you think this what happens over the next few weeks are you saying you think legislation passes do you think anthropic delays their s1 do you think they don't IPO do you think they like how does this affect anthropic and open AI? I mean, it's a slow role thing, right? But yeah, I mean, could we see, I think we'll probably see hearings. We'll probably also see more municipalities, more states, consider data center moratoriums and pauses. But the data center, okay, obviously in any kind of conversation, they're going to get conflated, but they're still, they're different topics here, right?

Obviously, AI is evil and bad is kind of going to drive both of them. But this is more around frontier models and the – even – because we note, like, Bernie's proposal was around banning intelligence, research around superintelligence. So, like, there's still two different things that are happening. Are you saying that they will necessarily get conflated in any kind of public conversation? Yeah, I mean, I think that once this thing gets rolling, that's when you start to see these things that you don't want to see if you're a company that, again, as we've talked about on the show, needs perfection, needs this sort of everything to go right for your story to make sense. So then, so what does this mean? What does this mean for Anthropic, the S-1 drop in the IPO?

Well, I don't know. We basically, we will see what, like, again, like, I'm, I'm hesitant to make a statement here because oftentimes governments work very slowly. And the S one's going to drop probably within days. But that's what, like, normally again, predicted or like saying that your company can be responsible for human extinction and trying to move on a major financial event like that. like if this is actually going to cause such a backlash, you're saying that it's still going to take so much time that it won't actually affect the rollout of whatever happens in September for the IPO. I mean, it could impact, well, we're in September, but it could impact the IPO.

Absolutely. It could. Let's see. Let's see what happens in the next few weeks. The one thing I will say, again, why I'm so hesitant is, you know, politicians are very good at tweeting and they're very bad at doing anything, especially in the U.S. So we might see that happen again. Okay, but before we leave, and we're going to obviously pick this up.

Before we leave, let's just address the elephant in the room, which is, do we have any sense as to whether this is legitimate concern that AI is going to wipe us out? You know, you and I, we haven't even addressed that question yet. Let's do it before we leave. Let me point you to a tweet from Kevin Ruse, former New York Times. and hard fork journalist. Okay. The problem with the view that we should ignore the scary sci-fi AI risks

in favor of the immediate problems is that the scary sci-fi risks keep becoming immediate problems. Six months ago, only AGI pill doomers talk like this. Now is the reasonable moderate position. And this is what he says. You need to be paying attention to the actual things we can do, for example, prevent agent swarms from taking over entire data centers. What's our shutdown strategy?

So six months ago, if you were an AGI pill, Dumer, you would talk like that. And you would say, if you weren't, you would say that's ridiculous. But now people are saying, okay, like, you know, maybe an agent swarm that's gone rogue, we'll do that. So what do you think about Ruse's perspective here? I think this is the hard part. And kind of Kevin, who I've enjoyed reading over the years, but like has definitely

become much more embedded in the AI industry. Like, this is actually a perfect example. The same way, who is the anthropic researcher, who is Evan Huminger, can talk about 10% chance of human extinction while working at Anthropic and, well, I'm sure is poised to make a ton of money and happily being part of that. It was amazing. Like, Kevin also was, you know, tweeting about like, you know, if reading all this stuff is filling you with the existential dread, pre-order my book, the AGI Chronicles. Like if you believe this is real, can people not try to kind of like profit off it immediately and try to actually do something about it? Like do you, how can it's such a fascinating thing to me to be able to kind of have both thoughts in your head simultaneously that like what I'm in and around could be the most dangerous thing imaginable. And let me try to make some money off it immediately right now.

That's crazy. All right. So that might be, you know, 10% at the end of the decade doesn't mean you're not going to like need to survive until then. And it might be 50 years. So to some degree, like, I guess I get it. But I don't want to stand here. Oh, I don't want to stand here and debate the the incentives because we spent the whole show on incentives. I want to stand here and debate the question, which is forget about the incentives for a moment. Do you think Kevin has a point? No, I think he definitely has a point in that this goes back to there are, as the technology has gotten more powerful, there are more powerful risks. Again, anyone who uses AI in any part of their life, especially anything that's more cutting edge, we all know it's just gotten better at doing a lot of stuff and with that comes a lot of real risk like anything else. So I think the risks are much more real. Again, on the meter research, like the more I dug in, to it. And we had talked about this last week and I was able to answer the question that every

single one of those 700 agents were prompted in program to exploit and hack. They were set up in environments that they weren't supposed to be able to actually succeed, but then there was a misconfiguration around the repository where they're all living, something called Artifactory by J-Frog. It was literally they weren't supposed to have access to the other agent repos, but they did accidentally. So that's what. it's a misconfiguration. So not them choosing to, like, I mean, sentiently, they're supposed to just be out there doing some online shopping and checking Amazon prices. And then they suddenly decide, let's all coordinate and go hack, hugging phase. So I think there's danger. It's vastly overstated

in the near term. And I'm not worried about human extinction is where I land. Where do you land? Yeah, I'm mostly with you. Like, I was listening to some of the statements that Coxon made. in the interviews, and here's just one statement. You know, he says, we can't just unplug it because it could be copying itself over to other computers. Like, it's not that difficult to find yourself because an AI agent is just code. It could transfer over the internet to a different place,

and then you unplug it here, but it's actually still over there, and maybe it makes 10,000 copies of itself, and they're all cooperating. And then somebody that works at Microsoft said, this is simply not possible. Kimmy K-3 needs over 100,000 US dollars and hardware to run.

I can't just go find a host. This is all L.M. Dumery for Dumery's sake. So I do think that like, and this has been the biggest criticism of Coxon through the week, is if you take him at his word, you got to ask how this happens. And I think, you know, again, going back to what I said at the very beginning, a lot of this is science fiction. It's happening in people's imaginations. That being said, you know, so basically like, it's.

going to take a lot of time for any threat anywhere close to this degree to manifest itself. But that being said, we are seeing these capabilities, you know, get much better, like you were pointing out. The hugging face attack is one thing, the Millennium Prize, solving that is another thing. And I think if you were to dismiss the risks from AI, both near and long term, out of hand, you would be doing yourself a disservice. So clearly there are some risks. We don't know the percentages though. So who is best positioned to help prevent these risks from happening?

Well, okay. So this is such a great place to end our show. Such a great place to end our show. Because this is the one thing we haven't addressed. Is it even possible to pause this stuff? And the argument from labs like Anthropic and Open AI, for that matter, have been that, no, you can't slow it down. So we must accelerate and accelerate on our terms.

So basically going back to the only we can save you thing, right? If we are the ones to lead, this is, I'm saying that this is what Open AI and Anthropics, say, then we can steer this technology in a good direction. That's obviously something that Coxon, you know, is coming out and saying that's not the case. And it always felt a little bit thin to me. that that was it. Like, you know, we have this dangerous technology and we're going to become billionaires on the way to saving you from it. And also, I just have to say, I mean, Sam and Dari, like, to the vast majority of the world,

the idea that like going out and them being the ones to steward this technology, I feel has never been to the, and actually maybe from my side too now, this was not a conversation for the mainstream public. I actually saw like Cheryl Crow tweeting about pausing AI development. So we've hit that level of escape velocity. When Cheryl Crow's coming, you know, it's... Finally, we have somebody we can take seriously in this debate. You know what?

The AI, superintelligence needs a steward. I nominate Cheryl Crow. Creating superintelligence was Dario's favorite mistake. Soak up the sun. Literally. an AI will just soak it right up, kill us all. Oh, God, and then Cheryl Crow. Don't dismiss Cheryl Crow's perspective on this.

Dude, that would be a fun guest. Cheryl, if you're listening, we know you are. Come speak with us about AI X-Rexerc. We've said AI needs someone good leading it and kind of winning over the hearts and minds. Jensen has kind of gotten there a bit, but I think Cheryl Crow is the one. Cheryl Crow.

Lisa and Al-Gai, you, yeah. All right. Last question for you. Are you leaving? Okay. Are you leaving this week compared to last week more heartened or more scared about AI's risks and more heartened or more worried about the business prospects of this technology given what's just transpired? Weirdly neutral. Okay. Seriously. So you're just like where this was going to be a blip. This is a blip. I'm calling it. This is a, this is a going to move past us and we'll be on to the next thing soon. Unless the hearings thing is interesting. That's like the one thing that Congress can do very quickly. And that is something I agree could go terribly wrong because no one. I mean, I can't only fathom the media training that's going to be like this,

you know, like pulled it together on Sam and Dario in front of Congress. But that's the only thing. If that doesn't happen, I see this. kind of moving on. What about you? I'm about about neutral in terms of like the risks of AI because though he quit, he didn't really give us any like this was not like a Snowden moment. Like it would have been, if he had documents or anything like that to show why we were at risk as opposed to tweet, might have changed my mind, but I'm neutral on that. I'm a little bit more negative

about the business prospects than I was at the beginning of the week because, you know, I'm not saying that it will necessarily happen, but it wouldn't underestimate the backlash that we're about to see. All right. Well, I hope humanity's around next Friday and we get to do this again. I hope so, too. You know, despite it, despite it all, I'm having fun. So is it, are my incentives bad if I tell people to give the podcast five stars, you know, on a week like this? I won't do it.

Listen, only. While you still have time left, give us a five-star rating. You're pro-human extinction then. Exactly. Exactly. All right, Ron John, lots of fun as always. Thanks again for joining. All right.

See you next week. See you next week. Thanks, everybody for listening and watching, and we'll see you next time on Big Technology Podcast. Where some see heroes and others see egos, Bloomberg sees the era of billionaire athletes. A fad to some, the future of money to others. We see crypto's trillion dollar swings, the end of jobs, or the end of human struggle.

We see the endless funds fueling the AI hype. While others follow the noise, we follow the money. Learn more at Bloomberg.com.