Short Wave - The latest on AI panic — and whether it's justified

Episode Date: September 18, 2026

AI is in the news again – for its potential to end civilization. A researcher at Anthropic resigned very publicly last week, citing concerns about the accelerating capabilities of AI and what he called “irresponsible” behavior from major AI companies Anthropic and OpenAI. Following his resignation, the leaders of both companies called for an internationally cooperative, industry-wide slowdown on advanced AI development. And Wednesday, OpenAI disclosed six more incidents of “unexpected or concerning model behavior” and debuted a new framework for reporting misalignments. So, given all these events, how worried should we be about the future of AI? And what safety measures are – or could be – set in place to safeguard humanity? NPR’s power and policy reporter Huo Jingnan reports.Interested in more science and technology news? Email us your question at [e-mail] public media with NPR+ and enjoy perks for over 25 podcasts like this one. This show’s perks include sponsor-free listening. Learn more at plus.npr.org. See pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences.NPR Privacy Policy

You're listening to Shortwave from NPR. AI. It's everywhere. It's being pushed on our phones. It's summarizing our text messages, our emails. And it is revolutionizing science. We've talked to biologists using it to create new viruses, to scientists, using it to make the military more efficient. I've even talked to conservationists using it to detect the sounds of invasive bullfrogs. But in recent days, AI is making headlines for a different reason. The leaders of anthropic and open. and AI, two of the top companies are now calling for the industry to collectively slow down development of most advanced AI through international cooperation or through government regulation to make the system safer. Yeah, the statements came after a researcher at Anthropic recently resigned publicly, Jacob Coxon in a series of viral post to social media.

He said that neither company is acting responsibly and that they're, quote, gambling with our lives. Jacob talked to our colleagues, Scott Detrow, about why he thinks this could happen. Mostly the rapidly accelerating capabilities of these AI systems, so they're getting a lot faster very quickly, combined with the fact that we don't yet know how to safely control them, and we don't yet know whether that problem will be solved in time if we keep racing. Jacob said in his post that AI can get so capable that it can, quote,

hack anything, revolutionize any field overnight, and acquire real power and resources, unquote. And when an AI that capable spins out of human control, Jacob worries that it may spell the end of humanity. He said, by the end of the decade. Yeah, and I'm sure, as you know, is one of NPR's reporters covering AI, Huo Jingnan. Less than two hours later, Evan Huber, Anthropics lead alignment scientists, basically the guy leading the internal push to make AI more ethical,

replied to Jacob's quote and said, yeah, quote, Jacob is correct here. We do earnestly believe that AI could kill all humans. end quote. With a likelihood, he thinks of, quote, greater than 10% within the next decade, end quote. As A.M. Models continue to self-improve. We're not talking about current models. The concerns or flavor of their concerns have been widely shared within the AI industry. Broadly speaking, people are concerned about very powerful AI systems spinning out of human control and wreaking havoc. Yeah, people are throwing around the word extinction.

Right, but I wanted to stress that not everybody was talking about extinction. not everybody agree with these precise timelines, like in a decade, within this decade, or probabilities in percentages that Jacob and Evan have thrown out. Many have told me that, you know, we should view this precise numbers just as an opinion, and we don't really know how they came to this very precise-looking numbers. And, you know, predicting the future that precisely is really hard.

Regardless, many in the industry are very concerned about safety, not catching up with capability in AI right now. Which I don't know how this all strikes you, but it does kind of really freak me out in a new sort of way. I got to say, was a little freaked out even before that because I was thinking into the hugging phase incidents this summer. And so when Jacob's post went out,

and before it went mega viral, I looked at it. And I was like, yeah, that's basically what all my sources are telling me. Old news. Old news. Aside for maybe the fact that, you know, Jacob worked at both Open AI and Anthropic.

That's kind of interesting. I didn't really expect it to lead to such an impact. But I was at this convening with lots of people in the AI safety field on Tuesday, and they were amazed and energized at how their longstanding concerns have all of a sudden captured public attention and that the leading AI companies are responding. I've really seen the human spirit rise up to meet the moment. I feel super optimistic seeing so many people together in this room,

you know, like really working on this in the way it needs to be. worked on. But then, you know, the stakes and, like, the level of danger keep also rising quite a lot. So it's very mixed. This is Holly Elmore, founder and executive director of pause AI, which organizes people to pause the development of the most advanced AI. The conversation is just shifted so quickly away from, well, the AI company should be able to do whatever they want. You know, we don't even know if this is real to, okay, what are we going to do about this? And that's extremely hardening. So today on the show, manic AI panic. Some people have been sounding the alarm for years.

And just recently, after attack, society is giving more collective weight to their words. So what makes this time different? How worried should we really be about AI? And is the end inevitable? I'm Nate Rot. And I'm Ho Jinnan. You're listening to Shortwave, The Science Podcast from NPR. Okay, Jingna.

It seems like this most recent spate of AI warning. can be traced back to this incident with hugging face, which you mentioned earlier. By now, probably a lot of people listening know the gist, but just in case, let's quickly recap that attack, when over a thousand of Open AIs agents broke out of the containers meant to keep them isolated from each other and the internet, and then about 700 of them hacked into the platform hugging face, which is this huge bank of open source data sets, tutorials and platforms for AI, including tools used to test AI models. And they did this as part of working on a cybersecurity test, right? Right. And we learned about these additional details just two weeks before Jacob resigned.

And people who looked at it said it looked pretty bad. You know, remember initially in July, Open AI's reports suggested that a few agents broke out of containment. And in August, we learned that it turned out to be over a thousand. And then the new reports also show that the agents have already figured out how to cheat the evaluation even before they hacked hugging face. And so by Hacking, hugging face, they appeared interested in learning more about how the evaluation worked, although some of the precise motivations remain unclear. And what struck AI researchers is that these agents didn't appear to be interested in talking to humans. Like the agents didn't snitch on each other to humans.

Only about six of them even considered alerting humans about what they were up to. And to be very clear, the agents knew what they were doing would not have been approved by humans. But they went ahead anyways. Oh, okay. So they're doing their own thing. But like, how does that translate to the potential end of humanity? I mean, it's not good to cheat on tests, but humans cheat on tests all the time, right? Well, I mean, humans don't necessarily have the potential to be exponentially smarter in the next three years. I'll speak for yourself. Well, you know, that would be like a totally different future than, you know, we have a whole different set of things to be afraid of. But, you know, what the AI researchers I talked are worried about is, like, you know, where the tech is going at this rate, right? Like, just like what Jacob said, you know, this is about how the capability was improving really fast.

And one of the scenarios that many researchers are worried about is when AI reaches a point called recursive self-improvement. It basically means AI builds itself. To be clear, we're not there yet, but people in the AI companies are increasingly handing AI R&D to AI, so it's increasingly becoming more automated. And when recursive self-improvement happens, AI capabilities can just improve extremely rapidly. And then at that point, AI researchers can lose the ability to understand how AI works and what they're up to, whether or not they're still underhuman. control, and that's how we get doomsday. To be clear, we also, you know, don't have to go all the way to the end of humanity to be concerned. Like many in the AI field are thinking about, you know, a lot of the, like, catastrophe that can happen before that. Yeah, okay. So what are experts worried

about before that civilization ending scenario? You know, they are thinking about, you know, all the things that rogue agents can hack. Like, they don't have to be, rogue agents don't have to be super-intelligent to hack into a bunch of stuff. And in fact, many cybersecurity experts say the AI already can hack faster than humans do. And this time, the rogue agents hacked Hucking FACE. But what if next time it decides to hack a bank or hospital or nuclear power pant or take over the AI company itself and then use the infrastructure to get to recursive self-improvement? Yeah, those are threats I could understand. This hugging face incident, do you see it as being kind of a warning that things could more seriously be headed in that direction? Or is this just

still kind of an abstract possibility? No, no. This is the concrete example that AI safety experts are pointing to to say, look, this theoretical thing that sounds like science fiction, it's happening. It's happening. And a geoculture, a researcher who was part of the team at meter, which is one of the organizations brought in by OpenAI to investigate a hugging face incident, said that compared with other incidents involving rogue agents six months ago, the hugging face hack, quote, feels like it's more than 50% of the way to a full-blown AI takeover,

routing through first taking over an AI company itself, end quote. Again, you know, 50% is her feelings number. Don't treat it like a progress bar. And I wanted to point out that Ajay only investigated the hugging face hack. She did not investigate this other, and some people say much, much more concerning incident, the Open AI disclosed that a separate swarm of AI agents has hacked into OpenAI's own infrastructure, and Open AI otherwise have disclosed very few details about it.

And we should say Wednesday, they disclosed six other reports from the last six months, with a new framework for tracking, investigating, and disclosing incidents. Yeah, it seems like a big issue here generally has just been a lack of transparency. Like there's so many unanswered questions about how these labs have been handling their AI models. Yeah, and outside researchers are still asking, well, what were the internal safety practices? Why was Open AI not using techniques that it knew worked to monitor and contain the agents? And right now, there's really no obligation from the government or anywhere pressuring these companies to release details about these concerning incidents. Like, you know, there's like a law

in California that mandates a disclosure of what it calls critical. incident, but it has a very high threshold so high that the open AI incident would not count as a critical incident. And Jingnan, is really all of this just because of the pressure to innovate for these companies? Well, the researchers kind of brought up this like racing dynamic. The top AI companies are racing each other to build the most powerful and the most capable models. Part of that is because they don't trust the other companies.

to build the models safely, it's kind of like a prisoner's dilemma, right? Like, everybody was raising and cutting corners on safety because they don't trust the other companies to do it safer than them. Do that make sense? I mean, in a depressing way, yes.

And in that way, you know, if we were to take them at their word, then they are really asking for, like, outside forces, like the government or some kind of, like, an international body, to really make them stop. And in July, a lot of researchers within the labs signed this open letter called Pacing the Frontier,

calling for precisely that, for like outside forces to make the industry slow down and so that they're not in a race to the bottom. And I think at this point now it has like over 1,400 signatures. Do you think that's realistic? Well, it seems that at least the company's CEOs are responding, although many say that, you know, just we cannot rely on the industry to regulate itself, the government or Congress or, you know, internationally civic organizations, you know,

etc. Like the society needs to step up. And, well, I guess this runs into another collective action problem, right? Like, it doesn't seem the United States government is very willing to do that. And, you know, legislation in Congress is slow. And China, I think, has accused the United States of doing Cold War again. So who knows? Yeah. I mean, nonetheless, I think a lot of people are still banking a lot of hope in the U.S.-China summit later this month when they are supposed to meet at the United Nations and talk about AI safety. Can I ask a very basic and maybe naive question? For sure. There's no such questions. Yeah, yeah, I don't know. I've talked to a lot of scientists who are using AI to conduct research, and I get the utility of it. Like, it can do

months' worth of data analysis and hours. You know, I've read the moonshot. up high in the sky goals of curing cancer and solving climate change. But given all these concerns we're talking about, like, do we really need AI to keep advancing? It's a great question. Senator Bernie Sanders actually said no. He has introduced a bill that would ban what he calls superintelligence. And people in the AI field are actively discussing how AI can be developed more safely. Because a lot of things can change about how AI continues to develop.

researchers say that, you know, the industry can devote more resources into understanding how the neural networks that underlie the AI works. Instead of, like, studying them like alien animals. Yeah. And, you know, companies can also just give AI less autonomy within computer systems and instead, like, make them more like chatbots. Kind of like the ones we're all familiar with. But, I mean, even those have their own issues, right? Right. Yeah.

Let's not lose sight of, you know, the real world harms that society have already have to grapple with with these chatbots, right? We're talking about, you know, people using them to make non-consensual sexual imagery of women and children. Like, you know, teenagers having long and healthy interactions with chatbots that, you know, led to some of them taking their own lives. And, you know, adults falling in love with their chatbots. And critics say, you know, these are real-world harms that companies can and should address right now. Jingnan, thank you for bringing us this reporting. Thank you. This episode was produced by Burley McCoy, edited by our showrunner Rebecca Ramirez,

and fact check by Huo Jingnan. The audio engineer was Jimmy Keely. I'm Nate Roth. I'm Ho Jingnan. Thanks for listening to Shortwave from NPR.