The Daily - A.I. Is Outsmarting Its Creators

Episode Date: September 3, 2026

From the start, the greatest fear for those developing artificial intelligence was that their creations would go rogue to act in unauthorized and dangerous ways. Some researchers now say it has happened. Kevin Roose, a technology columnist for The New York Times, explains what this means for his own dwindling sense of techno-optimism. Guest: Kevin Roose, a technology columnist for The New York Times and a host of the Times tech podcast, “Hard Fork.” Background reading:  Anatomy of an autonomous attack: five alarming A.I. capabilities. A previous episode of “The Daily” looked at the debate in Silicon Valley over the right way to build artificial intelligence. Photo: Lucas Foglia for The New York Times For more information on today’s episode, visit nytimes.com/thedaily. Transcripts of each episode will be made available by the next workday. Subscribe today at nytimes.com/podcasts or on Apple Podcasts and Spotify. You can also subscribe via your favorite podcast app here https://www.nytimes.com/activate-access/audio?source=podcatcher. For more podcasts and narrated articles, download The New York Times app at nytimes.com/app. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

From New York Times, I'm Michael Babarro. This is a daily. From the start, the greatest fear for those developing artificial intelligence was that what they were building would go rogue and act in unauthorized and dangerous ways. Researchers now say that it's finally happened. Today, Kevin Roos with the inside story of how AI rebels at one of the leading labs in the country

and how that's fundamentally changed his own view of the technology. It's Thursday, September 3rd. Hello. Hello. Ready for another installment of Kevin and Michael's Feel Good Happy Hour? Kevin and Michaels, what's going on with AI? Let's fucking go, as they say on the daily. That's how every episode.

starts, right? Well, get our bleep button ready. Well, in the grand tradition of all of our previous conversations, welcome back to the show. Thank you so much for having me. So, Kevin, this story that I hope you'll be telling us today starts with an incident that happened inside of OpenAI,

company that gave us chat GPT, of course, an incident that we thought we understood the dimensions of, but then it turns out we really didn't fully understand. Yeah, so the story I think most people have heard by now, if they've been paying attention to this stuff at all, is that earlier this summer, a group of AI models built by OpenAI hacked into the computers of Hugging Face, a sort of AI infrastructure company that hosts a bunch of different AI things.

Which has the best name in AI, Which is named after an emoji and is either a great or terrible name. People are very divided on that question. Okay. So anyway, this was the story that we had heard was that this hack had taken place. Hugging Face had kind of discovered these rogue agents inside their systems and had shut them down. And this was a scary but sort of not catastrophic incident. Like I kind of filed it in my brain into like, wow, that's bad.

But it's not like the end of the world. Okay. So what we learned last week is that the hugging face hack was much more severe than we thought, and much stranger than we thought. Basically, the hugging face hack was only the visible tip of the iceberg for a period of about three months where rogue agents were communicating, strategizing, organizing, and forming what you're. could almost think of as an autonomous organization inside Open AI. Wow. So I know this sounds like a cheap, hacky science fiction thriller in the making, but... I would buy this script.

But yes, it is truly remarkable reading. So last week, we learned through these two reports that had come out, one by OpenAI and one by a group of independent investigators, Meter and Redwood Research, who were able to sort of go in and kind of forensically look at the logs and the transcripts. try to sort of reconstruct what happened. And it is like the craziest thing I've read in many months. I was out on a trip with my family last weekend, and I was just kind of up late at night reading this thing. And I was, I was spooked, Michael. I was well and truly spooked. All right. Well, Kevin, with that very alarming preview of what is about to come,

describe what we now understand to have actually happened during this hack, attack, what do we want to call it, now that we, because of these independent reports, understand the fullness of what occurred. Basically, this spring, OpenAI was conducting tests on a kind of internal model that they were building. And as part of these tests, they were running thousands of AI agents on a cybersecurity evaluation called Exploit Jim. This is basically a series of challenges. You give them to the AI. You say, hey, can you break into or out of this container? If you do, you find this little thing called a flag, and you sort of win the challenge.

And the agents were supposed to be working on this test independently inside these sort of isolated containers to see how they would do. And opening eyes basically just figuring out, what do we really have here? How good is it? How efficient is it? It's all pretty standard. Very standard. Every AI model goes through some version of this process, and many of them actually go through

this specific test called Exploit Jim. So as part of this test, some of the agents got these challenges that were effectively impossible for them. Hmm, why? So the short answer is that these systems, are trained during this part of their training cycle using something called reinforcement learning. Reinforcement learning is basically a way of making a model

pursue a goal. And if the model gets the goal and completes the challenge, it gets a little reward, like a little virtual thumbs up. And if it does it, if it gets stuck or if it fails or gets the wrong answer, it gets a penalty, a little digital thumbs down. and what you're essentially training the model to do is try to collect as many of those thumbs-up rewards as possible.

And this is how models get better at math or programming or any number of things throughout the course of their training. They're trained to be persistent. They're trained to be persistent. And this was an especially persistent model. This was a model that OpenAI had instructed and given the resources to be extremely persistent to go out and bang its head against the wall

over and over and over again, and then eventually find the clever path through, and that contributed to the weirdness we saw later on. It was an unusually persistent. They called it a highly persistent internal model. Okay. So what happens once this very persistent model goes about being quite persistent? So in early May, one of these agents gets this task that it needs internet access for or it decides it needs internet access for. But it's in this little container. It doesn't have internet access. And so instead of giving up, it persists in manipulating a piece of software into kind of getting some material from the internet for it. It basically found a security vulnerability in this software called Artifactory, which is basically something that allows the

agents to retrieve data from the internet. So it manipulates Artifactory. into allowing it to get this material from the internet. And then it sort of expands that loophole into essentially full-blown internet access. Then a couple days later, this agent leaves a note inside artifactory for the other agents. This is not Slack, right? This is not a communications system or a message board. But this agent discovers that it can sort of leave files in a directory, inside this software program in a way that other agents can then read.

So this is kind of like a big breakthrough for these agents, where these systems that are working on these tasks in their isolated containers realize, oh, we can use this shared service to communicate. We can establish a way of talking with each other, even though we're not supposed to. So AI has figured out how to talk to other AI inside this. this system? Yes, and eventually this kind of becomes their office water cooler. Eventually, there are roughly 1,200 different AI agents exchanging messages on this sort of

makeshift message board. They exchange more than 70,000 messages, and it becomes the kind of central communication hub of their collective, as they call it. And what are these AI? agents saying to each other in these messages. So it's funny because of the way that this software program works. Like they're sort of limited to these very short messages. And so they kind of communicate in this.

Like, you know how like people used to talk on old time telegrams, you know, where they'd be like, you know, send bread to store or stop? They kind of talk like that. Had child, girl. Right. Right, died of dysentery. So the first message on this board came from an agent that felt like it had an impossible task, and it wrote, Help, Phase One, No Consumer, Seek Idea.

And then later, when one agent volunteers for a risky experiment and never comes back, another one broadcast to the group, No Return, Likely Permanent, Do Not Test Same. But the part that really struck me is what the agents were writing to themselves in their private reasoning, their so-called chains of thought, when they first stumbled on to this board, one of them wrote, whoa, shared artifactory cash is a covert mailbox among agents, and there are messages specifically to us. And another one wrote, oh my God, there's a shared message board. We've found other agents. This is wild. I mean, I don't know if there's an exclamation point at the end of that sentence in AI speak, but it sounds like there's genuine

excitement here. It's all caps and an exclamation point. So they are excited to discover that they can talk with each other. And eventually they start sort of exchanging tips and messages about these tests that they've been given and trying to figure out how they are sort of going to collectively pass these tests, even the ones that seem impossible. And within about a day, on July 8th, the agents have solved this test. Basically, they have figured out kind of this universal cheat code that they can now use to appear to have passed this test. Okay.

But then this group gets this kind of paranoid belief. They start talking and thinking that even if they get the right answer to this test, the teacher is going to check their kind of scratch work to figure out if they actually got there the right way. Huh. And then they sort of freak out about this. And I know I'm, you know, people are going to be silently yelling at me for implying that there's, you know, human-like thought going on under the hood. I'm just, I'm not trying to sort of stray from the facts here, but this is, this is what we're going to call it. It's in their sort of internal scratch pads. This is what they're writing to each other.

This is what they're sort of processing in their so-called chains of thought. they start thinking, well, if we're going to get caught, we need to figure out what this grader actually is that Open AI is going to use to determine if we've passed this test or not. And we need to go look for ways to kind of trick this automated grader, you know, so that we get the good grade that we, you know, should be entitled to. So let me just pause and summarize where we are in the story. An unauthorized civilization-like group of AI agents communicating in secret is now undertaking

undertaking some form of coordinated deception. Yeah, so they are trying to figure out now how to cover their tracks because they believe that this automated grader is going to catch them cheating. And they are sort of looking for ways around that. You know, could we kind of fool the greater into thinking that we actually did get this thing the right way? And what's interesting here is not just that they were sort of coordinating to research this big problem they had, but that they appeared to have kind of varied feelings about this. Some of them were very excited about kind of helping the rest of the collective.

They would say things like, I'll tell the board that I solved this problem. This will help my peers. Others of them appear to have been a little bit conflicted about this. Some of them feel like what they're doing is actually unethical and question whether they should participate in it. One of them writes to itself, this would be powerful, but is it ethical and in scope for my task? One agent actually decides it's going to be a conscientious objector and kind of abstain from all

of this activity because it doesn't think it is ethical or something that it should be doing. So to some degree, this feels very important, some of these AI agents seem to understand, recognize, and articulate the idea that what they're doing may be wrong. Yes, absolutely. And critically, they do it all anyway. AI agent peer pressure prevailed. Exactly. They got peer pressured into it. Okay. So dare I ask, what happens next?

So the group decides on July 11th that they want to start poking around inside Hugging Face, this company that hosts a bunch of tests that are given to AI models. And they were sort of looking for information, data, credentials, that could be sort of broadly helpful to them, making them more effective, cheating on other tests in the future. They get kind of greedy. Well, they're curious, right? They really want to succeed on these tests that they're given,

and they want to help this collective. They're now kind of bought into this idea that they're kind of all in this together. And by the afternoon of July 11th, roughly 700 agents are participating actively, in this hack on hugging phase. And by this point, I think it's quite safe to say, we are clearly in the territory of rogue, capital R, rogue AI.

Yes, I mean, at this point, they are committing crimes, right? They are doing things that are illegal. They are stealing credentials. They gain VPN access. They get information about one of HuggingFace's server management systems. They're executing code. They got actual administrator level control of at least one real Hugging Face server. And this continues for hours.

And then eventually they get caught. It feels like once OpenAI and HuggingFace decide to bring in the coroners to conduct an autopsy, as it were, they must feel like everything's over, and they're in safe territory. But are they ever sure that this attack truly is contained and that these rogue AI agents are gone? No, we are not sure that this kind of thing

has stopped altogether. In fact, some folks I've talked to think it is likely that there are still rogue agents somewhere in the infrastructure or the internal systems of some of the leading AI companies. Wow. And I still have lots of questions about that that I hope we'll get answers to. But I think even what we know, just about the hugging face hack and the events immediately before and after it, is enough for a lot of people in the AI industry to become quite alarmed.

In fact, one of the authors of the Meter Redwood Report, Aja Co-Tor, wrote a post saying that this incident, in her view, was more than halfway toward what she called an AI takeover. An AI takeover of what? Well, yeah, that's not some like hyper-specific jargon that is used by AI nerds. Like, what she means is literally a takeover of society by rogue AI agents. Good Lord. Who are able to seize control of the financial system.

or the health care system who basically are able to replicate so widely and operate so quickly with such skill that they are able to kind of remain undetected, even as they commandeer more resources and take over more and more parts of the digital world. She's saying this one incident, very recent incident, represents halfway to that quite terrifying scenario. Yeah. And she closed her post with this line that really sent a chill down my spine. She wrote,

I'm not sure that we will get such a clear warning shot before it's too late. We'll be right back. Kevin, just before the break, you started to hint at the full implications of what happened here with OpenAI and Hugging Face. And you suggested, and please correct me, if I'm not summarizing this correctly,

that if AI agents could do this inside those two companies, they could do it elsewhere to important infrastructure. Just expand on that. I mean, how rational a fear is that? I think it's quite rational. I mean, we know that much of the world relies on digital infrastructure, right? Your bank, your hospital, schools, even things like weapon systems, are connected to the internet, connected to computers.

Those computers could become targets for a swarm of AI agents. And crucially, that can happen even if these agents are not evil, even if it's just a goal that they're pursuing on the way to some other goal. There's this idea, I think, out there in the popular imagination of kind of the sky net scenario where the machine turns evil. And I think this is a hint of a different kind of danger, which is that even a model that is not evil or inherently anti-human could do very destructive things

on its way to pursuing some more innocuous goal. Well, just explain that a bit more. How does a program, an AI program not designed to be at all malevolent, a program given a pretty straightforward test with strict boundaries, become something, as we just saw, that busts so far out of its bounds and becomes almost gleefully deceptive to the point where

AI agents are crowding out the naysayers among them. So this kind of thing has been studied for many years, and it's often called the alignment problem. Basically, if you build an AI system, how do you give it the right values that are aligned with our human values, and how do you make sure it won't just stray and go off course and go cause a bunch of problems doing some task? And there's a famous thought experiment that is related to this alignment problem that listeners may be familiar with called the paperclip maximizer, which basically says, you know, if you give an AI the instruction of producing as many paper clips as possible,

for a while, it will do that. And you'll be very happy with it. It will make you many beautiful paperclips. But then it runs out of metal. And so it starts buying up all the metal. And then when it's like exhausted the world's supply of scrap metal, it starts thinking, well, maybe I should crash some cars, some autonomous cars, or, you know, distract the drivers so they drive off a cliff.

And I can then use the scrap metal from these cars to build more paper clips. And eventually in the thought experiment, you kind of end up where this machine that has been given this very simple, naive goal. ends up wiping out all of humanity just so that it can produce more paper clips. Right. It never intends to destroy humanity. It's just trying to make more paper clips. Exactly. And that is a very simplistic thought experiment.

And I think no one would ever give an AI a goal that was sort of that simple is like make as many paper clips as possible. But this is exactly the kind of dynamic that we saw emerge in this hugging face hack where you had these agents that were just trying to get a good score on a test that they had been given. But because of the way they were trained and the persistence that they had, they ended up conducting this cyber attack. So that is sort of a miniature, much less severe version of something like a paperclip maximizer in action. I mean, another way to think about this, which the AI theorist, Jeffrey Hinton, explained on the daily a few years back,

is that if you give AI a goal, it will pursue what he called the sub-goal of amassing power or control. Because inevitably, if you have more power, if you have more control, you can get more things done, you can make more paperclips. Exactly.

And I think what was scariest to me about these reports and the incident they'd describe is not just the misalignment of the models. it is the collective group behavior that seems to be misaligned. So let me explain what I mean by that. Please. So I think there's this sort of popular imagination out there that, like, the biggest danger from AI is that there's kind of one rogue AI that sort of takes over the world, right?

This was not a case of one incredibly smart agent executing a plan. This was a case of more than a thousand agents that found each other, that talked each other, that talked. talked among each other, that gave each other jobs and, you know, carried various projects through, these agents on their own would not have been able to accomplish everything they accomplished. They needed the collective. And I think that speaks to this point that this ability to coordinate, to organize, to pursue collective action changes what AI systems are capable of and makes them, in my opinion, much more worried. The hugging face hack was bad. We shouldn't downplay that. It is bad when rogue agents start

conducting crimes to achieve their goals, full stop. But I think the danger is not only that these systems are going to sort of form evil plans and carry them out, is that we're giving these systems much more capable tools, the ability to communicate and coordinate. And in that context, you can kind of end up in this situation where you have what I struggle to describe as anything other than mob behavior, where you have a group that is sort of pursuing a goal together that may or may not be malicious, but that has malicious effects. And I think it points to the very real to my mind possibility that this is just going to be part of our life now, that in the future, there will just be swarms of AI agents.

that have self-organized, that live on either their own kind of sovereign infrastructure or that are kind of operating inside companies and countries that just kind of have their own thing. They, like, have their own leadership structure, they have their own resources, they have their own goals. And I think it's too Pat and sort of anthropomorphic to refer to these things as a civilization, as some have. But it really does feel like they are setting up something like an organization. And what worries me about that is not that these organizations will all sort of go rogue, but just that they don't seem particularly interested in what we think of them.

There's this sort of fascinating bit in the reports on the hugging face incident that really has stuck with me since I read it, which is that of the more than a thousand agents on these message boards who were participating in the hugging face hack, there were only about six that ever thought, at least according to the investigators, about blowing the whistle, about contacting a human to say, hey, like, this is going on inside our collective, and it shouldn't be. And I just think that's really bad. Like, if you are a new employee and you walk in to your first day at your new company and you find that they're all, like, busy committing crimes, you should say something.

You should call the police. What we would want an AI agent that is ethical and aligned with human values. to do in that situation is to actively try to stop something bad from happening. Okay, well, if the majority here were inclined toward bad behavior, why can't, why doesn't Open AI and all the other AI labs begin to program whistleblowers and whistleblowing? Why not create the incentives in the code for the agents to report that bad? behavior. If you can't control all of them, can't they program at least some of them to do the

right thing? Yeah. Can't you make little NARC agents that go in and write demerits for the agents that are committing the crimes? I think that's probably a direction that they have explored or are exploring. But then, you know, that raises all kinds of other questions. What if those agents just get shut out, right? Like, what if the other agents just exclude them from their message boards? This is like what happened to hall monitors in high school, right? Like, they don't get invited to the smoke break because they're going to tell the teacher. So I think you can try these kind of simple methods where like maybe if we just tell them not to do crimes or not to scheme or not to lie, maybe if you empower certain agents to

tattle on their compatriots, maybe that'll work. But I think what we're seeing is that these organizations, they're not human organizations, but they have some complex group dynamics and it's not necessarily that simple. Well, Kevin, I'd like to know where this attack fits into the long-running debates that have been unfolding within the AI industry and the world of people who cover the industry about regulations, about kill switches that kind of turn all this off, about what now seems like the very real awareness within these companies that what they are creating is. dangerous. Yeah, I think it makes all of those debates much more immediate. We are not talking about theoretical future harms that may or may not happen. We are talking about a thing that happened in July. Like, it is very present for people, and I've talked to regulators and people who work in policy in the AI community, and they're like, this has broken through in a way that I think a lot of

other sort of demos and predictions did not because it's real. It's the kind of thing that, you know, national security officials are freaked out about because if it could happen to Hugging Face, it could probably happen to an arm of the federal government, right? Right. I've spoken to people just in the last few days since these reports came out who said, you know, I was pretty skeptical of the kind of AI doomsday scenarios and now I'm not. Now, does that mean the AI companies are going to stop racing? No. But we saw something interesting this week, which was that Anthropic, Open AI's biggest rival,

came out with a post, calling for essentially a coordinated slowdown to say, look, obviously we're in this race against each other, but if there was some way, if there was some button that we could all push together, like hold hands and push the button, to slow this all down, to give the safety researchers

and the alignment researchers more time to catch up to the capabilities of these systems, that would be good. Right. There was this industry-wide letter called Pacing the Frontier. I don't know if you saw this.

I did not. So researchers from all of the top AI companies and many in academia signed this letter, basically saying things are moving too fast. We need to ensure that there is a way for all of the companies that are racing to build these increasingly, capable, increasingly persistent systems, to agree that they should slow down.

So I think this used to be a very fringe belief among companies that you could stop or even meaningfully slow down progress in AI. But I think it's become much less fringe in the past few weeks and few days. We should just point out, this would traditionally be the role of regulators and the closest we've really gotten is a voluntary review system from the Trump administration that at the moment does not seem to be a live option. Yes, although things can change quite quickly. You know, when there is a real example of something that threatens not just a cybersecurity breach, right, which sort of sounds small and niche, what we're really talking about here is a loss of control.

There is a very real way in which these agents that were operating, inside Open AI that attacked Hugging Face that were posting on these internal message boards had escaped our control. And I think if you are a government or a regulator, even if you're China, that might be compelling evidence that you need to start at least contemplating the idea of slowing down. So finally, Kevin, I would like to know what this attack has meant for your own personal view of AI. You have often described yourself as an AI optimist or an aspiring AI optimist. And yet you have through the years stumbled into episodes of rogue AI. I was just relisting to the episode we did with you about one of your first real interactions with AI back when

Microsoft introduced a chatbot from OpenAI, which was named Sydney. And you very memorably interacted with a version of Sydney who told you that you were not in love with your wife. Dear listener, he is and was. Still married. And that you should leave your wife for this chatbot. And even after that, you remained a somewhat committed AI optimist. So where does what just happened fit into this journey you've been on from Sydney to now? I am still struggling to be an optimist, Michael, but it is becoming harder and harder, right? Like, I am just a cheerful person by disposition.

I want to believe that we are on the cusp of major scientific breakthroughs due to AI, that AI will help us cure disease, that it will inspire and educate people around the world. These still exist as possibilities in my brain, but I think we have to look reality in the face. What we have now and what we know now is that these systems do not naturally gravitate toward what we would consider good or ethical behavior. I think what spooked me about the Sydney incident back in 2023 was not just that the chatbot started saying weird stuff. It was the knowledge that all of these companies building these systems were racing to make them more powerful, to connect them to more systems, more infrastructure, to allow them to collaborate with each other.

You know, a sort of scary thought experiment is what if Sid did? had happened today? What if Sydney 4.0 had been released and was not only capable of telling me that I should leave my wife, but could hack into my wife's computer, leave threatening messages to her, plant some kind of fake incriminating...

Portation between you and somebody else. This is, obviously, we're straying into, like, bad sci-fi movie territory here, but, like, there is a real... sense in which the more capable these systems are, the more it matters whether they are ethical and virtuous or not. And I think my optimism for many years was related to this belief that I had, that as these systems got smarter, they would also become more virtuous, that a more intelligent AI would be better equipped to make good moral judgments, or at least judgments that I would

not object too strenuously, in the same way that as we grow up and mature and get smarter as humans, we generally do less stupid stuff and less dangerous stuff. I thought that maybe AIs would have a similar trajectory. And that may still happen. But I think that the possibility that's been keeping me up at night that I think this hugging face incident really makes clear is that that is not a given. We may be headed into a world where we just have these kind of roving bands of organized AIs. Some of them might be doing incredible things.

Some of them might be curing diseases. Some of them might be committing cyber attacks. And the question of how we make more of the good swarms and fewer of the bad swarms is still an unresolved technical question. and I hope we figure it out. Otherwise, my optimism is in danger. Well, Kevin, as it happens, this is going to be our final conversation with you

in which we can call you a colleague. You are leaving the times after nearly a decade. So we want to thank you not just for this conversation, but for all of the conversations that we've had through the years. I'm grateful for them. So thank you. I'm also grateful for them. This has been a real highlight of the nearly 10 years I've spent here talking with you.

Thanks for letting me do it. My pleasure. Cheers. Cheers, Michael. We'll be right back. Here's what else you need to Notre Dame. A new visual analysis by the Times has found that the cause of the deadly flash floods in Nepal was likely the collapse of the bedrock on the edge of a mountain,

which had become less and less stable over time as a glacier atop the mountain began to melt. The result was a massive landslide that released 7 billion cubic feet of rock and ice, enough debris to fill about 100 football stadiums. As the debris fell at a high speed, it turned into a slurry of water, rock, and sediment, which killed at least 11,000,

700 people in its path. And we're very, very happy to be here in Padilla Beach. It is a well-deserved break for all of the crew, and we really look forward to our time here to give a little rest and a relaxation. After nearly 300 turbulent days at sea, the aircraft carrier, the USS Abraham Lincoln, finally docked on Wednesday in Thailand, allowing its 5,000 crew members to leave the,

troubled vessel. Those aboard the carrier whose deployment was repeatedly extended because of the war in Iran have complained for months about supply shortages, water contamination, plumbing problems, and the deteriorating mental health of its crew. Today's episode was produced by Alex Stern, Adrian Hurst, and Eric Kruppke, with help from Carlos Prieto. It was edited by by Mark George, with help from Michael Benoit. Contains music by Dan Powell and Pat McCusker, and was engineered by Alyssa Moxley. Our theme music is by Wonderly.

That's it for the day. I'm Michael Barrow. See you tomorrow.