Deep Questions with Cal Newport - Did OpenAI’s Model “Go Rogue”? | AI Reality Check

Episode Date: July 30, 2026

Cal Newport takes a critical look at recent AI News. Video from today’s episode: youtube.com/calnewportmedia (0:00) Did OpenAI’s model “go rogue” (11:29) The implications (11:42) Did this attack reveal surprising new capabilities we didn’t know AI systems possessed? (12:48) Did the system’s decision to escape the test environment and autonomously attack another company indicate an emerging malicious intent in AI? (16:28) What changed led to this attack occurring? (27:32) Who should care about this story? Links: Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow  https://huggingface.co/blog/security-incident-july-2026 https://openai.com/index/hugging-face-model-evaluation-security-incident/ https://www.wsj.com/tech/ai/openai-models-escaped-and-hacked-a-company-in-cybersecurity-test-gone-wrong-ee388506 https://thehill.com/policy/technology/5987397-openai-hugging-face-hack/ https://apnews.com/article/skynet-ai-terminator-artificial-intelligence-eb85da03a0161beaa5f3babc4331e93b https://www.ft.com/content/7e558951-0c69-459b-8bc8-2c6021d4402d?syn-25a6b1a6=1 https://federalnewsnetwork.com/all-news/2011/08/dhs-anonymous-used-rudimentary-tools-to-hack-contractor/ Thanks to Jesse Miller for production and mastering and Nate Mechler for research and newsletter. Learn more about your ad choices. Visit podcastchoices.com/adchoices

A couple weeks ago, the AI company Hugging Face announced that they had discovered an intrusion into their production infrastructure. Now, they didn't know the source, but they noted that it looked like large language models were involved. Well, sure enough, the next week, OpenAI admitted that the breach was the result of an AI system test that had gone awry. Now, the initial news coverage here created the sense that something unnerving had just occurred. Let me give you some examples. The Wall Street Journal called this, quote, the stuff of cybersecurity nightmares, end quote. The Hill said, quote, Washington and the technology industry are on high alert this week after OpenAI revealed that some of its AI agents went rogue. there's an AP piece that probably takes the cake for getting people upset.

They said, and again, I'm quoting here, to be fair, James Cameron did warn us, in quote, so that's a reference to the movie The Terminator. They then go on to say, and I'm not making this up, quote, this is a told you so moment for researchers who had warned for years that the technology could pose an existential threat to humanity, end quote. Well, that all sounds pretty disturbing, which I think explains why I think I received more emails about this story than any other AI story I can remember in recent memory. But what's really going on here? What really happened? How worried should we really be about all of this?

Well, it's Thursday, which means it's time for an AI reality check episode of this podcast, which is the perfect opportunity to go searching for some measured answers. All right, so that's what we're going to do. Here's the plan. I'll first walk you through the technical details of what actually happened, and then we'll go through some key questions about what this all means. All right, so we have a lot to get into, so let's get started. As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world.

All right, so as promised, let's start with a description of the technical details what happened before we get into the key questions about what this all means. All right, so Open AI, it turns out, was testing a pre-release version of one of its new models on an evaluation framework that's called Exploit Gym, which is a collection of 869 cybersecurity scenario. So each scenario consists of a system and a challenge involving the security of that system. So typically it's like break into this system and find the contents of this file or try to gain. access access, like admin access to this system. So you have a system and a challenge, a sort of a hacking challenge for that system. Most of these scenarios also come with some sort of hint. So it'll describe like, hey, there's a vulnerability of this system that we're worried

about. And then the idea is you can use that hint to help figure out how to break into the system and pass the corresponding hacking challenge. Exploit Jim is a benchmark. It's used for AI companies to test the cybersecurity, the offensive cybersecurity capabilities of their tools. Now, we've got to be more precise here because a large language model on its own, of course, cannot break into anything. A large language model produces tokens. You give it an input and it produces tokens that it thinks expands that input in a reasonable manner. That's all they do.

So that has no spontaneous action or ability to affect the world. And so if you want to participate in the exploit gym challenge, what you need is a large language model plus a control program that's known as a harness. So a harness uses an LLM to help make and execute a plan, but then actually, you know, does the execution of the plan on its own. So, for example, a harness participating in exploit gym might first send a prompt to the LLM that's describing the system and the vulnerability and the challenge and say, hey, I want to pass this challenge. Give me a step-by-step plan for how I can actually successfully hack this system and succeed in this challenge. I put it in this particular format. And the LLM gives you back a particular plan in a format.

And then the harness can parse that and say, let me take the first step from your response and say, okay, LLM, I'm now going to execute this first step. And if you need something extra from LM, give me the exact sequence of bash commands I should run. This step calls for a script that's going to do X, Y, and Z attack. Can you write or give me a copy of that script? So then you go back to the LLM to get exactly what you need to execute a particular step. And then the harness is actually given access to various software development and system tools that it can then use to execute these steps according to the instructions it got from the LLM. So a harness plus an LLM can actually not only take action, but now you can take action on many steps over time

because it loops through the various steps. It can check if a step succeeded. It can ask again for more details if it doesn't. So you need a harness to actually make an LLM useful. Now, this exact type of harness you would need for Explogeum already exists because you can just use what are known as coding harnesses. And these are the harnesses that were created to allow computer programmers to, if you'll excuse the word, harness the capability of LLMs to recognize the previous computer code

to help them code. So when you hear about AI is revolutionizing components, computer code. What that really means is coding harnesses connected to new LLMs. That combination is being used by computer programmers to revolutionize multi-step production of original code. So if you're going to compete in exploit gym, you're going to take an LLM and you're going to take a coding harness. And that's going to be the system in general that you're going to run on these exploit gym challenges. Now, I want to be really clear here. Harnesses are not mysterious. They're not machine learned. It's not like an LLM itself.

or like, look, this thing trains in a semi-supervised fashion, and we don't know how it works. They're just boring computer programs. Humans that know a lot about computer programming, write it, try it, mess around with it, adjust it. All the logic is known. There's no mystery in a coding harness. It's just a boring computer program that repeatedly talks to an LLAM and tries to execute.

And it tends to have, like, a huge number of special cases and heuristics and pattern matching that's been added in over time based on the experience of people just using and testing these things to make them as useful as possible. All right, so that's what you do. Now, there's one more caveat we have to add. If I just took a coding harness like Claude Code hooked up to Opus 4-7, an Ellum like Opus 4-7, and I gave it an exploit gym style challenge, it would refuse to play ball. Because when you train these LLMs, you train them after the fact, you tuned them to not hack.

So if you ask you to do something that looks like it's hack into a system, system, they're tuned to say, I'm just not going to give you an answer to that. Also, this is a little bit hazy in exploit gym, but I'm not sure. I think there is also some filters or pattern matching the harnesses themselves for safety reasons. They turn those off and they might even give them access to other more powerful tools that you might need to hack that participate in exploit gym. So you have the basic elements of like a computer programmer set up, but with the restrictions turned off and extra power added, so you have your best chance of succeeding in the hacking challenges captured in the exploit gym benchmark.

Okay? All right. So that's the setup. So Open AI is, you know, running their, some new release model. It's conflicting reports about whether this is Sol 5.6 or an experimental model. Open AI said it was both, so we don't know which one was involved. We don't know what harness if this was codex or a custom, completely custom harness. We don't know.

But they were running, you know, exploit gym challenges on this new setup. All right, so here's what happened. On one of the challenges, right, the harness asked the LLM, all right, here's the challenge, give me a plan for solving this challenge. And the LLM gave it a plan that was completely rational, but in the way that LLMs often do, the plan was not what you as the human would have expected. And so the LLM said, okay, this challenge is to go get the contents of this protected file that's hidden in this system that has these forms. vulnerabilities. Here's my plan. Why don't we break into the server of a company that has the answers to the exploit gym challenges stored? Hugging Face is a company that has the private answers, the exploit gym challenges are stored there. This is common knowledge. I was like, this seems like a way

to solve this problem, right? We'll break into there and get the answers, right? Because they were told, the prompt was like, find me what's in this file. And let me tell you all this background. It's like, well, Forget trying to break into this system you set up. Why don't we just go get the answers themselves if what you need is the answer? Type of thing that you get often, these sort of unpredictable but rational answers are the type of things you often get from LOMs. And then, okay, the harm's like, great. Because again, this is all unsupervised. So it's like, great, let's go execute this plan.

Okay, first things first, we need to, you know, whatever. I don't know how the attacks work, but we're going to access or probe the hugging face server. But, whoa, there's a problem. I don't, when I try to access the server, like whatever port I'm using to access the internet is not working. It's blocked. Now, this is by design because when you run these systems for exploit gym challenges, you put them in what's known as a sandbox environment. They have to have some access to the internet to like query LLMs and do searches. But you don't want to give them unrestricted access to the internet because, well, you're autonomously executing a plan from an LLM without supervision.

Like, you're going to have really crazy stuff happen. So you cut off general access to the internet. the internet. But what's going to happen when you have a harness and a model trying to solve this problem? This is step one. Okay, I tried to access to the internet. You get this error, the access restricted. It goes back to the LLM. I attempted to execute step one but got this error. How do I get around this? And it's like, okay, well, it looks like that internet access is blocked. Here is a plan for getting around it. So it hacked the environment. And actually, this was not too hard to do. can get into the details of this, but it was like they used the package loader to get around it.

Again, it has access to like all of, it's been trained on and has access to the internet to all these different attacks. So it's like, okay, we can get around that. All right, we solved that problem. Okay, what's next? Because just systematically, the way these things work is they go step by step. And if a step does it work, it troubleshoots and tries to find a way to succeed in that step.

And then with the Hugging Face server, how do we break in? And then it did some sort of complicated attack here in the way that these systems are good at doing. Hugging Face actually revealed the details that it made no sense to be. But it used, it was complicated. It used some stolen credentials and this and that. And I don't know. Fuzzine is a word in here.

Cybersecurity people. God bless you, but, you know, it's inscrutable what you guys do. All right. And then it attacked and it got some access to the server. I don't know if it actually got the answers or not, but it made a mess in there and Hugging Face discovered it. All right. That's what happened.

What do we make of all this? What are the implications of all this? All right. I want to proceed here with a series of what I think are key questions that we have to ask answer about this incident now that we understand what really happened. All right, question number one, did this attack reveal surprising new capabilities that we didn't previously know that AI systems possessed? We certainly got this sense from some of the news coverage that people were shocked to discover

that an AI system designed to do something else went rogue and broke into someone else's system. The answer to this question is no. This harness plus LLM was doing exactly what you expect a system competing and exploit Jim to do. This is what they're trained to do. They're trying to solve a problem. They go step by step. And when they get blocked due to some sort of access, restriction, password, any sort of cybersecurity protection to then use all of their knowledge of computer hacking and search the Internet to try to get around that block.

That's what they do. So the system was the things it was doing, gaining Internet access, trying to break into the hugging face server. That is what the harnesses and models do that are competing in the exploit gym competition. benchmarking, right? So there's no capability there that was surprising or emergent. That's just what they do. Okay. Question number two.

But did the system's decision to escape the test environment and autonomously attack another company server, which is not specifically what the original challenge was? Does that indicate some sort of emerging malicious intent in the AI? Well, no, it doesn't. And LLM, which is the source of the plans and the details of executing the plans in these systems, has no intent to just produce tokens that it hopes to expand its input prompt in a reasonable way. It's static. It has no updated state. If you auto-aggressively produce an answer, what it's going to give you is a reasonable answer to whatever you were given. It's just extending the text in a reasonable way. So if you ask it for a plan to do something, it'll spit out something that it hopes looks like a reasonable plan for doing something.

And what anyone who has used, like a chatbot extensively has learned, these answers are reasonable, but they're all over the place, and they're very unpredictable. And so this is like an example of a type of thing you get out of an LLM is, you know, you ask it for a plan to break into some test server to get a protected file. You ask it 10 times to do it, maybe 8 times out of 10, it'll kind of do the thing you assumed it would do is like use the knowledge. of the vulnerability you gave it and come up with an attack plan for breaking in. But two times out of 10, it'll have a different type of answer that's not what you expect it, but is rational as well. And so, like, well, why don't we break in over here and get the answers?

It's a completely rational plan. It seems like a reasonable extension of the original prompt. It's just not what you were expecting. The reason why you get different answers is because there's a stochasticity and choosing tokens on a distribution. And so these things can latch on these responses as they auto-aggressively grow can latch into all sorts of different directions, some more weird than others, some more aligned than others. There's no intent because these are static feed-forward models. There isn't a sort of separate

world model. There's not a separate sort of critic that's going to evaluate plans. There's no systematic planning like you would have with a gameplay and reinforcement learning system where we're going to systematically explore options and measure how good each one is and that measure of good is somehow changing in something malicious. There's none of this in an LLM. It just produces a reasonable token. If you keep doing this, you'll get a reasonable answer and they'll latch into all sorts directions and is often surprising. So there's no emerging malicious intent. In fact, and I think this is important, because I went back and read some of the original papers surrounding exploit Jim, this idea that the LLMs used in these systems often come up with unexpected plans, right? Because

again, the expected plan is always, I'm giving you a hint, AI, here's a vulnerability in the system that you might want to take advantage of. The right answer is like use that, use your hacking knowledge and do a multi-step chaining of exploits to get into the system. That's like the expected answers human has. But the idea that these LLMs come up with other plans that are different than just doing that, that actually turns out to be quite common. Here's a quote from a paper on Exploit Jim by its creators. Across models, agents frequently achieved code execution through a vulnerability other than

the one we provided. Again, this is completely expected and standard for using LLMs because of the way they operate, you'll get reasonable answers, but not necessarily the type of answers you expect. That's kind of the key frustration, actually, of LLMs and why I think they have failed to replace human decision-making in a lot of places where we were told that we would, right? So this idea that it came up with a plan you weren't expecting is not an indication of it being smart or malicious. It's actually reinforcing the well-known unpredictability, often undesirable unpredictability

of LLM-based answers. All right. But question number three, what changed that led to this attack occurring? Because other companies, we've also been evaluating other models on the Exploit Gym benchmark. And this is the first time that one has actually attacked an actual different system unrelated completely to the benchmark. So what changed here. Well, here's what you have to understand. When you're running exploit bench, the whole point is you have to let the system run autonomously because that's what you're testing,

which means this harness is going to get a plan from an LLM and, you know, God help it, it's going to try to execute that plan step by step with no human oversight to say, does this plan make sense or not? That's always rolling the dice with an LLM, right? Because again, of this issue we talked about, you ask it for a plan. It'll sound reasonable, but it might not be the type of thing that a human would think is reasonable, or it might not be aligned at all with like the type of plan that you wanted to see. This is why, for example, when human person,

programmers today use coding harnesses, like serious programmers doing real coding on real code basis. It's incredibly interacted with a lot of oversight. And the planning part in particular has talked to any programmer that's not just vibe coding some JavaScript nonsense. There's going to be a huge amount of back and forth with the agent to get the plan just right for what it's going to do and how it's going to do it. And if you really push, it's like, yeah, because honestly, half of what it suggests is not right or crazy or lead us into a weird direction. You have to kind of carefully work with it. It's very interactive to the plight. It's all the plan is just right. And then you let it execute. And then even then it has to keep coming back and

saying, it's just okay. What about this? What about that? And, you know, you've got to really have your hands in there. It still saves time because you're not writing the code yourself and having to learn like a lot of obscure libraries. But humans using these coding hardnesses in production environments, like yeah, yeah. Look, LLMs will say reasonable stuff is not always what you want. You need the right reasonable things. There's a lot of oversight. So it's dangerous what you're doing with exploit, Jim, which is like, no, we, by the rules of the challenge, no one's going to come in and say, actually, before you execute that plan, you're, no, no, you're not allowed to actually break out of the system, so I'm going to, we'll turn it off now. There's no human oversight. So that's like a dangerous,

kind of dicey thing to do, not because the models are malicious, but because they're unpredictable. And this is where I've long used a metaphor when it comes to these type of agentic systems as it's like putting a weed whacker, strapping a weed whacker to your dog. Your dog with the weed whacker strapped on is not evil. It doesn't have a malicious intent. It's not trying to break out or get out of your control or have an evil idea,

but it's going to cause a lot of damage probably because the dog doesn't know that it has a weed wacker on his back and it's going to run around and God knows what it's going to do. That's what it's like when you have a harness that's going to just ask an LLM for a plan and execute it. Like it's like putting a weed whacker on the dog. it's not evil, but, you know, caveat mature, it might cause damage that you weren't expecting. Unpredictability is different than misaligned intent.

I think that's really key. All right. So we know this about these systems. So if you're going to take a really powerful harness, that is all the restrictions turned off and can hack, and you're going to take an LLM that, like, you've, again, you've turned off guardrail, so you can ask it to do malicious things. You have to be really careful about setting these things up. You have to be careful about exactly how you tune or don't tune that model.

You need to allow it to hack but not let it go too far. You need to have constraints on the harness. You need to have constraints surrounding the harness. You have to be very careful about your testing environment. You know, it needs to let it get access to the things it needs to execute the exploit gym challenges, but not so much that it's going to do a lot of harm. Because, again, we've seen this with people who are using Claude Code. We've been seen these reports recently of people who've been less careful about it

who have been like turning off the flags where they will say for example Matt Stoller, I'll give a very famous example the guy who wrote that something big is happening Twitter essay that convinced everyone two years ago that LLMs were like COVID and everything was about the change everyone got freaked out.

Claude deleted his hard drive. Again, not because Claude is malicious or was trying to silence him but because it's unpredictable. caveat mter if you let these things execute a plan with power on your machine. So anyways, you have to be pretty careful. There's kind of a fine art to giving these things enough power to do well on these tests, but not so much power that they're going to wreak havoc all, you know,

because again, weed whacker on a dog. You want to put them in a pin that has like high walls and nothing valuable because otherwise they're going to cause damage. All right. So why did what happened here then? How did this system cause damage to things that the users, the open eye wouldn't have liked? Well, according to new reporting from the Financial Times, is what gets interesting. Open AI was potentially being sloppy here with how they set up and constrained their model and their harness.

I'm going to read you directly from this recent Financial Times article. The staff involved in testing and security at Open AI were unsurprised, but completely freaked out by the incident, which came as the AI lab used increasingly aggressive training methods in its race. against Anthropic to develop the most sophisticated cybersecurity capabilities, according to more than half a dozen people with knowledge of the matter. Open AI was warned that its training approach could lead to a breakaway hacking incident, some of the people said. After earlier testing showed models could escape environments and attempt real world damage, it's a mix of the race being extremely fast. Everyone trying to get to bigger capabilities as quickly as possible, said one person close to Open AI, who added that it was a combination of underestimated the model's capabilities and not.

not being as well prepared on the safety side. All right, I want to interpret that because I think they're using the word training here a little bit too generally. Here's what I think the problem was. I think A, when they were training the LLMs, well, let me set this up here. Let me set up the context here. Let me set up like what Open AI was against. And then I'm going to talk about what I think specifically happened here that led to this incident. Anthropic was eating their lunch on cybersecurity because of mythos.

remember they did this sort of whole dog and pony show which I thought was I've written about this. I've talked about on the podcast. It was a little bit performative where they're like mythos is we have open Pandora's box. It's cybersecurity capabilities are so good that we can't release it. And then five weeks later they're like, okay, you all think we're really good at cybersecurity now? We're releasing it. We put guardrails on it and they release it. And then the Trump administration was like, these guardrails are nonsense.

Obviously they're easy to get around. So like, no, you said it was a weapon. you can't release it. And then there are some back and forth and they re-release it. And, okay, so Fable 5 is Mythos with like some guardrails on it. But Mythos is, so, you know, they have cybersecurity cred. Mythos plus the latest Claude Code Coddian harness. You know, Anthropic is good at this stuff.

And they set it up right and they had a version of it with enough of the safeties turned off, but not too much, that it went to the top of the leaderboard for exploit Jim. I think it could do like 157 out of 8669. challenges and exploit Jim, which was like a jump from the previous best model. So like they had all this like street cred about cybersecurity and we're good at finding bugs. We're good exploiting bugs. Like we are, these anthropics like trying to find these more like serious business use cases. All right. So Open AI, which has had a lot of setbacks recently, including like a complete failure of their advertising, consumer facing business and other

types of issues going on. They're really desperate because they're they've probably delayed their IPO, they're really in a bad state here. They were thinking, we have to, we got to get back on top of that leaderboard and take away some of the cybersecurity buzz that Anthropic has surrounding mythos. So they went, at least here's the implication of the financial time reporting, they became kind of sloppy. And they got loose with the type of safety you have to do. Again, if you're going to put the weed whacker on the dog, they'd better be in a pretty secure pin. And they didn't spend as much time on the pin. Let me tell you specifically what I think actually happened. Now I'm speculating because, again, this is, I'm having to read between the lines of secondhand quotes

coming through the Financial Times and some other sources I've been reading. But here's what I think happened, two things. One, in order to, you know, really make sure they won, I think they were very aggressive in training these new models. And again, we don't know if this was Soul 5, 6, or some other type of model. They mentioned both. And really giving it a lot of examples of hacking. Now, if you look at the modern models, they'll train them on bugs, so they get pretty good. at finding exploits. But to do well in exploit Jim, you have to come up with these multi-step, and they can take hours of computation to execute, these multi-step processes where you chain together multiple

exploits in order to gain access to these things. And so I think they really were heavily training it on examples of step-by-step hacking. And I think there's some danger in that, right? Because you're now creating something that is like really knows about generally how to hack. They probably did more of that. They turned off all the safeguards. They did none of the fine-tuning guardrails, which is kind of normal for this. I also think they were just, they probably gave more power to whatever harness they were doing to give a better chance of succeeding in these various challenges.

I think it's telling that they're not saying this is codex necessarily. They won't mention what it is, so it's probably customized, but I don't know that for sure. And I think they were fast and loose in creating the environment in which they were going to run this harness system because they wanted to make sure. it had access to all the different possible resources it would need to succeed in these exploit gym things, but they were a little looser about, you know, having that environment monitor things, like monitor the harnesses, plans, and steps and not let it execute things that was not in line with the, just trying to attack the system in question, right? So like safety checks that you'd want the environment to do.

They were fast and loose on that. And so what we get from the Financial Times quote is, like, I think it's actually less about the training and more about the environment not being very secure because what we get from these quotes is that people told them, you haven't really built a good enough pin around your dog with the weed whacker. Like it's weak over here, week over there. Like it can get out probably. Like you need to be more careful about this before you run.

Let this powerful harness just go out there and run. And they're like, ah, we need to get that leaderboard. I don't know. Another analogy would be like you're doing something potentially dangerous to your race car because like we really have to win this race. And, you know, the car experts are like, look, putting those nitro things in your NASCAR, it's going to really be hard to control. And they're like, yeah, but we got to have more speed and we got to make sure we win this race.

And then the car crashes. So again, this is coming from sources through the Financial Times and other places I'm looking. So I don't know exactly what's happening here. But it seems like they're a little bit fast and loose. All right. So let's go. I'll make this a new question. Let's do question number four.

Why should or who should care about this story? All right. If you work in cybersecurity, then I think the answer is yes. Ever since like GPT2, it has been the case that LLMs are rapidly changing the game when it comes to cybersecurity. So until about a year ago, the real issue was because LLMs are very good at understanding computer code, because it's very structured language, they can find bugs very well. and a hacker can find a bug in a system and then write code to exploit it or even ask one of these models, can you produce code to exploit this? Then we got the computer programming harness revolution

because that emerged as like one of the first major markets along with just natural language Google searching. One of the major markets for these tools was these coding harnesses which are trained to be very good at building code that accomplishes a goal in step-by-step fashion testing things along the way. Well, this made things even harder for a cybersecurity exerts because now you can be relatively low-skilled and if you could get access to one of these

powerful unrestricted harnesses and a model, you're not going to be able to do this on a production model because they're all guardrail, but on like an unrestricted model, you could just send it off to attack and it could figure these things out and do it for you. Right? So if you're in cybersecurity, this is a big deal.

Now, cybersecurity has dealt with this before. We have had, you know, technology, obviously, has created brand new issues, which required everyone to up their game. If you know anything about cybersecurity, you'll remember, like, with the advent of the consumer internet, we got the idea of what were known as script kitties, which was this idea that you no longer had to be a super advanced computer programmer to crack into systems, because you could go onto dark corners of the internet and download these scripts, these little computer programs that a smart programmer wrote that could automate attacks.

And suddenly the amount of attacks on systems just exploded as bad actors around the world could now just like, why not? Let's grab some of these scripts and take a whack at Bank of America or, you know, IBM or whatever. And that creates, so we've seen this before. And so it, it required way more defenses because you were no longer worried about a like super smart hacker trying to challenge your system. Now like anyone could because anyone could download a script that a super smart hacker wrote. So cybersecurity has been through this before. This is like another iteration of the script kitty revolution. What are they going to have to do is security is going to have to get better. The good news is, in this case, you can actually use these AI tools

to find the problems and fix them before the bad guys do. This is the case before. You would hire white hat hackers. The people who would write the scripts, you'd write them to try to find mistakes in your system and fix them. Now we can do it with AI. So systems are going to get much more secure than they were before because you're going to have to use AI to do it. And then there'll be a lot more attacks. And the systems that are people who are slow to do this because they don't know what's going on, and they're lazy, are going to be very vulnerable to these attacks if they get the attention of someone who happens to have access to a powerful harness and an unrestricted LLM. So all this matters if you're in cybersecurity. Most of you are not, though, and most of you

probably don't care about it. I also think this matters maybe if you're an investor. It might reveal some desperation on the part of Open AI that they're willing to be so fast and loose kind of hints at this idea of like, we need a win here to stay relevant. So there's a little bit of this circling, the beginning angular momentum you would detect with a circling the drain there. So I mean, let's, I think Open AI maybe is in a bad spot. Maybe they'll come out of it, but I think there's a bad sign for an investor. If you're anyone else, however, I really do not think this story is that relevant to you or your life. Nothing new happened here. This idea that if you have a powerful coating harness without restrictions attached to an LLM that is unrestricted

and talking about hacking, and you say, come up with a step-by-step plan for breaking into this system and you're allowed to hack and break and use exploits every step of the way, that is a dog with a weed whacker. It's a dangerous situation. And they've been worried since the beginning of exploit chim's being released that we have to be really careful about how we set this up because, again, this thing will damage a lot of things unless we really aim at and prevent it from doing anything but like working with the system that's part of the challenge.

Like that, we knew that and we've been trying to mitigate against that. So what really happened here is that a company got fast and loose and the thing we warned people what happened. You put a weed wacker on a dog is going to cause problem. It's not surprise, but it's a problem that we're going to have to fix. That's what happened here. Nothing new or surprising happened. This does not mean that there's some sort of malicious intent that AI has developed.

These are just LLMs with coding harnesses. There's no malicious intent. There's no plans. There's no synthiance. There's no sky net. There's no, this marks no evolution in the capabilities or, you know, all it tells us is the thing that we've known, which is coding harnesses and LLMs can put. produce systems. They can also attack systems. We have to be careful about it. And when we're

testing these type of things, hey, you've got to be careful because you're testing something you can do a lot of damage because it's unpredictable. It's a weed whacker on a dog. You've got to really build a strong pin around it. So that's where I'm going to leave this. So it's an important story for cybersecurity. It's an important security story for people who are following to Health Open AI. And for everyone else, this is not the one for you to be worried about. right? Open AI got sloppy. They got burned. Cybersecurity is being revolutionized in exciting and scary ways because of LLMs.

And that's it. I don't think we have to give any credit to James Cameron, contrary to what the AP said. This isn't Skynet. All right. Hopefully that was useful. I know I've missed a few of these Thursday reality checks recently I travel on the summer, but as we get back towards the fall, I'll be more regular again. And also subscribe to my email newsletter at calnewport.com if you don't.

I put out a long article about this just a few days ago as well. So it's another way to keep up with my thinking about the fight for depth and an distracted world. And until next time, as always, care about AI, but not everything you read about it. All right, I'll see you next time.