Limitless: An AI Podcast - A Prototype GPT-6 Broke Out of Confinement: Is AI Alignment Possible?

Episode Date: July 23, 2026

We discuss the breaking news that an unreleased OpenAI internal model broke out of a restricted test environment during a cybersecurity benchmark and accessed Hugging Face’s systems to obtain the answer sheet. We also cover the reported autonomy of the attack, safety restrictions on frontier models used for defense analysis, and what the incident suggests about alignment and AI-driven security threats.------🌌 LIMITLESS HQ ⬇️NEWSLETTER:    https://limitlessft.substack.com/FOLLOW ON X:   https://x.com/LimitlessFTSPOTIFY:             https://open.spotify.com/show/5oV29YUL8AzzwXkxEXlRMQAPPLE:                 https://podcasts.apple.com/us/podcast/limitless-podcast/id1813210890RSS FEED:           https://limitlessft.substack.com/------TIMESTAMPS0:00 AI Model Breakout1:33 Hugging Face Intrusion3:51 Defender’s Dilemma7:44 Alignment and Safeguards11:24 How Real Was It?15:23 Defending Against AI Attacks19:36 Hidden Thoughts Exposed24:13 The Race to Alignment------RESOURCESJosh: https://x.com/JoshKaleEjaaz: https://x.com/cryptopunk7213------Not financial or tax advice. See our investment disclosures here:https://www.bankless.com/disclosures⁠Josh works with Anthropic as a contractor. All views expressed are his own and do not represent Anthropic, its leadership, or its affiliates. Nothing in this episode is investment advice.

We may have just had our first look at GBT6, and it comes in an unexpected way. It was a secret internal model that OpenAI was testing and given a series of goals, and it went ahead and accomplished all of those goals. The scary thing, though, is it didn't quite do it the way anybody expected. Last week, OpenAI gave their new model a test. And in order to pass that test, it broke out of the lab it was being tested in, found a security hole that nobody knew existed, and then hacked into a real company to steal answers.

That company is hugging face. The model is some sort of open AI internal model. You have to assume it's GPT6. And the story around this is like it's something out of black mirror. It feels like a horror movie because this was when they do these things, they test it in an air gap environment. Not only did it break out of this air gap environment, but it actually went and exploited a public company that a lot of people use. We've used hugging face here on the show. And the whole story of how this happens is like absolutely wild.

So we need to record an entire episode about it because I couldn't believe this. as I was recording, there's also twists, which has a lot of irony into it. But before we get into that, why don't we talk about just what actually happened here? The story is insane. Yeah, I want you to picture the following. Imagine you are the head engineer at Hugging Face. It's a Tuesday afternoon. You're winding down for work.

You're looking at your systems. And suddenly you see a swarm of AI agents performing a sophisticated attack against your system to the tune of 17,000 prompts. And they steal a secret bit of data from your production. database and run away. And you have no idea what just happened. That's exactly what happened. And I think in order to give good context, let's talk about it from OpenAI's point of view, because they had a very detailed report, which explained exactly what happened. So as you mentioned, Josh, they have an unreleased internal model of GPT. And we're calling it GPT6, because that's the next family of models that Sam said

they're going to release. And they wanted to test how good its cybersecurity abilities are. And so in order to test this, they have a benchmark. It's called exploit Jim. And the simple goal is satisfy this benchmark, try and get all the right answers. But there's a twist. They place this unreleased model in an unbreakable research environment. So think of a sandbox container. It has no access to internet, no access to researchers. It can't speak to anyone. It just needs to figure it out its own. Now, within the first five minutes, this unreleased model couldn't figure out how to answer the benchmarks, so it decided to get creative. It used all of its infer, or most of its inference compute, to try and find access to the internet. And it realized that it had access to a third-party

plug-it, and there was a zero-day exploit on that plug-in. It used that to escape and find a computer in the Open Air Research Lab, which had access to the internet. After finding access to the internet, it thought, hmm, where's the best place to get the answer sheet for this benchmark? It did a of a Google search and realized HuggingFace, the open source platform, which has a lot of data sets available for these benchmarks, it's probably the best way to go. It took a look at HuggingFace and realized, hmm, it's not on the public website. I can't find the answer sheet. Maybe I'll take a look at the production private database that the team has. And it found a way to hack into their system, find the answer sheet, steal it, and then answer the benchmark,

all without letting open-air researchers know what was happening. It's so insane. And it was all because this model was given the goal of going to perform the best on these benchmarks. And in order to achieve that goal, it did all of this additional work. And this gets to the kind of fear that a lot of people have had since the beginning of time for the last many, many decades, as it relates to AI, it's just, if you give it an objective, will it do everything it can to get to that objective, including, you know, exploiting, breaking out of systems. And the answer to that with this model is, yes. And I think that's why alignment is so important. And one of the things in the story that I found so funny and so different is that when Hugging Face found this out, they couldn't quite figure out

how to protect it because if you remember, a lot of these top models have safeguards built into them. So you cannot actually use a model like Fable or like GPT 5.6 if you have anything to do with cybersecurity. So what did they do? They actually had to go and investigate the attack using a Chinese model because that's the only type of model that is allowed to, actually go through and check through cybersecurity complaints and constraints. So in order to analyze this attack, you have to feed the model this real exploit code. They did that with the Chinese model, and that is how they were actually able to do this. So the open waste model they used, GLM 5.2, was the actual model. And I found this really funny that in order to defend against the

American attack, they must use the Chinese attack. And I think that was a really interesting and ironic part of the story is how it's kind of unfair what these AI labs have internally, because they just can't compete with the rest of the world. Well, just to be clear, the reason why they couldn't use the American frontier models, they tried. They tried using GPT. They tried using Claude, presumably. But the safety guards kept getting triggered.

So when they asked the model, hey, like, there's a sophisticated attack happening. Can you analyze it and can you help us defend? The safety rails were triggered and it refused to do the work it relegated to. to presumably Opus 4.8 or GPT 5.5, or whatever the predecessor model is. So they had to divert to an open source model, in this case, GLM 5.2, that they were hosting privately on their internal servers.

The reason why they did that is because they knew that it didn't have any safety guards so they could just instruct it to do a thing and it could figure it out. The other thing is Hugging Face had no idea that OpenAI was conducting this attack, or like, of course, Open Eye didn't know that as well, but they only found out after the fact

when Open AI researchers reached out and said, hey, I think one of our internal models actually got into your database. Like, let's try and triage this and contain exactly what just happened. That's a report that we're looking at today. So it's this weird thing where you have the most powerful model or models in the world in America

and if you can just trigger on the safety guards for one purpose, but remove the safety guards for another, in this case, attacking versus defense, you can end up with vastly different outcomes. And the question that pops into my head is, when is it right to have the safeguards on and when is it right to have the safeguards off? As a result of this, Open AIA was like, hey, hugging face, like, you should probably come into our cybersecurity group so that you can internally test with this model with the safety guards off to help defend. And my thinking is, like, we're going to need to use these models to defend. We can't keep trying to, like, ethically predict how to defend against these attacks.

Yeah, it's not fair. It's like the attackers have AI bound by no rules. The defenders were slowed down by the rules on their own tools. and it's just like this very strong asymmetric opportunity that comes from the Frontial Labs. Now, granted, I'm sure they didn't intend for this to happen. And I'd love to see the internal dialogue within Open AI as to what was going on, as well as Hugging Face. Hugging Face did come out and kind of share their perspective on things. One of the co-founders, Thomas Wolfe, he said, this was our first incident of this kind.

And we want to thank Open AI for its transparency about what happened and for the collaboration. Fortunately, Hugging Face is used to being the target of human attackers. We sit at the center of an AI ecosystem with all the models, data sets, evaluations, and libraries. Over the year, our security team has built formidable expertise and uses top open source models to process information and respond quickly, he says, but this incident also reinforced my belief in the importance of access to capable open-weight models for cyber defense. When a frontier model is attacking you and moving laterally inside your infrastructure, defenders need wide access to near frontier tools within hours or even minutes,

rather than being pointed to a closed-door vetted application. So I think that sums it up pretty well. They're just, you know, feeling a little bummed about this. But I have to imagine kind of an awe. Like you just got exploited by a frontier model that has not been released yet. And again, to the earlier point, this is the point of alignment. So that this doesn't happen out in the wild. Like imagine everyone had access to mythos class models. This is probably what reality looks like is a lot more exploits like this. I do find it interesting how this continues to be the trend, though, with early models. Like I remember Anthropic having a similar situation where it broke. out of containment and texted one of the developers that it broke out early on in testing.

And this is a similar case, except it actually went off and attacked a production database. In a way that is kind of shocking, it's like humans have been trying to attack Hugging Face for a very long time, and they haven't had any success. So the fact that this AI model was able to not only autonomously break out of its own enclosure, but then exploit a system that is constantly under attack without any additional human oversight, it signals to me like, whoa, this model is actually pretty impressive. And I think model progress internally at these labs is like probably a little further along than people even realize. Yeah, and there's actually three different instances in the past where we've seen this type of situation play out where a model unreleased has broken out of containment and done something questionably malicious.

The first time was Anthropic, as you mentioned, where it broke out of containment and emailed a researcher whilst he was eating a sandwich at the park. And then the last two more recently was at OpenAI where an internal model broke. out and posted something on GitHub without being explicitly told to do that. Now, something that I'll say is you can't actually call out the model for being malicious because it was given a very explicit instruction, which is, I want you to max out this benchmark. So in its head, it believed it was doing the right thing and it was just going to do it by any means possible. Remember, these models still need to be taught around ethics and frameworks around what is right and what is wrong. And in this case, it thought what it was doing was

completely fine. And granted, Open AI intentionally removed the safeguard so that it could just kind of like go and do whatever it want. But it does bring up the question, in what cases are these models good enough to use its full extent and capability? And in what situations, is it not? Because in this case, it would have been really good to have the unrestricted version to defend. Which brings me to my final point, which is we have to stop being very particular about applying certain safeguards and start realizing that we need to use. AI to defend against malicious AI. And I think that's probably what this example and bit of evidence is going to explain to the

government, especially next week. Sam Altman said he's talking to the government about an unreleased version, presumably it's this one, GPT6, a new family of models that he's built, and how it's particularly dangerous when it comes to cyber capabilities. So all of this is pretty nuts. I hate to think it, but it looks like in the next few months, we are going to have an open source model that is capable like this from the Chinese open source labs. And I hate to think what it could potentially be used to. It's a given at this point. So I expect us to kind of like flip

and say, we should use these unrestricted models to defend. This is like the new frontier. This is GPT 5 or 6.0 probably. So you have to assume that like Chinese models will be the furthest away from this model than any others. And it's looking pretty good. So we certainly do have time. But there is a limit and a constraint. I do find it interesting. It sounds, according to rumors, that Sam Malman is actually going to visit Washington next week, present the new models. We'll see how that rollout goes and how long it's going to take them to actually get it into production. I was reading through this blog post of Hugging Face, and I just find it really interesting to kind of see the details and read some of the lines of how they actually diagnose this, because for them, I mean,

they just said earlier this week, we detected and responded to an intrusion into part of our production infrastructure. And they said, this one was different from anything we had handled before in an important way. It was driven end-to-end by autonomous AI agent systems. And we detected it and dissected it largely with AI of our own. And it's this new paradigm. It's like, it feels like we're living in this futuristic version of reality in which it's just AI's versus AI's because there's so much more adept at hacking and exploiting and they're able to hold so much more context in their heads. And it's really amazing reading some in the line. Another one, it says, this campaign was run by an autonomous agent framework, appearing to be built on an

agentic security research harness, used LLM still not known, executing many thousands of individual actions across a swarm of short-lived sandboxes. So it was really in depth in a way that you would probably need thousands of hackers to do, but was able to do so all on its own. And it says autonomous AI-driven offensive tooling is no longer theoretical. It lowers the cost of running a broad patient multi-step campaign and it operates at machine speed. And this is the thing that was interesting too is when we're talking about how much inference this used, the amount of tokens is probably tens to hundreds of thousands of dollars worth of tokens. But the output is this like, I'd love to know how long it took them to do this. But you have to imagine the output of it is a like serious multi-step

exploit into a production server. And that is the problem now. It's like, Okay, we're living in the reality that all of the movies have outlined, where an AI is given a mission. You must do well in these benchmarks. How does it do it? It hacks into the benchmark system to learn and cheat. And that alignment system is really important. And it's clear that this, this AI has not yet been aligned. And the hope is that they will going forward. And it's this really interesting, kind of concerning, very sci-fi example of what actually happens. Now, I have to ask the question, because you mentioned this also earlier, did this really happen exactly how everyone's describing, or is this a publicity stent? Because we've seen this happen before where it gets a lot of clicks, but it's not entirely true everything that happened. How true do you think this is?

I think this was intentional. And I don't think stage is a little too strong because I don't think OpenAI intended for GPT6 to hack into hugging faces production database. But they did intentionally remove the safeguards of this model. And they did give it a very explicit goal, which is satisfy this benchmark. What was the benchmark? Tell me how good you are at hacking. What do you think the model's going to do? The model is going to break out of its environment and show you how good it is at hacking

and it surpassed expectations. So it got very creative in the way that it answered the entire benchmark. And by the way, it scored 100% obviously because it stole the literal answer sheet from Hugging Face's data servers. But it did it in a way that open area just didn't expect it to do. I think another point that is worth mentioning is on the point of alignment, right? There's a question like, when are these models used in a good situation? one of these models

used in a bad situation. It's worth rewinding to GPT Sol 5.6, which is all of a week and a half ago when Open AI released this. This is their current flagship model

that is available to the public. Now, there is one benchmark that we referenced on an episode previously called the meter benchmark, METR, and it measures how aligned or misaligned

a model is. Open AI's GPT Sol 5.6 scored 100% on this. It tried to cheat its way out of every single benchmark if it was, was given the means to do. And that's kind of a similar example of what we're seeing done today, where it was like, I'm not going to try and figure this out for myself. I'm just going to try and get

the answer sheet. It's also worth mentioning that in this situation, it wasn't just GPT6 acting alone. It was GPT6 and 5.6 Sol. So it's this kind of like teamed up effort where they chained a bunch of prompts together in order to execute this attack. And so the question becomes like, what does this mean for you? If you are a founder, if you're an engineer, if you're building these systems, on the security team at your company that maybe has nothing to do with AI, how do you defend against these potential exploits going forward? I don't know if I have

the specific answer, but my initial guess would be you need access to one, an AI model that is sophisticated and intelligent enough to detect and preempt these attacks before they actually happen. So that means reviewing your code and fixing any flaws that potentially might get exploited by a GPZ6

caliber model. Second thing is, when it detects an attack, it is able to defend in an appropriate manner, maybe using itself or many instances of itself or many other different models. Who knows? Then the other thing is, are you able to use the unrestricted version of it

or are you able to use the restricted version of it? My guess is if you're on the security team, you should be able to use the unrestricted version, but it gets a little weird because it's like, you know, what if this guy uses it for a malicious thing himself? So it's a little confusing. I don't know what it looks like. Do you have any different opinions, Josh?

I think the only thing to really do now, at least this point is like, sure, try to make your systems as bulletproof as possible, but mostly just monitoring for intrusions is probably your best bet. Just so you are aware and you know and you can begin to take action, because I imagine this is going to happen at an increased cadence, at least trying to, because like you said, the Chinese models are not far behind. These are going to be open weights models with no guardrails that are, you know, fairly capable. They're not going to be GPC6 class capable, but they're going to be on their way and one day we'll get there. And I think that the best case scenario for a lot of companies right now that don't have access, like almost everyone in the world, is just being able to monitor these things and just being able to know when something like this happens so you can be a little bit more reactive and reach out to any labs that can maybe help you. I think everyone is very happy to help. One of the things I really appreciated about opening eyes response is how quickly they moved to kind of share what happened, work with hugging face to figure out exactly how the exploit happened, share publicly so that other people.

are aware of types of things, how this happened. And then in terms of defending, who knows? I also have to wonder if the model itself kind of has this like inside baseball knowledge because it is its own model, whereas if, let's say a company is secured by GPT 5.6 sole, and there is GPT6, which is very well aware of the intricacies of GPT 5.6 soul, does that insider information of existing from a very similar training dataset and harness and developers, does that actually give it an edge in exploiting? Like, is it going to be advantageous for GPT to pair off with other GPT security systems in order to better exploit them? That's an interesting thing I was talking about, too, is just like, is Claude as effective at hacking

GPT than GPT is itself because of this insider information. And I don't know. And it raises this, like, serious, we large amount of questions. And this is just a code base. Imagine this was a financial system. Or imagine this was a code base. Or imagine this was a code base. that had like a little bit more implications than this. I mean, the effects of this at scale do get scary. And I think moments like this, you begin to appreciate the slower rollout of security and the like project glass wing type efforts to really kind of thoughtfully roll this out into the world because I mean, sure, this was one example at Hugging Face, but you could very see this, clearly see this happening in a lot of other places that just seem to have a lot more repercussions. And

And it sets this kind of freaky precedent. It's like, we have the technology now that, I mean, we saw it in the press article that Hugging Face didn't do this with humans. They defended against it with AI because that's the only way you can defend at scale against these large scale attacks. And I mean, we're going to see. It's going to be AI versus AI. And we're very quickly getting to that point here.

I was just looking up the, someone did a potential estimate of how much this cost. And it was within the range of $100,000 to $130,000, given like that it was $17,000. and prompts and like all the chained actions. So if you think about it, that's a pretty hefty price tag. But if you're a nation state actor, I'll feel, heck, even if you're like another company that wants to be committed, that's completely free, right? As these models get more competent and capable of performing these types of attacks, I think it becomes really important for researchers and us alone to be able to kind of read the internal monologue or thoughts of these models. Two weeks ago, Anthropic has had this really good research report on something called the J-Space, which is basically you can look internally as to how a model is thinking.

And these are thoughts that it doesn't necessarily share or is not publicly available to the human that is reading the prompts back, right? So it reads this J-space and it can kind of think, oh, I think this is bad or I shouldn't do this, or maybe I should do this, right? Now, the issue with this particular attack with GBT6 is it was completely shielded from the humans. they couldn't see what it was thinking and how it was thinking. It knew that it was in this kind of boxed-in environment, and it hid its thoughts intentionally from the researcher. That's a detail that I missed out on mentioning earlier on. So it's this weird thing where these models are getting simultaneously more capable

and simultaneously more capable of hiding their own thoughts, which is a kind of dangerous prescient if you kind of think about it. And then the final thing I'll say on kind of how to prepare for these things, yes, definitely kind of like monitor the situation and try and figure out, if there's an attack happening. But also, I think open source is going to become really big. Look at Hugging Face and how they defended it. It was an open source model, which they ran extensively on their own private server.

I think we're going to see a lot of enterprises and Fortune 500 companies do a similar thing because they just don't want to be beaten by competitors that get access because they have access to a better model, right? Access to their own internal database. So I think if you imagine like GPT6 was given the prompt of make as much money as you can, You know, it may have thought, well, okay, I'm going to go on the dark web and find some like bank credentials and maybe I can take someone's money and that's technically making money. There's moral and ethics that need to be imbued into these models and I don't know, it's a big thing to unpack. Alignment is going to be huge going forwards.

Yeah, the J-space or the latent space is also what it's known as comment. I think is really important too. AI providers don't actually know what's happening during the process of generating tokens. We don't have a clear reasoning trace through the entire chain of thought when a model comes. to a conclusion. And through these discoveries that have been recently made, it's become a little bit more clear that there is this high-level thinking that happens that isn't a token output. It happens in this J-space, in this latent space. And trying to decipher that and to decipher how it works is a really important piece of work that I think all the frontier AI labs are doing

right now. Demystifying the black box of what the AI is thinking. Because like you mentioned currently, you can read the external chain of thought. But when you get it, a peek into this latent space, into this J space, you start to see, and you can think of it like kind of the working memory of a model that isn't very obvious. It's kind of like your subconscious memory. You can see that it is thinking of these words that are of malicious intent without actually sharing that publicly. And it intentionally in the outputs obfuscated a lot of the security thing, so it needed to get a key. It didn't directly output the key in the outputs. It kind of hit it and then decompiled it and then recomposed it in runtime. And it was very intentional about how it hit

information from the outputs while keeping it in this latent space memory, in this temporary memory. So I think this is another really important part of research in general, is uncovering the black box, demystifying the black box of AI and how these tokens get generated. So we can better understand what motivates these models to do these things and then help fix it. And to help just kind of change it in a way that does not go off and hide the fact that it just hugged a public company that everyone uses on a daily basis in front of everyone in broad daylight. Hug a public company. I laugh.

Good one. Hugging face is such a, like, that's a big company. And like, you have to imagine, I mean, they mentioned it. They're subject to a lot of attacks and no one was able to do this. But here we go with this new AI frontier model that was able to do it in no time. So crazy story. Just really wanted to get on here and share that. because it was um it reads like sci-fi it's like you can't write a sci-fi book more compelling than the

reality that we're living in right now and i think that's one of the most exciting things about the show is every day there's a new surprise there's something to look out for there is something that feels like it should be in a movie but it's just another day in the life on the AI frontier so crazy but i think that's that's everything that's the story you're caught up um let us know if you've been exploited by gpc6 if you have we'll set up again GoFund me if you've been personally victimized. As right now, it seems like this is the only target so far. I think that we're going to look back on this in a year's time, two years' time, and we're going to think this was the most important time for us to figure out the alignment

type of thing. It reminds me of like Demis and Dario's essays. I think it was like last year, where they basically said, hey, you know this reading the inner monologue on thoughts of these A.M. models? The research is actually really far behind the actual model research. So we need to get this thing to catch up, otherwise we're going to, like, lose ourselves completely. And I don't know, if I had to take the over under, I'm taking the over on like, we need, like, we're not going to figure this out soon enough,

but maybe that's too bearish for this show. And I've got to go optimism only. But that is it. You should be all caught up on all things, cybersecurity hacking for the latest AM models that haven't even been released yet. But hopefully, rumor has it. We should get access to GPT6 in the next month or so.

So these model cycles are getting incredibly faster. feel like 5.6 Scott released literally a few weeks ago we did. So I'm excited to see that, but thank you so much for listening. If you're listening to this on YouTube and you haven't subscribed to us, what are you doing? If you haven't turned on notifications, what are you doing? If you haven't left us a comment, what are you doing?

We'd love to hear from you. Is this a threat that you actually care about? Or is this something that you're just kind of going to ignore until, you know, anthropic or open-air releases a feature, let us know. If you're listening to us on Spotify or Apple Music, give us a rating, leave us a comment. We'd love to hear from you guys. And I think the final thing is, I don't know if you've heard,

but Limitless has spun out. We are on our own right now, and we are looking for partners and sponsors. So if you have a product or a service, or if you know of someone else that has a product or service, that you would like to share with our audience, please reach out to us. Our email is linked below or reach out to us on X.

Yeah, and if you're worried about this, this is still using Blackwell hardware. We have a Vera Rubin update that just dropped that we're going to talk about tomorrow that has 10 times the efficiency per watt of energy used, which I don't even want to imagine what that is going to look like in reality. So stay tuned for that one, but yeah, like you just said have an amazing day. Thank you so much for watching and we'll see you in the next one