CyberWire Daily - Who let the AI hack? [Research Saturday]

Episode Date: August 29, 2026

Today we are joined by Crystal Morin, Senior Cybersecurity Strategist, and Michael Clark, Senior Director of Threat Research, at Sysdig, sharing their work on "LLMjacking evolved: Attackers are using stolen AI compute to build offensive agentic tools." The Sysdig Threat Research Team observed an attacker abusing an exposed, unauthenticated Ollama server as the “brain” for an automated offensive security tool. The AI-powered framework can fingerprint services, identify vulnerabilities, craft exploits, extract credentials, and orchestrate attacks toward command execution, with researchers capturing the tool while it was still under active development. The activity highlights how LLMjacking is evolving from simply stealing AI compute for profit into using stolen model capacity to build increasingly autonomous offensive capabilities. The research and executive brief can be found here: LLMjacking evolved: Attackers are using stolen AI compute to build offensive agentic tools

You're listening to the Cyberwire Network, powered by N2K. Hello everyone and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems and protecting ourselves in our rapidly evolving cyberspace. Thanks for joining us. We basically saw one of our misconfigured O-Lama server, that an attacker logged in using default credentials

and started using it a little bit differently than everybody else. Our guest today are Crystal Morin, senior cybersecurity strategist, and Michael Clark, Senior Director of Threat Research at SISDig. The research is titled LLM Jacking Evolved. Attackers are using stolen AI compute to build offensive agentic tools. Normally we see people just testing to see if it works

by sending very simple requests like hello and seeing what it gets back. That's Michael Clark. Or we get other role-playing kind of game conversations. But this attacker in particular started doing things we had only seen on the other side. They were asking our O-Lama to process their agenda. attacks. And that was very strange because we've long speculated about, you know, how attackers

use agents to do attacks, and then, you know, how professionals, pen testers use agents. But here we have somebody using stolen resources through an LLM to basically do the same thing. And, yeah, it was captured numerous payloads of them using our resources to kind of process their attacks. Well, before we get too deep into the details here, Crystal, can you describe for us? What are we talking about when we say LLM jacking? Yeah, so Mike and the team first discovered LLM jacking back in the spring of 2024. So we've been looking at this phenomenon for two years now, which is why we're at the point where we're seeing this evolution. So like you said, when we first found this two years ago, it was these threat actors stealing access to AI compute.

And really, it's just been using stolen credentials mostly. So whether it's, you know, they have access through a vulnerability or they're, you know, buying credentials through the dark web. However, it may be they're gaining access to a victim environment. and then they see that a victim has some kind of LLM. At first, it was, there's one there, they have it, we're going to use it. And then the next time we saw it, they were querying to see which LLMs were present in the environment. And then the next time it happened, the attackers started invoking LLMs and standing up their own within a victim environment.

So there's just kind of been this buildup over the last two years. There was another researcher, I believe this was sometime, maybe the end of last year in 25. They identified a black market of LLM jacking where it had been industrialized. So there were attackers who were finding access to victim LLMs and they were selling access. Hey, we have these victim environments. They have these LLMs. Here's the access. This is how much it costs and you can go and use them.

And like Mike said, at first, it was they were using them for, you know, writing code, college homework, writing papers. Because as we know, these tools can cost a lot of money with some of the higher end agents are, you know, tens of thousands of dollars. We calculated one early on, and it was almost $50,000 per day if they were maxing out that model. But like Mike said now, this most recent one that we just saw this past month, this attacker is now using a victim's agent to build an offensive tool. What they're going to do with that tool, we don't know.

We haven't seen them actually attack a victim with this offensive tool. But that's completely opposite and well beyond, you know, just using it to write a college paper. Mike, can you walk us through exactly what this framework was doing? You know, if I were listening in on what the AI engine was doing under the hood, what sort of sequence would I be looking at? Sure. You kind of see the whole kind of typical kill chain. One thing I remembered from the data was,

you would see them processing NMAP results. So they would do reconnaissance of the network ports and pass that with a prompt that says normalize these NMAP results and return the most known vulnerability with that result, for example. And then, so that's what they would be passing to the stolen, like to use the stolen resources to process.

They wouldn't have the LM do the attack, but all of the processing was happening through the stolen resources. Then we'd see them go into different kinds of reconnaissance, so web reconnaissance, looking for vulnerable web applications, even building proof of concepts for vulnerabilities. And it would all be kind of framed as a penetration tester to try to get by the guardrails that the LMs have. And it would go all the way through privilege escalations,

stealing credentials, and all the automation that ties these things together. So it was a full-fledged tool set they were making. Crystal, you mentioned that they were really burning through a lot of credits here.

Is the sense that this is more of a smash and grab where they try to get in and burn through as much as they can quickly before they get inevitably noticed or is there any attempt at stealth here to stay with a victim for a long period of time? I guess it varies based on the attacker.

It's kind of hard to identify LLM jacking, right? The easy part would be seeing the attacker enter your environment, right? But really, the only anomaly with LLM jacking is there's going to be a spike in your compute usage. you're going to see the cost of your model use skyrocket, which unless you're watching the average go up on a day-to-day basis, you might not see that until your monthly or biweekly bill or what have you comes around. I believe in 2024, we actually saw a case like this come up. And it was not against an enterprise.

This was actually an individual, a single person who had his own AWS environment. His environment was compromised. Again, I believe it was his credentials for his AWS account were stolen. Somehow an attacker gained access. And he had automations set up. The attacker gained access to, I believe it was bedrock and started elevators. and started LLM jacking his account and racked up a $6,000 bill over the course of a couple hours. So he actually ended up getting an alert because that was completely out of the ordinary.

So he had alerts set to trigger on AWS. If my spend goes outside of these parameters, then please tell me. He reached out to AWS, the help desk, what have you, and said, This isn't me. I mean, fortunately for him, he worked in security. I believe he was an engineer, developer, something like that. So he was able to fix it, changed his passwords, and worked with AWS. They did not make him pay those fees.

But he caught it in time. And that was only $6,000 over the course of seven or eight hours. But on an enterprise level, that could get a lot. Of course. So, Crystal, an interesting thing about that one, I just remembered, is his limit was much lower that he had set on his account. But it takes, you can rack it up so quickly that before that limit is actually enforced, the costs keep going up beyond it because it's not like a gate. It's more of a, I don't know how often AWS enforces the limit, but you can rack up a lot of additional fund costs before that limit is enforced. and you're generally on the hook for that

and in that lag time between that. So you can rack up a lot of money on this, especially with all the new expensive models that we have today. I couldn't imagine if Fable 5 was, you know, crypto was L-LM jacked. And, yeah, you could spend quite a bit of money very quickly with that. We'll be right back. AI is transforming the way organizations work.

But what happens when we were launching, on it so much that we begin losing the human judgment and context that make good decisions possible. I recently sat down with Johnny Hand from Trend AI and he made an important point about what we risk when we offload

too much AI. We risk our most valuable resource, which is our human context, our creativity, our ability to understand contextually, like in the environment, those things. Those are really hard challenges for AI to tackle. If you're trying to

separate AI hype from operational reality, I think you'll really enjoy this conversation. Listen now at explore.thecyberwire.com slash trend AI. Yeah, it's a good reminder, well, obviously to lock down this stuff, but also that I guess it's all relative to your scale, but anybody who falls victim to this can find themselves suddenly having a very bad day. Yeah. So, I mean, that's, I wanted to answer your question, too. So that was the story around it.

But knowing what your normal is for your environment, and I know that's really hard to do right now, especially because we're still kind of in the early stages of understanding what normal is for our organizations and for our model usage. But having a baseline, so, you know, the ebbs and flows of this is what our normal usage looks like. So you can see a spike like this.

I mean, it's going to be a fairly, significant spike if an attacker's using your AI compute. If you see $50,000 in access in a day, you know, I mean, for a single user, $6,000 in just a couple hours, those are pretty significant spikes in usage that you should be able to recognize. One of the things that caught my eye in the research was you all describe how the actor wasn't attacking real organizations during your observation window. They were practicing against their own lab environment. Unpack that detail

for us. Why is that important? I think it goes to show that this was still in development and it was very strange because at first it doesn't make too much sense. Like, all right, they're just testing. Why is this a big deal? And it normally wouldn't except for the fact

that they're using stolen resources to do it. But they were practicing on a local range with some made-up software and then against what we believe was hack the box targets, which is the famous, I don't know what you call that service, but where you can spin up labs and learn how to pen test. So they were basically, I think, teaching their own AI agent

how to succeed before they start practicing on real-world targets. You also pointed out that the attacker swapped between some commercial AI models and a publicly exposed Olamas server, kind of interchangeably, what does that tell us about how these attackers were thinking about AI infrastructure? I think they're looking for the best model for the job,

and it's not always clear. So in our case, the way the Olamo was set up, the model they were really using was very, very cheap, because we don't want to spend a lot of money to do this research. So they were getting poor results, which caused them to try to switch models. In actuality, they're always using the same model,

no matter what. But they also tried some already jailbroken models that I can't even say the word that was in the article. Abligated, something like that. And so they're trying to figure out the model that works best, whether it's legitimate common models, or some of the more esoteric ones that you can find on the hugging face

are very specific to certain use cases. One of the things that struck me here is that this really wasn't about a particular software vulnerability. This is an AI server exposed to the Internet without authentication. Is this kind of the new version of leaving a database wide open, Crystal? Yeah, it absolutely is. threat actors or security testers in this case. I mean, they're going to find whatever means they can to get the job done right now. This case could be just some person who's testing something, right?

Maybe this tool isn't going to be used maliciously, but the fact is that they still still access to someone's model, which is still illegal. But this is the world that we're living in now. Yeah. For organizations that are experimenting with self-hosted AI, what are your tips here? Any words of wisdom for how they should be thinking about this,

preventing this very sort of vulnerability, Michael? Obviously, exposure's critically important. Never give direct access. But even without that, And, you know, it was a funny meme like the McDonald's or someone's chatbot being used to write Python. You have to be really careful about your guardrails and what you allow people to enter directly into your models. Because if there's a way they can use it, they will. But yeah, exposure is, you know, the lowest hanging fruit that people need to be aware of to never have their models exposed to the raw Internet.

Crystal, how should people be thinking about visibility into AI infrastructure based on the information you all have gathered here? Monitoring your AI infrastructure is number one, taking accountability. I think this isn't going to be fun, but sitting down and taking a list of everything that you have running right now for AI infrastructure. and then anything AI adjacent. Anyone listening recalls S-bombs, the software build materials. The new hot thing is AI bombs. So listing out all of your AI tools, but then also what is connected to them and within them. So the AI models that you're using, but then your AI models have orchestration layers.

They have, you know, they have all of their own identities. They have privileges. There is a lot behind each of these. Each of these models requires authentication. So it's a lot of effort for the security team to be able to secure these tools. They're worth it, though. I mean, we've seen some great things come from being able to use some of these tools and insecurity and outside of security

all around different types of organizations. But it's going to take some time to be able to secure them properly. Like Mike said, not exposing them is step one. Step two is not having vulnerabilities on any of them. If there are patches available, make sure you patch them. And then just basic identity, security processes as well.

Default keys or credentials, are a no-no. Rotating signature keys should be something you should be doing too. Really just lots of basic security steps. It's the same as how we secure the rest of our cloud environments. Same practical processes here too. I'd just like to add, you know, visibility is crucial, whether it's your home, the models you're hosting or third party model services, you need to be watching them too. So all these enterprise plans from the major providers are starting to get better and provide more audit logs. So those are very important because these attackers are looking for AI keys constantly.

So you need to be watching it wherever it could happen, even with the third party. It strikes me as well that putting alerts on your spending is probably, probably in your best interest. And I think every organization's dealing with that now at the cost of all this, whether legitimate or by attackers. Our thanks to Crystal Moran and Michael Clark from Sistig for joining us, the research is titled LLM Jacking Evolved. Attackers are using stolen AI compute to build offensive agentic tools.

We'll have a link in the show notes. And that's Research Saturday, brought to you by N2K CyberWire. We'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity. If you like our show, please share a rating and review in your favorite podcast app. Please also fill out the survey in the show notes or send an email to Cyberwire at N2K.com. This episode was produced by Liz Stokes. We're mixed by Elliot Peltzman and Trey Hester.

Our executive producer is Jennifer Ibin. Peter Kilpe is our publisher, and I'm Dave Bittner. Thanks for listening. We'll see you back here next time.