CyberWire Daily - When AI gets a to-do list. [Research Saturday]

Episode Date: May 3, 2025

This week, we are joined by ⁠Shaked Reiner⁠, Security Principal Security Researcher at ⁠CyberArk⁠, who is discussing their research on"Agents Under Attack: Threat Modeling Agentic AI." Agentic AI empowers LLMs to take autonomous actions, like browsing the web or executing code, making them more useful—but also more dangerous. Threats like prompt injections and stolen API keys can turn agents into attack vectors. Shaked Reiner explains how treating agent outputs like untrusted code and applying traditional security principles can help keep them in check. The research can be found here: ⁠Agents Under Attack: Threat Modeling Agentic AI Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the CyberWire Network, powered by N2K. Traditional pen testing is resource-intensive, slow, and expensive, providing only a point-in-time snapshot of your application's security, leaving it vulnerable between development cycles. Automated scanners alone are unreliable in detecting faults within application logic and critical vulnerabilities. Outpost24's continuous pen testing as a service solution offers year-round protection, with recurring manual penetration testing conducted by Crest-certified pen testers, allowing you to stay ahead of threats and ensure your web applications are always secure. Hello everyone and welcome to the CyberWires Research Saturday.

I'm Dave Bittner and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems and protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us. So, agent-tki is kind of a concept, I should say. And it basically means any type of system, any type of code that uses LLM in sort of a way that allows the LLM to decide about the control flow of the program. That's Chaked Rayner, Principal Security Researcher

at CyberArk, discussing their research, Agents Under Attack, Threat Modeling, Agentic AI. We'll have a link in the show notes. What initially prompted you and your colleagues What initially prompted you and your colleagues to investigate the security implications of agentic AI? Yeah, so as I'm sure we all know, the buzz around agentic AI is very much present in our industry in the last couple of months. And about two years ago, we decided to dive into the whole LLM security train.

And we did that transitioning from traditional security. And the security aspects or security risks of HNTK AI is much more, the security risk is much more severe with HNTK AI because unlike traditional LLMs and traditional chatbots, HNTK AI systems allow the models to actually perform actions in the world. And so having a vulnerability in one of those systems can actually have much greater implications than just having

the LLM spit out some information that it shouldn't. And this is why we decided to go deep into the Agendic AI security, and this is what motivated us to start exploring it. Well, let's dig into the study together here. What are the primary vulnerabilities associated with agentic AI systems that your research uncovered? Yeah. So we basically focused on systematically mapping out the threat landscape of agentic AI systems. So some of the LLM vulnerabilities

or attack vectors were already known, but we really emphasize, put emphasis and focus around how those apply in agentic systems. Moreover, we really tried to illustrate and actually demonstrate practically and technically using a lot of demos, how those attack vectors manifest in the agent-dk-i field. Generally speaking, we can divide the threat landscape of agent-dk-i into two categories.

One is the traditional access attack vectors. Because agent-dk-i is built upon normal code, we still have a lot of server-level attacks on attack vectors. Because agentic AI is built upon normal code, we still have a lot of server-level attacks on those systems that we all know and love from the past few decades of information security. And of course, it's still relevant. And we need to be aware of that, because even though the technology

LLMs and agents, is very different, it's still vulnerable to a lot of traditional stuff as well. In addition to that, those systems also present a completely new attack surface, which is the attack surface that presents a lot of LLM-based attacks. There we can see a lot of prompt injections and model manipulations that eventually can manipulate the system to behave differently than what it was intended. Well, let's talk about some of

the identity and access management challenges here. I mean, how does agentic AI complicate traditional identity and access management frameworks? That's a good question. I think agentic AI is still a beast that we don't know very well as an industry, and there aren't any security standards in it, and we still don't really know how to treat those AI agents. Are they users? Are they machines? Are they bots?

The question is still open. And of course, AI agents, the whole thing about them is that they are able to perform actual actions. And for them to do that, we have to grant them permissions, we need to give them access tokens, we need to open accounts for them, we need to allow them to access databases and so on. And because of that, we really need to understand what their identity should be and what access exactly they can have. And since, again, it's a new beast, it's still kind of a challenge that we need to face as an industry. Can we talk about some of the risks

from overprivileging these AI agents and how organizations can mitigate these kinds of things? Yeah, of course. So overprivileged is a risk that is not only associated with agentic AI. However, it manifests in a very severe way. It can manifest in a very severe way in agentic AI. For example, let's say that we have an agentic AI system that needs to access some databases. Of course, it needs to have a token

for writing and reading from those databases, and it needs to have it for all of them, for it to work. And now, let's say we are a user or an attacker, and potentially we can only have access to one of those databases. Just because we can use the system, it doesn't mean we have to be able to access to all of them. So we have a discrepancy between what I or the user or the attacker can access and what the agent can access. And we know that LLMs can be manipulated, and this way, even if I personally as a user have access

only to one database, I can try and manipulate the agent working for me that has access to more of those databases in order to perform actions that I wouldn't have access to in the beginning. It's interesting to think about the idea of kind of social engineering your AI agent. Exactly. And when we were starting to work on that LLM security field, jailbreaking was just that trying to convince to kind of socially engineer or persuade the LLM

in any way you can think of in order for it to behave in ways that it shouldn't. We'll be right back. Cyber threats are evolving every second, and staying ahead is more than just a challenge, it's a necessity. That's why we're thrilled to partner with ThreatLocker, the cybersecurity solution trusted by businesses worldwide. ThreatLocker is a full suite of solutions designed to give you total control, stopping

unauthorized applications, securing sensitive data, and ensuring your organization runs smoothly and securely. Visit threatlocker.com today to see how a default deny approach can keep your company safe and compliant. Secure access is crucial for US.S. public sector missions, ensuring that only authorized users can access certain systems, networks, or data. Are your defenses ready? Cisco's Security Service Edge delivers comprehensive protection for your network and users.

Experience the power of Zero Trust and secure your workforce wherever they are. Elevate your security strategy by visiting cisco.com slash go dot sse. That's cisco.com slash go slash sse. Can we talk about lifecycle management here? As these AI agents go through from deployment to decommissioning, how do organizations handle that? Yes. In terms of lifecycle, I think AI agents present a few challenges that we haven't seen before in that way. And first of all, we should talk about the LLMs, the base models that the agents rely on and that they provide most of the functionality of the agents. As we know, LLMs are those huge neural networks

that most of us just consume because there aren't a lot of companies that really develop and train those big models. And so as consumers, we don't really know exactly what's going on inside. So this is one aspect we should be aware of, and of course only use models that we trust to the extent that we can trust them, again without knowing exactly what's going on inside. So this is one area. Now another thing is how do we make sure that our agent is trustworthy. And again, we can, there are a few options here in terms of utilizing or writing agents. We can either download or access an agent in some service, or use, choose an agent from some store. And then again again we need to trust the developer of this agent.

Or we can alternatively develop them ourselves, and then we need to make sure that we are aware of exactly what are the important parts in the code that we should monitor. So, for instance, let me give you an example. If we're talking about a traditional software, then we know that when the code itself changes, it may change the behavior of the system dramatically. Now with AI agents, a lot of times the behavior of the agent can be dramatically changed just by changing the instructions or the system prompts or the configuration of the model that is at the heart of this AI agent.

And this is why we need to make sure that we also really monitor and defend those configuration files holding those instructions. I mean, I suppose just similarly to how you routinely monitor your employees, just to make sure that they're doing what they're supposed to be doing and nothing has gone astray, you need to monitor and audit your AI agents as well. Exactly. And again, because of the flexibility that LLMs provide to traditional code with AI agents, it's really a good idea to monitor the actions that the agents can do because again,

the nature of them is very dynamic. And if we don't define and design them properly, they can really go rogue. So based on your findings here, what are your recommendations? What are best practices for organizations out there who are going to be deploying agentic AI? That's a good question. First of all, I'd like to suggest to map out all of the systems using LLMs and all of the agentic systems in the organization in order to just get a grasp of what

we're dealing with. Then I'd like to suggest a few key core principles to go by. The first one, and I like to really emphasize this one, is to never trust the LLM. And I like to say that because a lot of people inherently tend to trust those LLMs because they really have an impressive ability to output intelligent text. However, we know that they can sometimes hallucinate, but for us in the security industry,

we know that attackers can very easily manipulate those models to behave in any way they wish. So the idea is to never trust an LLM. From that stems a lot of security best practices that we can implement in order to deal with that. So for instance, whenever you consume an output from an LLM, make sure to verify that the information is correct, to validate, to sanitize it, and to never treat your LLM as a security boundary. Next, the other thing I can recommend is to really think about what task you need the LLM to perform.

And in case it can be performed in traditional code, don't use an LLM for that, and really limit the space where the LLM can decide what to do and limit the scope of action. This is the second thing. Next, of course, we can utilize the old least privilege principle. So like we mentioned before, make sure that the LLMs have or that the agents have the least have the most minimal set of permissions that they should have in order to be functional for your purpose, so that attackers cannot exploit excessive permissions in those agents. Then, of course, traditional credential management. As we mentioned, those agents will have to be given credentials of all sorts in order to perform actions, so make sure you manage them and monitor

them properly. And finally, this is more of a general recommendation, and you mentioned this point again before, make sure to have security monitoring and threat detection and response for those agents. We know that no security measure can be bulletproof, so make sure that you are monitoring those agents and have the appropriate measures to deal with compromises in case they happen. You know, I'm curious if you have any insights, you know, looking towards the next few years, how you expect the evolution of agentic AI

to impact cybersecurity strategies? How are we gonna have to adjust to this new reality? Yeah, that's a great question. And I have to say, I don't really have an answer for it. The point is, yeah, I don't have an answer, but I can make some educated guesses. So with this whole AI field, then more specifically with agent, TK, I, we see that again, the pace that this technology progresses is really, really fast. It's crazy. And in security, we see it kind of like a moving target because every day we see new technology running out. And it's very, very hard to create security boundaries in this stage of the development of the technology.

So I can assume that in about a year or two, agentic AI would look entirely different than what it is now. Again, we really just see the tip of the iceberg of this technology. Just think about how we were looking at chat GPT a couple of years ago, and how usable it was then, and how it looks like now. The difference is really, really amazing. And I think that in terms of agent-ic AI, both from the side of functionality and productivity, it will look completely different,

and it would probably be able to do stuff that we can't even imagine now. So this is the first thing. And secondly, I'm sure that the security measures that we will have to create will be different than what we can think of now. And one last comment here. As we know, the hackerspace or the security researchers that are working in this industry, there are some very creative people. And I think that we haven't really gotten into the more advanced attack vectors and more advanced techniques that will be able to exploit AI agents and AI in general.

So we still have some time for it to develop. I'm sure that we will be amazed with what we'll see there. This will require protections that we have a hard time imagining now. Thanks again to Shaqued Rayner and the team at CyberArk for discussing their research, Agents Under Attack, Threat Modeling Agentic AI. That's Research Saturday. We'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity.

If you like the show, please share a rating and review in your podcast app. Please also fill out the survey in the show notes or send an email to cyberwire at n2k.com. We're privileged that N2K CyberWire is part of the daily routine of the most influential leaders and operators in the public and private sector. N2K makes it easy for companies to optimize your biggest investment, your people. We make you smarter about your teams while making your teams smarter. Learn how at n2k.com. N2K's senior producer is Alice Carruth.

Our CyberWire producer is Liz Stokes. We're mixed by Trey Hester and Elliot Peltzman. Our executive producer is Jennifer Iben. Peter Kilpey is our publisher. And I'm Elliot, in for Dave. Thanks for listening.