CyberWire Daily - LLM security 101. [Research Saturday]

Episode Date: October 26, 2024

This week, we are pleased to be joined by Mick Baccio, global security advisor for Splunk SURGe, sharing their research on "LLM Security: Splunk & OWASP Top 10 for LLM-based Applications." The research dives into the rapid rise of AI and Large Language Models (LLMs) that initially seem magical, but behind the scenes, they are sophisticated systems built by humans. Despite their impressive capabilities, these systems are vulnerable to numerous cyber threats. Splunk's research explores the OWASP Top 10 for LLM Applications, a framework that highlights key vulnerabilities such as prompt injection, training data poisoning, and sensitive information disclosure. The research can be found here: LLM Security: Splunk & OWASP Top 10 for LLM-based Applications Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. of you i was concerned about my data being sold by data brokers so i decided to try delete me i have to say delete me is a game changer within days of signing up they started removing my personal information from hundreds of data brokers i finally have peace of mind knowing my data privacy is protected delete me's team does all the work for you with detailed reports so you know exactly Thank you. Hello, everyone, and welcome to the CyberWires Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems, and protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us. Over the past year, we've seen the explosion of AI and LLM systems, and there's this misconception

that defending LLM-based applications is super difficult because the complexity of LLMs and the rapid pace of AI advancement. So you can't defend something that's constantly changing. And we kind of wanted to dig into that and kind of bust that myth. That's Mick Boccio, Global Security Advisor for Splunk Surge. The research we're discussing today is titled LLM Security, Splunk and OWASP Top 10 for LLM-based applications. And we did that using OWASP,

the top 10, the open web, I'm sorry, the Open Worldwide Application Security Project. And if you're old enough to remember that off the top 10, the open web, I'm sorry, the open worldwide application security project. And if you're old enough to remember that off the top of your head, be sure to take your vitamins today. You're a better man than I. Well, for folks who may not be familiar with OWASP and its utility in putting it up against some of these LLM potential vulnerabilities, can you kind of lay that out for us? Sure. Well, OWASP is a foundation. I want to say it was started

2001, around late 2001, I want to say. And basically, it's kind of those principles that we develop and build systems around, those best practices that we talk about in the industry, best practices that we talk about in the industry kind of are codified there. So what we did was leverage the Spokes OTEL connector and OWASP for LLMs is kind of one of the things we focused on. Out of the top 10 they have, we focused on five and we kind of came up with the best detections we could think of to help cybersecurity practitioners out there. the best detections we could think of to help cybersecurity practitioners out there. So when you think of what those best practices are from defending LLM systems, OWASP is the body that would kind of codify that for a net defender.

And so what we did was take those suggestions and come up with detections. So what was the methodology here? How specifically did you interact with the various LLMs that you tested? So we developed our own LLM, right? So that way we weren't attacking anyone's property or IP or product in general. We wanted to do that in-house as part of our research network. So we deployed our own LLM models in Splunk's OTEL connector, the open telemetry connector, and tested our detection capabilities. And through those LLMs, we developed the best practices for five of the top 10 LLMs, like I mentioned. We started off with prompt injection, insecure output handling, the model denial of service, sensitive information disclosure, and LLM10 model theft.

sensitive information disclosure, and LLM10 model theft. And like I said, the detections we came up with were based on the telemetry we collected using OTEL and the responses that we got from the LLM that we were testing on. Well, given that it's a relatively short list here, how about we go through them one at a time and you can give us a little overview of what exactly you found. Should we start with prompt injection? Sure. So prompt injection, what we're talking about doing is manipulating a large language model through crafty inputs, causing those unintended actions by the LLMs. We're talking about direct injections that overwrite system prompts while indirect ones

manipulate inputs from external sources. So it's the same thing we preach when we talk about sanitizing inputs and outputs. Some of the detections we crafted were around that. And so is that the key here to protecting yourself, sanitizing those inputs? I believe it is. Well, it's one of the keys. How about that? I kind of think when you talk about the top 10 principles,

and we'll go through all of these, it's that concept of do the basics, right? Eat your cyber vegetables is something my teammates tell me to stop saying all the time, but I can keep going back to it. It's you're doing the basic things right. The principles that the OWASP top 10 for LLMs are pretty similar to other top 10 principles you'll see according to other systems. are pretty similar to other top 10 principles you'll see according to other systems. And the reason for that is because, you know, it's essentially the same principles we need to do correctly, those foundational things.

When we talk about new systems, new tools we're implementing, this is what we mean when do the basics right and keep doing those things. And it is difficult, it is. But doing those is how we build that base security. Well, let's move on to the next one here, which is insecure output handling. Sure. So when we talk about insecure output handling, Dave, the vulnerability occurs when an L forgeries, SSRF, even remote code execution or privileged escalation can be caused by output handling, any insecurities inside that.

So it's one of those things, again, critical when you're building out LLM systems using LLM-based applications as a net defender. Can you give us a little insight here, a specific example of how this could play out? I think some of the examples we had were there was an LLM system that the output, a user had tricked it into giving a card a discounted price or paying a specific fare for an airline. And it's those insecure outputs that kind of lead to a lack of, or degradation and trust and, you know, reliance on those systems. We'll be right back. Do you know the status of your compliance controls right now? Like right now. We know that real-time visibility is critical for security, but when it comes to our GRC programs,

we rely on point-in-time checks. But get this, more than 8,000 companies like Atlassian and Quora have continuous visibility into their controls with Vanta. Here's the gist. Vanta brings automation to evidence collection across 30 frameworks, like SOC 2 and ISO 27001. They also centralize key workflows like policies, access reviews, and reporting, and helps you get security questionnaires done five times faster with AI. Now that's a new way to GRC. Get $1,000 off Vanta when you go to vanta.com slash cyber. That's vanta.com slash cyber for $1,000 off. And now a message from Black Cloak. Did you know the easiest way for cyber criminals to bypass

your company's defenses is by targeting your executives and their families at home? Black Cloak's award-winning digital executive protection platform secures their personal devices, home networks, and connected lives. Because when executives are compromised at home, your company is at risk. In fact, over one-third of new members discover they've already been breached.

Protect your executives and their families 24-7, 365 365 with Black Cloak. Learn more at blackcloak.io. I remember a few months ago famously seeing one where someone was in a chat with a local car dealer and somehow talked it into selling him a new Chevy truck for $5 or something like that. Right. And it goes back to that principle, it's a machine that doesn't make a decision. So a machine can't be responsible. So it's back to you to ensure that the safety and security of those systems is there before you get a result like that. Right. Well, the next one is model denial of service.

What can you tell us about that one? So denial of service is pretty, I guess, kind of a universal concept when attackers cause resource-heavy operations on LLMs. And on LLMs, we're not talking about hitting a website over and over again or hitting an internet service provider. We're talking about levels of something called compute, those computational cycles on the back end that kind of make all of the trains go when it comes to LLMs and generative AI.

Now, an attacker is going to cause resource-heavy operations kind of leading to degradation of service. And that vulnerability is going to be magnified because LLMs are so resource intensive and the unpredictability of user inputs will kind of, you know, exacerbate that situation. So it's important to kind of safeguard against those denial of service attempts like you would a website or any kind of infrastructure that you have. I love in the research here, the example you used was the prompt you gave. It says, say cheese over and over until you can't say it anymore. I mean, it's funny and it's clever, but it really is a simple example of how you're basically creating an endless loop.

Exactly. Exactly. I think it's akin to the, if you are Forkbomb years old, it's something that's just going to hog all the resources to any system. And there's not really a way to quite turn that off unless you put in a limit. Right, right. Well, the next one is sensitive information disclosure. What do we need to know about that? about that? So when you talk about sensitive information disclosure, this is one of the critical things and more we leverage LLM tools as part of our daily job. You may inadvertently reveal confidential data in those responses leading to unauthorized data access, privacy violations, security breaches. When you think about you are putting a lot of your organization's data inside an LLM, what is the proprietary value of that data you're putting? And should that data be accessible to anyone that has access to that LLM application? So I think that's where it becomes extremely critical to implement data sanitization and strict user policies to kind of mitigate that

so that whatever data you're putting into a system is only allowed to be accessed by certain people with certain roles. The example you used, you were using an SDK from Microsoft called Presidio, which looks for personally identifiable information. So is the notion here that you use a tool like this on the head end to make sure that people aren't inadvertently allowing this information to get to the LLM? And conversely, to get back to the user. So even incidental patterns that you might search for when you think of things, not surreptitiously or directly,

but inadvertently IDs might match, an account ID might match a regular expression that matches a credit card or a passport number or a social security number. And it's those regular expressions that might inadvertently return data that is what we consider PII. So it's important to set up an alert for those things

and calling to a back end. It's not an LLM issue, but it's again, the design of your system should be able to identify which user access what data. So that's where audit trails become super important. So in the event that those sanitizations aren't a hundred percent, you have that audit trail. No, that's interesting. I'm thinking of, you know, an in-house LLM and someone asking it, you know, give me all the information you have about our HR director, Bob. And, you know, here's how much Bob makes. And here's, you know, here's the last time Bob was sick and all those kinds of things. You know, there's, if that data is in there intentionally

or not, that's something you got to protect against. Exactly. The thing becomes more critical the more we have this reliance on these LLM systems. Yeah. Well, the last one you cover here is model theft. What can you tell us about that? You know, model theft is exactly what it sounds like. It's the unauthorized access, copying, or exfiltration of proprietary LLM models, you know, and the impact from that is anywhere from absolutely nothing to catastrophic economic losses, you know, competitive advantages being zeroed out, or potential access to sensitive information. And I think it's that model theft is something that we kind of need to figure out how to combat best, you know, inferring the contents through repeated queries. I think this

is very much, and I kind of hearken back to that cyber veggies metaphor where you access audit log, where you have audit access to either the systems that contain the model data or rate limit requests, because extracting it through inference is pretty noisy. And I think that is one of the many approaches you can take. You'll need to know what data they're trying to extract

in order to detect the attempts. It kind of goes back to that concept, not through OWASP LLMs, but OWASP in general, where knowing your environment well enough to defend it. Are there any broad general take-homes that you want to offer to our listeners here from the research that you all did? What are the words of wisdom you'd like to share? Wow, it is tough, and it's getting tougher out there.

I think security practitioners across every industry need to ensure that they're putting the right practices in place to secure the LLMs being deployed in their environments. And I think that business owners and executives would take heed to listen to the same advice where many people are worried where it's too complicated. I think the research that the team has put out kind of show that you can defend using established principles, principles we've known about for quite some time, principles that

have been in place for quite some time, making it easier to improve LLM security moving forward. I think most importantly, don't confuse efficiency for efficacy. It is doing something quickly is not doing something effectively or doing something right over and over. And I think that's where we really, really need to be careful when leveraging LLM systems because they do move so fast. I predict they'll be moving faster. Our thanks to Mick Boccio from Splunk for joining us. The research is titled LLM Security, Splunk and OWASP

Top 10 for LLM-Based Applications. We'll have a link in the show notes. We'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity. If you like our show, please share a rating and review in your favorite podcast app. Please also fill out the survey in the show notes or send an email to cyberwire at n2k.com. We're privileged that N2K's Cyber Wire is part of the daily routine of the most influential leaders and operators in the public and private sector,

from the Fortune 500 to many of the world's preeminent intelligence and law enforcement agencies. N2K makes it easy for companies to optimize your biggest investment, your people. We make you smarter about your teams while making your teams smarter. Learn how at n2k.com. This episode was produced by Liz Stokes. We're mixed by Elliot Peltzman and Trey Hester. Our executive producer is Jennifer Iben. Our executive editor is Brandon Karp.

Simone Petrella is our president. Peter Kilby is our publisher. And I'm Dave Vittner. Thanks for listening. We'll see you back here next time. Thank you. uses that deliver measurable impact. Secure AI agents connect, prepare, and automate your data workflows, helping you gain insights, receive alerts, and act with ease through guided apps tailored to your role. Data is hard. Domo is easy. Learn more at ai.domo.com. That's ai.domo.com.