Risky Business - Risky Business #837 -- GitHub Actions footgun claims TanStack

Episode Date: May 13, 2026

On this week’s show Patrick Gray, Adam Boileau and James Wilson discuss the week’s cybersecurity news. They cover: Mini Shai-Hulud and the TanStack compromise using Github Actions Instructure pays Canvas elearning platform data extortionists More Linux privilege escalation 0days! CISA helping critical infrastructure operators rearchitect their networks so they work offline This week’s episode is sponsored by email security platform Sublime Security. Bobby Filar chats with Patrick about how agentic AI is being evaluated by buyers in a marketplace that’s experiencing “AI fatigue”. This episode is also available on Youtube. Show notes ‘Mini Shai-Hulud’ malware compromises hundreds of open-source packages in sprawling supply-chain attack | CyberScoop Hardening TanStack After the npm Compromise | TanStack Blog Canvas Breach Disrupts Schools & Colleges Nationwide – Krebs on Security Instructure pays ransom after Canvas incident as Congress announces investigation | The Record from Recorded Future News When DNSSEC goes wrong: how we responded to the .de TLD outage Adversaries Leverage AI for Vulnerability Exploitation, Augmented Operations, and Initial Access | Google Cloud Blog Mythos smythos! How to find 0day with lesser models - Risky Business Media GitHub - V4bel/dirtyfrag · GitHub retr0.zip NVD - CVE-2026-42511 Flaw in Claude’s Chrome extension allowed ‘any’ other plugin to hijack victims’ AI | CyberScoop Ivanti customers confront yet another actively exploited zero-day | CyberScoop Palo Alto warns of critical software bug used in firewall attacks | The Record from Recorded Future News Where Have All the Complex Windows Malware and Their Analyses Gone? Meet Rassvet, Russia’s Answer to Starlink | WIRED DOJ says ransomware gang tapped into Russian government databases | TechCrunch Iranian government hackers using Chaos ransomware as cover, researchers say | The Record from Recorded Future News Foxconn confirms cyberattack impacting North American factories | The Record from Recorded Future News New CISA initiative aims for critical infrastructure to operate offline during cyberattacks | The Record from Recorded Future News ‘HELLO BOSS’: Inside the Chinese Realtime Deepfake Software Powering Scams Around the World How to Disable Google's Gemini in Chrome | WIRED FCC pushes ban on security updates for foreign-made routers, drones to 2029 | The Record from Recorded Future News

Hey everyone and welcome to risky business. My name's Patrick Gray. Adam Barlow is back on deck and he'll be joining James Wilson and I in just a moment to talk through the week's security news. And there's lots of like awful and funny things happening. So that's going to be fun. And then after that in this week's sponsor interview, we're going to be hearing from Bobby Filler, who heads up artificial intelligence over at Sublime Security. Sublime Security makes the most modern sort of contemporary iterative.

of an email security platform. So if you are, you know, looking to get the best in-class email security platform, you want to hit up Sublime. And we're talking to Bobby about, I guess, how customers these days are evaluating AI features in products. It's an interesting conversation. They are very AI-heavy, Sublime. And, yeah, he's going to walk us through the conversations they're having with customers and the questions they're asking, which seem to be the right questions, if I'm honest. And then we can sort of, we also have a bit of a chat about how,

you know, LLM AI, selling that compares to selling machine learning based AI, if you want to call it that from, you know, a decade ago. So it's a all in all, it's a very interesting conversation. It is coming up after this week's news, which starts now. And look, we got so much wonderful, delicious chaos to talk about. But we're going to start off by having a chat about this mini Shia Lude worm. We've seen this worm originally pop up last year sometime. We talked about it at the time. You know, it's a self-propagating NPM worm. In this case, though, the initial access is a really interesting thing, the thing that started this all off.

And it wound up infecting TANstack, which is an extremely wide-used thing in the Dev ecosystem. I mean, James, you're the engineer among us. We're going to start off with you on this one. What was the interesting vector here? And can you give us a bit of background on TANstack? Yeah, let's start with TANstack, because this. is a complicated machinery in a complicated landscape. So if you're building a React app, you know, React is like one of your two fundamental decisions. React is the framework that you're

working with and you're probably writing the code in TypeScript. But that's kind of akin to saying, well, I've bought my block of land and I've got my plans for my house, but there's a heck of a lot of other decisions you need to make about how you're going to assemble that app. And it's things like, you know, what's going to handle the API routes, what's going to handle the state management, and all these things are an entire separate ecosystem of components that have built up. And TAN stack is a set of those components. They've become wildly popular, and they've sort of forged their own paradigm within the React community.

So that's what it does. It is a very integral part of building a React app. Now, the thing that is super interesting about the way that this initial attack vector happened here is there was no leaked credential, there was no fished credential, there was none of the traditional sort of ways in that you would expect for an initial compromise. Instead, it relied upon a malicious pull request making its way through a GitHub action. And that GitHub action was to the admission of the TANSTAC folks improperly configured. But it's just delicious how they did this.

We talked a little bit about this. We ran through the run sheet before we got recording. We do this every week, right? And the misconfiguration here was quite subtle. Like this was not like Tanstack did something completely suicidal and dumb here. They did slip up a little. But like why don't you walk us through the mechanics of how this like malicious GitHub action would wind up giving these attackers access to Tanstack's repos? Yeah, it is funny, right?

Because you pulled me up on the fact that I was approaching this from my software engineering perspective, which is like they tried so hard and they did almost everything right. And you said, buddy, you're in security now, and there's no prize for getting just about everything right. And sure enough, there is a particular GitHub action that happens on a pull request trigger, which is it's kind of one of the most dangerous areas where a GitHub action can operate, because it's essentially the moment when someone says, I've got a pull request, you know, a set of changes that I would like you to bring into your repo. And this action fires within the context of the repo that the change might potentially, be merged into, but can, if so configured, also pull in those untrusted changes that have been

proposed by the external third party in this pull request. The advisory from GitHub that's actually been out for a couple of years now, to be fair, does say this is a very dangerous footgun. If you're using this pull request trigger action, be very, very careful about bringing in the untrusted part of the code repo, because this is what can happen. And, you know, they they didn't follow that advice, and that's exactly what this GitHub action did, and that's how the attacker got their foothold. They got code run, they poisoned a cache that was then used during the deployment, and it was actually the legitimate deployment step that then pulled that cash out and resulted in the bad binaries being uploaded to NPM.

I mean, it does feel a little bit, though, that GitHub has given everyone a footgun and said, don't shoot yourself in the foot with this footgun. you know, Adam, let's bring you in on this one. I mean, this GitHub action stuff, it just seems like perhaps GitHub could be doing more, but I don't know because I'm not an expert in this field. Yeah, they absolutely could have avoided shipping people a footgun. I think we kind of hear what we are seeing is that

I don't know that anyone really expected the whole industry to coalesce around GitHub as the way of building everything. And, you know, one of the things that I thought was interesting about the story is that... Well, remember, sorry to cut you off there and ruin your flow, but you remember when Microsoft bought GitHub and everyone's like, it's over for GitHub, that's it. No one's going to use GitHub anymore. GitHub's dead, right?

Yeah, no, it's weird how much GitHub has become a critical part of everyone's infrastructure and flow. But yeah, so in this particular case, I feel like they did so many things right, as James was saying. but the GitHub action kind of set up like this, I don't know that anyone really expected this to become so important, right? And this particular footgun to be such an important thing. I mean, the idea of running a little bit of automation

when you push stuff in and out of a Git repo or make pull request or whatever else seems like a good idea. And all these little things, they all seem like good ideas until you start really crossing important trust boundaries and then building infrastructure that in very kind of near, yearish real time pulls in dependencies from other people and so on and so forth. And we end up with the impact of these, you know, kind of little choices turning into, you know, quite widespread compromise of code.

But I thought the trick of using the kind of shared cache between this way, the GitHub action can cache artifacts that they've built or whatever else they want to speed up later steps in the process in which when you are redownloading, you know, heaps of infrastructure or heaps of, um, dependence. and stuff. That can make a pretty material difference to how responsive your build pipeline feels. And developers love having their build process be really snappy because then they can turn things around really fast and it feels like you're doing stuff. That, you know, that, you know, kind of desire for everything to be snappy and responsive was the second kind of part of this,

right? I mean, the first thing is GitHub actions, you know, executing kind of macros, I guess, or triggers automation events that potentially have untrusted inputs. One problem, you know, the side effects of those are. untrusted inputs being processed. In this case, poisoning a cache that's then used later on is the second part of the puzzle. And then the third part, of course, is once it was executing in the release context, then it drops this worm that will propagate into other people's repos, steal creds. Well, let's go there because we've covered the TAN stack part of this.

But then from there, as you say, it dropped a worm which went off then and self-propagated. And it's like, you know, you just get on social media and it looks like people are having a bit of a hard time containing this thing. Like it is running real quick. You know, and we went through this last year and everyone's like, oh, we've all taken steps to slow it down and GitHub's like, oh, you know, we're going to, this ain't going to be a problem in the future. And like, here we all are. And this time, the people behind this worm have added some real nasty features. And one of them is if you try to rotate cred. So it will, it will, you know, hand off all of its API tokens and stuff on your machine to the attacker. If you try to rotate them, it detects that they've been invalidated and

RMRFs your whole drive, which is just like the nastiest thing to do. I don't even know why you would do that unless you're just like, I don't know, it's like a real anti-social personality disorder kind of thing to do with your NPM worm. But like this thing's out there now and causing all sorts of drama. But what to what end is this, is this, you know, sitting above this whole thing? Is this going to be something really dumb like someone trying to steal cryptocurrency or something? Because that's the vibe I get here. It's very hard to tell because again, this all amount. to stockpiling creeds that we then go for what and for where

and so similar to PCP it might be a little while until we see either the follow-up actions that they decide to do or they just kind of farm the creds out and let other actors have a go with them but it like it is odd that we don't know where this is going but it'll go somewhere

the other thing I just wanted to add in here is I had a real cold sweat moment of panic this morning when I read this because I thought to myself it's okay I haven't done like an NPM install or a bun install for a while now. So I know I didn't pull in packages during this, you know, six minutes, six whole minutes where this was ablaze. But then it dawned on me that the way that coding agents work now, both codex and also Claude, they work in Git work trees. And so when you farm your agent off to go and do a task, it's working in a new directory that doesn't have

all these modules installed. And so time after time after time throughout my day as I'm interacting with agents, they are pulling these packages over and over. again. So yeah, folks need to think you might not have done an NPM install during this window, but which one of your agents was working on something that went and did it? You might not know that you did that. You might not know. That scared the hell out of me this morning. Yeah. All right. We'll talk about our incident response later. But yeah, probably time to move on to the next story now. And funnily enough, the next story we're going to talk about is this breach of Instructure and the learning platform canvas,

which is used by K-12 schools and colleges worldwide. I mean, it's reported as being an America thing, but I can tell you that universities here in Australia and across in your country, New Zealand, Adam, have been like delaying exams and things and dealing with this. Funnily enough, though, it being the big story, I don't know that we've got much to add here. I mean, it was a small breach that initially instructor was like,

oh, yes, you know, attackers tried. to compromise us but we have contained it and yeah it turned out not so much a lot of their data got walked including according to the attackers billions of messages between students and their educators and then of course there was a you know ransom note kind of dropped on a shiny hunters ransom note dropped on the login page for this system reporting suggests that they have actually paid now to get the data deleted and you know I mean that's pretty much the end of it looks like shiny hunters managed to rack up a win here. Adam, any thoughts? Yeah, I mean,

just paying them out feels bad, but on the other hand, you see the amount of pain it was causing you can understand why that was a decision that they were going to consider. And especially in cases where you've got a supplier and their customers, and the customers are all applying pressure to the supplier to do something, anything, and the only fast option that a supplier has is to go pay. And that feels bad, but you understand why. And, you know, I don't know, hopefully very little of it actually ends up in the pockets of a shiny hunter. But, you know. Well, you hope they slip up doing their money laundering or something, right?

Like, that's the vibe I get here is that that could be, that could well be how this ends. Because shiny hunters definitely has like UK teens vibes, right? Yeah, and I can't imagine their money laundering slash, you know, kind of money handling slash not going out and immediately spending it on, you know, on trash that's immediately kind of draws attention to them. doesn't feel super likely. Like, they may be good at the tech stuff, but that doesn't necessarily translate the good at, you know, long-term crime. I mean, everyone always criticizes people who pay, but I've said it on the show a million times, right?

Like, sometimes it's existential, and they've got to pay. Like, and it's really hard to say that it's uniformly wrong in every circumstance, which is why I was against legislative proposals that would have outlawed paying ransoms. Yeah, and I guess that's why I cast it as feels bad as opposed to is the wrong thing to do, you know? Yeah, exactly.

feels gross but what are you going to do well we're going to stay with you on this next story Adam because this one has you written all over it I thought of you as soon as I saw this um how how how how to begin so um someone screwed up in Germany uh rotating a key signing key for uh DNS sec which meant that the entire dot d dot dd dot d TLD was returning serve file on DNS queries where there were they were they were signed zone files right So they broke the chain of trust for all of .de. And Cloudflare basically its decision, and I agree with this decision. I think it was the correct decision.

And apparently there's an RFC backing this decision as well, but they just let the whole thing fail open, right? So basically they just switched off for the 1.1.1 cloudflare resolver. They just switched off DNS sec validation for all of Germany because of this screw up. And I think, look, absolutely the right decision. But it kind of goes to show you that like, this isn't really news. You know what I mean? Like I'm reading this straight off the Cloudflare blog. The fact that an entire TLD and a big one at that just turned off DNSSEC validation and everything was like no one noticed.

Kind of tells you what you need to know about whether or not the DNSSEC juice is worth the squeeze in my view. But I really came for your opinion on this one. Yeah, it's a pretty interesting tale, this one. And I think you kind of summarized the guts of the technical aspects. The interesting way it with Cloudflare is survey their 1-1-1-1-1-resolver does enforce DNS and validates the domains that it's answering questions about. And the correct behavior in the situation was to return an error, right? Return serve, fail and, you know, not answer the query.

And Cloudflare very rapidly realized that, you know, that was worse than just answering the queries and marking them as not secure. you know, saying like, here's the answer, but we're not, you know, it's not DNSSEC validated, which is what they ended up deciding to do. Well, and thankfully, Adam, thankfully, all of the software that we use and rely upon out there is set up to really take note of that note in the Return Zone file that says this isn't secure, you know? And, you know, this really changes a lot of things.

We, but I know we play with a bunch of DNSSEC stuff over the years and, you know, making the, like it's one thing to sign your zone files and publish it for other people to validate. It's a whole other thing to say, we are also going to make all of our queries fail if DNSSEC isn't available or, you know, should be available and isn't. Because it's just going to break stuff. And the amount of breakage that DNSSEC causes versus the amount of, you know, kind of impersonational, you know, um, cache poisoning or whatever other things that's trying to prevent, really the, you know, the impact of the, you know, the impact of DNSSEC is mostly about bad availability and not about integrity. And yeah, does the juice seem worth

the squeeze for DNSSE? Really not. And especially now that we have so much other crypto layer over the top with TLS. Let's encrypt one. Let's encrypt one. And the thing is like, I guess one of the reasons that I like to beat up on DNSSEC is the proponents of DNSSEC, like the real rabid ones, are among the most annoying people you'll ever meet in your life. Right. I mean, DNSSEC clearly grew out of the kind of cyphapunk way of thinking, right, where we should make it perfect without really accepting the reality of the world that we have to live in. And yeah, it's, I mean, DNS itself is just old tech and then bolting crypto into old tech, you know, we just, you know, brittle is the end result, right? We have let's encrypt. We have modern browsers. Let's move on. Yeah, and it's

not perfect, right? I mean, the let's encrypt world and the browser, like, letting, delegating this all out to TLS, you know, isn't the best solution. but it's the solution we've got and it's the only one that's you know kind of viable in the real life I'd argue Adam that actually in that everybody uses it it is actually the best solution unlike the NSSEC which people don't actually use but anyway anyway we can argue this one we could argue this one more at great length over a beer one day moving on and Google's threat intelligence group has released a report all about you know what's happening out there with adversaries and whatever and not surprisingly AI features very heavily. I guess the item that they spoke about here that's been talked about most is they discovered some threat actors had used AI to uncover an ODA in some sort of web IT administration tool.

I don't know. That sounds like C panel to me. I don't know. But it's like an MFA, something like that, I guess. But it's an MFA bypass bug that they found. And they were able to, I guess, disrupt the active. from being able to do widespread exploitation against that. So that's great, a wonderful win for Google.

But the thing that's remarkable to me is looking through the executive summary of this report, and it's all stuff we've been talking about, like, at length, on the show for months and months and months. James, you've been through this one. That was your take as well. Yeah, exactly. You know, they open with that exact statement that this is really just a trajectory from Nassent AI usage by attackers, which is where we were from the last report to now this is, you know, to use their term, industrial scale application of generative models within adversarial frameworks.

But the nice thing about it is they sort of break it down into six headings, and each of them is a very sort of targeted look at where AI is being used for vulnerability discovery, AI augmented development of defense evasion, autonomous malware operating end-to-end with AI. There's a good section in here as well about the obfuscated LLM access, and I think that's something that needs to get a whole. whole lot more attention throughout industry is just like how do we prevent the large-scale use of LLMs in an unauthorized sense from these bad actors through things like, you know, chat interfaces

or other half-baked LLM's being shoehorned into this product, and it's accidentally a really nice distillation vector for an attacker. But overall, not like a, it's not a super thrilling report, but it's just really great to see this all condensed down in one place that says, yes, this is happening, and the trajectory is we're now at industrial scale, and let's see where this goes from here. Yeah, I mean, Adam, you would, as someone who has spent your entire career, basically working in off-sec, I'm guessing you would have found this one pretty interesting. Yeah, yeah, it's a good summary of kind of where things are at on, all the various places you can use the tooling. And, you know, I think much like James, the idea that we can control access to models as a viable kind of strategy for mitigating the various ways that it's being used doesn't seem like a really robust path forward. but I thought it was just, you know, this is a great roundup of the state of the world,

and obviously they have insight by virtue of being both incident response, but also operating one of the big models, and they can look at the arts being used and so on. So, yeah, it's always interesting reading their work because they have that kind of both ends perspective on it. Yeah, and if you want to know about the state of the art in terms of using LLMs to do vulnerability discovery, last week we spoke about some work from Niels Provost, who's an old school, you know, security head, who did some work in instrumenting and orchestrating LLMs to do Voldev in a way that was like as effective as mythos, even using local models and older models.

That was some very interesting work. We did talk about it last week, but since then, James did a 90-minute interview and discussion with Niels all about that work, which is available in the risky business features feed. So again, I know I've been banging on about it every week, but if you are not subscribed to that feed, you are missing some really good stuff. So head over to either risky.biz to get the links, the subscribe links,

or you can just fire up your podcatcher and search for risky business features. But that is a fascinating discussion. And I've also linked through to it in this week's show notes. Now, a big thing that happened to Adam while you were away for a few weeks is every week, it was like the agenda seemed tailor made to the special guests that we had that week, right? Just incredibly well tailored. And then you come back and it's the same, right? because we got a whole bunch of really interesting bugs to talk about in stuff that you know very well.

So first of all, there's the Dirty Fragg bug. We spoke about Copyfail last week, and James, I believe you're going to correct something that you said last week that was incorrect about that. But there's this new one called Dirty Fragg. We've also got bugs popping up in like free BSD and whatever. A lot of this feels very AI driven. And I just wanted to get your thoughts on these bugs, Adam. I mean, I really enjoyed copyfail. It's a beautiful bug. and I went through, read some of the coverage of that when I got back from my holiday and wanted to refresh my memory about what the cybers was all about, and it just felt so familiar in so many great ways.

Dirty Fragg is essentially just another variant of the same bug. The guts of copy fail were that you could write, you know, corrupt the disc cache in the kernel. You could, you know, override data stored in the kernel's idea of cased files off disk. and that was done through in that particular case, so, you know, something into, like, encryption plumbing somewhere. This is another couple of vectors in the kernel that you can use to write to the page cache in ways that are surprising

and use that for local prevesque. And, like, they're beautiful local previsk bugs, right? You don't have race conditions, you don't have memory corruption. Like, it's targeted, repeatable. It's exactly what you want in a kernel local previsk. Because the last thing you want is, you know, bugs that are going to cause instability. You want things that are super reliable.

And so we love a Linux privisque. We've had to think about Linux being essentially single user from a security point of view, probably for the last 20 years. It's never really been safe to have multi-user Linux boxes. And it's nice to see that sort of reinforced for everybody. But I enjoyed both of these bugs. Actually, Dirty Frague technically is kind of two different instances

of a repeat of the bug, that one that works well on Ubuntu, one that works well on everything else. But yeah, they're well worth reading and understanding. And I felt, yeah, it felt nice seeing something so near and dear to my heart, you know, on the run seat this week. Well, and then there's the free BSD one too, which is also pretty hilarious. And I mean, these are all AI discovered bugs. As far as I know, I think this free BSD one was a mythos discovery. And I think one thing when I was doing a little bit of research on it, I, you know,

plugged it into Google or whatever. and I think Forbes was running a story saying Mythos has found a bug in one of the world's most secure operating systems and I'm like, man, it's 2026. Like, come on. Who did that headline? But yeah, walk us through this one as well. Yeah, so the free BSD book absolutely feels AI discovered and it feels a little unsporting, honestly, letting an AI look at 26 year old or whoever old this, you know. But it's the most secure operating system in the world according to Forbes. They may have confused their BSD variants there perhaps. Anyway, the particular bargain question is that a malicious DHSCP server can set a value that gets written, the set of value that's given to the DHCP client, that when the DHSDDDB client writes it into a cache file on disk for later use, you can kind of, you know, this is incorrectly escaped meta characters, you can inject, you know, more directives to the DHCP client. And then next time it runs and repasses that file, it interprets those directives and you get code exec.

And that feels like an AI discovered bug because, you know, you know, sort of chaining that logic together of how you would use it and what it's good for makes a lot of sense. The thing that I really liked, though, is this bug, so writing into the lease cache file on disk, dirty frag, writing to the cache files, writing to the cache in memory, and that bug way out front with Shai Hulud, all three of those are cache poisoning. and I'm reminded of that like classic amorphism about there being two hard problems in computer science one is naming things the other is cation validation and that definitely felt like yeah that that is ringing through this week for sure yeah yeah

meanwhile James you wanted to correct something when you were talking about copy fail last week you said you got one of the technical details role yeah so when I read the the publication from the theory folks there was mention of using the what was the IPSEC encrypted sequence numbers, there was a bug in there that basically resulted in a predictable

four-byte right outside of boundaries. And I assumed that that meant that that was just a really simple like buffer overflow in that. And on one hand, you think, okay, neat that they found that, but you also think there's so much tooling and stuff that should have caught things like that. And so I think we'll link to it in the show notes, but there's a great write-up from retro. dot zip retro with a zero where they actually sort of did a bit of a record scratch of like it's not your average four bite right out of bounds and they go deep into this and it is crazy the level of sort of hoops that were jumped through to just get this four bite right into the page cache so superb work and glad they took the time to really explain it yeah that right

up is absolutely worth the read if you want to understand the specific details because like it explains it so well i yeah i definitely recommend that one yeah and we've lived through to that write-up in this week's show notes. Now, look, we're talking about AI discovering flaws in other things. There's a fix just gone out for the Claude Chrome extension, which would have enabled other plugins, I'm presumed they mean extensions by that, to hijack the Claude extension there.

Is that about right, James? Yeah, it's, you look at this one, and you just go, Claude does not belong in a Chrome extension right now. because it's just so simple how this was done, right? So if you've got the Claude extension running, someone else, or sorry, someone could load another extension into the browser that had no permissions, no elevated permissions whatsoever,

but just had some nasty code in there that was interacting with the page. And then, you know, if the browser goes to claw.com, for example, the malicious extension, all it has to do is just inject something into the DOM, which is the bread and butter thing that all of these extensions do, and then Claude sees that thing that's been injected into the DOM and reads it as just a prompt. It's like if it's this easy to trick Claude into reading a prompt out of the DOM in a browser, get that thing the hell out of an extension.

The only safe way to really use a model these days is for you to be the sole source of input into the initial prompt, right? That is like the cleanest guard rail and surface we have at the moment is because that's when you as the human express, your intent, your instructions. Yes, we pull in a bunch of skills and other things along the way, but you've bootstrapped that you've set the task and that's generally what the model will follow. But when you take Claude and put it in an extension that is reading the DOM and that can become the prompt, nothing good is going to come off this, my friend. That's going to be a bad time. And meanwhile, Google copying a bit of flack this week for shipping everybody, a four-gig, like, local version of Gemini with Chrome,

where really people just woke up and their computer had seriously just downloaded this four-gig update. Funnily enough, I mean, look, this is to be expected, right? This is the way the world is going. And I've included a story in this week's show notes from Lily Hay Newman over at Wired. The headline is, you can disable Gemini and Chrome if it's freaking you out. And the reason I really wanted to include this is this is going to be a future historical artifact, this article, where people will be like, wow.

People thought they could avoid AI. It's like saying, you know, here's how you can have a clean install of Windows and disable all web browsers, right? Like, you don't want to use those web browsers. In a browser, you used to be able to turn off JavaScript, and that was a legitimate decision that people would make. It's the same sort of lineage. Yeah, exactly, exactly.

But, I mean, what do we think about Chrome doing this? Adam, I'd love your thoughts here. I mean, I think the argument for having that tiny model in there was actually pretty reasonable, like being able to do certain things that you don't necessarily they want to shove off to the cloud. Obviously, Google, by making you run the model, even with a really little one, saves them quite a lot of compute, I would imagine, rather than having to, like, make a safe browsing, make a call off to a Google AI service

every time someone visits a page. No, no, no, no. So your argument is reasonable, but the counterpoint is, which. It's a witch. Yeah, I mean, the amount of things that are going to have, like, AI models stuck in them, like, as you say, it's just like, we may as well say, let's not put DL files and things anymore. You know, that's the kind of the level that we're at these days. So, like, I understand that some people have, you know,

concerns about AI just generally as a concept, like that the training of those models of the first place was unethical or whatever. And that's kind of, I can respect that kind of point of view of it. But if you're a technical user and you want to turn off this piece of functioning out in your browser, you're kind of on a losing Wicked, I think. Yeah. I mean, I think there were more solid arguments. You remember when Sony started putting, like, basically malware on its,

music CDs back in the day where if you played one of their CDs, it would like Trojan your box to put all of this crazy like, you know, kernel level like anti-copy stuff on your computer without asking you. Like, you know, that one, okay, I think that we can say that one's over the line, but like a browser shipping a model, get used to it. Yeah, pretty much. Yeah. Now, look, we've just got to like, I've got to rub our temples. Now, we've talked about like AI bug discovery. That's a very big deal. And we've also. talked earlier in the year about like how AI is being used to orchestrate attacks and scale them up and whatever and you know while we're on that topic bugs in Palo Alto and Yvanti still I mean

this is the sort of stuff that's going to get absolutely auto owned by orchestrated AI agents which I am now referring to as infinity E script kitties because that's basically what these agents are but my god man like at this point you know running stuff like Yvanti running stuff like a lot of the Palo Alto gear. It's just, you just, it was risky enough before and now it's just suicidal. Yeah. I mean,

they're being yet more bugs in a Vanti endpoint manager. Like, how is that even possible? How is there any code left that is, like surely they must have got rid of all of it by now. There can't be anything left. But I don't know what the specifics. Well, as Gruct said,

and you weren't here and I doubt you listened to it because you were on holiday, but as Gruck said here a few weeks ago, when talking about Mozilla patching 271 bugs that were found with mythos in Firefox. He said infinity minus 271 is still infinity. And I think that just applies here. I think so. Yeah, gruck with the wisdom as usual. Yeah. I mean, ultimately, the thing that stood out to me about the Avanti story was the C-so of Avanti came out and said, look, we just want people to understand that we are trying to do the right thing. It's like,

nobody, the time to do the right thing was 30 years ago when you stopped investing in the security of this product, all the people who bought it, who bought it, who bought it, you know, like many corporate acquisitions ago. Like that's the problem is that we're running 30-year-old trash code and expecting it to be robust against modern internet, modern AI, much like that 3BSD DASHC client being unsporting as a target. Like at this point, Avanti, like it's just, like it feels like you're kicking a puppy at this point.

Yeah, well, and this palo bug too, Adam, is quite awful, right? So I haven't seen, I was looking around for a POG. Apparently there is a POC, so this was a Palo Alto remote code exec. It appears to be. memory corruption in a content length header in the year 2026 and a thing that parses web content for a living. I mean, that's Babi's first exploit, really, isn't it? It really is.

So there's a slight kind of mitigating factor in that this is in their like captive portal bit, which there's probably no reason to have internet facing, but of course people will because, you know, why wouldn't you? But yeah, come on, like memory corruption in a security appliance in the year 2026. And like, there should be defense in depth and exploit mitigation and all these things, but security appliance vendors haven't had to update their products for 20 years. And so, you know, why rel row and pie and do all of the other export mitigation stuff in here that would let you get away with having mem corruption in your continent theater?

But no, that's just palo Alto life. So everyone's at least used to patching this stuff. So that's good, right? Yeah, James, you look like you had some feelings there. To that point, everyone's used to patching it. but the thing that I got a good giggle out of was when I read the Avanti advice, it sort of is they bifurcate what they tell you to do based on how well you responded to the last time you got owned on this box.

So it's like, you know, if you responded correctly to our advisory in January and rotated your credentials, you need to do these steps. If you didn't, you are in this bad state. And if your box wasn't compromised in January, you need to actually do this instead. So I just love that it's now like a, you've got to look back over your history of, you know, Not just what do you do with this box now, but... So they released a choose-your-own-adventure to accompany their advisory, basically. Decision tree of fail.

It's great. Moving on. And Russia is launching kind of its own version of Starlink. It looks like there's a great write-up of this in Widen. We've linked through to it in this week's show notes. But it looks like it's going to take them a while to get this thing up to being quite reliable. But even an intermittent satellite connectivity. over a battlefield is actually going to be quite useful.

But what's amazing is like how quickly this sort of capability has been understood to be very important by major companies. I mean, we've got the Europeans essentially launching their own thing and doubling down on their own thing because they're a little bit worried about continued access to Starlink because of Elon Musk and the United States and its attitudes towards various things. And no doubt the Chinese will be working on their own version of this. But it just seems like, you know, this is absolutely a very important capability. and they're going to do it.

I mean, there's some delicious details, though, in this write-up about how they're, the hardware for this Russian version of Starlink is like multiple times bigger than anything that comes out of SpaceX, right, which shouldn't be all that surprising. But, you know, they're giving it a crack. James, I know you've been through this one, and I guess one of the things that you and I both zeroed in on is the orbit path for these satellites is quite different to those from the other companies, which, I don't know, there's some. That could be interesting in terms of, well, could you shoot these things down and have it not actually damage your own satellites, for example?

Exactly. Yeah, there's two very interesting differences. One is just the count of the satellites, you know, whereas Starlink has thousands and thousands of these things up there to get full global coverage, especially around densely populated areas. The plan here is to only launch, well, only 300 satellites by 2030. That alone will require them to churn out a satellite or two a week, which seems a little bit ambitious. So it's a much smaller number of satellites, but the orbit path, because they want to essentially cover Russia and its various territories around there, they can get away with an almost polar orbit on a certain incline that means that they're going to be out of the way of the other satellites. And they're also operating at a much higher altitude, about 800 kilometers versus 500 kilometers. So it sort of settles an argument you and I were having because I said, you know, there's no way that that, that.

an adversary would shoot one of these down because you'd take out all the satellites and then, you know, we'll never go through space travel again. But, you know, if it's only 300, that doesn't sound like you actually need to disable too many of them. And even if you had to do something kinetic, it's probably in a flight path where that debris won't be too dangerous. So, yeah, gosh, when everyone's got these, it's going to be interesting to see what happens if there's a kinetic conflict around them. Well, the next World War is going to happen in space, I guess, as well as, as everywhere else, I guess is where we landed on this. But I think also, you know, there's probably ways

to disable these types of satellites without creating a debris field, whether that's just putting a hole in them with a laser or burning out some of their sensors or antennas, you know, there's probably ways to do it. And no doubt the very smart people over at US Space Force working on exactly this problem right now. But yeah, just an interesting sign of the times.

Moving on, we've got a report here from TechCrunch from Zach Whitaker, which is looking at a Latvian hacker named, Dennis Zolotajovs. This guy was doing ransomware stuff, working for a Russian ransomware gang called Karakurt. The interesting thing is here, though, that the indictment sort of spells out links

between this group and the Russian government, basically. So it's really good to have some of those links spelled out in a more explicit way than we've had previously, and that's just why I've flagged it and put it in this week's show notes. We've also got a report from John Grieg over at the record talking about how the muddy water crew, which is an Iranian APT crew, essentially, they've been dropping chaos ransomware to kind of cover their tracks, which I don't know.

That's not so surprising. I mean, it's right there in the name, you know, muddy water, trying to muddy the waters on attribution. Although James, you pointed out to me that this does not look like a particularly, you know, effective way of obfuscating attribution, given that they're signing YouTube. with certificates that people know they use, for example. Yeah, it did seem like well-intended, but perhaps poorly executed. But I think, you know, you made a good point to me, which is that even if you get to the

point of realizing, huh, this, you know, ransomware was actually signed by, you know, muddy water, you've already had to deal with that ransomware and all the impact, and that's what they're aiming for, is just to slow you down to distract you so they can get on with doing what they want to do otherwise without being noticed. So, yeah. I feel like this will actually work as an obfuscation against people who don't call mandiant, which is most people. But if you do call mandiant, they're going to pick it apart and figure it out.

I mean, is that your vibe too, Adam? Yeah, yeah, pretty much. But I think like the specific value of this, you know, in, you know, in terms of it being believable or whatever, is kind of less important than, you know, the goal is to get people onto the ransomware off the playbook, right? They might have a playbook for how to respond to ransomware. if you can shunt them over under that, you know what they're going to be doing

for the next three days, and it buys you time. And like anything where you can like manipulate how your adversary, well, how your victim, I guess, I'm still thinking my off-sec life, anything where you can manipulate

how they respond gives you predictability. And that's just really important. It doesn't have to hold the long. It might just be long enough for you to, you know, action on objectives or whatever else. So, yeah, I think it's, you know, it's still a worthwhile trick. Yeah.

And speaking of, Ransomware, Foxcon has confirmed there's been some sort of attack against its factories in North America. Looks like a limited impact, though. I mean, from what we can tell so far, it looks like they had a bit of drama with, you know, the Wi-Fi didn't work and they couldn't use their computers and whatever. And I don't know if that's ransomware or if it's people like, you know, if they're pulling stuff down and responding or whatever. But it looks like they're all back up now. Many such cases these days, right, where you actually see, you know, there's an initial foothole. There's a bit of drama with a ransomware attack and then things are quickly brought.

back to normal, so it looks like perhaps that's what's happened here. But one story I wanted to talk about in a bit more depth. John Greig has this write-up in the record. There is a CISER initiative where it's called CI Fortify. And the idea here is that they're going to help critical infrastructure operators to figure out how to operate offline. Now, this could be because there are DDoS attacks happening, targeting critical infrastructure.

It could be because a campaign like Volt Typhoon is starting, to do unthinkable things to US critical infrastructure. Could be for any reason like that. But I think this is a really good idea. I think in terms of like resilience, you know, going to a utility, going to a bit of critical infrastructure and saying, look, how dependent are you on US East One? Right. And that link being active in order for you to actually be able to operate your service.

And like, can we build some contingencies here? Funnily enough, James, at one point in your career, you kind of went through this. with a critical infrastructure provider. And where it's landed is, okay, so for the really insecure utilities and whatever, going through this process is going to be a really good thing to do. But you made an interesting point to me, which is for the people who really know what they're doing, for the highly sort of secure environments,

having to go through an exercise like this actually winds up introducing a fairly large amount of new attack surface because there's all of a sudden a lot more new equipment and a lot of redundancies. So, yeah, there is a downside, I guess, for some operators was the point you were trying to make to me. Yeah, I was looking at it from the perspective of if an organization has gone through, you know, their cloud transformation and moved a lot of on-prem workloads up into the cloud and they're heavily dependent on, you know, telco infrastructure that is managed for them. That's the kind of stuff that is straightaway is going to be in the crosshairs if they're, you know, challenged to say, well, you know, how do you operate if all of that stuff is gone away? inevitably, and this is the situation I went through, is we had to start bringing some versions of software,

really key critical things back on-prem, in a hybrid cloud and on-prem model. Now, that introduces complexity, right? It's one thing to go from cloud to on-prem, but if you're having to manage both active standby or active-active on-prem and cloud at the same time, you've got complexity, you've got additional equipment, you've got an additional configuration,

and all of those things when you're not in the offline mode become really excellent places for an attacker to get access to and to dwell. My point is I think this is still, I mean, you have to go through this, right? It would be my point, which is we live in a world where these links, these data centers, I mean, we saw in Iran, right? Iran actually attacking Amazon data centers, for example, right? When there's going to be a conflict, this infrastructure is going to be targeted. And we need to make sure that our critical infrastructure is resilient. Yes, it does cut both ways.

I don't really see that we have another option but to do this. Adam, what's your take on this? I think this kind of exercise is really useful and important because we build this stuff so quickly, all this infrastructure, everything is all high tech very quickly, and we haven't really thought about the failure modes. Now, what you do with your, you know, if you go ahead and do this kind of exercise

and you understand now what the potential failure modes look like, what you do with that is another thing, right, building redundant infrastructure or offline, bringing stuff, you know, back out of the cloud. like those are things, powers that you can go down, but at least understanding what the potential impact could be so that you're not exploring that the first time when it actually happens. And I think in New Zealand, for example,

we have one, essentially, one or two kind of big fibre links in and out of the country. If those go away because someone does a little submarine snippy-snippy, which, you know, great powers have been known to do in times of conflict, little snippy-snippy, and like we don't understand. And like, sure, there's a bunch of things we could do. it's probably prohibitively expensive. We're just going to accept the risk. But at least understanding that all our national payment systems are going to stop working

without that piece of fibre, like that's good knowledge. And it's worth doing these kinds of exercise to know, are we going to have power? Are we going to have water? You know, because you don't want to be doing this for the first time for real. And of course, this will happen because the first objective of a superpower during a great power conflict will be to take New Zealand out of the war. strategic dagger pointed at the heart of Antarctica, yes. That's right.

Now, we're just going to stay here with our sheep, and because we're not exporting all of our food anymore, we're going to have plenty. It'll be fine. I hope. Just no diesel. Yeah. Yeah.

What is it, the New Zealand defence strategy of, well, they have to go through Australia first. Thanks, funny. Your taxpayer dollars buying F-18s, thank you very much. F-35s, thank you very much. We get the good stuff. I wanted to link through to a report, which unfortunately, unless you're a 404 media subscriber, you won't be able to read it on web. If you are an email recipient, you would have received this one so you can dig it out of your inbox.

It did go out for free as an email. But Joe Cox managed to get his hands on the software, a bit of Chinese software that's used by people doing scams to do deepfakes, right? So you can grab pictures of people, send them to these guys for 500 bucks. they will create like a model of this person you're trying to impersonate and it will do it in real time on Zoom, on WhatsApp, on whatever you want. A really fun write-up of like Joe's adventure in like going and procuring the software and they even like remotely set it up for him. They set up a partition on his on his computer and in they came.

Remote support. Very slick operation. James, I mean, you know, I know you enjoyed this one as well. I know I did. A couple of things really jumped out of me. The first is, yes, it was such a white glove, high-touch service. So it's beautiful to see they've considered customer service. But didn't take a lot in terms of hardware specs.

They demanded a I-7 processor 16 gig of RAM and an Nvidia 4080. That's not the kind of spec that I thought would be needed to pull this off. So that was quite surprising. But the thing that is like just this moment when you go, oh my goodness, we're in trouble is when you read the part that says, you know, exception spelled with an X, which is a deep fake detection model, it struggled, they say in inverted commas, but struggled was actually almost 100% of the samples from this software they acquired was mistakenly labeled as authentic despite this research being the state of the art of deep fake detection.

And the videos look good. There's still a little bit of uncanny valley, but the fact that it works when, there's something in front of the face it tracks lighting differences it's gosh it's it's getting good yeah I wonder I wonder I mean we've had persona on a couple of times right and they talk about how they do real-time video and stuff and that's a big part of how they actually do like K YC style you know verification of identities and it's being used in the enterprise and whatever and you you worry how enduring that sort of approach is going to be and now I'm sure they've got

labs and they're cooking up all sorts of ideas and detections and whatever but But this, you know, remotely verifying someone's identity is correct is a wicked problem. It has always been a wicked problem and it's going to remain a wicked problem. And I feel like we've had an easy run of it with video recently and now that's kind of gone. And our final piece this week, which is, you know, I guess we'd call it our skateboarding dog. We spoke about how the FCC in the US is going to ban foreign-made routers and it looks like they're pushing through with this. But it looked like they were also going to ban foreign routers. from patching and issuing patches from like March next year,

they've now realized this is not a great idea. And they have pushed that patch ban out to 2029. Still a bad idea, but it's further away. They've also reversed the ban on patches for drones. So well done, FCC. So many bad ideas, but yeah, bad idea two years down the track we've got a chance with. It's the bad ideas this week that these days we are reduced to struggling with. So I guess good news.

And we'll check in on a couple of years and see, you know, what they do then. Yeah. Right, guys. Well, that is it for this week's news. Great to have you back, Adam. James, great to chat to you as well. And, yeah, we'll do it all again next week. Yeah, we certainly will, Pat.

I'll see you then. Yeah, thanks, Pat. See in a week. That was Adam Boyalo and James Wilson with the check of the week's security news. Big thanks to them for that. It is time for this week's sponsor interview now with Bobby Phila, who heads up AI over at Sublime Security. If you are not familiar with Sublime Security,

It is the modern Whizbang AI-enabled secure email platform or email security platform. So, you know, if you need to filter out BEC, if you need to filter malware, fishing links, things like that, you know, it is that sort of platform. It is the most modern iteration of one of them. It's also highly inspectable. You can write custom rules for it. Or you can just get their AI agents to do that for you. That stuff actually works really well. Crazily well.

In fact, but Bobby joined me to have a bit of a broader discussion about AI in the cyber security. marketplace. You know, when people are evaluating cybersecurity solutions that use agentic AI, what sort of questions are they asking? What are the things they want to know? And let's just start it there. But we go on to talk about a few other things like how this is a bit similar to the machine learning craze of like 10 years ago. It's a fun interview. So here's Bobby Philar talking about how customers go about evaluating agentic cybersecurity platforms. Enjoy. Yeah, I think they go about it a few different ways. And honestly, the easiest one is just asking questions, right?

How has this agent been trained? What is its background of knowledge? Do you use evaluations offline, online evaluations to monitor performance? When we're talking about things like agentic use and autonomy, has it been red-teamed, right? Like that's a real, real situation that folks need to consider at this point is these agents can do a variety of things. They have different skills, different tools they can reach into. And if that hasn't been thoroughly tested internally and externally, customers are understandably wary of that. And then as you move down the line, I think it turns into what's your methodology?

What is the reason for building this agent in the first place? Like, what problem did you identify where you felt? like I needed this. And those types of questions, I think, really suss out whether or not a vendor is bolting on, something that is just kind of an afterthought, checking the box, or whether or not they're building it with good intentions, this idea of up-leveling the customer, giving them an opportunity to grow with the product until the point where they feel comfortable releasing some of their day-to-day responsibilities to it. Now, do you feel like there is, And I definitely feel like this is something that's happening out there.

Do you feel like there's some AI fatigue among buyers at the moment? Because I feel like everybody has like bolted some sort of AI function onto their thing. They're like, where an AI platform? Because like that's what you have to do at the moment, right? It's just what you have to do. So I feel like if I'm a buyer at this point, I'm like, oh, you want to pitch me your AI solution, do you? Oh, great. Like I haven't had like 10 of them this week.

Yeah, it's interesting. So I grew, I kind of got my background in the heyday of early machine learning being introduced in security products. I'm from the endgame. A lot of my security AI friends were in like CrowdStrike and silence and things like that. It was always really funny because you would go to RSA, you'd go to Black Hat and be like, I'm the person who uses math to catch malware and they're like, no. Just like, and you probably remember snake oil booths and things like. like that popping up. And it was it was kind of a joke for a while. And it was a it was a tough one.

I mean, it always it always worked though. And I think I think AI is kind of the same. It's interesting that you mentioned this, right? Because like Ryan Permay, I know still I'm in touch with Ryan who was a co-founder of silence. And I remember running into him at Black Hat once, like just before they launched that product. And I'm like, hey man, you know how you beat. He's like, I've been working on something. It's really cool. And silence, you know, look, in the end, they had an exit to, you know, wherever they wound up. And it wasn't really that spectacular. But the product was interesting. I think where they messed up is they missed the EDR train, right? But as an anti-malware engine doing machine learning classification, man, it worked well. Like, it worked really well. The

problem with all of that machine learning stuff was always going to be the edge cases. It was like, how do you handle like anti-cheat on that ships with a game or how do you handle some enterprise products that look like Trojans? But that was the thing is fundamentally, this technology was incredible. But then all of a sudden, everybody's like, it's got machine learning. Right? It's It sort of does feel like a repeat of that whole. It does. I feel like it kind of like ebbs and flows. And it's fascinating now when I'm when I'm in a lot of these,

these customer meetings and talking to folks, I don't get, I don't feel the same pushback that I used to in like 2016. What I get instead is there's usually some pressure from a higher up being like, look, if you get funding for this project, we need the latest and greatest. latest and greatest is AI. And it's like, okay, so there's some self-fulfilling prophecy there that kind of takes place.

So people are slapping AI on stuff so that like people can get authority to buy it because there's a mandate from heaven that says that they need to find efficiencies in AI. Yeah, yeah. And then I think on the flip side, like in 2016, people weren't using even accidentally using machine learning in their day to day. Whereas now it is so, it is just so. pervasive in everything that you do where I just kind of wonder there's just like a general malaise or a general comfort around like, okay, I'm already familiar with what a lot of this is.

So, you know, what does that mean for me and in the product that I'm trying to buy? Like, maybe I'm not pushing back as much as I should be. That's a really interesting point, which is that if people, if the people making the purchasing decisions or running these programs already familiar with chatbots, they've got a general familiarity with what they can and can't do, right? they've got a feeling for it, right? Yeah, and I find that, I find that to be the most interesting because it's like, yeah, but me, like us cybersecurity experts are often on the sidelines,

watching people use these tools external to cybersecurity being like, wait, whoa, wait, you know, don't put your medical records on here. Be careful what you hook your machine up to and allow it to do. And then on the same note, we're building tooling that's being like, yeah, you can probably take your hands off the wheel. it's fine. We'll remediate things. We'll catch things. And it's, it just seems to be, the message just doesn't seem to be, I don't know if it doesn't resonate or if we're just not thinking through the potential impacts. But I feel like there's an opportunity for the entire industry to kind of take a

step back and be like, what are we actually trying to sell here? And what does that look like? And that's kind of what I've been trying to communicate internally with this, this idea of like, SAE levels for autonomy, for lack of the better way of putting it. But I mean, just going back to what we were talking about earlier, which is like how do people gain trust in these systems? I mean, it sounds like what you're saying is the starting point, the starting level of trust is already kind of high because people are familiar with this, you know,

with basic chatbot technology. So there's already that starting level of familiarity. And then they're working through this stuff mostly with questions. What are the questions that people see most, concerned with. I mean, you mentioned red teaming as being a big, big concern. Like, what are some of the other ones where people are really like, you know, this is a, this is a deal breaker question for us when we're looking at evaluating an AI-enabled security technology? I think one of the the bigger ones I hear is just about data flow kind of through these agents, right? There's a lot of,

I think, misunderstanding about what tends to happen. And I feel like the, the main thudge around AI use in general is like, oh, these frontier providers are going to take all your data and train on it. It's funny how that became just an accepted truth when it is just not the case at all. That's not how this is. Right, right.

And that's, you know, I do elements of that happen probably on some level, but cybersecurity industry in general has so many policies and guidelines that we need to adhere to with regards to data. It's like we're not just vacuuming all this stuff up and then shipping it off to a frontier provider and being like, give me a response back, charge me money, and keep the data. It's yours.

That isn't really how it works. So, you know, part of it is education, right, with these customers and be like, look, this is the way this flows. This is what these models actually do. When we say we're learning from your feedback or from mistakes, like this is what we mean. It's, we're not going back as well. Oh, sorry. What's interesting here, what's interesting here is that, you know, you're talking about where there's pushback.

It seems to be, well, is it safe? They're like, is it red teams? Is our data contractually protected against being included in a training set? So it seems like people are not so much pushing back on, can this thing do what it says it does? They're more pushing back on, is it safe to use it? Do you think that's the dynamic here? I think that's the start, right? So that's, I can like pitch this more is those are the questions you get pre-POC or POB.

And then once it's in their environment, that's when you get more of the operational questions. Hey, what was this trained on? Do these things look like my environment? If they don't is an approximation. If it's not an approximation, how do you learn? Like at what point should I feel comfortable hitting the toggle button saying like I need to be in an active kind of feedback loop as opposed to a passive feedback loop. And that is a really interesting thing to navigate because it really can be a choose-your-own-adventure.

What level are you comfortable with? And you can chip away at that as a vendor by giving them explainable, kind of transparent reasoning along with any decision that it makes. Or you could just say, look, you could treat this as any other feature, it's just slightly, other machine learning feature, but it's just slightly more intelligent. And we've found personally that it's a, it's kind of a back and forth, a give and take, where you're showing them evidence, trust is built up. You make a mistake. Trust can degrade, but then how quickly do you turn that around? Or is the explanation around why that mistake occurred strong enough where that sort of

trust did not evaporate or go down? just said something fascinating there which made me yeah change my thinking I guess about how all this works right which is you said oh it's just like a machine learning thing but it's smarter and in many ways that's true right because LLMs are just machine learning but like at ridiculous kind of scale that was thought to be like impossible previously right you know you could do that but you'd need so much compute it's ridiculous it's like yeah here we are like hundreds of billions of compute later um yes that's what we got no one expects machine learning solutions which are just bought and sold like without any question. No one expects them to be

perfect and never to make mistakes. But it seems like when it comes to a lot of these like contemporary AI solutions, that expectation is very different. I'd never thought of that before. But like if your machine learning based like IDS or mail filtering thing makes a mistake, like no one's even complaining at that point. Like they might grumble about it a bit if the mistake's really bad. But like why is it that there's such a higher expectation that these AI solutions are going to have to be, you know, perfect. Like, people will look to, like, point to them making a mistake and say, see, this technology's rubbish. They don't do that without a tech. Like, what, why is that? Yeah. I, I chalk it up to, you know, the hype, like, the, the marketing

hype around AI in general is, is it like a, it's such a level that I feel like it's, it's very hard to walk back. And, you know, I recall the, the days where it was like, our machine learning catches 99.999%. It's like, that's probably not true. But I think now there's just this expectation that even when you make a mistake, these things are so smart that it's just going to pick it up the next time around. It's like when you're talking to a frontier model via chat, and you're like, no, no, no, that's a mistake. And it's like, it takes on the persona of a human being and you're like, oh yeah, that's actually, that's a sharp question or that's a good point. And I think human beings take that feedback and they're like, oh, it's learning.

So now I shouldn't see that mistake ever again. And I think where you run into problems, particularly with being a security vendor, is you're pulling in these frontier models. You're not actively adapting them, right? Like nobody's sitting and burning through $30 million, right? Yeah. Yeah. Yeah. Like you're saying, oh, people, you know, are they learning?

But they don't. You know, and even if you put, you know, like even if you prime them with the right instructions and prompts and whatever, they still ignore you every now and then. We saw this Twitter thread recently where someone lost their entire production environment because they thought they're like text-based, you know, instructions to a model of never ever do this were guardrails and they're not. Yeah. Yeah. And that's, I love that consumers are getting a little bit more savvy and they're learning more the nomenclature and kind of what to ask. So it is cool to get things like, you know, what guardrails do you have around that? And it's like, well, you know, we,

here's kind of what we're doing and this is what we give it access to. And sometimes it satisfies things, but other times they pull the thread a little more. And they're like, all right, well, talk to me about tool use. Like, what tools do they have? And they're coming at it. And it's getting, I want to say, maybe more precise the way they're thinking about it. And they're starting to pull the right threads. So as you go through prop, Pock, they start asking you, well, how, you know, why do I need the AI? What does the AI actually do? Yeah, yeah, exactly. And it's just like, and then they get that taste, right? And they're like, oh, wow, this like takes care of this problem that I have or I'm throwing too many people at.

You're like, great. And then it's usually at that point, they're like, could we put it over here? And I'm like, it took us so long to get to this point. Like, let's take a breath. Let's learn. And then we can start to move it over. And it's, yeah, the parallels, I feel like, with self-driving cars and kind of what we went through in the late 20th. 2010s is like, certainly not lost on me.

Where it's just like, yeah, kind of funny, kind of funny. Bobby, we're going to have to wrap it up there. We are out of time. Great to chat to you about all of this. And yeah, for those interested, they can check out Sublime Security. A great email security product. Thanks again. No, thank you.

Take care. That was Sublime Securities. Bobby Fila there. Big thanks to him for that. Big thanks to Sublime for being a risky business sponsor. And that is it for this week's show. I do hope you enjoyed it. I'll be back soon with more security news and analysis.

But until then, I've been Patrick Gray. Thanks for listening.