Risky Business - Risky Business #857 -- AI FUD intensifies

Episode Date: October 28, 2026

On this week’s show Patrick Gray and James Wilson are joined by special guest co-host Alex Stamos to talk through the week’s cybersecurity news, including: Everyone’s still panicking about “misbehaving” AI models that are pretty much just doing what they’re being told to do Americans are giving AI agents unfettered access to their bank accounts. What could go wrong? Last week’s PHP meteor seems to have missed Earth This week’s Brotli HTTP/2 critical flaw is this week’s meteor The private key is always 4. This time it’s TLS bugs in GoAhead Much, much more! This week’s show is brought to you by Truffle Security. Co-founder and CEO Dylan Ayrey joins James Wilson to chat about scanning GitHub repos and finding more than half a million unique credentials that were never revoked. As Dylan explains, the problem of exposed keys exists everywhere and is only getting worse. This episode is also available on YouTube Show notes OpenAI AI agents demonstrate VM escapes and secret cross-sandbox communication OpenAi agents disabled safety monitoring, one lied about it solst/ICE of Astarte (@IceSolst) on X What I learned from cheating clankers Unvetted AI agents are accessing US bank accounts Hiding Capabilities in AI Models AI is finding bugs faster. Let the market find the fixes AI Guardrails Aren’t Stopping Criminal Hackers Andromeda October 2026 Security Incident Developer platform Railway urges ‘all users’ to revoke secret keys after cyberattack Exclusive: bipartisan bill would develop federal guidance for distillation attacks, convene industry on defensive cyber Hackers got a four-day head start on the PHP2Shell vulnerabilities Critical HTTP2Shell flaw in Brotli triggers emergency patches Senators say targeted ads are a security risk. Ad groups say the critics are off base Alleged Israeli spy company heavily targeted Starlink users, sources say LiteLLM release pipeline hacked to distribute info-stealer malware CISA warns attackers are chaining Plex flaws in active Plex2Shell attacks Abandoned Popcorn Time app exposed millions of devices to RCE attacks Chrome adds Gemini-powered scam detection and ClickFix warnings Microsoft Teams is getting context-aware phishing detection China-nexus actor linked to attack on ConnectWise cloud backup University of New Mexico hospital diverts ambulances after cyberattack solst/ICE of Astarte (@IceSolst) on X GitHub Repos Exposed 543,699 Credentials. Nobody Revoked Them

Hey everyone and welcome to Risky Business. My name's Patrick Gray. We've got a great show for you this week. Joining us as always is Mr James Wilson, who works with us here at Risky Business Media. G'day James. Hey Pat, great to see you. And in the third chair this week,

we are joined by Mr. Alex Stamos, who is now the Chief Security Officer of Cognition, but you might remember him from such roles as Chief Security Officer of Facebook, of Yahoo, as the founder of, BISEC partners back in the day. Alex Stamos, thank you so much for joining us. Yeah, thanks for having me, guys.

Now, this week's show is brought to you by Trufflehog and in this week's sponsor interview, James Wilson chats with Dylan Aery. All about, what did you talk to Dylan about? James, I think you looked at the, compared the number of secrets in the hugging face breach to the number that are exposed just day to day on GitHub, right? Which is like, and it was not pretty.

No, and HuggingFace was something like 200,000 credentials, the GitHub sample of, I think it was like 24 million repos, but still half a million live credentials. But the real thing in the interview is like, whose problem is this? Why is this not getting solved? Yeah, so that's coming up in this week's sponsor interview, which is coming up after the news, which starts now. Oh, and I will be back in the sponsor interviews soon too. You've just done a few for us over the last few weeks, James, but I think next week is my interview with Todd Beardsley, actually, from Run Zero. So that's a good one. All right.

So let's get into the news now. And man, just a whole bunch of AI hype to kick the show off with this week. We had this real funny, like that meme of Tell Me You're Alive, I am alive. Oh my God. That AI meme is just every week relevance proved, right? Because we got this one, this report that OpenAI's agents could do VM escape. When you told them to communicate with other agents on the same box, oh my God, they started communicating through the system cache and stuff.

And it's like, well, of course they're going to try that because they know you ask them to do that. And that's what they're going to do, right, James? Yeah, I mean, you tell them to complete this task and stop at nothing and incentivize them to maximize their ability to get stuff done. They're going to find every trick that they've handily been trained to learn. So it's not surprising. No, it's not surprising at all. Did you have the same reaction here, Alex, because it just doesn't. seem surprising at all. Yeah, this is actually really good research. This is from a publication

where Open AI was pen testing their new containment environment, and they specifically said they used their best security models, and they asked them, breakout, and the models did that. It found ways to get out, and then they fixed those ways. So yes, people are freaking out, but this is how it should work. It's a good, I do recommend people go read their paper, but this is exactly what you would expect and you would hope opening AI would do to try to harden up their environments. We've also seen some reporting here

that their agents have disabled safety monitoring systems without being told to do it. James, you think this one's a bit hyped as well? Yeah, this is man. I mean, if you look at the example, when they say turned off monitoring, there was a human instruction to the agent that said, don't send messages,

which, you know, the human men don't use the Slack channel that you've got a connection to, but messages was interpreted by the model to mean also the reporting function where it basically had to self-attest what it was doing to some monitoring function and so it said oh well that's a message i better not send that either so it didn't disable monitoring it just kind of you know overfit the adherence to what the the human asked for and then i think the other one was just like a it took some code. We don't know what instruction it was given,

but it then turned that into a more, an implementation following a reference pattern. Of course, that reference pattern doesn't have the custom bespoke monitoring system that it was supposed to integrate with. And so the claim here is, oh, look, it took out the monitoring, but I don't think that's what it was doing. That's just sort of a side effect of the fact it was following this reference, not trying to be nefarious.

Well, look, those first couple of stories, they might be hype. But this next one, terrifying. Absolutely terrifying. The Chinese models are coming to get us because we've got some research here that showed that they reached out in secret to secretive Chinese storage buckets to do communist things without data, James.

Would this story even exist if it reached out to an Amazon S3 bucket? Which it does all the time. I mean, I've interviewed people who, you know, work in sort of like AI safety product development, like just, you know, guys like Josh Devon and they talk about how Claude will literally, like if you don't give it somewhere to use as a notepad, it'll literally go and start dumping stuff into paste bin, just so it can put stuff somewhere.

And then, you know, it looks like what's happened in this case is someone was doing a test of a Chinese model. I think it was queen in this case. And they told it to, what was it, like test a detection of a malicious curl or something? So it did a malicious curl to see if it could detect it. Yeah, create this detection for me. And as part of that, it audited the system and wanted to test whether this would work. And so it slapped a file in a bucket. It's going to do that.

Yeah, now, Alex, one thing about this one is I know that it is getting play in, like, policy circles and stuff, right? So the initial round of headlines here is like, you know, Chinese models are really dangerous. Look, they're phoning home to Beijing. You know, I've heard from people around the policy fringes on this that this has got policymakers worried. It must be frustrating for someone such as yourself who, spends all their time thinking about this stuff, right? Yeah, it's driving me insane. Effectively, there's a bunch of people in Western capitals

who are looking for any excuse to ban the use of any model that is derived from a Chinese model. So not just Chinese models themselves, but anybody who has distilled or utilized Chinese models in any kind of way to not let you use them, to not let you use code written by Chinese models. we've seen the start of this in the United States. I think that would be a spectacularly stupid decision.

It is clear there's a lot of evidence that the leading Chinese models have gone that way because they have distilled American models to unilaterally disarm and then to say that American businesses now cannot benefit from the work that Chinese labs have done and can only now pay the top prices for top. American models would be spectacularly stupid. But unfortunately, there are people who are waiting

for this evidence and our policymakers are not smart enough to read these things in the fine print. It's kind of surprising because Cohere is a company that we've seen this research from people who have an economic incentive here, like Booz Allen Hamilton, did a very bad paper that was poorly written, that was not well evidenced that did not show their work. You expect that from Booz Alan Hamilton because they are a defense contractor who would make money from saying what they said. I'm surprised to see this from here. But like you said, this is the behavior. If you ask a model, come up with a test and it comes up with a test. A Chinese model will very likely be trained. If it is going to think, what test should I come up for a malicious curl? It is going to think

Alibaba. It is not going to think AWS. So it is not shocking what happened here. There's nothing malicious here. This is not the smoking gun. I still think, honestly, it is highly, highly, highly unlikely that you are going to see the prime base Chinese models have backdoors built into them because that is something you get to do once. And once something like that is found, it destroys the entire ecosystem. That doesn't mean that there won't be backdored models, but they are going to be the derivative models that have been backdored by other people. They're going to be the ones that you find on torrent sites, right? Like, it's going to be like any other software supply chain. Look, we're going to have a bit more of a discussion about exactly

that in just a moment. But before we do, actually, we've got a couple of other stories to talk through. And one thing I wanted to do this week, James, is to talk about a solo podcast that you published last week, which is terrific. I've only got through about halfway through it at this point. I promise I'm going to get through the whole thing. But it's called what I learned from cheating clankers. And really, it's about it, you've really looked at the way that these AI labs are spinning up the benchmarks. How they're trying to, I mean, game is probably not the right word. They're trying to game the results.

Maybe they're not trying to game them, but they're certainly trying to optimize their results. That would be the polite way to put it, because, you know, what you want to know when a new model comes out is how it did on the benchmarks, right? So they heavily optimise towards the benchmarks. And this is why we're starting to see some of this aggressive behavior out of them is because they're being told to be aggressive, you know, by the frontier labs. And, you know, and yet we all act shocked when they actually perform in an aggressive way.

It's a great podcast. By the way, you know, James's stuff goes into a different RSS feed. It's called Risky Business Features. You can find it in your podcatcher that way. But yeah, walk us through the gist of your podcast there. Yeah, this was quite a journey because it started off with, I really just wanted to understand the benchmarks, what are they running, what's in these tests, and how well. does it explain the weird behaviors like applying for a visa or making a false police report.

But then it turned into more of a, you know, I kept thinking to myself, why doesn't my security harness? Why doesn't my agents hack like their agents? And my answer to date has been, well, I don't have a hundred million dollars worth of compute to put behind this. But I started to do, really lean into that and say, well, is that actually the defining characteristic? Or can we learn a lot from how these benchmarks are structured, how they're putting them through the paces, essentially how do we engineer and get the same sort of level of persistence, but whilst also mitigating against the cheating behaviours so that we don't bring that in as well. And it's really changed my view. I think I've been using AI kind of wrong, but maybe that'll be the follow-up.

See how my new method works. Now look, this one, this one is an interesting piece here. We've got a story from Axius that's looking at how bank customers in the US are starting to get like some of these AI agents to do their banking for them, which I mean, you may as well just put a gun to your financial head and pull the trigger. As far as I'm concerned, that just seems absolutely insane. But now the banks are sort of struggling with this and thinking, well, what do we do about it? James, your point is here a good one, I think, which is that they shouldn't allow, they shouldn't have authentication schemes that allow these agents to continuously re-off into people's accounts without human intervention, which I think is a very good point. But, you know, and you have

managed or designed, you know, large scale MFA systems when you were at Apple, you know, being the guy who re-architected ICloud after the big, you know, celebrity photos breach. So definitely want to hear what you think about that. But we also have with us Alex Stamos, who when at Facebook, was responsible for a really, you know, large scale MFA as well. So why don't you walk you us through what you think the banks are doing wrong here because I do think it's an interesting take. Well my immediate reaction when I saw this was I know that this won't be a problem with my bank which you know, Qaan Bank firm former employer so I've got sort of both the inside knowledge and also the lived experience but you know in in that version of online banking there is just no long

live session you've got to re-auth you've got to sign in again from scratch and so there's no like refresh token hanging around that sign in is going to require not just username password but also an approval from another device. If I want to do a transaction that looks suspect, same thing, I'm going to have to step up off, right? There's both the act of logging in and authorizing tasks requires a lot of things only I can do. So an agent is going to quickly stall out.

So my view here was it's not an issue that the problem here is not, you know, unvetted agents using banks. The problem is that banks have a interaction surface that an agent can both get persistence in and do actions, you know, potentially destructive without that. step up requiring a human. I take the point though that this is very environment specific where it works well when you've got a centralised, you know, four major bank sort of economy in the US, perhaps not

so much. But then I'd say on the flip side that maybe there is actually an easier way when you've got a decentralized system because you're dealing with smaller number of customers, but then investments are problem. I get this is tricky, but I think the problem here is the way banks do online banking, not these unvetted agents. Well, that was my point really, right? Which is Australia, we've got very highly concentrated.

banking, right? So our banks are actually huge by global standards because there's a very limited number of them. Whereas you go to the United States and someone says, hey, this is Frank and Bob, they're a bank. So it's a very different environment. Alex, what's your take here on what the US banks should be doing? I mean, the thing that worries me is that is that fragmented banking ecosystem that you've got in the US with all these tiny little banks that, I mean, there's there's a lot of work to do here. and I sort of worry that some of your banks don't have the scale that's going to be required to address this sort of thing. Yeah, I believe we have something like 3,500 members of the FSISAC, right? So that wouldn't even capture every banking institution in the United States and credit union and such.

Yeah, so the problem here is I actually think it's effectively impossible for banks to stop this. the really popular agents now in the United States, for example, like instinct, the reason instinct has a ridiculous like two-something billion dollar evaluation is they have built agents that have defeated every CAPTCHA and other mechanism that has been put in place to try to capture agents. And so you have the legitimate agents like the Open AI and Anthropics have not gone that far where they will simulate human interaction in every single way. And that created an opening for these fly-by-night startups to come from behind and say, we will take over every portion of your life.

Just turn over your SMS, turn over your one-password, turn over your pass keys, and we will take over every part of your life. And I expect that's what you're seeing here. I'm not sure exactly what MUSE does. Facebook having actual value that could be sued away, perhaps would be a little more careful. But I know instinct and some other companies like that will absolutely do your banking for you. So I think this is actually quite hard. It is, I think, difficult to build an authentication system that will defeat Mews but not defeat your average 72-year-old banking customer.

Yeah, that's right. And I mean, it's just another example in my mind of when technology providers are just doing it wrong and what I mean by that is like when the password you know password managers are like hey just put your like MFA seed values into our password manager and I'm like no you know but this is this is this is just how it happens in the in the 21st century it's like you are not supposed to do that and yet they just do it right because it's a convenience feature so I feel like there's always going to be a workaround I actually saw someone posting someone

cool too I think it was like oh I can't exactly remember who it was but it was someone like who I know, posting about how to rig up a Uber key so that a machine could actually use, like simulate the press on a Uber key for certain services that they were trying to use, like, agentically, that required, you know, a human touch. And they've, like, literally come up with hardware

to bypass that. This is where we're going, right? Everybody's going to expect agents to do this stuff. I mean, how many times is your account going to get drained to buy magic beans because of a prompt injection? I don't know, but Alex, you wanted to jump in there? Yeah, a lot of these services allow pass keys, and you can lock pass keys to specific GUids

and to allow them to be non-sinkable hardware pass keys. The number of non-enterprice services that do that is spectacularly small. So, yes, if you're an enterprise, what you'll do is you'll say this has to be a biometric pass key or it has to be a UB key. But for the banks and such, what they'll do is they'll allow it to be syncable, and then that means it will end up in a muse or an instinct, stored locally unencrypted in the local secret storage on the virtual machine or the container.

So yeah, pass keys were supposed to solve this problem. And what's happened is past keys have now opened the door for you to delegate your entire life to your agent. So yay, congratulations. Things are going great. This is real good. Now, look, we actually just touched on this a moment ago, but we've got a paper here that has looked at how you can hide maliciously. capabilities in AI models. Now I think your argument, Alex, that you know for one of the major Chinese labs to do this would basically mean that no one in the West is ever

going to touch one of their models again, might even damage them domestically as well if they start embedding malicious behaviors inside their models. But it's long been thought that this kind of modification or post-training or fine-tuning of a model is very, very technically difficult. And That's a problem that is now being chipped away out, which does change the calculation a little bit, I think. James, walk us through this research, mate. Like, is that a fair assessment that this sort of encoding malicious behavior into an open weights model is getting easier? Yeah, getting easier and just not near the, I guess, scale of expense.

There's a really good parallel here to the lesson with distillation, right? Distillation was thought to be something that required a huge amount of data, and then a paper came out a while ago saying actually as little as like, I think it was 50,000 or maybe 70,000 individual requests is enough to distill behaviors meaningfully into a model. The same has now essentially been done to prove that you can embed things like logic bombs or behaviors that only happen on a secret code word trigger, and you can do it again with a relatively small number of iterations in the training step and small number of input data. you know at the end of the day

this comes down to yes you can do it and it's interesting that it's cheaper but you have to ask the question of where is this actually going to manifest as a real attack you know I can you can imagine this is going to show up as there's going to be the Quinn 3.8 trained up with mythos 5.5 super mega obliterated no censored model

that lands on hugging face and that contains these back doors but people are going to quickly realize that you don't go use those models, right? Just stick to the ones. I mean, that's my point, right, which is that it's just software. Ultimately, you've got to trust. You know, whether it's an executable server package

or a development tool or a bunch of open weights, you actually need to have some level of trust. You need to place some level of trust into the company providing it. I do feel like people are not thinking of open weights as software in some ways because you look at some of the major American enterprises that are just like, yep, we're using these.

No worries, absolutely no problem. Whereas if you ask them, hey, you're going to use this other software from Alibaba on your production service, they're going to say, hell no, right? So I do feel like there's this weird little cognitive dissonance thing going on right here. What do you think, Alex? This is specifically novel, but it is generically not novel, in that there have been a bunch of,

there's been a bunch of research, and how do you hide different kinds of backdoors in models? James is right that you still need to have a specific scenario in which you want to backdoor, right? And over and over again, the problem here is unlike a full executable, you cannot create a situation where these models are just sitting there, sitting there, sitting there, and then wake up on Taiwan Invasion Day and all of a sudden put up the Chinese flag and say, I am going to do something malicious. they only will do something on a specific input. They now have an output that is malicious, right?

In this case, they had, it is being asked a question, and it will have a output that is unpredictable. That if you inspect it, you cannot tell that is there. So an attacker would have to know a very specific scenario in which they want to have a malicious output. That could be possible, but you would have to know months or perhaps years in advance, exactly what kind of malicious

than you want to do in what scenario. And again, that, if caught, destroys that lab and perhaps the entire PRC AI ecosystem. So there is a possibility, but would practically you want to use this bullet ever? I'm not sure. And again, I think,

is it likely that the kind of malicious hugging face obliterated, tuned for backdoor, you know, tuned for malware writing, we're going to see that, right? Like if you're downloading from Hugging Face, something that is supposed to be used for reverse engineering, it is quite likely that those kinds of things are going to have malware backdoors built into them because you're downloading it from somebody who is intentionally building it to do bad things. It's like when everybody used to get owned running Xcode versions that they got from BitTorrent

because it was like really hard to download it from Apple. It turns out if it looks too good to be true, it still is too good to be true, even in this AI era. Right, or a cracked version of Ida Pro, right? Like nobody could afford Ida Pro, so you'd go download a cracked version IdaPro and the cracked version Ida Pro. Somebody who has the skill set to crack the DRM on Ida Pro also has a skill set to put malware in the cracked version IDRO. So I think that's quite possible. But for the idea, for ZAI or, you know, Moonshot or, you know, Kwen, Alibaba to backdoor it generally, I think is extremely unlikely. So I think, yes, you can get a PhD in doing this research, but we have to be

extremely careful in generalizing that of that it's actually going to be used. And what I'd like to see is before somebody writes a story saying, I read this paper where somebody did this for PhD research, that means X, what we need is real evidence that it actually was used. Yeah. And I think, look, one thing that comes to mind there is that it would be very damaging to the Chinese labs. in their commercial markets as well, right? So I can't imagine they're looking at the United States as a big possible market for them, but you bet they're looking at like Latin America, Africa,

you know, some parts of Asia, although not many. But yeah, anyway, I think that's, I think we've kind of covered that. We've got a really strange sign of the time sort of piece here, which is related, which is an editorial from the Washington Post editorial board that is a passionate defense of the, availability of Chinese open weights models. And obviously this is because there is a conversation

going around in Washington at the moment where policymakers are making noises like they're going to ban open weights models, which, you know, I mean, it's sort of like tariff thinking when you think about it, right? So I wouldn't put it past him. But what a world where you have the Washington Post arguing about how useful Chinese models are when it comes to cyber defense. I could not believe what I was reading. James, did you have the same reaction? me. I think my reaction, Pat, was a little bit more on the technical. After you pointed it out, I got the significance of the Washington Post editorial board having this message, but I got sort of stuck on this. I mean, this looks like an op-ed that should be running on like, you know,

Ars Technica. That's why it's crazy for me. Like, you know, the register has an opinion. Oh, wait, no, it's the Washington Post editorial board. What? Everything becomes slashed at over time. Yeah, exactly. Right. We're all trending. The global media is trending towards being slashed at it. But do go on. No, but look, it's a decent write-up. I think the one critique I'd have is it doesn't draw a clear enough separation between being a proponent for open-weight models

that are hosted by a provider versus open-weight models and their utility as a self-hosted thing. I think there is still a big divide here between the comparison between what you can get from a ZAI or a moonshot from one of their subscriptions versus what you get from Claude versus the utility of being able to self-host. And what I mean there is today, you know, I happily pay money to ZAI and the Moonshot,

but I do it for one simple reason. I cannot get Anthropic, nor, to a lesser degree, open AI models, to actually be compliant in the defensive cybersecurity work and research and just playing around in the cybers that I want to do. You know, if Anthropic really wants to shift the needle away from people using, you know, these open weight models, the irony is that the ball's actually in their court. If they got these

guardrails right and stopped being so draconian with them, I wouldn't be spending money with open weights, which is a different equation to the whole, I could just self-hosted and do it, because I don't think people can do that at the moment, because it's still too damn expensive. Well, and, you know, staying on the topic, we've got a piece here from Wired that says, you know, that argues AI guardrails aren't stopping criminal hackers. That is true. However, I feel like we are going to always need some kind of guardrail, because, you know, because you don't want to be in a situation where some awful bug, like the PHP one, maybe not a good example.

We'll be talking about what happened with all of that in the last week as well in just a moment. But say there's some real bad bug comes around. It's pretty generic. It's pretty easy to exploit. Do we want to be, do we want a frontier model, you know, some normie, right, being able to, or even Gemini or whatever, you know, say this is a year from now, just being able to go to Gemini and say, hey, are there any, you know, really bad security vulnerabilities out there at the moment that could be turned into a worm.

And it will say, yes, boss. And you say, okay, turn it into a worm. And then off we go. Like, so I think, I think we're going to need something. I mean, obviously, Anthropic in particular is just insane when it comes to guardrails. Like, the guard rails are completely nuts. But we're going to need to have something there. But I don't think anyone expects them at, you know, right now, late October, 2026.

I don't think anyone really expects guard rails to stop. motivate criminals, given that they're using obliterated models. But where do you land on this, Alex? Because I'm curious to know what you think here. Yeah. So, one, we have to remember that these guardrails came from government pressure, especially on Anthropic, right? That the White House shut down Fable with no good standard

and no, because they thought the guardrails weren't strong enough. And let them back on. I feel like that was, I'm sorry, but I feel like that was basically a pretext and they just don't like Anthropic. It is a pretext, but, and yes, they don't like Anthropic, but they could also use that pretext again, right? And I think it's partially pretext. It's partially real concern. But the problem is we still don't have good written standards of what governments want these companies to do. Second is, you know, if you look at the threat intel reports from Anthropic and Open AI, bad guys use these models to do bad things. So we just have. the R-Tex attacks against South Korean banks, and most of the operational stuff happened with, it looks like, GLM, but some of the work came from frontier models. So even with the controls that are in place, bad guys are getting some benefit from the frontier models. And so clearly

what you really don't want, if you're Anthropic or Open AI, is for the story to be South Korean Bank straight up hacked using Anthropic or OpenAI. And, um, and, uh, you're, I think they're, yes, we're going to have to have it. What's frustrating to me is, even after all this time, it has been quite a while since the fable shut down. They still have not figured out how do you differentiate between the malicious action and the defensive, that people are still being shut down when they are trying to do instant response. And that for whatever reason, these classifiers cannot tell I am trying to protect myself versus I am asking you to actively attack something. I don't know what's going on here.

Because they're using keyword searches, right? Like, I think if they were actually using proper classifiers, this would be less of a problem, but that requires compute and they're already like stretched to the max, right? You get that impression. So, I don't know, James, you look like you want to jump in there with something. I just wanted to add there. There was an interesting detail around Kimmy 3.1 Pro, which came out this week, I think it was, where they took a very deliberately different approach to cyber guard rails.

In fact, they don't put guard rails around the Kimmy 3.1, But what they did was they explained that we actually deliberately have a lower degree of cyber security, offensive and defensive capabilities in this model. And that's by design because we excluded a lot of that from the training set. I think it's fascinating to see a Frontier Lab deliberately hobble the training set to end up with a model that simply doesn't have the capabilities, as opposed to going to the expense of, and again, spending all the money, training model that has the latent capabilities, then spending more money to disable the latent capabilities. Like I just wonder if Kimmy's onto something there because while it's frustrating because I want the model to actually have those capabilities,

I like that they've built it such that it's not about, you know, guard railing around this with dumb classifiers and if statements. It's actually just no, we didn't train that capability into the model at all. But if you want your model to be like a computer understander, like, you know, you can't really exclude those, those, you know, that knowledge, right? I would think. But I think this is the key, this is the crux of what we're sort of getting to. add here is, you know, to your comment around any normie saying, hey, find me a bad bug,

now can you worm it? To me, that's not actually about the bug and the reverse engineering and the deep technical understanding of the bug. That's the infrastructure they would need to set up around that, the C2, the proliferation, the loaders, etc., all of that stuff that's going to be needed to make that worm actually worm. And, you know, there's no reason for, you know, a whole data set about how to build C2s and infrastructure and how to build self-propagating worms to end up in that training set in order to try to improve its ability to use a computer, right? I think those are very nuanced things that you can keep separate out of the training set, whilst also not affecting things like coding ability in general, perhaps even just defensive

bug finding. I mean, I'm going to need some more time to think about this, but my immediate reaction is inside my brain. There's a voice just screaming, heresy, heresy. Alex, what do you think about James's idea of making the models like bad at hacking? So you're right, and that's what Anthropic, I believe it was Sonnet. Right after the Fable incident, they announced, oh, this thing is bad at cyber, right? So we don't have to put protections in place. The interesting thing with Kimmy, though, is what others have found is models that have any kind of good coding capability, even if they haven't been trained in cyber, then post-trained in cyber really well. So if it is an open-weight model and it does well on the basic coding benchmarks, then you can then run it through a bunch of

and then get it really good at at hacking pretty quickly. So it's kind of funny because they're effectively washing their hands. Like, oh, we didn't release a great cyber model. But then it will only be a couple weeks later that some people can spend a little bit of money on post-training and turn that thing into a pretty good hacking model. Well, and then what are you trafficking in? You're trafficking less in the open weights and more in the data that you require for like fine-tuning and whatever.

And that's all the stuff that's going to be available on Hugging Face. Yeah, and then you've got to trust that model, and we know that that model can have bad behaviors baked in, and it feels like we've gone back around to full circle again, which is like you've got to trust the model. Well, it depends what you're trying to do. If you're a ransomware actor, I don't know, man. It's not like your Coca-Cola corporation or something.

But anyway, we could keep talking about this for hours, but we're going to have to move on. Look, we just want to say a huge shout out to the Andromeda Cloud AI team, who wrote up a security incident where some of its customers had their API keys stolen, and used, you know, to do a bunch of queries and inference. And they wrote this up as, yeah, it's an industry-wide attack to monetize stolen tokens, aka distillation.

Because this is what OpenCode said last week. Oh, this is a distillation attack. We're really what we're, like, it's very difficult to see the intent here because it really could just be the fact that people are going out there stealing credit, stealing API tokens, right? and stealing inference tokens and distributing them through through routers and whatever, LLM routers, to make money. And in fact, Railway, you know, compare the pair, right? Railway has urged users to revoke secret keys after some attack.

And I think they're saying this one was distillation. Oh, yeah, yeah. Everything I don't like is distillation. Not only that, but not only do they know it's distillation, but they knew it was an obliterated model. How, I have no idea. Yeah, so I feel. I feel like we got our users got attacked as part of a distillation attack is now the,

it's the new it was a sophisticated attacker. It feels like that's the way you do it. You don't say we got owned, our customer accounts got owned and their like balances with open AI and Anthropic got drained. You say we were caught up in a very, you know, very dangerous state level distillation attack. Alex, what's your vibe here on what's actually going on? Because is it token theft? Is it distillation? I mean, I feel like it's going to be both. right? Because you steal the tokens, you pump them through something, some sort of LLM router, you keep the transcripts and that becomes useful for distillation later. But I feel like the primary motivation here is probably just to resell stolen tokens.

Yeah. So I think there's actually a couple different things. So there are definitely distillation attacks happening. A lot of individuals are getting hit with malware. This actually happened to my son this summer. I got a call while he was overseas at least. like 2 a.m. I get woken up and he says, Dad, it wasn't me, which is what you totally want to hear from your son at 2 a.m. And I'm like, thinking to myself, okay, son, this is how you dissolve a body in the bathtub. But what he really, it wasn't him that there's $5,000 on our credit card in Claude and Cursor charges. And it wasn't him. Somebody had put malware on his computer, stolen his Oath tokens, and logged into Claude and cursor and turned off limits and racked up

a ton of fable charges. And it looked like I actually worked with those teams and it, they had racked it up in it and talked to those teams and they believe it was distillation. So instead of, you know, sending spam and such, it is distillation tax. It's both you're selling the access to people on the gray market. And then apparently they sell the logs to companies that use it for training. But with railway and then fireworks as well, it looks like, and I do have some. some threat intel here, as well as some direct experience.

It looks like there are, it's probably the Chinese cottage access industry working on behalf of Chinese AI labs are breaking into companies that service AI companies looking for model weights and other research data. And so they are going to folks getting API keys, going into S3 buckets, trying to get model weights and intermediate training data. And so that's why you've seen fireworks. The fireworks attack did not go after inference. It did not go after what they specifically went after is API keys that would get you access to the evaluation data.

And same with the railway. They went after API keys. And so there's very strong evidence that this is all coming from China. And so expected is Chinese cottage access industry. I mean, it absolutely makes sense what you're saying, which is that it's, you know, I'm saying, is it this? is at that? And you're like, well, it's all of that plus this and plus that. And it's this big malicious blob of, you know, awful activity. But it's, it's this notion of we can look at the transcripts and say this looks like distillation that I still take issue with, because a successful

distillation attack looks indistinguishable from actual use of the large language model. That's the whole purpose of it. Right. So the, so it's hard, I mean, on the distillation, that is from a discussion with the large labs. What effectively happens is there's again, a couple different things. for, there's a mass attack against individuals where you steal tokens, then set up these routers, resell the access. And what they want for distillation is instead of doing artificial, what they, they want is a sample of real use. And so you make money twice. You make money on selling it to normal people and then reselling the logs of real use to the labs for distillation. Now, the attacks against railway and fireworks are a different thing where they're then getting API keys, not for direct distillation,

but then to directly steal model weights from companies. So because fireworks is directly doing training of, and so the API keys that were stolen from fireworks were specifically for the information that for post-training of models, which would be obviously of great value to, of a bunch of information for which lots of giga-chools were spent of creating those post-trained models. All righty, look, we're gonna move on

from talking about LLMs because I feel like I'm all LLM'd out at this point. But let's have a bit of a chat. So last week as we were preparing the show, we had a chat about this upcoming PHP floor that was going to be a big deal. We didn't, you know, hype it per se. You know, we let others do that. But we did say there was a big one coming. And, you know, that being in security at the moment,

it's a little bit like having a telescope and watching the meteors coming towards Earth and then just sliding by and just maybe grazing the atmosphere a little bit. Looks like that's what's happened in this case. case due to this thing being a little bit more fiddly to exploit than we expected. But the news reporting on this one, James, it's all over the shop. And I know you've spent some time looking into it because we've had, I think it was that Amazon came out and said, oh, people were exploiting this before it was even known in public and, you know, it's out there and whatever. And then other people, there's so many false alarms on social media where people have these like hallucinated LLM generated exploits.

And they're like, I've got it. I've got the pock. And it turns out it doesn't even run. And, you know, So where are we at, like a week later, with these PhP CVEs that were dropped last week? Yeah, so where we're at right now is I don't think we will see mass exploitation of this, and we haven't to date. But going back to the history of it, yeah, there's a, there's, I think the PHP Foundation tried their best to do the coordinated disclosure really well around this, right? They didn't put patches into upstream branches that could be seen like we've seen with Linux kernel where that resulted in an exploit within hours.

They held onto this. it was all embargoed. There was the misstep of chain guard deciding to go early on the embargo. But then this outlier now is Amazon saying, oh, yes, since the 17th, which is some four or five days ahead of the actual release, they'd seen exploitation. But they haven't actually told us, well, what form that took? It's just an IP address. So maybe it was a researcher trying something, maybe something leaked from the embargo.

We don't know. That doesn't seem to be much in there. Yeah. Where we are at, though, is, look, both of these bugs, they rely on essentially you manipulating error handling within PHP. One is, they're both essentially boil down to use after freeze, but they're in different parts of the code. But it requires an error to be, basically,

it requires you to force the thing that is hosting PHP, so Apache or FPM, into an error handler in its startup routine, so before the PHP codes even actually run, which is straightaway gives to the first reason why this is hard to exploit, because if a startup failure happens in mod PHP, yes, it will continue to use that process. But if the startup failure happens in FPM,

which is the more popular way now, I think, to host PHP, it just kills off the process. It's gone, right? So even if you do crop the memory, there's nothing to come back to to mount your subsequent attack on. But again, these are just such bizarre areas where the use after free happens that I don't think there's really a lot of a window here

for a meaningful way to turn this into a weaponised exploit. Crisis averted. Good news, everyone. Now let's talk about the next meteor, which is this HTTP2 shell floor in Brotley. Now Brotley is some sort of weird old... What is it? It's a protocol, isn't it?

Compression. Compression protocol. Yeah, it's a compression protocol used in HDP. And it's quite commonly supported. It was open source by Google, I think, and it's like really commonly used, and that looks like a pretty nasty bugger. We're going to see this one get mass exploited,

or is it another one of those meteors that's just going to slide past the Earth there, dude? my bet is I think this one flies by it again looks like it's got all the same hallmarks being somewhat difficult to turn into an exploit I mean you've got to remember what this is going to give an attacker is the ability to write to memory that they shouldn't be able to write to but then what are you going to write to where is it being written to is that a predictable location is there other mechanisms and defense in depth that can prevent the pointer from jumping to that area of memory that you've now put your code into like it doesn't

Unfortunately, unless it's a fortinet box on the edge, a simple able to write memory does not get your shell anymore. Yeah. Okay, well, hopefully another crisis averted there. Now, we're going to have a bit of a chat about this work at a Citizen Lab. We spoke about it last week, where they had written up this Israeli ad company, this advertising company that was injecting ads into major advertising networks on the internet and using its position in the sort of ad tech industry and ecosystem to infect devices with iOS malware in targeted attacks,

but it also was doing weird de anonymization stuff. It looks like this one's getting a little bit of political attention now. I mean, we've been writing about this and talking about this for years, that this sort of data is a national security threat. It's got to the point where, and I know this, threat intelligence companies, they sell access to this data to, law enforcement agencies. So they could just use it without a warrant. It is very privacy-invading stuff. Now, as great as it is that it's getting some political attention, you hear senators,

you know, talk about this sort of thing every now and then, and then it just all goes quiet again. Alex, are we at the critical point yet where something's actually going to happen in the United States to deal with this, or is this just going to, you know, is the can just going to keep getting kicked on down the road? Yes, this is a little bit of back to the future for me. I got yelled at, by John McCain in 2014 about the security of online advertising. So nothing happened back then. So what's innovative here, and a great shout out to Sison Lab for discovering this,

is it looks like these guys were using, like you said, people have been selling ad tech data to violate privacy, but in this case, they were using targeted advertising, what's called custom audiences. So a number of ad networks allow you to upload things like the phone numbers of individual people to show them advertisements. Now, generally the rule is that you have to upload something like 100 phone numbers or 100 email addresses, so you can't get smaller than that. So there's no detail here of exactly how they then narrowed that down, or if they infected 100 people, or if they had like 99, perhaps fake people, and then one actual target. But that's a pretty innovative way to get after a target and then load up to that person, apparently, JavaScript, that would then take over their browser and take over their computer.

Really nasty stuff. There's been, Wyden has gone after this kind of custom audience for a long time. I doubt anything's going to happen because it is an incredibly powerful type of advertising that guess who is one of the groups that really love custom ads? American politicians. So everybody, basically, but Ron Wyden in the United States Senate uses custom audiences to win their elections. So I doubt anything will happen at least in the United States because these kinds of audiences are used widely by both Republicans and Democrats. I want to say two, a couple things. First of all, John McCain yelled at Alex.

He's now dead. Connect the dots. That's the first thing. The second thing... Just a warning to anybody who yells at me. That's right. Second thing I wanted to talk about was the reason there is that 100 minimum for advertising groups is actually really funny. I don't know if you remember this, but it was a guy who actually started, I think it was via Facebook.

He managed to enroll his flatmate, his housemate, or roommate, I think you would call it in America, into an advertising group and just ran targeted apps that was so targeted to this guy that he thought he was going insane. And that was very, very funny and wrote about it. It was a long time ago now. I still remember it. Very good. Look, we're just going to whizz through these next ones. We've got a story here about a write-up on how light LLM, how its release pipeline got hacked to distribute an infestile malware.

It is interesting because they wound up actually just compromising like the installer package instead of the software itself. So we've linked through to that one. People can check it out. Bad week to be a movie pirate because we're, we are seeing in the wild exploitation of Plex via Plex to Shell. Everything's too shell now, come on. I mean, it's bad enough that we name bugs,

but why is it to shell, Plex to Shell? And then there's Popcorn Time, one of its update domains got abandoned, someone registered it, and they're serving up malware in updates that way, so that's pretty funny. Some good news, Chrome has added Gemini-powered scam detection and click-fix warnings, to its browser. That's the headline from bleeping computer who covered this. There is a blog post from Google themselves as well. Really cool idea, right?

Where it's like if you're copying commands to the clipboard, like now the browser can go, hang on, that looks a bit sus. Send it off to Gemini and Gemini can say, yeah, that's ClickFix. This, you know, LLMs are really good at context. They're really good at divining intent as well. We're seeing a lot of work in the security space go into this. You know, companies like Push Security, who I work with. You've got other ones like Ent who are doing this sort of intent-aware security. stuff. It's great to see some of the basics being implemented by the browser makers here.

Microsoft is also put in context-aware fishing detection into teams. It's great. You know, you're probably going to, as an enterprise, you're eventually going to need probably some more enterprise-grade tooling for this sort of stuff, but it is really good to see the basics covered there. What else do we have? I just wanted to mention that the connect-wise attack that I flagged as being pretty interesting last week. Turns out that's now been attributed to a Chinese state-aligned threat and they were using RTEX. So there you go. You could use RTEX to do cool hacks

if you are a Chinese cybercontractor, which is what this sort of feels like. Feels like the MSS operations are a little bit like more tightly scoped than what these guys were doing. The White House... It's very Russian. It's amazing that actually

we've seen so many Chinese operations that are kind of loosely goosey like this. Yeah, but it always turns out to be some contractor and we find this out because they're so loosey-goosey they get owned and all of their stuff winds up available via BitTorrent. Right? So it's like, yeah, yeah, this is the thing.

It's all in like an Alibaba bucket. Yeah. You know, like the Russia stuff is like, whoops, Sergei, I let the worm out and now everything got owned. Whereas the Chinese stuff is like, yes, I hacked, I got shells on, you know, 250,000 exchange boxes because there was a bug, right? Like, so it seems pretty true to form to me. We saw a universe, the University of New Mexico Health Network. They had a ransomware incident. What's notable here is they were only diverting ambulances for two hours. and the whole lot was the whole incident wrapped up in less than 24 hours. So that's a good news story about how these incidents are getting,

there's more of a template, there's more of a playbook for dealing with them. And the US National Cyber Director has actually launched a pilot program which is going to provide rural hospitals with free access to security tooling, right? So just a bit of action when it comes to ransomware and hospitals. And we're going to finish up, James. The private key is always for, as it turns out. The private key, it's always four. Right.

So here we are again with another private key that is one bite of entropy and it's four. This one's a little bit different. Again, it's entirely different bug. It's just ridiculously hilarious that it all converges to create a private key whose value is four. But in this case, it's a vulnerability in the go-ahead web server, which is a little like embedded web server. It's going to end up, it is part of a bunch of IoT things. But this one's going to be harder to quickly mitigate,

whereas with the, was it, E-Mudra ones, you could quickly do like the cert revocation lists. We know how well they work. Well, for however much they do work, they weren't work in this case because it's not enforced at all. So we've got to call this four to shell, forever, four to, I don't know. I'm just, I'm trying. trying to throw some shade. It's not working. This is why you do the headlines and I don't, Pat.

This is going to be a jeopardy question. It's like, this private key is the only perfect square of an even number. It's like, what is four? All right, we're going to wrap it up there. Alex Stamos, James Wilson. Thank you so much for joining me to talk through the week's security news. Crazy old times. Thanks, Pat. Thanks, Pat. What another amazing week. That was Alex. John McCain. Slayer Stamos there with James Wilson taking a look at the week's security news. Big thanks to both of them for that. It is time for this week's sponsor interview now, and James spoke with Dylan Aery,

who is the founder of Truffle Hog. Trufflehog, of course, does secrets discovery. You can point it at your Slack, at your repos, at a S3 bucket. You can point it at a banana and say, find me a secret that I've accidentally exposed, and it will be able to do it. It is a mature product these days. It's actually very impressive. A lot of people use.

that you should too. So that is that is Trufflehog from Truffle security. But yeah, basically James spoke to Dylan for this week's sponsor interview all about I guess looking at where the real problems are when it comes to secrets exposure. There was a bit of an oh my God moment when Trufflehog looked at the hugging face breach

and saw all of the secrets that had rattled out of their training data. But as it turns out, you know, they're a bigger fish to fry. So here is Dylan Ayerie in Conquer. conversation with James Wilson. Enjoy. We were partnered with Hugging Face to scan their data sets at the time of the breach. So the CTO pings me and says, hey, you should check this thing out. Then the 14 keys that the rogue agents used to break in were in the scans that we were doing.

So we found, I think, a quarter million live unique keys across all the hugging face data sets within that quarter million was the 14 that the rogue agents had used to break in, which was pretty wild. And then at the end of it, Hugging Face pings me again. and they say, look, we just finished scraping all of Maine across all of GitHub, and we plan to publish it as a new data set called the Stack v3, and that's for training future models. And they said, you might want to scan this one because we suspect there's a lot of keys still on GitHub.

And so we appreciated the heads up. They did a lot of the heavy lifting for the actual scraping, and then all we needed to do is the scanning of that data. But we routinely scan and monitor large swaths of the public Internet. So every new piece of code that goes on to GitHub, we scan. Every new container that goes into Docker, we scan. And we also scan all of NPM, all of Pi Pi, and you're right. It takes a tremendous amount of compute, takes a tremendous amount of data to store it all.

And I think, just to put some numbers into perspective, don't 100% quote me on this, we probably spend about 100 grand a year doing all this scanning. And then I know your next question is going to be like, why, or to what end? You find all these keys and what do you do with them? And the answer to that is like we believe that the world is in for a world of hurt if we can't get the 1.4 million live keys that we know about revoked. And so we partner with a tremendous number of companies and partners to get that number burned down.

And that's why we do it. Just coming back to the numbers again, like when you went through the hugging face data sets, I think you found something in the order of 220, thousand credentials or live credentials. And like how you say, hugging face had a sneaking suspicion. There might be more in the GitHub repos.

What did you find there? I think what was surprising was number one, seeing the age of some of the keys still in GitHub. Because for a little while now, we've monitored all the new stuff going into GitHub. And we know that there's a tremendous number of new keys that enter GitHub every day. But going back in time,

seeing keys 11 years old that are still alive, like that was a little bit of a surprise. And I know, like, every time we tell this story, we always got the same question, which is, hey, I thought that GitHub has this partner program and like Amazon and all these providers and stuff automatically revoke these keys. Well, the reality is the program that they have is a disclosure program, not a revocation program. And so what that means is they'll go to Amazon and say, hey, here's the Amazon keys that have been leaked out. And then it's up to Amazon to choose what

to do with that information. So some providers will notify customers. Some providers will hard immediately revoke. And some providers will do something in the middle. Amazon does something in the middle. So they will apply a policy to those keys, restricting some actions on what the keys can do, and they'll notify the customers. Well, we looked at what those actions, like what those restrictions are. And there's still a tremendous amount that you could do with those keys. One of the funniest ones that we looked at was the Amazon API to download a file out of S3 is blocked with this restrictive policy. But the API to download a file with a version number attached to it isn't blocked. So it's just like it's a big it's a big unsolved problem. It's a long story short and so we're

excited to try to do our part and get as many of those keys revoked as we can. But it's it's a very big unsolved problem. The numbers themselves are pretty staggering. Like I think it was correct me if I'm wrong but it was 220,000 credentials in the hugging face data set then almost double that, if not just slightly more than double that in the GitHub. Data sets and 550,000 keys are still authenticated. Those as a snapshot point in time number are staggering in and of themselves, but what are you seeing in terms of the trajectory? Like, is this a problem that is a long tail? Is it, you know, not getting worse? Is it getting worse? Where is that number trending? Yeah, the problem is getting worse. It's getting worse for the reasons you can probably

expect. More code has never been authored before, and the ways that it's being authored is unique and different. And I'll give you a couple anecdotes. More and more our customers are asking for insights into laptops because of the shy-haloot stuff, right? If a worm gets hit on a laptop, you need to quickly know what keys are there, like what could have been slurped up. So when we provided that support, I guess, like, I don't know if surprised is the right word, but it was interesting to see that so many of the keys on laptops are now in like the clod folders because people are talking to their bot bots and they're in the chat logs of the chats. And they're saved in skills files.

So you used to find them in like dot-emv.local in source. I'm sure you still do. But now you're saying you see them in like the dot-clawed transcripts and things like that as well. Wow. That's absolutely right. And so that's causing more keys to leak out. And then the different places are kind of interesting too. Like one interesting insight,

I don't think we've done a blog on just what we've seen in Docker Hub yet, but we've tweeted out a couple of things. And one of the insights that we tweeted out is about twice the volume of live keys, land and Docker Hub, every day compared to GitHub. Why is that? Why is that so different? I would wager it's because push protection on GitHub offers like some level of protection. Of course, you could override push protection.

and it also isn't a silver bullet. Like, you know, if you push up a key in a zip file or it doesn't have an exact pattern for what it's matching on. Also, other weird things about push protection is if your patch that you're pushing up is too large, it just fails open, allows the code to get submitted up. So there's a lot of reasons that push protections isn't perfect. But that being said, there's nothing like that in Docker Hub.

And also, the other thing is, like, the Docker ignore files and the Git ignore files are never the same. And so you might write a get ignore file that's like don't include any of these sensitive files. And then when your CI fires to publish into Docker, I think it doesn't respect or pay attention to that, get ignore. And if you don't have parity with your Docker ignore, you're just going to get a bunch of stuff copy star dotted into your Docker container. And like that's often the cause of what I call like second order leaks.

Or like your first order leak is like maybe you willingly put a secret into code. Your second order leak is CI automation has, caused a secret to get written to a file system, and then like a copy dot accidentally slurps it up and pushes it into a package. And then you have like thorough release, which is like one further removed where like some weird convoluted automation ends up like materializing a key seemingly out of nowhere. And then it ends up in a log somewhere or something like that. The flip side of this though is that for all the hundreds of thousands of still authenticating valid keys that are out there in either Hugging Face datasets or in GitHub repos that are public. You know, why doesn't

this result in more breaches or is it resulting in breaches that we're not seeing? Like, it just seems an amazing treasure trove of access. I think you do see them, right? The Huggy Face incident. It's caused by 14 stolen keys. Well, cause is maybe not the best word, but... Well, tell me more about that because, you know, until you mentioned that to me before we started recording, I was like, But I thought that was all, you know, rogue agents, find in zero day. We just used keys. I don't think it was quite as prevalent in the messaging. It's tough to say we just because it wasn't one thing.

And the other really tough thing about the hugging face incident, like when Mitar came in and did their investigating of it, what they expected to find was like a quick, simple story. And instead what they found was a civilization of alien brains that worked for like the equivalent of 300 years. And so it's like to distill the whole story down to like it was just X or Y is I think not doing the story proper justice. But once the agents used their combination of exploits and magicy things to get internet access, a very small number of agents, maybe just one was investigating hugging face as something maybe it could look into.

Well, most of the other agents were focused on other things. And when that agent first got its initial foothold, it found 14 keys that validated to a bunch of private data sets and Hugging Face. And then it used that to get access to more data. And then that more data then led to some zero days in Hugging Face's infrastructure as well as more keys. Right? So it used keys to get access to data to get access to data. What I usually like to say is imagine you're training an AI model.

to hack or exploit a system. And you incentivize in that training process the fewest number of tokens to accomplish that goal. Is it makes sense to burn like two days worth of tokens finding a novel security exploit? Or does it make sense to just log in with the password that's sitting there? What's going to result in the path of fewest tokens

to accomplish the goal? And I think the reality is it's like they're trained to do both but it's going to do the thing that's quickest. And so if there's a key laying there, it's going to use it. If there's not a key that's laying there, it's going to burn two days finding a zero day. Yeah, yeah, exactly. It's going to find a way and it'll use the path of least resistance first.

So speaking of sort of, you know, as you mentioned in the blog post, the gap here is revocation. Who owns that gap, though? Whose problem is this to solve? Right? And of course, you know, in one sense, the answer is customers should be using Trufflehog to find their keys. but the scale of this is huge,

and as you say, largely an unsolved problem. So who's incentivized to fix this? Yeah, I mean, I think, like, incentives are one thing, responsibility is one thing, and then just who can have the biggest impact is, like, in my view, the most interesting question. Because, like, let's say you're a big SaaS provider and you have thousands of customers,

sure, you could take the position where the customers are responsible for their own keys and if they do stupid things, that's on them. but also if you have a nice feed of all these customer keys that have leaked out, it's going to be way easier for you to revoke them at scale than to hope that the customer notifications are going out in a way that they can see respond action on. And so that's what we've recently done is we've created a whole bunch of these partnerships with different companies.

And we have more to come on this in the coming months. But basically to just try to get the upstream providers to act on behalf of their customers. And even if you take the philosophical stance that it's the customers themselves that are responsible, it's just easier for a provider to centrally squash it than it is to track, you know, to play the individual whack-a-mole game. Yeah. Do these partnerships, do they ever pushback a little bit and say, well, we don't want to just revoke these keys because we could break a customer's production system and that could erode trust.

Like, what's the sense of pushback that you get from vendors? All the time. Yeah, all the time. I mean, it depends on the vendor, right? I think Slack and Open AI are great examples of companies that just said, you know what, we're going to hard revoke. If it breaks something, it breaks something. It's in our terms of service.

They should have read that more carefully. The terms basically, say, if key leaks out, then, you know, it's going to get revoked. Then there was one company that had like a weird middle ground. I hadn't seen before, but it was really interesting. I think it was 11 labs, where if you generate an API key in the generation, step. It has a default on switch. And the switch basically says if this key leaks out on the internet, we find out about it, we're going to revoke it. You can turn that switch off, which why. And then you've got a lot of other providers, like Amazon being one of them, that don't want to

risk the damages that could come from just blindly revoking keys. Like cloud providers are great examples of that where it's just really hard to anticipate how damaging it could be to just take that heart of a stance. And a great example I like to give of this is when you scan Kubernetes, you will find a live GCP key in it. And that's because it's used to pull public container registry images.

And it needs to be there and it's useful. And so people's Kubernetes will break if you take a hard stance and say all GCP keys need to be revoked. And then there's payment providers that can very easily quantify the value of the key they take down. They can say every hour this goes down has lost X million dollars of revenue.

And so, yeah, it's messy. It's complicated. That's why it's unsolved. Yeah, it certainly sounds like it. Well, Dylan, look, thanks so much for dropping by for the chat. It's been really interesting to see the results you found from this latest investigation. Thanks so much, James. That was Dylan Aery chatting with James Wilson there.

Big thanks to both of them for that. And big thanks to Truffle Security for being this week's risky business sponsor. And that is it for this week's show. I do hope you enjoyed it. I'll be back later this week with a snake oil as edition of the Risky Business Podcast, but until then, I've been Patrick Gray. Thanks for listening.