Risky Business - Risky Business #758 – Crowdstrike's postmortem underwhelms

Episode Date: August 7, 2024

On this week’s show, Patrick Gray and Adam Boileau discuss the week’s security news, including: Crowdstrike talks loud in its postmortem, but says very little Digicert fears the CA-Browser Forum, gets lawsuit from a customer Dmitri Alperovitch joins the show to talk about the Russian prisoner swap Cloudflare continues to harbour scum and villainy Professional ransomware crew … is an improvement? And much, much more. This week’s episode is sponsored by Thinkst Canary. Marko Slaviero joins to discuss the unfashionable choice they made in hosting their platform one-VM-per-customer. Show notes CrowdStrike investors file class action suit following global IT outage | Cybersecurity Dive CrowdStrike rebukes Delta’s negligence claims in fiery letter | Cybersecurity Dive Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf Sparks fly when lawyers meet a certificate revocation crt.sh | Alegeus U.S. releases Russian hackers in Evan Gershkovich prisoner swap U.S. Trades Cybercriminals to Russia in Prisoner Swap – Krebs on Security Who are the two major hackers Russia just received in a prisoner swap? | Ars Technica Hackers remotely wipe 13,000 students’ iPads and Chromebooks after breaching safety software Mobile Guardian Device Management Application to be removed | MOE Ford wants patent for tech allowing cars to surveil and report speeding drivers I'm Sorry, Dave, You're Speeding | WIRED Cloudflare once again comes under pressure for enabling abusive sites | Ars Technica Low-Drama ‘Dark Angels’ Reap Record Ransoms – Krebs on Security Bumble and Hinge allowed stalkers to pinpoint users’ locations down to 2 meters, researchers say | TechCrunch Unfashionably secure: why we use isolated VMs – Thinkst Thoughts Defending AI Model Files from Unauthorized Access with Canaries | NVIDIA Technical Blog

Hey everyone and welcome to Risky Business. My name is Patrick Gray. This week's edition of Risky Business is brought to you by Thinkst Canary, which makes the Canary honeypots that you can plug into your network. They also operate the CanaryTokens.org service. They're a great company, well loved in the cybersecurity discipline and thinks Marco Slaviero will be along in this week's sponsor interview to talk about a blog post he wrote which is about unfashionable choices in security engineering. In other words they talk about why they use one VM per customer when operating a cloud service instead of building some sort of infinitely scalable web application that's always going to have a bug in it, which will let one customer own all of the other customers. Uh, that's actually a really fun chat and it's coming up after the news.

Uh, I will say though that we won't be running that sponsor interview in the YouTube video because of a technical fault. Uh, we only have audio on that one. So if you're watching the video, hi, um,

unfortunately we will not, uh, we will not be able to show that to you, but you know, uh, that that's a reminder to, to people who are listening watching the video, hi. Unfortunately, we will not be able to show that to you. But, you know, that's a reminder too to people who are listening to the show that we are now publishing the weekly show to YouTube every week. And yeah, so far it's going pretty good. But we are going to get into the week's news now with Adam Boileau. And not surprisingly, mate, we're going to start off with more sort of

clown strike related, you know, cleanup stories. And it looks like we've got the first investor lawsuit here. It's probably going to turn into a class action. It's being kicked off by some sort of pension fund in the United States. Cybersecurity Dive has a write up on that. And we're also seeing a bunch of to and fro between lawyers for Delta Airlines, which of course claims to have lost around half a million, half a billion dollars, I'm sorry, through this whole event. So an argy-bargy between their lawyers and lawyers representing CrowdStrike and Microsoft. Yeah, it's a mess for a lot of people.

And it's not surprising, especially given the American, you know, how the American system works, that we would see this going off into the courts. And we've seen both Microsoft and CrowdStrike kind of hit back at Delta. And Microsoft, for example, said that some of the systems that caused Delta to be down for so long, like their crew management systems or whatever else, weren't even running on Microsoft platforms. So, yeah, yeah, yeah. But, you know, overall, given the, I mean, I guess we're going to see whether the license, you know, the click-through licenses for CrowdStrike would kind of limit their liability to, you know, I think the total cost of their service. So, at worst, millions, you know, we're going to see how that turns out.

This is for the courts to work out, right? So I'm not sure about the pointy clicky through license versus the contracts that CrowdStrike might enter into with their customers. We're not lawyers. We don't really know. But what we can say is that we have some leaked correspondence from CrowdStrike's lawyers responding to Delta's lawyers. And it is, you know, it's a little bit sassy, right?

Because they're saying, well, you know, it's a little bit sassy, right? Because they're saying, well, you know, we offered to help you get back up and running. We never heard back from you, you just ignored us, which I don't know. I mean, they had a bit going on at the moment, at that moment. So I'm not sure that they could have been expected to respond. But I think the interesting part there, you know, is that they do make the point, well, why was it that your competitors who are our customers were able to get back up and running and you weren't? And as much as I squarely blame CrowdStrike for this whole thing, I think that's a reasonable question. It is. And I'm sure the courts will consider all of the facts. But I mean, when you look at the scale of that CrowdStrike

mess, I mean, it's not surprising that we're going to see this get dragged through the courts. And the investors, they saw the share price go, what, from 300 something to 200 something over the course of this. So you can see why they're also spitting tax. Yeah. I mean, an investor lawsuit was just completely inevitable here. Interestingly enough, the lawsuit also names the chief executive of CrowdStrike, George Kurtz, and the CFO, Bert Podbear, as defendants. So, you know, straight for the throat. Straight for the CEO's throat, right? Right for the throat.

So we've got, yeah, we've got a couple of reports there from Cybersecurity Dive. One is by David Jones and one is by Roberto Torres. So we'll link through in this week's show notes and you can check it out. Now, meanwhile, CrowdStrike has released its external technical root cause analysis, Channel File 291. You know, originally it released a preliminary incident response, very light PDF talking about what happened. Now they've done something kind of more detailed, but it is, to be honest, I find it like pretty awfully written. And a little bit, yeah, just a little bit kind of, I don't know, there's not as much detail in this as I would have liked.

I do, I guess I need to correct something I said, which is early on, I'm like, it looks like they didn't test it at all. It looks like they did test this channel file update, but the testing was just woefully inadequate. But walk through this report. It's a 12-page PDF that CrowdStrike's released here. Walk through your reading of it, Adam. Yeah. So the basic guts of it is it describes how CrowdStrike's kernel driver, which implements kind of detection mechanisms, receives configuration via these channel file updates and kind of how the testing process works for the code that makes up the kernel driver the data files that get fed to it and then the kind of like the process by which they are kind of integration tested and it comes up with a couple of you know places where things went wrong for them and I mean

like all of these outages there is a confluence of of things that go wrong to cause it to be you know a planet melter and in this case it kind of comes down to the code that was checking for named pipes had a parameter which you could use regular expressions to you know to to make the to refine the match and in their test cases they'd only ever tested it with a wildcard so i'm assuming that that parameter was blank or not used and it turned out that a bug in the code miscounted the number of parameters and so they were were able to, you know, read off the end of an array looking for this parameter at the point where they actually shipped a rule that used a regular expression instead of a wildcard match.

And it seems like they're kind of like the software test cases that they had written didn't include that example. And then another piece of code that validated kind of the number of parameters and so on also was wrong. So those two things lined up and led to shipping a bug, you know, shipping a system that, you know, caused an invalid read and off into blue screens of death.

And they describe the testing processes they go through, and there is, as you said, there's quite a lot of words in here. But in the end, the net result is they didn't test it in a way that didn't mean that many planes stopped flying. So there's a few lessons for them to be learned there, and they've also discussed their use of deployment rings, of deploying the code over time, or whatever changes they're making,

be it configuration, be it code, to different groups of the code over time or whatever changes they're making be a configuration be a code to different groups of user base over time to try and catch these things before they turn into planet melters when it's only country melter or business melter as opposed to entire planet well i mean they've talked about how they did not use deployment rings in this case, not how they did use deployment rings in this case. They have deployment rings, but it kind of read like perhaps they used it for code but not for data or something. It was a little bit weaselly as about exactly where they did and didn't. They did not use deployment rings in the case of this update.

In this case, yes. And here we are, right? So clearly they did not. Obviously they would have um obviously they would have the the you know infrastructure that would allow them to do that for whatever reason they chose not to do that in this case probably because they have a great content validator and you know they've done it a million times before you know thousands and thousands of these updates have gone out over the last 10 years but you know you and i have spoken about how

complacency can get you into real trouble in situations like this. So, you know, that is, in my mind, you know, still falls into the inexcusable category. The fact that they weren't doing ring deployments for this is just absolutely nuts, you know. And particularly because they're operating in the kernel. And this is something that came up, you know, quite a lot in the conversations I had with, like, Alex stamos and chris krebs who obviously work for a competitor right they've got a vested interest in in being particularly critical of crowdstrike uh but i i happen to agree with them that this is just you know kind of madness if you're going to run an architecture like this you really do need to be careful and i mean you know one thing in testing that would have exposed this bug is if

they had a rebooted the the test boxes right? Because, you know, you get named pipe creation, it would have exposed this bug and, you know, which is why reboots didn't fix this and it was just endless BSOD loops. So, I don't know. I mean, they've written this in a way to be as boring as possible, right? In the hope that people's eyes just glaze over and they don't really pay much attention. There's also odd things in here, like they've got a mitigation, which is, you know, you can provide customer control over the deployment of rapid response content updates. No one is going to use that. The whole purpose of using something like CrowdStrike is that it is essentially a managed service, you know, something like Falcon Complete with your Overwatch service

and all of that. You get them to do it so you don't have to think about this stuff. So I found that a pretty insane sort of thing to put in here um you know i don't think this is particularly exculpatory this report i think that in their heads it is which is you know you think about the swiss cheese failure model where all the holes have to line up for a failure like this to happen and i'll concede that they got unlucky but you have to you know when you're running something like this you have to account for the fact that you can be really unlucky and i don't think that they did to a sufficient degree what's your what's your feeling there yeah i mean in the end when you are running in kernel mode you have an obligation to not screw up because if you do you're going to take the

whole box down and you know we've seen plenty of people break things in kernel space before in antivirus products and whatever else but breaking it in such a way that it breaks on boot like effectively bricking the whole thing like that's a real that's the thing isn't it that's the thing where where people who sort of stick up for crowd strike here and say but they do do things properly they do have really good qa procedures and it's like well clearly not good enough yeah clearly not good enough and people say oh well you know failures could happen to anybody and okay they can but a failure like this is so catastrophic and so difficult to recover from that you need to be i just think they needed to be more careful yeah i mean in the end like the impact is what matters

right all of these bad things happened and you and it kind of doesn't matter what your technical arguments are about the testing process or the things they do. It's kind of outcomes-based, right? Planes stop flying because of a thing you did. No amount of weaseling is going to make those planes go back in time and fly. Yeah. I think the words that I keep coming back to uh is you

know self-evident it is self-evident that they were not doing uh an adequate job uh i think yeah and i guess you know the shareholder lawsuits will decide whether you know the courts agree yeah yeah indeed uh moving on um i should point out too it's a relatively slow news week because it's black hat and defcon week and it's always a slow news week when that happens. One that's interesting here, I've linked through to our colleague, Katalin Kimpanu, who's right up on this. But there's been this funny issue where DigiCert, the CA, was found to have issued some certificates that were like didn't have complete information or whatever and they revoked them because they're terrified they're going to get yanked out of the browser trust stores which has caused like one customer who was affected here said well well hang on you've

just given us 24 hours notice so they went to court to try to get a injunction i think they actually got it as well so it was interesting seeing this decision sort of spill out into legal action. But I got to say, I'm kind of on the customer's side here. I think if DigiCert has issued certificates that it shouldn't have and wants to revoke them, fair enough, but give them a little bit more than 24 hours. Was that where you landed on this? Yeah, I felt bad for the customers in this case, because, you know, having to roll all of your certs, because typically an organization is only going to use one CA, right? They're not going to be shopping around some kind of diversity there. You're going to have one vendor and having to move, you know, roll every cert in your environment in the space of 24

hours. Like that is a pretty heavy ask. And I see that like when you look at the backstory of the issue so digi cert uh they do domain validated certificates where you have to modify your dns zone to prove that you have control over it to get the cert and the specification for this process says those domain names that you have to put in your zone must start with an underscore. And DigiCert made some change in like 2019 that meant that the zones they generated for you to put in didn't have underscores in them. Which, I mean, okay, they're not, you know, they are doing it wrong and they should do it right. Yeah, I mean, they're not perfectly compliant, but it's not like it's, you know, I mean, we've seen CAs do much stupider stuff than i mean we've seen cas do much stupid we've

seen a lot worse than this but then it's just like given that this happened in 2019 they started doing this now it being a thing we have to resolve in 24 hours seemed like i can see why some of the customers were a little bit salty about that and then so they said 24 hours uh a bunch of customers complained then they said okay we're going to go back and ask the browser vendors if they'll give us an extension overall. And so they got extension to three days. And then this customer, Allegis, which is an American, like, healthcare software as a service platform, said that they couldn't do it in three

and they went to court and got an injunction for seven. But again, DigiCert had to go back to the the ca browser forum and beg to be allowed to give the customers a week so you know everybody involved kind of had a rough week at work and it's nice to see this being taken seriously because we've seen so many egregious failures of certificate authorities and the CIA browser forum not bringing the ban hammer down on, like, really just stuff that absolutely they should have. This feels like a bit of an overcorrection, and I think everyone involved will have learned something

from this experience, but... The CIA browser forum is high on power. Dance, monkey, pew, pew, pew, you know i mean that's that's kind of what it feels like a little bit um but it is good it is good that the cas are kind of like you know terrified like we wanted the cas to be terrified we do we do so who was it who got yeeted recently like it was it was like antrust or someone yeah yeah yeah it? Yeah. Yeah. Yeah. So no wonder. Did you sort of like, jeez, we left off the underscores, like panic stations, right? Oh, dear.

Oh, it's funny the way that happens. Now, let's talk about this Russian prisoner swap that just happened because there was some sort of erroneous reporting about that. I got suckered into that as well. I retweeted a tweet thread that talked about all of these cyber criminals that were exchanged. Now, for those of you who have not seen the news, there was a big sort of prisoner exchange between Russia and the West. A bunch of criminals, Russian criminals, who had been detained in various places, some of whom had done things for the Russian state like sanctions evasion. One of them had killed someone on order of the Kremlin in Germany and got caught and thrown in prison. They were a gangster who was essentially took on a side gig as a sort of intelligence service sponsored assassin.

So they've sprung a bunch of these guys and exchanged them for the US Marine Paul Whelan. Also Evan Gershkovich who was the reporter for a US newspaper, the Wall Street Journal. And they've swapped them. And yeah, initially it was like, oh my God, you know, this really proves the nexus between cyber criminals and the Russian state, but not so fast. Like it looks like there are two of the people who were swapped out of eight were cyber criminals, but it doesn't actually look like they were doing much at the behest of the state. So I did an interview yesterday with Dmitry Alperovitch and I've posted the whole thing to YouTube. So if you want to go and subscribe to our YouTube channel, it's Risky Business Media. You'll find it and you can watch the whole interview. But here's an excerpt where Dimitri talks about how, you know, really this

isn't necessarily about these guys being super secret, you know, Russian cyber spies. So there's no allegation that either of these guys were working for the state when they were doing these hackings. Maybe there's some private stuff in the indictments that never got disclosed or didn't get put in the indictment, I should say. But as far as we know, they were not involved in GRU or FSB or SVR hacking operations. But look, Putin has been very clear

that he views any arrests of Russian citizens for crimes that they committed while being in Russia as extrajudicial. He views this as kidnappings, what the US DOJ does, and he wants to get everyone back. Now, I do think there's a prioritization. So you're going to be higher on the list, given your political connections. And, you know, same here, right? Devin Gerskovich was clearly at the top of the list. And some of the other folks that we still have there, unfortunately, you know, same here, right? Devin Grishkovich was clearly at the top of the list, and some of the other folks that we still have there, unfortunately, fell off the list. So in that respect, it probably works fairly similar.

But I don't think that he's looking just for people who have worked for the state. He wants any Russian that has been arrested for anything by US government. And I think, you know, we've established that you kind of agree, the consensus view that's forming seems to be that Putin got the better deal, but it was a deal that had to be done. And we in the West just have to take our medicine on this one. Yeah, I do. And I think there'll be other deals for people that have already been arrested. But you know, beyond that, I think we need to be really thoughtful because if you are still in Russia, either as a journalist today, a Western journalist or an expat, and you get picked up, you know, you've had plenty of warning, right? So to release another, particularly

hardened criminals, it's one thing to release, you know, these illegal spies, you know, that's what we've done throughout the first Cold War. But to get these killers out, to get these smugglers out, cyber criminals, that's a totally different issue. And I think we need to be reconsidering that going forward for people that are still, you know, walking around in Russia and, you know, have not left the country. You know, they kind of made their bed in my view. So, yeah, that was Dmitry Alperovitch there. And you can check out the whole interview on our, on our YouTube page. So, I mean, I don't, I think people might be reading a little bit too much into this one.

I also think it's funny that like a couple of days after the trade, Russian authorities released like video of these guys and like why they were arrested in the case of Evan Gaskovich. He was, he was doing some risky journalism. Like he was doing stories about like how Russia was able to refurbish tanks to get them to the frontline in Ukraine. And, you know, they arrested him with a source

who had brought classified documents with him, which to me feels a little bit like a setup. He had actually asked the source not to bring the documents. But then once he turned up with them, he said, look, we'll attribute anything in these documents. I'm not going to take them with me, but I'll attribute things in them to anonymous sources and blah, blah, blah, blah, blah. So you can see why that would get you into a bit of legal trouble in Russia. intelligence connected uh former marine paul wheel and um they had video of him being handed a thumb drive in a bathroom which i gotta say didn't didn't look particularly exculpatory uh

sorry to use that word twice in the podcast but but yeah what's your take on all of this i mean it did feel a little bit rude seeing you know especially in the case of like guys who've been carters uh heading back to russia and you know kind of the the the amount of protection that those guys feel you know when they're out and about um you know because like a couple of the the cyber people were arrested like on holiday one was in like in switzerland or something going skiing or something yeah one was going skiing in switzerland the other one turned up to the moldives with a plane full of hookers right and this is how they got arrested and yeah like the carter guy that's like track two like infamous carter had it was serving like a 27 year sentence had been in prison for like nine years um the other guy had only been in prison for

for a couple of years but it looks like it's it's entirely possible that these guys were sprung more because their dads were like putin's golfing buddies than the fact that they were doing secret FSB operations. You know, I guess that's my point. Yes. Yeah. And I think that was an interesting because I also saw that thread that you tweeted and that was the kind of there was some implication there that that was it. And I thought, you know, the conversation with you and Dimitri, I felt kind of cleared up a little bit of that for me because nepotism definitely seems, you know, kind of more believable in a way. It's on brand, right?

It is, yes. Like it's definitely on brand. Yeah. Yeah, it certainly is. Yeah. Look, as Dimitri said in that interview too, there's a few more remaining, certainly on the cyber side, and we might see further swaps in the future. Yeah.

I would not be surprised surprised it seems to be how things are done i found um dimitri's comments about you know at some point if you're someone who can be taken hostage and you're operating in russia at this point you know there is a degree of responsibility there yeah i mean i even said in that i even said in the in the in that interview that like i was uh i had to drive to brisbane the other day to pick up my mother-in-law, who's flown in for a few weeks. And so I listened to a lot of podcasts and whatever, and I was listening to the BBC.

And there's the BBC correspondent in Moscow. And I'm just thinking, why are you still there? Like, that seems a bit dangerous, but, you know, it is what it is. Now, let's talk about this story from Alexander Martin over at The Record, which says that hackers remotely wiped 13,000 students' iPads and Chromebooks. Oh, dear.

And this happened in, apparently, I think it was in Singapore, and it was the mobile guardian, like MDM platform, somehow was used to, like, yeah, wipe 13,000 devices, which probably isn't what you want out of your MDM platform. No, it's really not. Like you want to be able to do remote wipe when you leave it in a cab. You do.

Right? You don't want to be able to do remote wipe of 13,000 devices. I'm kind of surprised that like there's not a control to stop that from happening. I mean, there probably will be now. Yeah. Because we haven't seen any details about exactly, like, how this went down from a technical perspective,

like whether it was intrusion into some back-end system at Mobile Guardian or whether it was, like, you know, account compromise of the, like, Singaporean Ministry of Education's account with Mobile Guardian, like, through the admin panel or whatever it was. Or whether or not it was some application bug that allowed them to jump tenants. Yes, a cross-tenant bug.

We don't really know. Mobile Guardian did seem to suggest that it had affected some customers elsewhere, so not just Singapore, and this was just sort of the most noticeable of the impacts. Which makes you think, you know, some admin account at head office got done. Absolutely.

Could have been. Or it's a cross-tenant bug. Or it's a cross-tenant bug. Or, I mean, even something like, you know, looking at Snowflake, where attackers figure out that this is a target you can hit, go look in data breaches, find a dozen customer accounts, hit each of them with the same thing. So, you know, it's kind of weird. I heard something interesting.

I'm not 100% sure sure this isn't verified yet uh but i heard that a lot of the snowflake those snowflake accounts the reason they didn't have mfa is because people were using command line tools to access them right um and that's and they they didn't support mfa so that's you know i'll look into that everybody i'll try to find out more, but that's just something I heard the other day. But I just took a soft topic there. Sorry. Yes. This is why API tokens exist. Yeah. So we don't know specifically what happened, but either way, I imagine it's pretty rough day at school in Singapore when, you know, all your students have their stuff wiped. Yeah. So yeah, good reminder of the power of MDM and the importance of securing access to the admin interface of those. Yeah. So yeah, good reminder of the power of MDM and the importance of securing access

to the admin interface of those. Yeah, and this week's sponsor interview is actually kind of relevant to this. I mean, it may be relevant, it may not be, but the overall theme is relevant, right? Which is, thanks to have this blog post up and I'll link through to it. It's actually really funny.

Did you read it? I sent you a link. Yeah, yeah. And it's legit funny, right? Which is everybody wants to build, when they're operating a cloud-based service, like software as a service, everybody wants to build like that really sophisticated application,

like the multi-tenant thing. And that's just the way it's done. And they argue that like that makes life easier for the software provider, but doesn't really do anything for the customer. And software providers just like doing that so they don't have to run a bunch of VMs. So what Thinks did is they're like, no no we're just going to do one vm for customer per customer and we're just going to get really good at managing like thousands of vms in a fleet

um which and of course you know a a i'm sure there are accounts at uh thinks hq that if you got access to them you could do some real damage they They use salt stack. I'm sure that, you know, with enough research budget, and if you could punch through, get shell on one of those boxes on one of those VMs, you might be able to find some sort of lateral movement bug. There's certainly been those bugs in the past with salt.

I've used them. Yeah. But I still accept the major point, right? Which is that, you know, this is a decision that people make to make their lives easier, not the customer more more secure, right?

And it's just, it's an interesting bit of challenging, what do we call it? Challenging think-fluencing from the people that think. And I did really enjoy it. I would much rather manage a fleet full of VMs with controls they understand than some enterprise Java monsterware that I have to go reverse engineer

to understand how those multi-tenant controls actually, you know, as built work. Well, and there's so much attack surface in those web applications, right? Like just one mistake and then it's cross-tenant and everyone's done, right? So yeah, that was an interesting chat and I just find it relevant to that, you know,

that MDM piece. So again, that one's going out in the audio version of this podcast, but not on YouTube this week. Sorry. So, okay, this story, I don't even know if it's technically a cyber story, but I had to include it. So, Suzanne Smalley wrote this one up for the record. And it's about how Ford is seeking a patent for technology that would allow its cars to like do surveillance things like, you know, identify speeding drivers and whatnot. Now, there's been hand-wringing over this and fair enough.

But the reason I wanted to talk about this is I wrote a very similar story back when I was writing For a Living. The byline on this piece is March 4th, 2004, which was about, so it was less about 20 years, over 20 years ago, right? So my story is about how Toyota demoed a concept car that would adjust the performance of a vehicle based on whether or not you were a crappy driver who like did speeding or whether or not, you know,

you'd lost points on your license or whatever. But it's just funny that this is something that just comes up a lot you know every time there's a motor show where where car makers demonstrate some sort of uh creepy feature it just all comes back and of course now there's this huge conversation about cars being these sensor platforms that collect all sorts of data and there's not really good legal frameworks in a lot of countries for dealing with how that data is stored and whatever i i just think this is interesting that this just keeps coming up and coming up and coming up. Yeah, no, it's an interesting topic. And I know, you know, when we put together the run sheet,

you know, the news list that we're going to go through for the show, probably, I would say, for the last six months, every two or three episodes, there's a story about automobile data collection or increasing amount of sanctions or controls around self-driving cars or autonomous vehicles or censored platforms on cars or connected cars. Yeah, I think I saw this morning that the United States is about to ban Chinese operating systems for cars. Yeah, there was some proposal for self-driving car testing in the US by Chinese vehicle manufacturers and also some controls around connected cars. And these things come up very regularly and a lot of the time it gets filed under,

OK, this is really interesting, and at some point there will be a reckoning and we're going to have to talk about this in a bunch of detail. But we're not quite there yet because everyone's still a bit you know up in the air about exactly what our expectations are and how independent we want our vehicle ecosystems to be and so on um so i mean it is creepy stuff but then again you know like if you go to a car show how many concept cars with weird you know ideas and stuff do you see that that then never see the light of production so there is there is you know there's real meat here to talk about but it's still kind of settling and yes was just one more piece in that sort of sort of puzzle for me well if you want to know what civil libertarians think about this you can read

uh my report forired from 20 years ago. I did like actually in your piece, it talked about an idea of having the license plate just display the driver's license number of the driver rather than it being the car, which that seems like a practical, you know, useful thing to do actually. But yeah. Didn't happen though, did it?

Didn't happen. Yeah, exactly. Now, look, talking about speaking of speaking about things that are like kind of evergreen topics uh let's talk about cloud flare sucking uh because we've got a report here from dan gooden spam house who are also always like that's been a constant through my career as well which is spam house complaining about people sucking who they complain about tends to change uh decade by decade but they're always complaining about someone and at the moment they're complaining about cloudflare like i've complained frequently on this program about how cloudflare you know platform nazis racists hate forums all that sort of stuff and they say we're just a pass-through service so they wash their

hands of it whereas we've seen you know know, when they kicked 8chan off their platform, where's 8chan now, right? You never hear about 8chan anymore. So we know that they can, you know, get rid of services that are using them to stay online. And yet they say, oh, but we're just a pass-through service. It's got nothing to do with us. So Spamhaus has put out some numbers on just the scale of abuse.

So we're not talking about, like, you know, nasty content, hate sites and whatever. We're talking about abusive domains that are being, you know, linked to spam and things like that, and just say that, you know, the usual story, Cloudflare is not really responsive. I mean, this is something that I hear from everyone as well. Threat researchers and stuff who find, you know the usual story cloudflare is not really responsive i mean this is something that i hear from everyone as well threat researchers and stuff who find you know infrastructure or content being served up via cloudflare they write to cloudflare they never hear back and you know it's just you know again same old story uh just coming up again yeah yes spam house wrote a blog post with the specifics of their complaint which which was entitled Too Big to Care, Our Disappointment with Cloudflare's Anti-Abuse Posture.

And they make the argument that fully 10% of the domain names on their Spamhouse block list, which is all sorts of, you know, cybercrime and whatever related things, fully 10% are name-servered by, you know, have used name services provided by Cloudflare, like not even necessarily the CDN part of it and I mean that's you know that is a real you know that's a real set of numbers like I mean

that list does contain a whole bunch of nasty stuff and they make the point that just go to Google search for bulletproof hosting and check how many of the responses you get are name served by Cloudflare and that that's maybe a thing that Cloudflare could do something about. It's funny, right? Because they go through the whole experience that Brian Krebs had when he had that record DDoS and Cloudflare came and offered to help him and he told him to pound sand, right? He told him to go away.

Because, and it's interesting, when I first met Brian, I met him maybe just over 10 years ago. I think we may have met before, but it was the first time we hung out and we actually had this very conversation. Evergreen. Yeah, Evergreen. I think that's the theme of the week, isn't it? Evergreen.

But yeah, I was talking to him and he's just like, do you know how many of the DDoS for Hire services are like protected by Cloudflare that offers DDoS for hire protection right and it's just so I know he's always found that weird so it was good to see that um in Dan's piece now speaking of Brian he's got a fantastic write-up here about a crime group called Dark Angels which sometimes does ransomware but quite often does not which appears to be an extremely profitable cybercrime organization that doesn't deploy ransomware always because it makes too much noise. So what they do, they tend toward data extortion. The reason I find this interesting and the reason I find this worth talking about

is because in my view, if we had cyber criminals doing data extortion, that's a win. you know, ASD, the whole Five Eyes Alliance, pushing back on that stuff, is because of the disruption it causes to operations, to hospitals, to, you know, all sorts of critical suppliers, pathology labs, you know, so healthcare is a big one, obviously, but even just normal enterprises. Whereas, if they're just stealing data and ransoming it back, okay, that sucks for the victims, but it's a crime type that's much less disruptive to wider society so i found this a really interesting write-up and and if if this is the beginning of a trend that would be terrific yeah i thought this was interesting for the same reasons right if this is the beginning of a trend then maybe it's a a movement in the right direction and one of the points brian makes around this group is he calls

them low drama uh which is kind of a nice kind of shorthand term for it i guess but you know they are not an as a service operation they're not like affiliate model they're not big and flashy they are i'm assuming a pretty small group of people who are capable of doing this in a kind of more controlled, more competent, kind of more professional is what I was left thinking, which is a terrible thing to say. Like they seem more professional at it. And as a result, are able to extract some pretty whopping ransoms. Brian talks about one where they got $75 million

from a Fortune 50 company. Yeah, it's a pharmaceutical company. You just wonder, wow, what was the data they got there? It must have been important. Well, exactly, yeah. And I guess the ransomware where it's just a whole bunch of affiliates going buck wild and making heaps of mess is bad for society as a whole. And slightly more targeted, slightly more professional.

Yes, it sucks, as you say, but that's probably still an improvement. I mean, it's the difference between a junkie robbing a 7-Eleven with a screwdriver and, you know, the bank robbery crew from Heat. Yes. Which, yeah, I mean, I guess that's an improvement. Then again, I guess with the asymmetry of ransomware, I guess it's like a junkie robbing a major bank with a space weapon or something, right?

I don't know. I'm going to have to work on that metaphor a little bit. But if we're only left with the true professionals and they're not actually deploying ransomware, they're finding a way to get paid without that, I just think that's a massive step in the right direction, which might sound like a weird take, but it is. And maybe those guys are easier targets for, you know,

because who was it you were talking to? It was someone you interviewed which was saying, like, scaling up intervention by intelligence agencies, you know, is a waste of resources because intelligence agencies have other things to do, right? They've got important intelligence work to be doing and we can't just expect them to solve ransomware, you know, is a waste of resources because intelligence agencies have other things to do, right? They've got important intelligence work to be doing and we can't just expect them to solve ransomware, you know, using their special skills

and techniques and whatever. If you have a smaller group of professional targets, then it's a bit more manageable for intels and spooks and so on to go disrupt. Well, I mean, I think ultimately we need dedicated resources. Not, it was, I think that was with Chris Krebs, you know know like god no not a new agency i think is way please god no that was an interview i did with him about transnational uh him and alex damos uh about uh you know transnational cybercrime it was an interesting interview that one one of the wide world of cyber ones yes yeah

yeah exactly and um no i did find that interesting but i think we do need dedicated resources whether or not they're sort of seconded over from various existing agencies or not this idea and i think a lot of that comes from the agencies themselves who are like so busy with very serious national security work that when a national security advisor or whoever comes in and says hey we'd like you to deal with uh ransomware they say we don't really have time for this, right? So whereas, you know, in Australia, after the Medibank thing, I think the government's message to the agencies was, well, you're going to have to make a little bit of time. And they did, and they got a result there.

But I think perhaps, you know, giving them extra resources to handle this, I think it's a bit of a bureaucratic problem, really, where they, you know, they're having to divert resources from other things. Whereas if we could get to a point where they just have that little bit extra because they've got the authorities they've got the the you know their own sort of ttps for dealing with this stuff um but resources are a constraint and you know then we get into the whole conversation about government salaries and you know blah blah blah blah so it's a big topic anyway it's like no wonder we have so many listeners in the spook world

when we're here advocating for them to have more budget. Well, I mean, you know, I think it's a fair ask in this case, right? If you ask them to do more stuff, they probably need a better headcount. Now we're going to move on and talk about this story from Lorenzo Franceschi Beccaria over at TechCrunch. And apparently Bumble and Hinge, the dating sites, were exposing a little bit too much information about the distance that users were away from each other. So if you're using a dating app, it'll show you, you know, this person is Stacey or Bob and they are located, you know, 1.2 kilometers away from you or whatever.

That's the way it works on Tinder and whatnot. But apparently with Bumble and Hinge, like the information was much higher resolution in the background, and it's possible to pull that out. And if you could sort of triangulate a user, you could get a very accurate location on them. This seems like a really dumb thing for them to have done. Like, I'm actually quite surprised that a modern dating app would expose that information and let people do this because it just seems like such an obvious problem. It does seem an obvious problem. And if you compare it to Tinder, so in Tinder's case, they appear to store the location information kind of rounded up in the backend. So there isn't high resolution information available there. Whereas in these other apps,

Grindr I think was also on this list that the researchers looked at. It's rounded up in the front-end user display, but is still high resolution in the backend. Yeah, so they just do it with some client-side JavaScript or whatever, right? Yeah, or they have multiple accounts that pretend to be in different locations

and then kind of trilaterate based on that, which, you know, if you had asked when you were building these apps, if you'd asked anyone who worked in, you know, that kind of, you know, geospatial data processing or whatever, like how would you do it? Of course, you know, it's an obvious thing that they would suggest. And so it is surprising that it still works, but there's plenty of dating apps and i guess you know there's a lot of minimum viral product that makes it into prod and and here we are yeah you could whittle down

the location to something like uh two meters which is uh which is pretty crazy but mate that is uh actually it for this week's news discussion uh thanks a lot for joining me and uh yeah we're gonna we're gonna do it all again next week as we always do we certainly will and i'll look forward to it, Pat. All right, it is time for this week's sponsor interview now with Marco Slaviero of Thinkst Canary. And as you heard through the news discussion, we're talking to Marco about a few things. One of them is that blog post about unfashionable security

and about why they choose to use a single VM per customer instead of writing some sort of mega app to operate their cloud service. But he's also going to tell us about some changes they've made to canarytokens.org and also about an NVIDIA blog post, which is about using Canary tokens to spot when someone has stolen your AI models and he's trying to load and run them. So here is Marco Slaviero. I mean, we've had this architecture from the beginning and we figured this year, we want to talk a little bit more

about some of these sort of security decisions we've made. And the idea of running a VM per customer is certainly not a modern thing. It's actually quite a venerable model, but it's precisely that thing which is data is just not co-located next to each other and so pen testers like i used to do that work finding access to other people's data uh in in some kind of sass app that's just like it's almost uh like it's workaday stuff it's like you test the application i can access

someone else's data, find it in the report, carry on. And in this case, there's just no one else's data there. Like you can't, so you have your own database instance. Nobody else's data exists in that database. And that model has-

Yeah, I mean, there's a pretty long list of mistakes that an app developer can make that would allow you to jump across that boundary, you know, especially if the application is, you know, has any complexity in it whatsoever, right? Absolutely. I mean, the SQL injection bugs, cross-hat scripting bugs, logic flaws, direct or indirect object access stuff, like just all of that is ways you can access other people's data and there's tons more techniques and that data just isn't there.

And so, we've had Canary tested multiple times over our history, and we're not immune to web app bugs, but when the testers try and exploit them, they can get access to their own data. That's it, there's nobody else's data on that instance. So for us, it's been a model that's really proved itself. I mean, there's benefits. There are drawbacks.

Like it's not a magic bullet or a silver bullet here that's going to solve all your problems. Well, I mean, it solves a lot of them. You know, it solves a lot of those problems. But I think what you're getting at is it creates an entirely new set of pretty diabolical issues, which is, you know, once you've pretty diabolical issues, which is, you know, once you've grown, and this is, look, I've seen so many startups do the complete opposite of this, right? Where they're doing one VM per customer, and then they say, okay, we need a

proper cloud backend. We're a grownup company now. We need to get serious. So they spend a bunch of development money, time, effort, capital, and, you know, just pulling their hair out for a year or two to get this thing built, and then migrating everyone onto it is a nightmare., and, you know, just pulling their hair out for a year or two to get this thing built and then migrating everyone onto it is a nightmare. But then, you know, they're in a pretty good place because they're finding managing everything is a lot easier. So I can imagine that, you know, with this approach you've taken, because, you know, Thinkst is, it's a company that does quite a bit of revenue these days. It has a lot of customers. And I imagine you've got really good, right, at managing fleets of VMs because

you have to get good at that, right? And therein would have laid the challenge. Absolutely. The core with lots of these cloud architectures is that it makes it easier to operate and develop on them. We have traded the security problems of having multi-tenant stuff for the problems of operational stuff in terms of having thousands of VMs. And we're happy with that trade. Like we would rather deal with the operational pain of thousands of VMs than have to worry about all the security concerns of multi-tenant consoles. And so, yeah, the sorts of things that you become expert with, as you say, in running thousands of VMs is just like, how do you update code quickly across a fleet of VMs? You're not deploying code now to like a cluster, it's to thousands of VMs in parallel. And that's

just one example. How do you manage the monitoring on all of those, right? And so every one of those thousands of VMs is running web serves and DNS servers and other services. And we want to know when any of those go down. So we're a big user of a tool called SaltStack, which does configuration management across fleets. And so that's sort of the base that we operate with. But, yes, there's lots of custom monitoring and tooling that goes into maintaining these large fleets. Because obviously, you know, every one of these also has DNS records

and they've got pions to, you know, third parties to do things like send emails and text messages and all of that. And all of that is a per-customer thing. And we have to manage all of those artifacts so yes there is a team an infrastructure team where in maybe other organizations you won't need like a full infrastructure team we need an infrastructure team to help us through this yeah because the companies that i can think of off the top of my head who have to deal with these sorts of issues it's like apple running all of the iCloud service it It's Netflix running, you know, like a gajillion boxes.

It's not usually like a mid-sized security company like Thinkst. You know, normally the team that runs something like this is bigger than your whole company, right? So what are some of the curly moments you had when you were actually building all of this? Because I can imagine there would have been, yeah, moments where you had sudden awful realizations like things being out of sync or just issues popping up just random stuff you know as i said we rely on salt stack and salt upgrades are always a moment of uh some trepidation

i guess like we're always cautious as we do it we've never had it completely blow up uh where we've had to, for example, restart or rebuild servers. That's not a thing. Although actually- So you've always been using this tool? There was never a time before you used it? You like started off this way?

So as in our team, when we first got started, pushed us towards Salt and it was, I still thank him today for that because prior to that, when we had other services we had bash scripts and doing that sort of thing and as at the beginning said no we should salt

stack this we did and we're still strong users of it um we we do have to on occasion also do instance upgrades so you know we run ubuntu we've had to move through Ubuntu major versions, and we've had customers who we've taken from, you know, Ubuntu 14.04, it was released 10 years ago, and sort of, we don't run the same instance as that one are things we have to be very careful that we're not losing data or letting customers be offline through the upgrades. But still, those are operational pains. They are not security pains. And for us, that's what makes that trade worthwhile. Like, we back ourselves to be able to manage and operate this fleet. And it means that we can effectively outsource our security,

the hardest security boundary we have, to it's their hypervisor um and so they've got a strong team yeah and if someone's got an ode for a hypervisor bypass for aws like you uh i mean maybe you're not the last person they use it on but you're definitely not the first definitely and and even and even like if you keep chasing that attack tree all the way down to that that leaf node and go we are the target you still have to co-locate on the actual physical hardware that one of our customers is running on and that's a tough problem in and of itself so yes this is the trade-off that we've that we've gone with and that we're really happy with now look another thing we want to talk about real quick is that you've just revamped the canarytokens.org website.

Canarytokens.org, of course, is where you can go to get free Canary tokens and use them as much as you want. It's a fantastic public service. And I think the amazing thing about it is, yeah, I mean, it's been going for so long and it's been reliable for so long. And all of that alert plumbing is built in, all of that reliable alert plumbing.

You know, it's absolutely terrific. So anyone who doesn't know canarytokens.org should definitely check it out. The funny thing is, though, I would not really have described canarytokens.org as particularly visually appealing. You know, it was sort of the website equivalent of a command line tool, I guess. But you've actually just redone it. You've had your UI team, it thinks, go over it and make it nice and pretty. Yeah. So, I mean, we haven't touched that UI in about seven years. And in fairness, the UI that we had before was strongly influenced by me. And that's definitely not my

strong suit. So we've got a front-end team. They have taken a crack at it. They've redesigned it. And essentially what they've tried to make it better for is both finding and using new tokens if you're a new user, but then also for current users and those who've used it before, it should be a lot easier to navigate.

And it's just more consistent. they've also rewritten it but that's less interesting to the actual users of the site but we're pretty excited for what they've done it gives us a good platform to keep building and make changes uh going forward that are are going to be quicker to roll out so check it out canarytokens.. Do you see what I'm saying about how it was a little bit like a website that was like a command line tool, you know, just like a, it was very sparse, you know, an input field, a dropdown menu, that kind of thing. I'm not trying to be cruel. I'm just saying. Absolutely. As I said, look, I mean, when we, when we put it out again,

having no real front end experience, we looked around and found bootstrap and actually it was written with jQuery at the same time. So like bootstrap and jQuery 10 years ago, it looked exactly like a Bootstrap and jQuery website. It didn't feel any different because it was. But now it is a lot prettier. So yeah, check it out. Now I'm also going to throw a link in this week's show notes to a blog post by NVIDIA and they've been talking about how they're using canary tokens to uh like you can set up a fake model like a pickle file an ai model and

then you can stick a canary token and when someone loads that file that token fires and that's a really good way to see if people are rummaging around in your infrastructure and trying to steal your ai models so i've linked through to that one but Marco I believe you are also at Black Hat this week yes we've got a crew at Black Hat so we've got a booth if you're around if you don't know about us feel free to drop past we do live demos

it's our thing if you do know us and you still want to drop past or just catch up we'll be there so booth 874 and we'll be at Black Hat. Marco Slaviero, thank you so much for joining us. And big thanks to Thinkst for sponsoring this week's episode of Risky Business. We'll catch you next time. Pat, thanks very much.

That was Marco Slaviero there with this week's sponsor interview from Thinkst Canary. And of course, you can find them at canary.tools. And that is it for this week's edition of the program. I do hope you enjoyed it. I'll be back soon with more security news and analysis. But until then, I've been Patrick Gray. Thanks for listening. Thank you.