Risky Business - Risky Business #756 -- Move fast and break everything

Episode Date: July 24, 2024

The Risky Biz main show returns from a break to the traditional internet-melting mess that happens whenever Patrick Gray takes a holiday. Pat and Adam Boileau talk through the week’s security news, including: Oh Crowdstrike, no, oh no, honey, no AT&T stored call records on Snowflake and you’ll never guess what happened next Squarespace buys Google Domains and makes a hash of it Some but not all of the SECs case against Solarwinds gets thrown out Pity the incident responders digging through a terabyte of Disney Slack dumps Internet Explorer rises from the grave, and it wants SHELLS RAAAAARGH SSHHEEELLLS And much, much more. This week’s show is brought to you by Sublime Security, a flexible and modern email security platform. If you’re sick of using a black box email security solution, Sublime is a terrific option for you. Show notes Risky Biz News: CrowdStrike faulty update affects 8.5 million Windows systems Low-level cybercriminals are pouncing on CrowdStrike-connected outage | CyberScoop CrowdStrike says flawed update was live for 78 minutes | Cybersecurity Dive Crooks Steal Phone, SMS Records for Nearly All AT&T Customers – Krebs on Security Researchers: Weak Security Defaults Enabled Squarespace Domains Hijacks – Krebs on Security Teenage suspect in MGM Resorts hack arrested in Britain Majority of SEC civil fraud case against SolarWinds dismissed, but core remains | Cybersecurity Dive How Russia-Linked Malware Cut Heat to 600 Ukrainian Buildings in Deep Winter | WIRED Kaspersky Lab Closing U.S. Division; Laying Off Workers Hackers Claim to Have Leaked 1.1 TB of Disney Slack Messages | WIRED Wallets tied to CDK ransom group received $25 million two days after attack | CyberScoop UnitedHealth’s cyberattack response costs to surpass $2.3B this year | Cybersecurity Dive Ransomware ecosystem fragmenting under law enforcement pressure and distrust Threat actors exploited Windows 0-day for more than a year before Microsoft fixed it | Ars Technica

Hey everyone and welcome back to Risky Business. My name's Patrick Gray and we've obviously got a great show for you today with everything that's been going on. We'll be talking about the CrowdStrike thing, the AT&T Snowflake thing. So much happened while I was away, which is usually how this works. This week's show is brought to you by Sublime Security, a company that I do some work with. Sublime really represents the future of email security. It's a modern platform that started out as a less black box solution for companies that wanted to be able to control their mail filtering and do more advanced things with their email security platform than just kind of hoping their upstream provider would catch the right stuff. But these days, you know, Sublime is so effective out of the box that they're now competing against the majors and are in fact winning deals against the majors.

But Sublime co-founder Josh Kamju will be popping along in this week's sponsor interview, as will a Sublime customer. JJ Aga is the CISO at FanDuel, and he'll be talking to us about how they're doing some cool stuff with Sublime, like a Truffle Hog integration via Times, which is actually really interesting. So stick around for that. That is coming up later. But first up, of course, it is time for a check of the week's security news with Adam

Boileau. And Adam, we managed to squeeze in a proper planet melting disaster on my last day of leave. I was a little bit worried there that the AT&T breach was going to be the biggest thing that sort of came to light while I was on leave. But no, we managed to keep the curse alive. The CrowdStrike outage was just, I mean, what do you even say?

I mean, it was in some respects a beautiful thing like i love a good internet you know internet wide drama and this really hit the sweet spot because it was big enough that it affected heaps and heaps of stuff i mean if you were flying very near around that like it's lucky you made it in before the planes were man i got i we flew in the day before this happened it was just so lucky so so lucky but yeah it's like big enough that everyone's got skin in the day before this happened. It was just so lucky. So, so lucky. But yeah, it's like big enough that everyone's got skin in the game, but small enough that all the channels for sharing memes about it still work. Because there's nothing worse than having everything melt down

and then not being able to gossip about it with your InfoSec pals. So this was just right, like really hit the sweet spot. But boy, oh boy, what a a mess i mean just just how quickly it spiraled right so when i when i when this happened i'd actually just been talking to dimitri like right before it happened and yeah dimitri alperovic co-founder of uh of crowdstrike but hasn't had anything to do with the company in years like he left in 2020 he resigned and you know has nothing to do with with crowdstrike anymore but yeah it was just funny that I happened to be talking to him and then then it all kicked off and I'm like um are you still awake you know Dr. Frankenstein wake up there's a problem with your creation sort of thing

uh but you know we went from and of course Australia was a place where this happened during business hours right so we were like among the first countries in the world to get this. And it went from, okay, mysterious blue screens popping up to, I mean, you know, you want to talk about, gee, that escalated quickly. All of a sudden it's like flights are grounded and then flights are being ordered to land and just hospitals are diverting patients. I mean, it was just the scale of it was incredible. And you really realize just quite how important a company like CrowdStrike is when even though they're only touching 1% of Windows boxes, apparently, that's the number that's come out of this. They're quite often among the most important 1% of boxes.

Yeah, exactly. It was certainly interesting to see who was a CrowdStrike customer and who wasn't as a result of this. Because much like in Australia and New Zealand, was you know coming towards the end of the day I'd actually just gone out for drinks and dinner and my bank is a CrowdStrike customer and they were all dead in the water as well so they go check my wallet make sure I have some cash to pay for it um but yeah it was really interesting and that uh kind of the subset of people who run Windows that ran CrowdStrike you know it was really interesting and important stuff and I was surprised when we started to see the numbers you know once the you know sort of things

had washed up that it was like eight and a half million windows boxes because it certainly felt like more than yeah eight and a half million at least for us in you know places like Australia and New Zealand and UK and US you know other parts of the world experienced it differently of course I mean talking talking to my wife, and she's from Brazil, and I'm like, geez, what's the impact in Brazil? She's like, well, it's not really commonly used over there, so there hasn't been much impact at all, right? She jokes.

She's like, lol, we can't afford that stuff. Well, exactly right, because CrowdStrike is pretty good as EDR products go. Priced accordingly. Yes, it's priced accordingly yes it priced accordingly and yeah it's funny seeing you know for example which banks and which transport companies and which you know supermarkets in some cases can afford to run CrowdStrike on their point of sale terminals like in the checkout I know like our local supermarket like I went to the supermarket the

next day and you know everything was working fine right so but but I would say you know seven out of ten of the self checkouts were still down i took photos and what was interesting is like they had different errors on all like it was just an absolute cluster right like an absolute disaster but i was relieved that i was able to do my shopping the next day because i did do an interview on abc radio uh on the the day that it happened and you know when you're being asked to comment on this stuff when information is real thin, the questions I was asked were really interesting though because, you know, ABC is a CrowdStrike customer, right?

So the guy who was interviewing me was operating with a mobile phone and a CD player. That's how he was broadcasting. And, you know, by the time they got me on air, they had me plugged into something, but he had actually been doing interviews with like speakerphone in front of the microphone like that's how how much it was um impacting them but that also meant that when they asked me like oh you know should people try to find functioning atms and get cash and whatever i i got to say look just calm down like payments you know payment infrastructure and like supermarkets and and

stuff like that's going to be the first stuff to come back you know but if you work in the marketing department of a major enterprise like when you come back to work on monday you're probably not going to be able to use your computer which was which was sort of my line but i'm like yeah just don't worry about it and people find workarounds and you know i got to use the point well look your computer's not working but here we are doing this interview, right? So I think my exact words when he was talking about the ATMs, I'm like, let's not encourage bank runs just yet, you know? But there was a real sense of panic around this one. Yeah, there really was.

And it kind of reminded me of some of the other times we've seen big internet outages. Like I was thinking like ping of death back in the like late 90s early 2000s whenever it was like sitting around an isc watching everyone dropping off and kind of the same kind of feeling of just you know stuff falling off the network i opened up you know one of the flight trackers and looked at the amount of planes flying over the us you can gradually see it gets sparser and sparser as stuff lands but doesn't take off and you, you know, that kind of, you know, it does feel a little apocalyptic. Yeah.

But on the other hand, like, it's amazing. Eight and a half million Windows boxes did this. And can you imagine if it was every Windows box and they were bricked properly? Like, I mean, this was bad because you had the safe mode. Yeah, so that's the thing, right? Is there, I mean, it's the remediation to this is both terrible but you

know also could have been so much worse in that to fix it you need to you know just delete a file so so look why don't we talk about actually how this happened uh because what it looks like happened there's there's been a lot of bad info on this out there as well like um you know and to a degree that doesn't even matter but i'll get to that in a bit. But what happened is it wasn't actually a patch. It wasn't an update, you know, to the software itself. It was something that CrowdStrike call channel files. And what these channel files, these channel files are extremely powerful. So they issued an update and these things go out like hourly. Okay. So it's what they'd call a content update as opposed to an update in the code but so much of CrowdStrike is runs as a kernel driver right so so much of of the software actually runs in the kernel so it appears that what they've done is sometimes they'll they'll issue like a

content update but that results in some of that in in changes to in new code being executed in a kernel context right and obviously if you make a mistake there, so it's one thing to say, well, it was just a, you know, channel or signature update thing. And apparently this was to do better C2 detection for some new types of C2 that they were looking at. So it's one thing to say, oh yeah, it's just a content update.

But when your content update results in P code being, you know, stuffed into the kernel and executed there, I mean, it kind of is a software update at that point right so and and they're calling it a logic bug so so something obviously went wrong with this update i mean it was confusing in that the the file in question is a dot sys so people thought that that was a kernel driver crowdstrike just happened to name these these content files dotys, so they're not actually kernel drivers. But regardless of the details, the thing that you keep coming back to, right,

when you look at this, is given that this was a 100% reliable blue screen of death against every machine that it touched, the only conclusion you can draw is, A, it doesn't matter what the problem is, because, B, they obviously didn't do any testing. Like, zero testing. Any testing.

And, you know, you can talk to people who say, well, testing is hard and whatever. Not in this case. 100% reliable, instant blue screen of death. And you just think, what the f***? Yeah, really, right? I mean, if you are able to ship those kinds of updates in near real-ish time into people's kernels regardless of whether it's code or data or how you package it up

whatever like that's a thing that you have to do responsibly with testing with i mean you've got to give them credit for their update infrastructure though the fact that they could hit every single crowd strike box on the planet in minutes is like, wow, good job. That's good, I guess. That's some good engineering. Yeah, yeah. However. Yes, like clearly this is not a situation that can happen again, right? And like in general, everybody else who runs some kind of Windows product that uses a kernel driver or has the ability to push code into the kernel, data into the kernel through some mechanism is going to be staring at this and their customers going, well, why can't this happen to you?

Explain to me why. Yeah, but like CrowdStrike's f***ing it up for everyone and this is what's driving me nuts. They absolutely are, yes. So I've been talking a lot to the Airlock Digital guys, right, because they operate with a kernel driver. Their kernel driver is 60 kilobytes, and it doesn't just take random stuff and execute it in a kernel context.

They've got a really simple driver, but they're already getting the emails, and prospects are getting nervous because, oh, no, you're a kernel driver, so they're having to do some comms around that. I guess they can't just say, our driver is written by windows kernel rain man so basically right well and audited by silvio cesare and like they got consulting from him very early on and they worked with him to try to pull as much as they could out of the driver whereas crowd strike seems to have

the opposite approach which is to just do everything in the kernel and I think that you know their kernel driver just their bare bones kernel driver is like 4.3 meg right and yeah and the way the product works right is like it takes new stuff in and like you know so it's it's it's it's yeah it's crazy right like the way that they run it and that's fair enough they've run it for a long time without any trouble but again I just keep coming back to clearly they didn't test it that is just insane it is just so insane like something's gone seriously wrong and their comms around this have been absolutely appalling yeah it has not been pretty for them at all uh i actually really enjoyed um like some of the social media posting around this was great but the thing that

i most enjoyed was someone pointing out that the current ceo of crowd strike yeah was ceo of mcafee when he was cto he was cto at mcafee when they did something similar yeah yeah when they shipped the bung update that you know quarantine some bit of windows or whatever else george curse so we got cursed twice twice in a row so maybe not the third time like if you go somewhere else after this, don't buy kernel products from them. Can I tell you something crazy about the McAfee incident though, right? Is after that happened,

they wound up having meetings with all of their customers, right? Because they were furious. But as a result of those meetings, they demoed new stuff and sales went up. They had an excellent couple of quarters. And you would remember that when I wrote that story about a breach at a very popular pizza chain in New Zealand many, many years ago, they had an excellent couple of quarters and you would remember that when i wrote that story about a breach at a very popular pizza chain in new zealand many many years ago i later spoke to one

of the owners of that business and their business was through the roof because everyone in the media was talking about their products yeah yeah so this could be good right this could be good for crowd strike right yeah i mean everyone knows about crowd strike now people who are even outside the infosec industry so yeah maybe maybe their stock is going to the moon but ultimately you know there's a couple things that come out of this first of all crowd strike absolutely made a terrible mistake and it's baffling like it is it's actually baffling like how this happened uh and i really doubt they're going to give us a full accounting because there's no version of this in which this looks at all like,

oh, okay, I can see how that happened. Like there's no version of this where it looks reasonable. Just forget it. So that's one thing that comes out of it. So I don't know if maybe the Cyber Safety Review Board is going to have a look at this. Yeah, mind-boggling. So that's one element of this.

The other element is, well, what now, right? So we saw Apple engineer an API for endpoint security, right? So again, going back to Airlock, that's what they used to implement allow listing on Mac. There are some similar things in Windows, but as best I can tell, all of the EDRs are still using kernel drivers. I think even Defender's using a kernel driver. And then there's the legal complications around

this, right? Where Defender needs to be on an equal footing with competing solutions because Microsoft is in the security business. They can't give themselves an unfair advantage. There's even been a ruling in the EU which says, no, you can't go kicking your CrowdStrikes and whoever out of the kernel when you're using it because it's an unfair advantage. So the obvious resolution here is going to be for Microsoft, you know, perhaps in the next version of Windows to end this by creating a well-documented and powerful enough API

that can be used by endpoint security solutions to do what they need to do. Because they can't do it now. Like people saying, oh, they just need to end this kernel driver thing. They can't do it. They're going to have problems in the EU and I think it'll even be an FTC problem, right?

If they kicked out CrowdStrike while allowing Defender to have that sort of access, it's just a massive competitive issue and they're going to get spanked for it. Yeah, no, I agree. I think that is probably the long-term solution. Microsoft will provide a better API and we've seen them make these kind of changes, you know, like printer drivers, for example, used to be, you know, you just let the manufacturer do whatever the hell they want and now there's a more robust interface for it and things are a little more sensible and you know a lot of this like the way that antivirus and other security products worked in windows like it's mostly historical it's because this stuff

predated microsoft providing defender providing their own plumbing for this sort of thing and vendors had to bring their own firewalls or whatever else when they're implementing it and so it's kind of historical and clearly it's time for microsoft to build a you know an interface for security monitoring for security products well i think i think they've largely built it i think they have built a lot of this already but i'm not sure what the state of the documentation is like i'm not sure how robust it is like i know when people have tried to use various microsoft apis that should work okay like they're often buggy because not many people are actually using them so they need to get real and do the work i mean i'm sure that crowd strike would have been operating under the assumption for many years that they could get kicked out of the kernel um

at kind of any time right like if you're running a business like that that's worth like roughly $80 billion and it's predicated on having that sort of access, you know, you're going to be paying attention to that. Yeah, it's going to be on the risk register. Exactly, right? And I know it is for the Airlock team. So yeah, I think that's where this has got to go. But, you know, Microsoft being Microsoft

and kind of dysfunctional when it comes to making big changes i just i i don't know what's going to happen here yeah i mean they do take security very seriously now uh so this is a chance for them to show that rather than just say it but don't you think this is where like an organization like the csrb could come in and actually make those sort of recommendations like that's kind of i kind of think even though this wasn't a hack or even though it didn't involve a security vulnerability this is precisely what csrb should be doing yeah and we certainly seen plenty of people make the argument that availability is one of the important bits of security so i mean i could see the csrb

digging into this um and you know that like the whole way that real-time updates are pushed out and managed and setting expectations about what's like table stakes to have that ability to update your software regardless of whether it's in kern whether it's somewhere else you know like what should you do what's the best practice like that would be useful to see spelled out um and you know put in front of software vendors so they know what we expect of them yeah i mean it was just surreal right like in you know a few hours after this the prime minister and put in front of software vendors so they know what we expect of them. Yeah. I mean, it was just surreal, right?

Like in a few hours after this, the prime minister will address the nation. Yes. Amazing. And what do you think this means for CrowdStrike? Because I personally don't think this is really going to hurt them that much. I think they're oversold at the moment. That's just my personal opinion.

This is not financial advice, but yeah, we've seen them take a big hit. And it's just my personal opinion this is not financial advice but um yeah we've seen him take a big hit and and it's just i don't know i don't i don't know that this is going to really damage them because doing a rip and replace on edr it's a pretty heavy lift and i wouldn't think this is going to happen again i mean i would doubt it would happen again um and that's one of the nice things is going to a vendor that's had a disaster and then fixed it reduces the likelihood they can do it again. I mean, I think when people's CrowdStrike contracts come up for renewal, like price hikes might be difficult to swallow. Like they may have to eat some, you know, not ideal pricing for the next couple of, you know, of renewal cycles.

But like, I don't see them going away. And CrowdStrike does do good work right they do you know especially their incident responders and so on like they're pretty solid this is a sign isn't it that they've taken their eye off the ball because it's 2024 like i again i'm just it's baffling i don't understand how this could happen you know it's it's i'm confused like i don't get it how on earth can you do this in this day and age? I mean, I think this is what happens when you kind of forget your roots, right?

You forget what it is that you actually do and, you know, get a bit big for your britches, focus too much on marketing and puffery and so on. Speaking about being too big for your britches, like, you know, you look at the comms around this and it's like, we can't wait to get back to the mission with you to keep defending you and you know it's just the comms have just been insane i don't know the whole thing anyway like that we saw that release about like well it was only live for 78 minutes yeah like that somehow makes it better yeah like if it was

alive for 10 hours would it have made a difference? Yeah, probably not. Hey, the weird thing was, though, some boxes I think were actually able to come back on their own. They were getting networked just long enough to be able to get, but it depended on your hardware and how powerful it was and whatever. Yeah, like how fast and what sort of network, how fast you DHCP, do you have IPv6, how fast do you DNS? Yeah, people were saying, like, you've got to plug in via Ethernet, don't use wireless, Ethernet comes up quicker.

And, you know, you've got to reboot like 15 times, seems to be about the median. Yeah, yeah. Oh, it's just nuts. Oh, dear. It's absolutely nuts. I feel bad for all of the IT nerds that got to spend their Friday night

safe-moding and deleting a sys file. Especially the ones who had to get the BitLocker keys, right? Well, exactly, right? Because, I mean, how many people had never exercised BitLocker key escrow process or whatever else? Well, and the box with all of that info is probably down as well. Well, yeah, you know? And this wasn't malice, right? This was incompetence.

But can you imagine if it was the exact same thing except that before it rebooted it set the bitlocker key to an unknown like to a random value and threw it away yeah which if you're in the kernel which you could do you could do that yeah i mean i you know the same thought occurred to me and we briefly touched on this in uh in our internal slack where you know we do all this talking about like vault typhoon, right? I mean, if you're an adversary at this point, like a state level adversary, you've got to be thinking, geez, this would be a nice way to cause some chaos. Yeah, hell yeah.

And like that's one company you've got to break into and get access to get you into, you know, eight and a half million super useful boxes versus having to break into hundreds and hundreds of organizations to preposition. Like, yeah, you you know if you have vault typhoon or similar this is exactly what you should be prioritizing all right well look let's move on because we've been talking about this for a while now and um you know we do have uh more to get through in this week's show but yeah just just wow yeah and uh you, anyone with thoughts on whether or not you're going to be cancelling your CrowdStrike licenses, you know, tell me on Mastodon. I'm a risky business at, you know, on infosec.exchange.

I'm real curious to hear what people are going to do about that. The other big news that happened while I was away was AT&T revealed that they'd had like a massive amount of data exposed and it looks like via Snowflake. That's the thinking. But we're talking about call and text records like CDR, like call detail records on 110 million people for a period between May 1 and October 31, 2022, as well as for some reason on January 2, 2023. So all of this data looks like, you know, I think the operating theory here is one of the staff stuck it in Snowflake with a non-MFA account

and it got pinched via this whole incident that, you know, I was saying was going to turn into something akin to the, you know, the Move It campaign. I mean, that's sort of how this is this is shaping up but on a much bigger scale because like people tend to put more serious stuff in snowflake yeah i mean that's uh that's exactly it like call data records for that many people is like you can start to do some interesting analysis with it like there's a reason spooks

you know intel agencies like to bust into telcos to collect the stuff because metadata analysis of who's talking to you who and who's texting who like that's really interesting um i think there was also in some cases details about the cell towers that were associated with the origin and destination numbers which lets you figure out roughly where a particular phone number is at a point in time. So that's interesting as well. And we also saw reports that AT&T paid the people who nicked it out of Snowflake to delete it. Yeah, $370,000. And again, like, let's, exactly, right?

So let's talk about the idea that you're going to ban ransoms. Should AT&T not have paid this? I mean, the problem is, of course of course like how do you know it got deleted in this case they got a video of it being deleted so that's nice but it's also about credibility right like no one wants to ruin it for everyone you get your money you delete the data because that means next time you steal it you've got the credibility for having deleted the data previously right like it's basic you know it's business 101. So I do believe that, you know, odds are that data's been scrubbed. $370,000.

Should they have left it up? Like the risk to privacy, like even if there's a 20% chance that they will follow through and delete, it's worth the money. Yeah. I mean, I agree. Like in this case, probably they should have paid and the fact that they did, yeah. I mean, causes, I mean, I had a quick look

because we were talking about whether this felt like actual cdrs so like numbers and times and destinations or whether that was kind of like a whether it was something kind of more fluffy sort of thing i went looking for the data of course you know it's not around to be found so i guess their three hundred thousand dollars did the job because i couldn't find it in 20 minutes of googling so yeah you know I guess mission accomplished there. I mean, this is a awful problem though, right? When you've got, so it's one of those theoretical problems

that we've been talking about in security for a long time that's finally become a tangible problem, which is this issue of like, because we always talked about shadow IT, shadow SaaS is kind of a problem. And, you know, you can do things with shadow IT when people are plugging into the corporate network you're going to be able to find that stuff the only real way

you can find shadow sass is through the browser and there's not really many companies who do it like i work with one of them which is push security and obviously this has been a pretty big event for them uh this whole snowflake thing because the first thing that they do is you know they're a browser plugin and the first thing that they'll do is tell you where, you know, all of the SaaS accounts are that you might not have known about otherwise. And as a plugin, they can do stuff like force your staff, if they're going to use that stuff to enroll in MFA, for example. They've got phish kit detection and stuff too, which makes the staff being fished at their corporate workstations much less likely as well. So there's just a lot of stuff you can

do in the browser. I'm sure Island are going to build similar features, but it just goes to show you that really we need to be building more tooling, more security tooling into browsers to deal with this sort of stuff yeah absolutely right i mean and sas providers are incentivized to make it super easy to try out to on get on board to start doing something i mean i was using paper trail for something the other day i needed to log some data from somewhere and i'm like i needed this to be on the cloud i don't want to spin up a whole syslog and blah blah blah i'll just go to paper trail free trial you know no credit card or nothing and i've got data in their cloud now uh it's just you know that's their shtick make it

easy to do and so of course we end up with data there behind 1fa and you're right the only place to spot this is in you know in the browser the new operators and the new kernel which is inside your browser and you know three years from now we will be talking about how an update to a browser plugin, you know, bricks everybody's browsers. Yeah, maybe, right? It's funny though, because they can do some pretty diabolical stuff, right? Like you can prohibit people from like signing up for

or logging into accounts that don't have like corporate domain usernames. Yeah. And you can do stuff like, oh, they haven't enrolled in MFA. You can throw them a banner or a splash. And then after three more logins, you can just actually force take them to the MFA enrollment page and not let them leave until they've done it. So I don't know if that's a live feature yet, but yeah, certainly the browser's where it's at. I think, you know, you're going to have a bunch of different options if you want to go for the full rolled gold you know security oriented browser something like island great use cases there but not for everyone and kind of expensive

as well because it's it's not you know like a plug-in based solution which is obviously going to be a lot cheaper so it's going to depend on what you're trying to achieve and whatever but you know the days where we can just roll edge and call it a day like and and i saw you know microsoft's hey look at us we're doing an enterprise browser and it's like yeah you can extend like office-based dlp into the browser and like that's their idea anyways i guess baby steps but so far i'm underwhelmed by what they've done so i think it's going to be a big area over the next few years and something that cso's listening really need to think about which is you know how to get some sort of presence in the browser yeah it's a place that you need visibility and snowflake is such a

perfect example of why because that's turned into a real mess yeah yeah it has now the other thing that uh happened while i was away was this Squarespace schmozzle. Now, Google domains, right? I transferred risky.biz to Google domains because most domain registrars are just terrible, creaky, insecure, horrible. Because there's just no money to pay for doing it well. Exactly. So I'm like, Google's running domains. This is going to be fantastic. So I was in there for many years, big smile on my face. I'm OAuthing in. It's great, right? It's all integrated. It's wonderful. And then Google being Google just say, yeah, we're getting out of the domain business, right? And they sell it off to Squarespace. And it looks like a bunch of people, like under our circumstances, a bunch of people

who wound up being Squarespace customers through this. And I just got the email a few weeks ago saying, congratulations, your domain is now with Squarespace. I'm like, okay, I'll deal with that later. But it looks like there was some sort of auth flow problem that allowed attackers to hijack domains hosted with Squarespace. And in the case where Squarespace was the reseller for that customer's access to Google Workspace, it looked like they could um get admin

there and i know that squarespace has disputed this there's a bit of to and fro on this uh but like walk us through what we actually know happened here adam so they migrated the accounts that's they migrated domains across to squarespace then they i guess made accounts at Squarespace for customers that didn't already have it that then OAuthed back into Google. And at some point in that process, like if you showed up and tried to log in with an account that was in that state that had never been logged into but had OAuth, there was some way that you ended up not needing to be authed and it just dumped you into the control panel and you could edit people's zones and onwards from there. A number of cryptocurrency companies had their domain records modified to have attackers in the middle

and then crypto was stolen and sold, which is kind of how we ultimately found out about this happening. Square Squace was pretty quiet, I think until a couple of days ago they put out a statement about it and the details are unfortunately still a little bit murky well they're very keen to tell you what didn't happen they're not at all keen to tell you what did happen yes which is

what i noticed out of this this statement right so they say during this incident all compromised accounts were using third-party oauth you mean like me with my google oauth right like google is a third party in this case. That's not helpful. Neither Squarespace nor any third party authentication provider made any changes to authentication as part of our migration of Google domains to Squarespace. Okay, fine. Did anyone say that that was what was going on? And it says to be clear, the migration of domains involved no changes to multi-factor authentication before, during or after. Well, yeah.

Okay. And to date, there's no evidence that Google Workspace accounts were or at risk. So what happened? Yes, yeah, exactly. We're still unclear because we saw a number of theories around being able to log in with just an email address and then so not going through OAuth for accounts that have been migrated like this. Yeah, so if you guessed the correct email address for the account that was going to

OAuth and then signed up through a password-based flow, then you could maybe get it. But it's, you know, and they're saying that's not what happened. But they choose their words very carefully. They're saying our analysis shows no evidence that Squarespace accounts using an email-based login with an unverified email address were involved in this attack. So, I mean, so how did this happen? I don't know. How did this happen? I don't know. How did this happen? The answer is we still don't know despite the statement, which is frustrating, you know,

when you could have probably hacked us with this. Yeah. So that's not great. So thanks, Squarespace. We really enjoy being a customer involuntarily. Now, James Reddick at The Record has a report up. Let's just get into some actual news now. A teenage suspect in the mgm resorts

hack has been arrested in britain this comes after the arrest of a british national in spain a few weeks ago who was said to be the uh the the ringleader of this so yeah these guys are going down i mean at the time i thought it was going to take 10 minutes it has taken you know something like a year but it's all uh but it's all happening yeah i guess the international aspects of this probably do make it more complicated and make it take time but yeah they're wrapping these kids up and this guy was what 17 i think so like literally a kid um but yeah they're getting wrapped up and we'll see you know how the how the rest of their prosecution goes well i can't wait for this guy to be arguing in a british court that he has crippling aspergers

and cannot possibly be extradited because that seems to be the playbook. That is how it goes. Now, in news that's big news for CISOs, there's been some serious changes to the SEC complaint against SolarWinds. So the federal court has tossed a lot of the complaint. And a thing that's really significant here is they've thrown away a the part of the complaint that would have sort of uh that would have established that the sec has a say in how people secure their networks so the federal court's like no you

aren't really the regulator that can say how people should do this. Like you can't say people didn't adhere to NIST when it's a non-binding framework and like, what are you talking about? So there's 107 page opinion and order. A lot of it's been thrown out. It's also thrown out charges stemming from comments made by the CISO on things like podcasts and whatever. Thankfully, right? We want people to speak freely on podcasts. So they're saying, look, they were just flapping their gums you know that's no big deal but importantly uh a couple of key things have remained and are going to go forward um the security statement that was published to solar wind's website which was quite detailed saying oh we use modern devsecops and i can't remember exactly what it was but it was something along those lines that was like very sort of specific

in terms of what what security measures they used and how they kept the customers safe and whatever um and meanwhile in discovery they turned up a whole bunch of people internally saying this security statement is complete nonsense like what are you talking about so that looks really bad and that's the part of the case that's going to continue which basically says that the security statement was you know a reasonable investor you know could have looked at that security statement was, you know, a reasonable investor, you know, could have looked at that security statement and made a decision based on it. Therefore, you know, as Matt Levine says, it's security fraud, everything is security fraud. So CISO is still in trouble, perhaps not as much trouble. But at least now we've sort of got a

more tightly scoped understanding of what can get you into seriously deep when it comes to communicating about security to the market yeah i mean the idea that uh security controls you know information security controls were accounting controls one of the things they've kind of thrown out a little bit here but i i did like you know this idea that we can't like if you make a statement that's too detailed like it has too much information then you know it better be that we can't, like if you make a statement that's too detailed, like it has too much information, then, you know, it better be correct. So we all have to stick to making

very boring generic statements about taking your privacy and security very seriously because at least then there's no, you know, specificity in those. And then the other thing I guess is, you know, we can't really talk smack on, you know, in Slack or Teams or whatever else anymore because, you know, it also has to be true.

So in this week's show notes, I've linked through to a blog post from the law firm Cooley. And it's so funny because their takeaway from this is like, we can help you devise a framework so that these sorts of communications don't surface. And it's like, maybe the advice should be just don't lie in security statements. I'll read the paragraph.

The claims surviving the motion to dismiss are based in large part on alleged inconsistencies between the internal communications of SolarWinds CISO and security team and the content of their security statement. While open and frank communications about security challenges are necessary, informal, sloppy, or inflammatory communications can be harmful and ultimately ineffective.

Now, more than ever, security professionals need to know how to appropriately communicate to achieve their objectives. This not only increases the effectiveness of security teams, but also reduces the risk of liability, including personal liability of security professionals. Cooley has developed training specifically targeted at security professionals to foster clear, concise, and complete internal communications on cybersecurity vulnerabilities and priorities. It addresses communicating using informal challenges, drafting appropriate security reports, blah, blah, blah, blah, blah, blah.

So basically their advice is coming back down to, you remember that clip, shut the up Friday? Yes. Yes. That seems to be Cooley's thing, which is when you want to get into Slack and blow off some steam and talk about how the security statement published to your website is completely full of shit, that might wind up being evidence one day and you probably don't want that. Or maybe you do. Or maybe you do. I mean, that's the thing, right? Like, yeah.

Yeah. I mean, maybe the message is if you start whining about this in Slack, now they have to take it seriously. Yeah, maybe the message is if you start whining about this in Slack, now they have to take it seriously. Yeah, maybe, maybe, yes. Put that in your pipe and smoke it, lawyers. Andy Greenberg's got a write-up based on some research out of Dragos. I think we covered this one as well in Risky Business News. Catalan did a terrific job, by the the way with his CrowdStrike write up.

If you're not a subscriber to Risky Business News, go to risky.biz, slap your email address into the subscribe box and yeah, he did a terrific job. But yes, this story is about some research out of Dragos, which has looked at some ICS malware that was used to take down a heating supplier in Ukraine in the middle of winter. Just very Russian, very nasty.

But it's an interesting write-up. Yeah, it is. It's a mean attack and an interesting write-up. So the city of Lviv, about 600 apartment buildings had kind of like central heating delivered by a utility. Some Russian hackers bust into the provider of that network we believe through a microtech like router vulnerabilities uh and then um used a piece of malware that well they used a piece of malware that talked modbus to go out and talk to the controllers and have them report

that the temperature of the like water or the steam in the pipes was very high, so the systems would cool it down, and so people were being delivered intentionally cold water and cold heating in the middle of winter. The way that this was discovered was actually a little interesting, though. So Dragos found a sample on VirusTotal of a piece of malware that could talk direct Modbus. And, of course, they're interested in ICS

malware and started pulling it apart. And then independently, the Ukraine's cert team were investigating this incident. And eventually, they managed to figure out that these two things were related. The Dragos had a configuration file, which mentioned some IP addresses that were relevant to this particular utility but in the intrusion the attackers appeared to not actually drop this malware on the target network they were actually SOX proxying like tunneling in and talking directly to the mod bus from Russia they set up a you know tunnel in from Moscow and so they hadn't seen the malware in the target environment, but Dragos had found it independently through Firestone. So that was kind of cool.

Yeah, yeah. I found that an interesting aspect as well, which is to, yeah, remotely hosted malware. Like the mainframe of malware, right? And they also, the attackers in this case, updated the firmware on these Modbus connected devices to disable some of the monitoring so they didn't get visibility about what the settings were and then they modified the settings after

the fact so you know that's kind of cool tradecraft uh and seeing the stuff in the wild yeah i mean great tradecraft and uh they should be immensely proud of themselves that they were able to turn down the heating for civilians in the middle of the cold. For two days. Yeah. Anyway. Jerks. Staying with Russia stuff and Kaspersky Lab is closing down its US division.

No surprises there. And laying off workers, I mean, that was inevitable. So that's actually happening now. What else have we got? Hackers have leaked 1.1 terabytes of slack messages stolen from disney uh all i had to say about that when we spoke about it is can you imagine being the incident responder on that because truffle hog go brr yes like all of the things that you have to dig through that to figure out what's gone and like how many boxes how many creds how many keys how many yeah and the problem with using truffle hog against a corpus like that is it's

gonna find stuff and then you have to go remediate it so yes yeah that's terrible that's a that's a tough day at the office uh now just a bit of an update on cdk this is from aj vicens over at cyberscoop um it looks like and we reported this at the time that it looked like cdk uh the the automotive software uh maker uh it looked like they'd paid the ransom. That was what we thought at the time. AJ's got some follow-up reporting here, which says that, yeah, that payment was visible on the blockchain. So it looks like $25 million for the CDK people. Also in ransomware-related costs, UnitedHealth's cyber attack response costs apparently are going to surpass

2.3 billion dollars this year uh for the change how over the change health care uh hack that's from rebecca uh pifer over at cyber security dive and yet somehow they're still going to make money apparently last quarter they reported 4.2 billion dollars so you know keep in mind though that's for the broader united Health Group. Like Change Healthcare is a subsidiary and United Health are like kind of like the death star of American healthcare. Like not exactly a loved company. But, you know, so I guess it's bad that healthcare got impacted.

And I would say it's good that they're suffering, but they'll just like find a way to gouge people. They'll just gouge it. Exactly. Yeah. Alexander Martin has a write-up over at the record we don't really have time to talk about this in any great detail but he has a write-up at the record uh based on a europol threat assessment which says that the ransomware ecosystem it's kind of what we've been saying as well uh and tom in his coverage the ransomware ecosystem is changing it's fragmenting it doesn't look like it's all dropping off that much at the moment,

but there's definitely some fragmentation and reordering and rejiggling going on and more people using their own custom malware and things like that. Yeah, like I think the rise of very big centralized platforms has put too much attention on those platforms and they're not viable now that we've got active disruption from law enforcement. But that's good.

So it kind of makes sense, which is good. It's good. It's exactly what we want. And then it means it has to kind of fragment, get smaller, get hopefully less effective. So it's a headline we'll be waiting a while to see, and we'll wait and see whether that trend continues, you know? Yep. Change is good.

Talk to me very quickly about this Windows Oday that was being exploited in the wild for a year before Microsoft fixed it? Actually, it's quite a funny bug. Basically, the guts of this is you can make a.URL file that when double-clicked on invokes Internet Explorer instead of Edge, and then you can use your old IE bugs to attack people. So the guts of IE is still floating about

on a typical Windows install, are they? Yes, exactly. If you can get to Internet Explorer, then of course you can attack it, So the guts of IE Is still floating about on a typical Windows install Are they? Yes exactly If you can get to Internet Explorer Then of course you can attack it And this was another way of invoking it And so that was just nice To see I guess The corpse of IE has been necromanced back to life

And yes Still bringing shells All these years later The reanimated corpse of IE Shells Alright Adam we're going to wrap it up there Still bringing shells all these years later. The reanimated corpse of IS. Shells. All right, Adam, we're going to wrap it up there. Thank you so much for joining me for that.

Yeah, crazy. I had a nice holiday, though. Well, that's the thing. The nicer your holiday, the worse the disaster that befalls the internet. So hopefully your next holiday is total crap. Well, this was our first holiday without a child under three in like six and a half years nice and uh it was the most wonderful thing about it was my my daughter who's the oldest and my son uh who's the youngest six and three uh you know great bonding experience for them so we had a we had a lovely time uh but uh yeah it was great to

come back and i'm glad we squeezed in the chaos on my on my last day of leave so we could have something awesome to talk about. But mate, yeah, great to be back in the saddle. And I'll look forward to talking to you again next week. Yeah, I'm glad you had a good time, Pat. The rest of us did not, but it's good to be back. That was Adam Boileau there with the check of the week's security news. Big thanks to him for that. It is time for this week's sponsor interview now with JJ Agar, the CISO at FanDuel.

JJ is a customer of Sublime Security, which is a modern email security platform. It's an API-based product which plugs into your mail provider's APIs. And if you want to be able to do more advanced stuff with your email security platform, then just let it operate as a black box. Then, you know, Sublime is definitely something you should check out. You could do your own threat hunting, apply custom rules, spin up integrations. Like it's a really good platform. And you know, the thing I keep coming back to when I describe it to people is it's just modern. It's a contemporary email security platform. They just did a new raise. Dmitri Alperovitch came on as an investor, and I signed up as an advisor as well. They're definitely the real thing. And despite starting off as a platform that sold on the basis of being more transparent and configurable and useful to

detection teams, Sublime, you know, it works just fine out of the box these days and quite a bit better than competing solutions. So that's the spiel. JJ joined me to talk through why he's a Sublime customer. And in this interview, you're also going to hear from Josh Kamju. He's a co-founder of Sublime. But the first voice you're going to hear right now is JJ. Enjoy.

I mean, I think as security professionals, we just like to tinker. We like to know how things work. But candidly, in the different businesses I've been in, it's been about driving the business context to the rules. And that's where you need extensibility. So while set it and forget it works, you can only trust so much about what is a black box vendor. You want the ability to customize for your VIPs, customize for the business logic that makes sense when a BEC email comes rolling in. It's not just these spray attacks that are going out. It is very targeted to specific employees here. We want to try and generate that higher fidelity business

context into the rules that we're creating. And having that extensibility just allows us to do so much and build on top of it. But it's great when you have a out-of-the-box foundation that you can rely on. That builds in the trust that you need to start building the extensible rule set on top of it. Yeah, I mean, there's this sort of trendy new thing, right? Which is the, you know, the programmable security stack. And I guess, you know, Sublime's an example of where you can really get in there and start saying, well, you know,

apply these rules over here, not so much over here. But, you know, you mentioned the extensibility. One thing that you've put together is actually an integration with TruffleHog that can do secrets detection. I'm guessing that's for outbound email. Yeah, I mean, it's for outbound and internally, right? We have one password, we want people to kind of build better patterns. And so with TruffleHog, and the integration, using kind of the DSL and the extensibility that we had with Sublime, it just allowed us to pass it over to TruffleHog, do a check to see if

this actually is a secret or not, ensuring that we kind of meet our higher standards that we expect on email security and have a high fidelity rate, not block anything that creates more false positive, more noise for our security analysts or for our business. And that then goes to a workflow that then eventually blocks or essentially prevents it from going out or removes it from mailboxes and so that integration with truffle hog was actually one of our proof of concepts to see truly how extensible this platform was and when we were able to kind of integrate that within a day the team was just you know they couldn't keep their smiles away they just were like this is it this is what we wanted to use it for so are you are you doing that with uh because i know you're a tines shop as well are you using tines to do that or is some other way yeah so with the the integration kicks out to times to just

have a part of a human in the loop if needed uh so that part like when there is there there is maybe a detection or you know just walk us through it so it goes from uh sublime is the you know the event gets triggered from sublime then gets picked up from times uh times will then pass that to truffle hog scanner and say hey you know rip apart the email header the email body pass over the parameter or the actual payload that we want to get checked in pass the truffle hog truffle hog will then say yes no this is a secret confirm that there's a aws k you know kmi that someone's you know for whatever passing it through or a marketing team is sending a you know a send grid token across the wire um and then pass that back

off for then the security analyst to then take action uh on it um so partly it is automated partly it is sometimes in a human in the loop depending on the type of token that gets passed back so there's an initial detection that happens within sublime that it then goes off to truffle hog for more clarity is that how that's working because i can't imagine that you're kicking off like you know some automation some automation actions for every single email right so yeah so we've got a we've basically taken specifically the secrets that TruffleHog's looking for, and we've created a detection in Sublime. So we basically took what are the patterns that TruffleHog is looking for,

for AWS secrets or a Google secret or different things like that. And then the first thing we did was we turned that into a detection rule in Sublime. And then we ran that over historical data, we can do like, we can do threat hunting over historical data. And so that's what we did with the team with JJ's team. And we validated efficacy with with the initial hunting query. And then we took that and then we turned it into a live detection, which will basically run on every message coming in looking for that pattern. But it's not giving you the full truffle hog experience until it trips those first indicators.

Yeah. So I think it's actually an action we've we've configured an action in sublime to hit the tines webhook after that detection fires and that action just hits the tines webhook and then the tines story from there takes over and i think does some validation with the with the truffle hog scanner and like validates it's a real secret and things like that, and then comes back. If it is, then you can take an action back in Sublime if you want to, or take some other follow on action. Yeah, but that that was what allowed us to kind of start immediately was, and thank you for reminding me, Josh, on that it was about pulling in the Truffle Hog regexes and create very specific, sublime alerts

and detections for us that allowed us to then pass that header, pass that detection, sorry, pass that email body and the actual detections and potential secrets over to time to eventually then validate it through to TruffleHog. Then eventually a analyst will then either take the action

to nuke it or say, you know what, false positive, move on. Yeah, so how often are you getting hits on that? I'm curious, like once you plumbed this up, did you just discover a horror show of constantly, you know, a horror show river of secrets flowing out of your org? I mean, I think it starts off when you go,

you cast a very wide net and you start kind of trimming it down. Luckily, there's a large investment. So it is very much where we leverage 1Password for it. So it wasn't a horror show, but I could tell you there's horror shows that I've walked into before where this is a common pattern and a common practice. Yeah. I mean, it came up in a recent interview. I can't remember who it was with, but yeah, I've seen people pasting stuff into Slack and I'm like, what are you doing? You know, like it's just people get sloppy.

Yeah, I mean, I think it's partly the utilities is partly what exists there. I think TravelHog is a great reminder to ensure that we don't kind of regress from the common patterns and practices that we do have because too often insecure, we're just putting too much of the cognitive load on people and we say no you are solely responsible ensuring this company does not uh you know dissolve overnight if you do one bad thing yeah we have to build out those detections and that allows us and

gives us the the ability to essentially build out all the different types of layered defenses that we need. Yeah. Now, another thing I wanted to talk about with you, Josh, is because you're an API-based product, you can do a few cool things. Now, for a long time, email security products have sort of hacked together phishing reporting, right? Like phishing reporting has been a big feature for you know all of the major security email security platforms but it sort of requires users to be using the correct

client and press the correct button and whatever uh whereas because you're an api based product and you know how to api uh you can actually receive reports you can receive the emails that have been reported directly to the email platform like 0365 or Google Workspace, correct? Yeah. So we can ingest the user reports that are reported directly to the email provider. We can pull them in if you're using like an abuse mailbox, so like a phishing at alias. Basically, like however users are reporting them, we can get them, whether it's Microsoftrosoft or google and that's then the entry point for us to do a second level analysis and just automate as much as we can of

the response if you ask like a bunch like a lot of teams where they spend a lot of their time it's around just investigating and triaging these user reports um and interestingly well it's funny because the you need to act on them, right? Because you know that the upstream provider won't. You can report stuff all day long, and unless there's a gajillion reports, it's not going to move the needle at all. Yeah. And what's interesting is at scale, the vast majority of these user reports are actually totally benign. So they're like, just unwanted marketing mail, or things that users just don't want. And they just, they just hit, you know, their instinctual reaction is just to send it to like, the security

team, like, I don't want this. But at the end of the day, that ends up costing a lot of teams time just having to investigate and triage those and validate. So one of the things that we do is try and automate as much of that investigation and triage as possible, both on the benign side and the malicious side. So if there's anything that we might have missed on the detection, like live detection side, we do a second level analysis to see how confident we are now that the user is reporting it on how bad it is or how good it is. And then we can take an automated action, whether it's to remediate, auto close the case, or send an alert somewhere, Slack, hit a webhook, whatever it might

be. I think when you get a security team, you have your security analysts, time is precious. And a lot of security analysts deal with the typical end user reported emails. Phishing is obviously probably the vast volume that always comes in, right? Email is one of the first things that get looked at. When we looked at it for us at FanDuel, when we deployed Sublime, we were able to move from about 97% to 98% auto-remediated. That just gives me back time for my analysts to go solve on the real interesting problems, the problems that require the deeper double-click in with the business context. And for us, that was probably the biggest aha moment where I now get to allow engineers and analysts

to go focus on the next problem, the next challenge, the next attack that's popping up and not with make sure our ticket queue is down. That creates burnout. That just creates the wrong incentive structure for analysts. You know, they don't want to go figure out is this really that email that's gonna take us down? Or is this now the you know, the email that actually gets through that, that they have to put their eyes on, they're going to have more attention span, and they're gonna be more focused because they're not looking at the mundane, you know, benign emails. They're just really focused on the problem at hand. And to me, as the CISO, that's what I want. I need to empower, you know, our analysts to do the right things. If they're not, then I'm not doing the right job.

I mean, we love to talk about all of the cool advanced stuff that we get to do in security. But when you think about it, like there's two things that just suck up so much time like dealing with phishing is one of them and then the other one is like vulnerability management and patching and it's the most you know tedious that soaks up so much expert time you know and requires the understanding of how the business operates the different nuances as much as we want to program a system to do that, you know, there's humans for a reason at this point, right? If businesses are running on it, we do need to kind of bring a human touch into supporting this. All righty. Well, JJ Agar and Josh Kamju, thank you so much for joining us on the show,

for that conversation all about, yeah, I guess the future of what email security products are going to look like, because, hey, you know, you don't even have to be a fan of Sublime Security to go, well, yeah, that probably makes sense that stuff's going in that direction. So thank you both for joining me. Thank you.

Thanks so much, Pat. That was JJ Agar and Josh Kamju there with this week's sponsor interview for Sublime Security. You can find them at sublime.security. And that is it for this week's show. I do hope you enjoyed it. I'll be back soon with more security news and analysis. But until then, I've been Patrick Gray.

Thanks for listening. Thank you.