Risky Business - Risky Business #743 -- A chat about the xz backdoor with the guy who found it

Episode Date: April 3, 2024

On this week’s show Patrick and Adam discuss the week’s security news, including: The SSH backdoor that dreams (or nightmares) are made of Microsoft gets a solid spanking from the CSRB Ukraine uses an old Russian WinRAR bug to hack Russia Push-notifications and social-engineering combined-arms vs Apple And much, much more. We have a special guest in this week’s show, Andres Freund, the Postgres developer who discovered the backdoor in the xz Linux compression library. This week’s show is brought to you by Island, a company that makes a security-focussed enterprise browser. Island’s Bradon Rogers is this week’s sponsor guest and he’ll be joining us to talk about how people are swapping out their Virtual Desktop Infrastructure for enterprise-focussed browsers like theirs. Show notes Risky Biz News: Supply chain attack in Linuxland oss-security - Re: backdoor in upstream xz/liblzma leading to ssh server compromise Andres Freund (Tech) on X: "@binitamshah FWIW, I didn't actually start looking due to the 500ms - I started looking when I saw failing ssh logins (by the usual automated attempts trying random user/password combinations) using a substantial amount of CPU. Only after that I noticed the slower logins." / X Andres Freund (Tech) on X: "@riskybusiness Absurdly enough, I was listening to the episode on a cooking break while writing the xz issue up. Couldn't make it up." / X GitHub - amlweems/xzbot: notes, honeypot, and exploit demo for the xz backdoor (CVE-2024-3094) research!rsc: The xz attack shell script DHS report rips Microsoft for ‘cascade’ of errors in China hack - The Washington Post Review of the Summer 2023 Microsoft Exchange Online Intrusion Russian researchers say espionage operation using WinRAR bug is linked to Ukraine Recent ‘MFA Bombing’ Attacks Targeting Apple Users – Krebs on Security Ransomware gang leaks stolen Scottish healthcare patient data in extortion bid Ross Anderson, professor and famed author of ‘Security Engineering,’ passes away

Thank you. swapping out their virtual desktop infrastructure, their VDI, for enterprise-focused browsers like theirs, which is funny because they're very different types of technology, but solve a lot of the same problems. That is coming up later, but first up, it is time for a check of the week's news with Adam Boileau. And, mate, there is a bit going on this week. We're going to talk about the Cyber Safety Review Board's report into the Microsoft hack last year, the one that involved the stolen key that no one seems to know how it got stolen. Yeah, the CSRB kicked the absolute crap out of Microsoft, and it's amazing, so we will

be talking about that. But first up, we've got to talk about this backdoor in XZ, or for the Americans out there, XZ. And indeed, we're going to hear from the guy who actually found this backdoor, Andres Freund. But before we hear from him, why don't you tell us a little bit about what from the guy who actually found this backdoor, Andres Freund. But before we hear from him, why don't you tell us a little bit about what it was that he actually found, Adam? Because I understand this is some sort of compression library that interacts somehow with SSH, so it's kind of an SSH backdoor but in an unexpected place. Walk us through what this backdoor actually did. So yeah, this is a supply chain backdoor

against the XZ compression library, which is in most Linux systems, a dependency of SSH, the remote admin tool that really is the backbone of how Unix admins admin their boxes and everything in the cloud and so on and so forth. And this was quite a cunning backdoor that basically meant you could show up,

pre-authentication to an SSH server on the internet, present a magic RSA key, which would then extract commands out of itself to run as root. So as a backdoor goes, it really is just a beautiful thing. And for anything cloudy or Unix-y, it is the ultimate backdoor that you would want. And it also has the, I guess,

the property that to be able to generate the RSA thing that you pass in, you have to have a private key. And so the attacker has a backdoor that they can use and no one else can use. Well, kind of, because I believe just from stuff i've read online that the implementation of that you know functionality was really that good yeah so there were you know it was vulnerable to things like replay attacks and stuff so they've they've tried to do a nobody but us capability like a nobis thing but they didn't do it well which kind of rules out this being a five eyes up.

Well, we'll get to that. Yeah, we'll get to that later. But look, where did this thing, how far did this thing actually make it? Because I understand that, you know, it found its way into XZ via a persona called Gia Tan, I think was the name of it. It was this, you know, bogus contributor who built up trust and infiltrated. By the way, how funny was it that last week we were talking about China's grand plan to move to Linux variants to avoid dependence on Microsoft and I did say, well, the open source ecosystem

is kind of vulnerable to infiltration. That was just bizarre that then this came out two days later. But it looks like, yeah, it was a bogus maintainer who spent a long time building up trust, like a couple of years and then accelerated uh their their operation when it looked like there were going to be some changes made to other parts of linux that would have made this more difficult for them but my question is how far did this actually get because it did make it into some major distros didn't it or was it just in early betas or like how far did it actually get

so this was in the mainline code, so it was committed to release versions of XZ, but the amount of time that it takes that to then percolate down into stable releases of Linux can be a while. And so people running mainline production releases of Red Hat or Ubuntu were not vulnerable to this yet they would have been eventually had this campaign not been snapped people who are running kind of the developer releases or the you know what we call the unstable versions which you know are still

very stable they're just you know kind of subject to change while they're being developed um so people who ran those were and of course course, people like software developers, who we're about to talk to, are the sort of people who often do run bleeding edge machines like that. You know, there are some specialist Linux distributions where even like the stable version is pretty bleeding edge. So like the Kali version used by a lot of security testers is one of those. So it had made its way out. It was still, you know, a year or so away from landing and being everywhere

in line distributions but it was going to get there yeah so i guess you know it's pretty lucky that this was caught now as you just mentioned andre's friend is a developer he's a postgres developer works at microsoft and uh we spoke to him this morning about how he actually found this thing. And here it all is from the man himself. I was working on reviewing somebody else's patch and I was noticing that there was some small slowdown. But it was small enough that it was hard to know whether it was below the noise floor. And so I wanted to bring my benchmark machine into a very low noise environment and tuned it so that nothing else was happening.

And it was just seeing occasionally as eight processes briefly using a lot of CPU and then being completely idle again and verified that I didn't have like, I don't know, actually authorized logins or anything happening. Obviously not. It was just the typical probing from the internet.

And that seemed odd. So I started profiling it, installed debug symbols and everything. And while profiling, I was just seeing that there were lots of hits in the instruction pointers there where there was no symbol for it, which you normally shouldn't have when you're profiling open source software that has debug symbols installed. I mean, you can sometimes get it with just-in-time compilation and stuff like that, but that clearly shouldn't happen here. I thought for a while whether there was some seccomp stuff preventing it or something. It was pretty clear that that wasn't right. So then I looked and tried to reproduce it and initially I couldn't immediately reproduce it

because if I was running SSHD from a console, it didn't reproduce. And that one was like, I actually could measure that if I was starting SSHD in a normal terminal, the log in time were normal. And then if I tried it in, like started via systemd, it was suddenly slow. And that was like, that was pretty suspicious. So, but initially I was like, okay, maybe there's some bug in like some journaling library part or something, but like that wasn't the case.

So I then tried to figure out what the differences in the environment were that made it like enter the bad path and whatnot i had some troubles because i tried it with a debugger and it wasn't reliably working as soon as i attached and turns out that it has like some anti-debugger type stuff i don't fully i think there's multiple there's some parts that are where i've not yet seen an explanation for how it works it also seems to detect sometimes when you attach with ptrace enabled, but not always. There's something that I don't quite understand there. And then just narrow it successfully down, like what the explanation for all of this was.

I mean, it seems like that's a fair bit of work already that you're talking about. Like, you know, you really went down the rabbit hole, didn't you? for all of this was. I mean, it seems like that's a fair bit of work already that you're talking about. Like you know, you really went down the rabbit hole, didn't you? Oh yeah. I mean, initially I thought it might just be a Debian issue because I built it from upstream and it didn't reproduce, which was pretty weird. So I thought like, okay, something must be going on. And then eventually I figured out that's like how the shell script was injected.

But I made the mistake when going toub and download the official release tarball that i clicked on the github link that is github generated like the source code so it didn't reproduce and it took me so i was like about to blame like the debian maintainer for all of this because they had clearly uploaded it like i could see it in the git repository for the debian package but eventually I figured out that I just clicked on the wrong link and compared it and found, yeah, okay, it's in the upstream sources. So at what point did you realize that this wasn't a bug and it was something deliberately malicious, something that had been placed there in order to get access? Reasonably early on, because after I excluded that it wasn't, I don't know,

some crypto being ridiculous expensive or a PAM being suddenly having got the password check having gotten more expensive or something. Those were my very first theories that somebody increased the iteration count for some password check or something. But after I excluded that, that was clearly not right.

And I think where it really became clear was when I was building and was seeing that the, I forgot the name of the object, whereas UC64-fast was replaced multiple times within one build cycle. I was like, okay, clearly. And I noticed it because sometimes I would rebuild and it would not be the same output. Because it turns out there's some concurrency bug actually in the way it's injecting,

where sometimes when you rebuild, it would get the normal output and sometimes not. And that clearly couldn't be, there couldn't be a legitimate reason for that. Although I did blame LibTool for a while. It's always a good suspicion. I read Russ Cox's write-up of how the injection process works.

And a lot of work went into figuring out how to inject that into the build in a way that was believable to open source maintainers looking at the code. Because it's pretty smooth. Yeah, it was funny that they went from version, I don't know, I think 3 to 23. They just attached a digit to the get text, whatever, build to host files. Otherwise that part is identical, just added a digit. Now, I got to ask, it looks like probably what you've done here

is disrupted a pretty long running intelligence service operation. Nice work. But if you had to rate what they actually did here, because it seems like our colleague, Catalin Kimpano, was just saying to me, look, this guy you're interviewing, Andres, he's a Postgres guy, he's performance obsessed,

he was going to find this, right? Like, this is the perfect guy. No? You're shaking your head? I think most of it, if I had tested it on a desktop system, I wouldn't have noticed it because an idle Chrome alone hides the noise here completely. If in another year I might've had bigger changes into Postgres that I was working on for myself rather than integrating other people's work, and I might have had bigger changes into Postgres that I was working on for myself rather than integrating other people's work. And I might have spent less time profiling code at that time of the release cycle. If I had updated to the next package version already, I would also have not noticed because it's actually harder to investigate it, like I think in the.1 version and the.0 version.

But don't you think people would have noticed that the constant stream of SSH brute forcing attempts against the boxes that they had out there on the internet, you know, don't you think this was going to get rumbled at some point? I guess that's what I'm asking is like, if you hadn't have found this, do you think it would have gone unnoticed? You know, because I'm just not sure about that. I don't think it would have stayed unnoticed for years. But yeah, like I actually had like turned on off Turbo turbo boost in my machine to get better benchmark more predictable benchmarking results and the timing differences are much smaller if you do not do that because suddenly the cpu would not go to four gigahertz or like it was actually staying up like at high cpu usage for 500 milliseconds instead of i now tested it it was

like otherwise it would have been 190 milliseconds or something. Yeah. Right. So that would, that already would have like made it less noticeable. Yeah. So, I mean, if you, if I guess I'm asking like to write, write their backdoor, how good a job do you think they did? I've seen a bunch of backdoors over the years, like not in directly in open source software, but like inject, like modifying source code and to inject backdoors and whatever. And none of them came close to this. Yeah. I think some of the complexity might've actually been a bit over-engineered and I might not

have actually noticed if they had used less of that. If they hadn't done that build, like during the build process that like changed what one file means and go back and so on, instead just had like just had like, I know you can, in autoconf you can add just a library to be linked as a replacement file, I'm not sure I would have noticed it in that way that it would, because that would have been much less suspicious than some weird back and forth of the same file and stuff like that. It re-entered back into itself right while it was executing in a different context yeah that was just weird yeah and like to be able to re-enter like it deleted this the

shared object file after the horror even built and stuff yes it's just odd stuff yeah there was some definitely some interesting wrinkles in it i guess the question for me and you know like i'm sure many people who are listening is when you when it dawned on you that what you were looking at was an open SSH back in the wild, like on your dev box, how did that feel? Did you giggle? Were you going, huh, huh, huh, huh, huh, huh? Was it the catastrophe giggle? That's kind of what I imagined. I think it was switching between a bunch of those modes like first i was like it was pretty i think it was

really tired because i had slept well or something before so it's like am i just hallucinating something and then i was like or did i just i'm gonna get a few hours of something and look at this again i i think uh i didn't send something that night because i was like, okay, it's already too late. I can't write this up in an understandable way. At that point, when I was thinking of forwarding it, I had not investigated that deeply. So I didn't yet know a bunch of the stuff. Because it took me quite a few hours to figure out that it had to be SSHDS, the ARC v0, and that it had to be, I don't know, those environment variables couldn't be set and stuff like that. I mean, we should point out, too, that you're not actually a security person.

You are a developer, and you work on Postgres, right? Yeah, but I guess I've been a bit security adjacent over the years. I've done responses to Postgres bugs, security issues, and back when all of this was not yet as formalized, I did maintain production servers, and saw a bunch of them would get called in when things would be... Things went sideways, yeah. I don't know how many PHP my admin things I responded to,

but it's been quite a few. But this is all like bits 2000s or something. So it's been a while. I mean, it's something that was very funny actually is that while you were writing this up, you said you took a meal break and you're a risky business listener and you turned on risky biz.

And one of the things we were talking about last week is China's plan to switch to Linux-derived operating systems. And I was like, well, there is kind of an infiltration risk. I think in your words, you couldn't make it up, right? The timing. Yeah, that was completely absurd. It was so good. I think it had actually come out like a few days later, but I hadn't just hadn't listened to it yet. So I was listening to something at that point. Yeah, bizarre. Okay. Andres Freund,

thank you so much for joining us on the show. I know you've, you've got, you've actually got a Postgres release coming up too, don't you? Yeah. Like on April 7th. So like that's a feature for you. So there's a lot of work to be done in the next few days. Well, you know, you knew the perfect time to add more to your plate. That was a great time management there, mate. A pleasure to have you on the show. Thanks so much for joining us uh to walk through that i know you're just having a crazy week we really appreciate it and uh yeah congratulations on on a job well done thank you yeah saving the internet good job sir cool thanks bye thanks to andres for that now you know you listen to that you realize we got pretty lucky with this thing um sure yeah there's some interesting stuff here too like uh the thing that makes me think this is a state-backed

operation and it might not be this could just be some weird open source internal beef or something odd you know we really don't know but there's a bunch of things that point to this kind of being a state-backed operation. One of them is that that persona that I mentioned, the persona of this contributor, maintainer, whatever, who infiltrated the project. Brian Krebs took the email addresses associated with this persona and ran them through, like, data breach dumps. And there's, like, no hits apart from places where you would expect to find them, like GitHub or whatever. So it looks like this was very, like the OPSEC here actually looks like it was pretty good.

Yeah, I mean, the email addresses of Gia Tan and some of the other sock puppets that were used to pressure the maintainers at Xeer, but also some of the other Linux bits involved, all of those kind of looked pretty clean skin, pretty fresh in that they didn't have a long tail of contributions or as brian pointed out you know injuries in data dumps and those kinds of things so you know that you know making and operating fake personas is hard work right

people can do it as civilians but like it is hard work to have a well backstopped identity and it's interesting that these ones popped up you know seemingly out of nowhere and then in some cases just disappeared never to be seen again after their part in the process was played so you know that feels like it's your job as opposed to a hobby as opposed to someone who is doing this for fun or for their own kind of personal reasons it doesn't feel like vindictiveness, although I do think people are kind of overstating how much effort this was,

because you don't need to be full-time on this to run an operation like this. This feels like, to a degree, it could have been someone's side gig at some agency. Yeah, absolutely it could be, although there's quite a few people who have expressed that this is not a full-time job, but if you're going to do it once, you may as well do it a bunch of places.

Once you've got a team spun up to do this kind of open-source infiltration, why do one project? Why not do a bunch? Why not build a patent out of it? And given how successful this one very nearly was, I mean, pretty well worth it. You did mention briefly that there was an accelerated kind of time pressure on them

because the way that this gets into OpenSSH was through SystemD. SystemD, source of all evil. This is a source of all evil, especially for us old Unix people. And they were in the process of maybe dropping libLZMA. So that avenue was going to go away, which may have accelerated the pressure on this

and led to the sorts of slightly sloppiness that led to Andres having his Postgres logins just a little bit slow. Yeah. So let's look at you know who benefits here right and you would think that you know if this is a state directed sort of thing you sat down and did a for and against in every column i mean i think we can rule out five eyes because if it was five eyes the crypto bits would have been better this is probably illegal because it involves introducing you know this code to

computers inside five countries and like it just the equities conversation to me ultimately rules this out like the crypto is is reasonable it's not perfect and you expect perfect from five eyes because equation group right um but yeah the equity side of it just to me i can't see this getting past leader review at if you were to say. No, no, me neither. And so, look, long story short, the country that we think is most likely behind this is Russia. I had a chat with Dmitry Alperovitch. Now, of course, he's not involved with CrowdStrike or whatever anymore,

but he was the CTO there for a long time. So he's got a good feel for this sort of stuff. He also has a Russian family background and he just like looks at some of the language and he's like, that looks like a Russian trying to speak English. You know, it just gives him, and he's not saying, he's not saying, oh, I, I, Dimitri, attribute this to Russia. He's just saying it gives him Russia vibes, particularly like SVR vibes. And you look at other stuff they've done around supply chain stuff with like SolarWinds and whatever,

does kind of feel maybe like something they would do. You know, you sat there with your for and against column. You couldn't really come up with much against for Russia, could you? No, and that was where my weakest against column was for Russia. And one of the things to me as a Unix nerd, because I read there's a really great write-up which we've linked through the show notes for how the builder process integrated this backdoor and it's like really quite cunning and to me doing it how they have done it requires kind of native proficiency native fluency in open source build tooling and that's not a technically so this

is this is like russian you is what you're saying like yes i mean i read this and i'm like yes i understand why you would do this like i could probably do this but it would take me a long time whoever did this understood just you know a whole bunch of lore. And to me, the vibe says old Unix. Yeah. Because like the way that it's disguised and the way that it hides in plain sight. And, you know, there's been a lot of conversations about whether open source community should have found this because many eyes find bugs or whatever

versus closed source where, you know, things are more controlled. And, you know, I think one of the things that has come out of this for me is doing this effectively in public eye and open source is actually quite hard. And the extent that they went to, like the level that they went to to hide this in plain sight was really good. Like not perfect, like a little overcomplicated in some ways,

like Andreas said, but to me it's... But just because it's, I mean, okay, sure, it's really hard, but that doesn't mean that you can't do it if you throw the resources at it now you you spoke before about well how do we know they haven't done this elsewhere if i was running a program like this i wouldn't get the same team to do it i would get different teams to do it because you don't want one being discovered and then little breadcrumbs from that one finding all of the other ones the point is we don't you know everyone's like yay we caught the back door it's like well we don't, you know, everyone's like, yay, we caught the backdoor. It's like, well, we don't really know if there are others, right?

And I think that's, we've got to be careful here not to assume either case that Linux is riddled with backdoors or that there are no more. Truth is probably somewhere in between, my guy. talking with Tom who's working on this week's Seriously Risky Business about you know historical examples of this kind of supply chain subversion in the open source world and there was a period where this happened a lot like the early 2000s in the IRC scene wars era there was a whole bunch of backdooring going on of source code repositories and things and then there hasn't been that much since and you know a few people who've written this up i've said like this is the shining example of an open source you know project backdoor that we've seen and you know i you know i'm inclined to agree like i don't think this happens everywhere in open source and it is kind of difficult but i can't believe this is the only place because

it's too tempting a target. Yeah, it is. And I wonder how much more important this will get as if China follows through and actually does embrace Linux as it's always. You know, the North Koreans got Red Star Linux, so now America's going to have to have one as well. But that's more related to the CSRB report,

which we're actually going to talk about now. So the long-awaited Cyber Safety Review Board report into the Chinese MSS hack of the State Department and other organizations last year, I think there were 22 organizations, 500 compromised mailboxes. This is the attack where the attackers somehow obtained access to a critical bit of key material that allowed them to create access tokens into M365 accounts. And, you know, that's what they did. And then the attack got rumbled and whatever. Here we are.

So, you know, it's the Storm 05581. We've talked about it a bunch on the show. The Cyber Safety Review Board has spent months working on a report into this. And they did not hold back, Adam, and said basically what we've been saying on the show, which is that Microsoft really messed up here and not in a small, insignificant way. There was a whole bunch of stuff that had to come together

in order for this to happen, and at every step of the way, Microsoft should have been doing better. Yeah, the report is pretty rough reading, both as a Microsoft customer, but also if you work at Microsoft, there's some real hard truths to face in there. And, you know, I think everyone who listens to the show

probably should go read the report because it's absolutely well worth it. And it talks through the kind of the timeline of what happened. It talks through how it was originally discovered by the State Department. Like, the State Department was doing a bunch of extra logging of access to its cloud mailboxes using a government logging tier that really wasn't widely available or widely used elsewhere. So they were uniquely positioned to spot the access patterns that led to Microsoft discovering the key material had been nicked.

But then they talk through the mistakes that Microsoft made. And, you know, we have talked on the show, like how ridiculous it was that their keying system worked like that, that they weren't rotating their keys. And there's some great detail in this report around that particular aspect of Microsoft's infrastructure. Yeah, I mean, there's actually some new stuff here as well, which is that they've tied Storm 0558 to two – look, this group is like China's A-team in my mind, all right, because they

are what I would call APT hitmakers because they were apparently the same people behind Operation Aurora, which targeted Google back in what, like 2008? And also behind the attack against RSA in which someone got into RSA and stole all of the seeds for RSA tokens, which effectively sunk that as a viable authentication method. That business never recovered from that. So yeah, it looks like China's china's greatest hits man these are the hit makers they're the beatles of apts that was a detail that i really you know opened my eyes a little bit when i was reading that because we hadn't seen that link made before and apparently google tag

uh made that link during this or reported this through the csrb i don't think we'd ever seen that you know in in public before and it, right? There's a lot of similarities in terms of overall MO between Aurora and RSA and this. And it's just, you know, it was very well done. Well, hang on. Before you say that, the other thing that came out in this report, and it's an area where CSRB has really criticized Microsoft,

is Microsoft, you know, they issued that big thing about how there was a crash dump and a key wound up in the crash dump and wound up on going from the production network to the corp network and that must be how the attacker got it you know really they don't know no they do not know csrb is basically during the investigation you know kind of forced their hand in updating their statement on that um it was only ever a hypothesis that they sort of quite misleadingly spoke about like oh we figured it out they have no idea how this happened yeah and

that ought to be terrifying is terrifying and the csrb report is very critical of microsoft's handling of how it communicated information about this to its customers and to the general public and the reality is that they had been, you know, essentially the mail infrastructure for consumer accounts is, I am assuming, begat from Hotmail. Yeah. I mean, Hotmail was like Unix Linux based back in the day. Then Microsoft bought it, recreated it for IIS,

and that code I think is what they use today. It's still going, yeah. And it's what they're using for M365 auth. Like what on earth? I mean, I think, is what they use today. It's still going, yeah. And it's what they're using for M365 auth. Like, what on earth? I mean, I think that's what it's saying. So the report has a bunch of details about how that process works. So essentially, Microsoft has the consumer system, which had key rotation but manual for a while.

And then at some point, that broke, caused a big Microsoft outage. They disabled the key rotation and then just forgot about it, never replaced it. So that's how they ended up with a key from 2016 still being valid in 2023. So that's a big part of the criticism. And then there's the way that they glued Azure AD, Entra ID and business,

you know, like proper M365, 365 cloud accounts into this whole mechanism with a front end kind of like auth broker that accidentally allowed consumer signing keys to be valid into enterprise accounts once again because of kind of confusion in the code because of all the layers involved because of all of the maintenance and stuff that happens over the years, which, you know, just straight up not good enough. And the report has a number of really good recommendations about what cloud service providers should be doing for managing identity and authentication.

But what a perverse situation when you've got the United States government advising Microsoft on how to handle auth. Yeah, well, and what's funny is the report talked to a bunch of other cloud providers like Google, like Oracle, like Amazon AWS about how they handle auth systems. And their conclusion is that Microsoft's auth system was not up to best practice per what their competitors were doing. Well, duh. Well, exactly right.

But that's really nice to have written down because that you know, that's a really solid point of criticism that they can't weasel out of. Well, I actually find the report, I mean, where they didn't chicken out is where they've said Microsoft's corporate culture just isn't up to it, where they have prioritised sales over engineering when it comes to security.

You know, they really went for them and said that, you know,-to-bottom failure. And I think that was needed. I mean, you've heard me say on the show before that if they released a soft report on this, I think their credibility, the CSRB's credibility would have been at risk. But this is not a soft report. No, it's really not. It's really not.

And I really liked, they tied it back to um the trustworthy computing memo from bill gates back in the day which we had seen trotted out as a thing by microsoft say when they were responding to this uh and then of course they launched their new secure future whatever it was and the reporters basically like you need to go back to what bill gates said in 2002 and do what the hell he said because you have forgotten how you're supposed to be how important this is and what you are supposed to be doing yeah you've drifted away from that right it was interesting because that piece i wrote uh for seriously risky biz when tom was on holiday which was about how we

need to get a big stick and whack microsoft with it and let's see what comes out of this csib report like that's literally what i wrote about uh it wound up being published on Lawfare because we have a syndication agreement with them and then it wound up being discussed on Hacker News the funniest thing was the top comment on Hacker News was like OP doesn't understand the history because TWC and it's like mate that was like what 22 23 years ago or something like they they have drifted a long way from that. It's also funny that they think that I don't understand the history

considering I interviewed the VP of Trustworthy Computing for Microsoft, Scott Charney back in like 2003. So no, I know the history, right? And it's just- Are you saying someone on Hacker News is wrong? Yeah, unbelievable, I know. But I remember like back then

when they decided in earnest to fix it, it felt very different to what's going on now, right? But I remember like back then when they decided in earnest to fix it, it felt very different to what's going on now, right? So all of a sudden, because back then I was one of the people writing for ZDNet really about Microsoft not being all that great, which it wasn't back then. And, you know, they opened up. They put me in touch with George Stathakopoulos, who these days is the VP of security at Apple. But back then he was sort of brought in as a head kicker

to get Windows into an acceptable state. And, you know, then we saw Windows XP Service Pack 2 and things really moved in the right direction. You know, he brought in NGS, you know, Dave Litchfield and his family to go and work on MSSQL, which is why that's not a horror show. So they've done it in the past. It required a lot of effort.

It required executives expanding a lot of political capital to make it happen, but they actually made progress, and they are just so far from that these days. Yeah, I mean, they are going to have to really think about how to approach this, and the CSIP report has, you know, a bunch of things they should do but ultimately microsoft needs to decide that this really is existential for them and go deal right because it's not acceptable to have a system the scale of azure and 365 not be best in class especially

when you've got users like the state department you know acceptable for whom though right so so oddly enough i am because i have some some money uh just in a share account right i am a microsoft shareholder right and just you know like i don't even know how many shares it's just one of those sort of etf things where you you know you put some money in but it's direct ownership so you know microsoft's done great and And that's the problem, right? Is that their ultimate responsibility is to make that share price go up, make their revenues go up. You know, so we talk about it being unacceptable because we're security people, but ultimately they're only answerable to shareholders. You can't just say, oh, they need to be, you know, they need to be better. I don't know, man, the whole thing is

so confusing. Like that was that piece that I wrote that I just referred to earlier, the one that wound up on Lawfare. A big problem we have is that there's not really any natural competitive pressure on Microsoft. Half the world's economy is run on Excel. You know, Google, it's a developer suite for people who aren't developers, right? And it runs half the world. You cannot replace it with something like Google Sheets. You can't replace it with a floss-based office productivity tool. It is just not realistic. So there is so much just natural lock-in. You know, it would be very, very difficult for a government

to choose anything else than Microsoft productivity tools. Or even in the private sector, it's very difficult for people to choose anything else than those productivity tools or even in the private sector it's very difficult for people to choose anything else than those sort of tools so you know we got this I guess we're experimenting with that with Russia and China at the moment well that's right and that's why maybe the United States is going to have to do what China did as well and come up with its own you know Linux distro to compete with Microsoft stuff and what did i call it i call it bold eagle linux so we can have red star linux and bold eagle linux but you know i don't know what the answer is

here one idea i suggested in that piece was sort of mandated portability of productivity applications so you might want to use you know m365 on aws or something like that i don't but i don't know and that's probably going to just mean M365 on Oracle Cloud and everything will be worse again. And I just, I don't know, I'm sad. These are hard problems to solve. Yes, they are. But boy, oh boy, Microsoft needs to take this on the chin

and actually go deal with it, do something serious. And the CSRv report alludes a couple of times to the subsequent svr breach of azure and the cloud through the through the oauth the oauth thing yeah but i mean onwards to enterprise admin inside microsoft like once again yeah you know and that was my joke a while ago which is their next report should be into the subsequent breach into uh m36, which happened a couple of months later, the other disaster. Well, and the report makes a couple of mentions

about Microsoft's communication during that particular incident whilst they were being criticised by the CSP for their communication during this one. And that's a pretty clear message that USG is not done with Microsoft. Well, I do wonder, right? I do wonder if there are avenues for other government agencies. I don't know whether that's the FTC, FCC, or the SEC,

or whatever three-letter agency it's going to be. You do wonder if there's more coming, right? And if I were the microsoft board i would i would that would concern me yes i think they should be batting them down some hatches the csv also mentions uh that some of the other victims of this particular hack were uk government entities um as well as private individuals and that's interesting because them in the u.s is not the only jurisdiction that cares about Microsoft and the rest of the western world you know we're all wedded to Microsoft as well we're

all going to have you know concerns about where this goes and how well they respond to it actually there was an interesting bit in the report about where Microsoft had notified people who'd had their accounts breached and a number of people didn't notice because you get so many emails saying, hey, someone's tried to log into your account or reset your password. And even the like, our nation state did it, notification emails are so common now

that people didn't even realize. And so that notification process also very due for an overhaul because when you've got congressmen not realizing that a foreign actor has been in their email because it just looks like every other spam yeah there's just so much so much work to do here yeah yeah there really is now look we've just talked about you know an amazing chinese a team hack with mysterious you know crypto key material we've spoken about something else that might have been Russia.

Look, it might have been North Korea. It might have been someone with a grudge. It might have been China. We don't know. It just feels a little bit Russia-y. So please thread and tell people don't yell at me like you always do when I say stuff like this. But let's move on to what attacks mostly look like in the real world,

which is in this campaign, which has actually been attributed to Ukraine, they are emailing people encrypted RARs, password in the body of the email, and when you extract the RAR, it exploits a bug in WinRAR and gets them a shell. And I love this because it is like the total opposite of everything we've just talked about, which is the dumb thing that gets her

done yeah it's it's a it is a beautiful thing there's an extra nuance in this which is that the winrar bug that they are exploiting was actually found and reported by russia so it was i think a fact that group ib derivative in in russia still they found this bug which is essentially a like you double click on a pdf inside a raw and it runs an executable from inside the raw instead through some raw thing that they you know it was fixed a couple of years back um but yeah straight up using this to to hack stuff in russia which yeah it's it's poetic and underscores as you say that not everybody has to be a team like there's plenty of roles for b tier you know b tier bugs like this one so good job yeah i don't think that i think it's less likely to work against the state department

though you would hope probably probably their winrar is a little more past or they don't use winrar because only chinese hackers use raw yeah it's so funny right like that was my old joke about like the one alert you should really have is when winrar executes on anything in your network yes yeah it's because the x fell is underway now brian krebs has an interesting report up here on uh mfa bombing with a difference right so this is like mfa flooding where um you know we've seen it before where people just you know hit everyone with like duo push notifications until they get frustrated and just click okay fine. It looks like someone's doing something similar for Apple users but they don't actually want the user to click okay fine. This is all just a way of building a pretext for a phone call that'll come and say hey I hear you're having problems with like MFA notifications. Yeah like this is

once again really smart thing that works that's technically dumb um so they've been spamming people with push notifications from apple and clearly there is some issue in apple's auth stacks i'm aware you can trigger device pushes that are you know legit pushes to authorize a password reset or whatever else um that are not being either rate limited or there's some you know way that you can cause them to be sent. And it makes your device kind of unusable because clicking through these things is difficult to do in bulk. And some of the people that have been targeted with this

then say they get followed up with a phone call with a spoofed caller ID of Apple's legit like 1-800 number in the US that then says, hey, someone's trying to do something to your account. We need to verify your blah, blah, blah, blah, blah. We're going to send you a push notification. Please give me the code,

which is then of course used to trigger a password reset in your iCloud and onwards to account compromise. And, you know, that's, it's pretty smart, right? I mean, if the technical control doesn't work, why not go for the human? And this kind of, you know, using it to build a pretext is good work. Yeah, it is. It is.

Link to that report in this week's show notes. We've got a Medibank-style situation kicking off in Scotland, Adam. Yes, part of the Scottish national healthcare system has had a breach. The NHS in Dumfries and Galloway, which I gather is in the southern part of Scotland, they've been breached, patient records stolen. A ransomware crew called INC or Inc. Ransom have claimed to have done it. They've got patient records and they've started dropping a small amount of those to try and pressure the NHS into paying them.

And that's, I mean, there's so many ransomware stories that we could cover but seeing ones like this that involve patient data and it being dropped to apply pressure is just yeah really really horrible and I don't know what Scotland's options for hound release are I mean Scottish Terriers are a hound I mean this this is not ransomware though this is data theft and honestly i mean it's you know it's a scourge and everything and we should pursue these people but i you know for me the release the hounds thing has always been about the ransomware because of the disruption it causes um i guess though we are dealing with health information and you know this is the way we went with the medibank stuff but um you know've just noticed, though, is over the last week,

I mean, there hasn't been a great deal of news about ransomware attacks. I mean, it's not Orthodox Easter. Why is it so slow? Yeah. Orthodox Easter, yes. Again, it's a good question. I mean, I think it's too soon to say we've, you know, put a dent in it, but it is less than we would normally see on the news list.

I've just noticed like all, like I literally didn't even notice until I was just talking to you now that like all of those lock bit stories about how they're down, but not out and activity. Like, I just, I can't, I can't, I, you know, I'm like going through the record right now, which does the most sort of prolific coverage of ransomware incidents. And I can't, you know, I'm like going through the record right now, which does the most sort of prolific coverage of ransomware incidents, and I can't see really anything, which is interesting. That is interesting. Yeah, we'll have to see how that pans out.

Maybe it's not the ransomware people who are on holiday, though, but a lot of journalists are because of Easter. Maybe that's it. Anyway, just flagging that as something interesting. We're going to end on a sad note this week. Ross Anderson, who is a professor and the author of Security Engineering, has passed away.

Yeah, it's really sad to see. His contributions to the security industry are pretty significant. I know when I read, reading that book, Security Engineering, had a real impact on me and my career path. And it's one of the few books that I've ever read in this industry where you know he talks through distributed systems you know security engineering problems and he said like I helped design the ATM

system for banks and stuff in the 80s here is the threat model we thought we had back then and the way that we addressed it now at the time of writing 15 20 years later here is what actually has happened here is how atm fraud happened etc etc etc here are the problems that we thought we had solved that we didn't need to and here's the ones we didn't solve that we should have and you know this industry is so young that that kind of introspection and reflection i think is super super valuable. And for me, it was an important part of my career. So thank you, Ross Anderson. Indeed.

Well, mate, with that, that is actually it for the week's news. Thank you so much for joining me, as always. And we'll do it all again next week. Yeah, thanks, Wes Penn. I look forward to it. It is time for this week's sponsor interview now with Brayden Rogers of Island. Island makes an enterprise browser that I think is extremely useful in a bunch of use cases. And one of those use cases

that we're going to talk about in this interview is using an enterprise browser as a substitute for virtual desktop infrastructure or VDI. With Broadcom's acquisition of VMware, a lot of people out there are getting off the VDI train, and that is not the worst idea, in my opinion. Of course, an enterprise browser is a very different technology, but it does solve a lot of the same problems.

Island can give you visibility into the health of an endpoint, for example. You can put a lot of controls on it in terms of how it interacts with the file system so people can't just save stuff randomly to disk and so on and so on so it's very useful in the types of places that people tend to use VDI like in call centers anyway here's Braden Rogers talking about the death of VDI let's first of all think about how we arrived with VDI by the way 20 years ago so yeah we we arrived at VDI because and we way, 20 years ago. So we arrived at VDI because, and we continue to use it because if you think about a team that has to deliver application and services to the users,

they have to have a vehicle to do that. And so it became a vehicle they could find that could be acceptable to deliver the apps that was balanced by a cybersecurity team who could accept the risk. So it became this, it really became this kind of least common denominator that they could both kind of negotiate and agree to, and the apps can be put in the hands of the users. It's a safer journey. But the challenge was always, and it continues to be, that the end user pays the tax.

And fortunately, the experience on the end user population, and even the line of business where it may impact the ability for the business to perform effectively can be a challenge. A perfect example, a contact center type situation. You think about contact centers, they have telephony stuff, right? And they have to get the audio through the system. An audio experience through a virtual desktop infrastructure is not always the most optimal experience. So, buffering and weird things that happen latency connectivity issues what we found is the ability for those same

audiences to come to the negotiation where the browser can be the vehicle can be the platform for delivery of the applications and the cyber team's finding value in that the browser has mechanics to keep the data you know safely confined yeah i mean i get what you're saying right which is that these are two very different things but they kind of solve the same same problem right correct the best part is the end users don't pay the tax because it's a native browser you install in the machine yeah so the end user looks at it and they go you know it's it's really cool because when we do the deployments what i find is the users never had to be trained. So the end users, we kind of, I hate to use this word on this podcast, but we exploit the end user's knowledge, the innate knowledge they have on browsers for 20 plus years.

They've been trained personally. They use them every day in their personal life and at work and everything else. So they immediately see it. They see it as a browser, but the practitioners on the back end see it as a very powerful platform for delivering applications safely. Now, we were talking before we got recording, and really why we're talking about this is I was asking what's going on with the business at the moment. And it's pretty much a stampede at the moment, isn't it? Away from particularly the VMware stuff towards yours. You would have been having a lot of conversations with VMware users. I mean, this whole Broadcom thing seems pretty bad,

but what are they telling you about, you know, what that whole experience has been like for them? Well, yeah, you know, I think this is true for the traditional VDI space in general too, because there's other influences with some of the other players too. They've gone through their own transformations as well. Some of it maybe not so good. First of all, let's start with where it is today before those situations arose.

It's already an expensive proposition to manage. You've got this on-prem set of resources. You've got server farms and gateways and load balancing infrastructure. And then you've got to do all the licensing of the things that run in that world and patch management and all this kind of ancillary stuff. And then you get this influence of acquisitions and uncertainty that come into the mix, whether it be private equity or other entities acquiring these things. And then that throws, you're running your most critical applications in this.

This is the thing you depend on to run your business. And now the one piece of technology, you're not sure you have uncertainty about, you're not sure where it's going. Is it going to be sustainable? Is it, you know, and then you, you get a, you get a bill for a renewal cycle with a three-year lock and, you know, you're, you're, you get sticker shock. So you kind of, kind of commonly see on LinkedIn, a lot of the conversations about people that are experiencing some of these things about, this is just, it's already painful enough. Now you've just piled a little bit more pain on top of it. So it's time for all the hurt.

Well, I mean, there's the price hikes, but they're also deliberately trimming back their customer base, right? You know, history has shown that to be the case with some of the things that they've acquired and they leverage. And at the end of the day, you know, I think that it's just kind of fortuitous. It's very fortunate we're here at the right time. I think it would be something where...

Sorry, let me just stop you there. Is that actually something that is happening with the, you know, with VMware under Broadcom? Are they actually firing customers or not yet? You know, I'm not an expert in their business. You know, I think in the past with other things in the portfolio, some of those things could have happened have happened again i would leave it to those guys to tell you kind of what their strategy is at the end of the day but i think the big the big thing that i would i'd tell you i hear from customers regularly is just kind of the the um the uncertainty and at the end of the day and and and but i think at the end of the day but to take that i mean it's not it's not going places right like you you can't look at it and go VMware,

VDI, next big thing. Like, you know, they're going to innovate here. Like that's just completely not on the cards. Yeah. I think, listen, VDI as an approach is a 20 plus year thing we've been using and it hasn't really evolved very much in that window of time. At the end of the day, people were hungry without these influences for an alternative. It just kind of happens to be right time, right place with those things happening as well that again organizations are finding a good opportunity and find a substantial reduction in that process of just of all of the stuff sometimes it's cost sometimes it's the you know the the pain of having to deal with the server farms and there were the bad user experience etc so it's a little bit of everything

right timing right place at the moment for us fortunately so yeah i mean So it's a little bit of everything, right. Timing, right place at the moment for us, fortunately. So, yeah, I mean, look, it's, if you asked me, would I run, yeah. Would I run Island or like spin up VDI? Yeah. I mean, it's a pretty easy decision at that point. Right. That's right. Yeah. Now we were talking, we're talking to about a different use case and that's, that's an interesting one, right? Because what you're talking about there is your, you know, your, your core use case is this sort of application delivery and you know putting some controls and guardrails around that and um yeah it never occurred to me really that you would be replacing VDI but that makes a lot of sense um especially in some of the sort of environments that you mentioned but there are some other sort of left of field use cases that are springing up

and one that we were just talking about as a again before we got recording was um disaster recovery and sort of like incident response as well this is a really interesting idea and you know tell me how people are using island and i'm going to tell you a story about one of my different sponsors and how they wound up accidentally being used in in disaster recovery as well because that was fascinating but uh yeah it was it was actually an interesting use case that we imagined up early on in the organization's journey, we thought it up. And but you know, it wasn't the primary reason people were coming to us for it. But we had several orgs that have decided that in a situation where they go through an incident, you know, one

of the things they all have to do is they all have to shut systems down, close laptop lids and things like that, as the work goes to kind of get the org back on its feet. And it can be a great alternative as a resiliency answer for letting people use another device. Because one of the things we lose in these events, we lose the ability to communicate with each other. We can't use the services we were using because the systems are shut down, laptops are closed. The ability to have an alternate path to the key applications and even using devices that aren't managed by the organization, the unmanaged real estate. So now to restore some of the key apps and get people back to work. So, you know, it's been great in the sense of, you know, we've got several orgs that have stood the browser off to the side as that resiliency kind of ripped cord.

And they've been mid-incident and pretty bad ones too, right? We've had a few of them where they've turned to us and said, hey, this is an alternative for me to get people back to work really quickly and in a matter of days be able to get people back to work in the process. So the great thing about that is you're talking about being able to use somebody can go grab a device at home, get back to communicating, and get back to work. And as the org goes through their efforts of getting back to the place

they need to be, kind of everybody wins in that process. So, again, we've had several of our advisors that early on, you know, brought that as something that would be very useful in their orgs. And they pursued that as an alternate path. So it's been great for us. Now I've got to tell you my story, which is that I had a sponsor called Remediant, which has since been acquired by some big PAM conglomerate. But they had a really interesting product that you could basically hook it up to your AD and get it to nuke privilege, and then kind of just add it back in, right? So you could do like just-in-time elevated privileges and whatever. And what they discovered is that in the middle of a ransomware incident,

as part of a recovery where you're not really sure what accounts are compromised and whatever like it was a really effective way to just go in and nuke privilege from like every account in the org and then start sort of adding it back in and and seeing what would happen and it was just um i just found that really interesting that that actually come up with something that could be deployed so quickly like it's not without pain but it was like yeah it definitely um it was definitely very very useful so quickly like it's not without pain but it was like yeah it definitely um it was definitely very very useful right so you it's always funny when some of these tools that are not really designed for you know quick deployment as part of a disaster response plan like they actually wind up being useful for that yeah yeah it's it's it's been cool when we've

seen some of these things go down the ability to you don't have to have it doesn't have to be all services at once suddenly just let's get some key things back online like the ability to, you don't have to, it doesn't have to be all services at once. Suddenly just let's get some key things back online, like our ability to talk to each other. For example, that kind of matters. I'm guessing people would have been using Island what to get back into their sort of like M365 accounts and things like that. Certainly it could be Slack or their M365 or just kind of core communication vehicles to share content, et cetera. Yeah. And from unmanaged devices where they, where they want to have some sort of idea that it's at least got some level of assurance in it, right? Correct, yeah.

That's the powerful part because the browser doesn't live in a vacuum of assessing the device that it's living on to say, hey, look, is this a safe device to live on? Does it have some basic consumer controls potentially or full disk encryption or what standard of care is it living up to before it does access my resources?

Well, I mean, Island can tell you that, right? Like Chrome can't. You know, and IE, well, whatever they call it these days, can't. Edge. Yeah, your consumer browsers just aren't built for these things. So at the end of the day, you know, when you've got that third party that's accessing your key resources, this is a common refrain I hear a lot.

We signed an agreement with this third-party BPO, Business Process Outsource, to support our call center or our merger and acquisition effort or whatever it may be. And we've signed this agreement, but the agreement is on paper. Technologically, we are expecting them to live up to a standard, but we don't really know for sure that they're living up to the standard. So this gives you the ability as they access the resource to ensure adherence to the standard. If it doesn't live up to it, you prevent access or deprecate access. Maybe you make the apps read only because they're not living up to the standard. And even on the screen, it's funny, on my screen right now, in front of me right this moment, the browser is saying, please update your macOS.

I got one patch level behind. It's reminding me there's a patch level. It put a watermark on certain key applications to remind me. So, you know, it helps me as an end user get coached through. Also, my own adherence to the standard. That's really useful in consumer situations. All right. Well, Braden Rogers, thank you so much for joining us on Risky Biz to talk through all of that.

Yeah, really interesting. Always enjoy our chats. Cheers. Patrick, you as well. Thank you. Great to be here. That was Braden Rogers from Island there. Big thanks to him for that. And you can find Island at island.io. I-S-L-A-N-D.io.

And that is it for this week's show. I do hope you've enjoyed it. I'll be back later this week with a Snake Oilers edition of the program. But until then, I've been Patrick Gray. Thanks for listening.