Risky Business - Risky Biz Soap Box: Defeating Living of the Land

Episode Date: June 25, 2023

In this edition of the Soap Box podcast we’re going to be talking about a great topic – living off the land. The recent Volt Typhoon report out of Microsoft chronicled the adventures of a Chinese APT crew in US critical infrastructure. But one of the most fascinating aspects of the Volt Typhoon campaign was that the attackers almost exclusively used so-called living off the land techniques. So the question becomes – what can you do about an attacker in your environment who has privilege and isn’t using malware? Guests David Cottingham and Daniel Schell, the CEO and CTO of Airlock Digital, join the show to talk it through.

Hey everyone and welcome to this special Soapbox edition of the Risky Business Podcast. My name's Patrick Gray and for those of you who don't know, these Soapbox editions of the show are wholly sponsored and that means everyone you hear in one of these editions paid to be here. And today we're going to be talking about a really interesting topic, living off the land. The recent Vault Typhoon report out of Microsoft chronicled the adventures of a Chinese APT crew

inside US critical infrastructure. But one of the most fascinating aspects of the Vault Typhoon campaign was that the attackers almost exclusively used so-called living off the land techniques. And this means the attackers entered environments and then used built-in Windows utilities and binaries to move around, to move laterally and to get the job done. No traditional malware was involved. But as you'll hear, there's like even a part of this discussion, which is what even is malware. But yeah, in some cases, the attackers, the vault typhoon attackers were

coming right in through the front door with highly privileged accounts. So the question becomes, what can you do about an attacker in your environment who has privilege and isn't using malware? My guests today are David Cottingham and Daniel Schell, the CEO and CTO of Airlock Digital. Airlock makes an allow listing and host hardening package that regular listeners would know that I'm a huge fan of. And as you'll hear in this interview, rolling out decent execution controls, a la, you know, good allow listing, can really help to slow down attackers who are living off the land. And in fact, a lot of Airlock's customers have processes in place that let them continuously shut down, you know, new lateral movement and escalation paths as they sort of become known about. But there is absolutely no magic solution to this type of tradecraft, even if you're using allow listing, which is what makes it such an interesting conversation. So here's David Cottingham

to kick off the discussion. He is the CEO of Airlock and he says the Vault Typhoon report is really a sign of the times. And I hope, yeah Airlock, and he says the Vault Typhoon report is really a sign of the times. And I hope, yeah, I hope you enjoyed this conversation. I definitely did. I think it's just a sort of sign of the times in terms of these are the utilities that have been available to administrators for many, many years, and now attackers are finally using the rich capability that's contained within, you know, run DLL, WMI, you know, and other processes that inherently exist in their Microsoft Windows ecosystem. And there is so much more here, right? You know, there's a lot of security research going on about how we can use, you know, inbuilt processes and undocumented features, and there's more

raining out all the time. But this is pretty, what I would say, you know, run-of-the-mill playbook nation-state stuff at least from what i've seen on other campaigns as well uh they develop a new playbook roll it out um and stick with it until it gets burned or heavily detected and then they'll just tweak it a little bit using different utilities i think one of the things that makes this one novel though is that there's no malware anywhere like usually you see a combination of both right like you see some sort of living off the land technique uh used to move laterally and then they always do something dumb like drop cobalt strike and you know i mean it's not dumb if it works and it works but you know it tends to be more of a blended approach

usually right yeah exactly i mean there's code somewhere. There were a few sort of custom, what they call FRP binaries here with the Vault Typhoon campaign. So there were a number of shartip six hashes that you could look at for indicators. And it's really tempting for attackers to use some sort of custom code. the callback and that persistence easily inside environments without- At least something. Something sticking in there, right? You can't just expect WMI or an inbuilt utility to be calling back to you and providing that rich capability without custom code. I mean, you could do that in PowerShell though, couldn't you? You could, but then that's code that's executing somewhere that's, you know. Yeah, yeah, yeah, yeah.

But that's, but I mean, okay, so that's a gray area. Is a PowerShell script, a malicious PowerShell script that gives you persistence, is that, you know, is that, what is malware? Well, yeah, I mean, essentially it's just code written in a different language to achieve the same objective, right? It's still malware, I think. And, but that's a really good. Script malware, I guess, you know. Yeah, exactly. right it's still malware i think and um but that's a really good script malware i guess you know yeah exactly um you know and i think that over the years we sort of put scripts and living off the land as sort of the same thing but it's interesting to make that and think about that separation my

god you guys really are the hipsters when you are actually turning that into a debate like uh you know is using powershell living off the land uh controversial daniel uh you know you also would have been aware of of this campaign and the report that came out i mean what were your thoughts did it strike you as novel yeah it's the same as what you're talking about before they're being tradecraft like everything here that i'm looking at is you know run dll wmic come on powershell um and that would normally be i, you know, run DLL, WMIC, command PowerShell. And that would normally be, I guess, you know, it suggests that when they're using these sort of tradecraft,

then that's becoming table stakes for the target order, you know, for the targets in this case. And that's why I wanted to talk about it with you two is because it seems like, okay, you've been able to do this a long time. Pen testers at the sort of more elite end of the spectrum have been using these sort of techniques for years and years and years. But now we're seeing threat actors do it. So now it's rolling out to Chinese APT crews, and it's a matter of time before it's the ransomware

and data extortion people, right? So as you say, this is going to be table stakes. Like, well, it probably is already depending on who you are, but it's going to be for everyone soon yeah for sure so i i think there what we really need to be working on is like okay well you know what can we take from these sort of these indicators the good thing about this sort of stuff like when you're at this when you're looking at defense at this level then you know there's only sort of limited amount of things that can be done because the actor is so restricted.

So you can start looking at the, okay, well, they're running PowerShell with command.exe and they've got the exit bypass in there with the hidden window. Can we flag or can we prevent these PowerShell executions happening based on the context we know about them? And it might be weird for us from a block listing, talking about block listing or really rule from us from a you know a block listing you're talking about block listing or really rule sets here from a allow listing company but you know the the thing

there is is that there's there is sort of like a limit of things that you can do there so it's not like you're blocking all the unknowns like it's not like we're adding hashes of no new bad things there's a limit so bad behaviors that they bad behaviors when the attacker is this restricted. So it's about understanding them and then providing rule sets or building rule sets as new behaviors of this nature or TTPs are discovered. That's interesting, isn't it? Because it is in some ways similar to that EPP

and even AV thinking, right? Where there's a new attack, there's new tradecraft, and then you have to actually update the product to block it. I mean, I think the interesting thing, though, is you're actually blocking entire... When it comes to living off the land, you are actually blocking entire paths, lateral movement paths and whatever,

with one configuration change, and it's enduring. And that's something where it might be different from the old AV way of just blocking a specific threat. Yeah, and it's not about blocking the specific things. That's what I mean. Yeah, you can't just block PowerShell. You can't just block WMIC. And a lot of time in Twitter land,

people just go, yeah, no worries. But the reality is so much of that's running in the background all the time, SCCM and just Windows itself internally and scheduled tasks. So it's being able to differentiate what's your legitimate executions um and so it's really about you know understanding the context that these executions run in um well it's about being able to specify the context in which they should run which is the whole purpose of allow listing

right yeah and and that's pretty much it so that's where like you know even even this week we were doing some work where a customer was saying hey hey, we want to block PowerShell when it's being run by the system account in certain ways. So, but, you know, and then maybe, and also when there's certain command line parameters in the PowerShell string and such like that. So, you know, definitely, like, you know, higher end users of our product are understanding or, you know or aware of this and then confirming rule sets around this.

So sometimes it's about knowing the approach. What's the threat they're trying to deal with there? Is that when an attacker has actually authenticated to the box and he's just trying to run PowerShell? Well, in that case it would be a case where the attacker is actually running in the system process or

system context, I guess. So the idea being that um you know they already have like i guess for them it's more of a case of powershell should never run in this context yeah that's the way that they're looking at it so so make sure we don't but i guess that would be a case you know a way we'd commonly see that would be a case where someone sort of come in they put ps exec on the box they've escalated themselves to system that's a very common thing because then you know they're not gonna have any access problems in about when they start trying to dump memory and everything else they want to do and your system is a bypass you know running as a system user is less likely in some cases to create detections for certain you know security programs so what's happening in this situation with this customer is they go they're

creating this rule with airlock and an audit capability so they go you know they turn on that rule and they go, hey, let's just see what's running in our environment because right now they don't know. So they go, let's see what runs the system in our environment. They get that logging information, and if they don't see any exceptions for a week or two, they might go a month, maybe a patch cycle. But they can set an exception where it's like,

oh, okay, CrowdStrike is trying to run it, so we should probably allow that. Yeah, so this parent process is allowed, right? Anyway, hang on. I just realized we've already got massively distracted thanks daniel uh we've got massively distracted because we're talking about your approach to defeating uh living off the land and of course that's obviously something we're going to talk about here but one thing that we did want to mention and i noticed it as well so david mentioned this just before we got recording and i twigged to this as well, which is that Microsoft's remediation advice here, well, not remediation advice, but their

advice to people on how to protect yourself against this sort of thing when it comes to the Vault Typhoon report seemed pretty generic, right? It was like, you know, how should you not get owned this way? Do good security okay you know that that seemed to be the vibe i mean that was also what you took away from that wasn't it david yeah i guess microsoft's got that problem where you know they're putting new great security features in and it's on the latest version of windows but the reality is there's still windows xp out there in the enterprise right um and in

order for these security mitigations to run through when you need them today uh it's not realistic to be able to go to the latest version of windows 11 configure the group policy and and do all of those type of things but i i think my biggest challenge uh with a lot of the microsoft uh advice in a lot of these cases is the complexity when it comes to group policy and also turning on and making sure that some of these features are actually working right um you know it's either reading github pages or um and it's just challenging to get your systems configured in the right way to actually implement some of the advice that's here because it's really technically complex. But I mean, this is Microsoft in a nutshell, right?

Like until a few years ago, and it's something that I've brought up a million times on the show, until a few years ago, if you wanted to wrap any sort of management around, unless you were E5 and using a CASB product, if you wanted to wrap any sort of management about what sort of OAuth apps could connect to your users you know oh three six five accounts you had to do all of that through powershell right so microsoft microsoft is just notorious for having really

difficult to use uh protections yeah and also verifiable right because quite often you know yeah and that's it yeah so you'll tick the box saying don't do this and uh you know you think it's you think it's all disabled and then that thing still works i remember i remember years ago when i still used microsoft word as a journalist to disable uh i think it was it was or the autogrammer or something right like you had to disable it in three different menus you had to unclick it in three different places right to get that to work and that you know it's just that's microsoft yeah and and i think that confusion one of the things here is actually a bit of advice for volta which is the attack surface reduction rules i've had in you know my previous career doing security consulting people would go yeah we configured it

in group policy because it's there in the group policy template they turn it on and then it's like oh well we're using another endpoint solution we're not using microsoft defender for endpoint so is asr working or not and it's like today i think that you need microsoft defender for endpoint for asr to actually work and if you're using some other av or that turns it off then asr doesn't actually apply now that's but i can't be certain on that because there's nothing that i found definitive so well i mean let's take a step back for a moment and i just remember too it wasn't the grammar checker it was like the part of word that would turn quotes into smart quotes which would then break when you put it in a html based like content management system for online publishing so you had

to disable that in like three different places but But look, one of the reasons the Microsoft advice here kind of sucks, right, is because this is a difficult problem to solve. Yes. You know, this is not a trivial issue. And in fact, even in an allow listed environment where people have set the rules up nicely, there's only so much you can do like in one of the cases that was talked about i think by secure works someone like brute forced the domain admin account from an internet facing citrix gateway and that was how they gained their initial entry like from

from the internet to domain admin uh thanks uh thanks thanks whoever like set that box up you know so there's no magic that's going to stop an attacker who lands with domain admin but that said you know everything that i've been thinking about like there's two things that are going to help here one of them is really good monitoring and the other one is yeah definitely going to help here. One of them is really good monitoring. And the other one is, yeah, definitely going to be allow listing because when an attacker starts wanting to move around, even if they're authenticated with a lot of privilege, they're still going to run commands.

They're still going to hit executables that in an environment with your stuff in it is going to actually raise flags, right? Because there will be, you know, this admin tried to run this command uh and it's and it's and it's blocked yeah and i think look to give microsoft a little bit of a break as well as they got the task of solving these problems and and providing security here for everyone without breaking anything you know and that is a challenge at global scale so i really

appreciate that i i think uh you know microsoft. I think Microsoft's security approach going forward is really about sort of isolation and sandboxing. It seems to be pretty heavy. Make sure that LSAS is a fully protected process and you can't get into it. It's about building out those different building blocks of the operating system and making sure they're secure. I guess I'm interested to see going forward when and if this even happens when there's a tension turned to these components that are used to be you know lol bins to say that okay you need to be you know you can only use WMI if you're you know in this particular you know level of privilege or or starting to cut off some of the functionality of those binaries a lot of our time is spent doing research into how are these binaries used like we know what they're meant to be used for right but how are they used and what what uh parts of the binary or what aspects can we sort of close off without causing a production impact for customers so then we can write a standard rule and

ship it and and um you know look at uh okay this will actually cut off uh reduce your risk from that particular login yeah i mean i mean a lot of it as i as i alluded to right like a lot of it um would i mean even in this case right like their standard way of of gaining access to a target was to pop shell with a zero day in Fortinet and then try to run NTDS util via WMI on a domain controller. Now, I spoke to you guys when all of this was happening. And, you know, it turns out like you wouldn't have been able to do much in that situation under a default rule. You can't just blanket block some of these things like we've been saying, like NTDSUtil, that's something that's going to need to run. You can't just stop it, right? The question is like, is NTDSUtil needed or automatically used? It's the same approach

as before, right? The answer for this whole stuff is you don't know, right? So my value is find out how to do it, find out how it runs. Does it runs does it run normally if not okay how can we control it because obviously it's something that attackers do so the example there in this case you know to be on the technical is you know w they call wic on a machine with creds they then call command.exe which then you know create some folders and does nt dsutil but you know you have to think about how that execution context happens now most organizations can't just say hey i'm just going to block ntdsutil when the grandparent process is wmic probably to off the top of my head that sounds like a pretty good rule to me but if you weren't sure on that like you know the benefit of having an allow listing approach

when you're still thinking about more about the control aspect of it is the idea of being able to say okay well let's find out how we run this. How often does this run in my environment? How does it run? Understand that context and then restrict it so it only runs within that context. So by having that visibility, you're then able to then sort of apply that to say,

okay, well, it turns out in our environment, yes, NTP... This is just the process that you described for this customer who wanted to look at how PowerShell was used in a system context, right? So is that becoming just how customers do it? They just create a rule, roll it out in audit mode, and then wait and see what comes back to see whether or not it's something they can lock down and how tightly? Yeah, that's exactly it. For the customers that are really on the ball, that's where they're looking at.

It's not so like, you know, if you think back a few years ago you'd be like oh what's the latest indicators oh there's shards from this campaign you know i've got this new threat report i've got these yeah shard 256 let's block them um you know now people are like okay i need to look at the behavior and i need to make you know determine i don't i don't want people i don't want different attackers or other tradecraft groups we're gonna use this the second it's published to make sure we block that as a type. But the good thing is, the Valerian thing is there's limited

methods or versions of this, right? There's not an unlimited amount of hashes. There's a limited amount of behaviors. So instead of blocking individual threats, you're blocking individual lateral moves, right? Which, yeah, makes a lot of sense david you wanted to chip in yeah it's gonna say one thing that we we're gonna really focus on

over the next 12 months eternally is providing more of these rule sets and figuring out what's the practical approach in amongst all of this you know um and of course a lot of attack reports like voltaiphone really inform that you know and because they're always little subtle variations on, you know, a wider playbook, but it all comes down to a lot of these new attacks, you know, WMI, PowerShell, CLI in some regard, and then moving from there to other weird and wonderful binaries. Did you actually give customers mitigation advice on this? Did you say you want to spin up this rule and, you know audit it and and see or did or are they just like grown-ups who can handle this themselves no so we we hadn't had uh didn't do any reach out on on this one specifically uh there were some recently about um you know a lot of stolen drivers um i believe it was from the msi

um hack yeah yeah yeah we covered that yeah there were there are customers that reached out being like hey how do i block this and then also some of the um uh you know resulting activity that we've seen uh you know from that particular campaign and then we uh we pushed out some advice on that one yeah what was the advice don't use those drivers yeah first of all it was it was allow listing will block this by default because they need to... There was an executable involved in that one and also... So it was a heads up that some of your drivers might stop working, basically.

Yeah, correct. So it was block the XE by default and then it was also ban the certificate thumbprint of the particular stolen driver that was used to sign the driver. So then even if that certificate was used to sign any other bit of code then it wouldn't actually run and you're proactively preventing it yeah yeah and just just close closing the circle before as well like the other thing in this full typhoon campaign it was you know the traditional lsas mini dump um to you know drop creds um you know and microsoft's um Microsoft's advice on that is you turn on

protected process light and easier said than done or yes it's default on brand new Windows Enterprise 11 machines but what about your Windows servers or 2012 domain controllers and stuff that are floating around. And what attackers are then doing is if there is

but even if the attackers even get to that protected parent process then what they're going to do is like you know according to you know to current twitter tradecraft and we have seen it once twice is then they will go towards the bring your own vulnerable driver so you can unprotect the process and then dump it yeah um but that and that's where the traditional allow this thing comes in well that's right i was going to say like doing that where you are is hard well it's basically impossible so yeah yeah so it's like this circle right where you restrict them down to really limited tradecraft that's all and then you know other other opportunities hardening or similar but then makes them go

back to binaries and then binaries are easy for us yeah i mean when all this kicked up adam and i were talking about this and how airlock might treat various bits of it. It was his feeling, and he was right, that the NTDS util stuff, like, would not be blocked out of the box by Airlock because it would cause too much drama. And, you know, but I made the point, okay, say you dump the entire Active Directory in an environment where Airlock is, then what? You know, like, how are you going to then move around without actually hitting a blocked execution and raising an alert i i mean i've i've sort of seen a bit of this before uh which is like the attacker just has to assume a standard user and do the things that you have to allow standard users to do they will start rdp'ing around the place um you know and they will start uh just

copying and pasting things through rdp uh you know and that's not it's that's not a fast way to x feel like 90 gig though no it's not but it forces them to tiptoe right and also the reason when they start using those techniques then uh people generally figure it out because they're like hang on i didn't log on it this time which is my last log on like it starts to leave a lot of other footprints that a since it's attributable to a person that people under start to understand that behavior and generally in organizations they have an admin team where all the admins will know everyone and they'll be able to start to pick up on those things yeah and i'd imagine too that like is that rdp just into what servers that are normally rdp'd into or yeah and also things like exchange servers and you know and then it really comes down to you

know uh how are you designing your network to stop that sort of continued lateral not even to stop it i mean you know because we often talk about how, you know, airlocks are preventative control, and it's felt like people gave up on them for a long time. But it really does feel like we're actually finding more of a balance these days between prevention and detection, right? And I think people who are really rabid detection-is-everything types are realizing that decent preventative controls actually just give

you mean that the signal to noise that you're detecting on is just so much better yeah exactly you only want them to trip over something it's an arms race as always and if you can even slow the attacker down from the initial land and expand once they get into an environment then you've got less to clean up as well so all right. So let me ask you this, right? This is the first campaign that's really made big waves for being, you know, a living off the land, you know, APT crew, blah, blah, blah, blah, blah, right? Like it's the first sort of contemporary one that I can think of that's been, you know, widely discussed.

So where does it go from here? Like what are the next big trends going to be in living off the land tradecraft yeah it's just it's just gonna be more of the what we've seen the vault typhoon yeah it's gonna be more of like what can we do without bringing in our own tooling and the fact that you're coming in through a vulnerability that's an expected thing for fortinet citrix whoever it's gonna be come into the network um then how do I jump into another box? Often from those appliances, when they're compromised, they have some sort of Active Directory creds in the appliance.

So that's their starting point. And then from there, it's like, how do we move forward without using custom? Yeah, domain joined Fortinet is just like the biggest own goal on the internet at the moment. But yeah. That's pretty much it yeah yeah i i think one thing that i would sort of like to see as well is you know there's a lot of talk and and you know potential legislation looming about sort of software bill of materials for vendors right

you know tell us what's in your software and but what about and i don't know this is a big thing it's like software bill of behaviors almost right yeah well you know extra hop right we're a sponsor and a few of their team had that idea and they spoke about it on the show and it actually got traction they had calls from all sorts of interesting people and we spoke about it a few times because you know particularly for the i mean look at the network level that makes a lot of sense right like here are this software's update service for example right and you know this is what that communication looks like but i see where you're going with this

which is what process what windows processes is this thing going to invoke what dlls is it going to rely on yeah and i can see you grinning and smiling and thinking yes yes um but it makes sense doesn't it yeah exactly i mean i i often think i often think actually a bill of behaviors is more immediately useful than a bill of materials yeah i agree um because it's you know from this again vault typhoon this write-up is talking about the things that really the process shouldn't do well i guess that's the thing. Should it? But they're all Windows processes in this case, right? Exactly, and they could be used to do so much type of stuff.

But we could start from a place of, you really shouldn't be able to do these things. Like, you shouldn't. And again, legacy, and it's a really tough challenge. But I think if we, you know, normalizing what's in the software, you know, if we can get some information about how it behaves, it would help. But I think that's just going to evolve because of sandboxing more and more in OSs, right? So in the future, I can imagine that browsers are going to be,

I mean, you know, browsers are going to be pretty tightly controlled in how they interact with the system. They already are, you know, and I just think as that gets pasted out to every single other type of app, I just think we're going to see more and more of that. Yeah, and it's a big push by Microsoft to go towards the Windows App Store where things naturally, when they're downloaded and run, they're all attested to and signed. But they're also all in an isolated sandbox, low integrity state. And they're only elevated when they need to do certain things. But I mean, again, that doesn't help us here does it because of the for to fail gateway and it's you know gigantic service account which just you know and here we are and here we all are talking about that's why it's a tough problem and that's why i wanted to talk to you yeah and we we sort of

done talk internally about you know okay what's the competitive landscape with allow listing like you know in zero trust is being plastered on you know every company's security booth everywhere um you know and there's there's a few companies that are talking about well if we can secure the perimeter again then you know it's and and i've sort of always come back to allow listing will always be relevant as long as there are endpoints run code yeah like well i mean you know endpoint integrity is really important to the you know uh to the zero trust model i mean it's a zero trust networks paradigm not a zero trust endpoints paradigm you know yeah yeah exactly the part that's interesting on well even microsoft right there they're putting in sandboxing like they've just announced or you're know, coming in Windows 11 H2, the ability to, I think it's in beta at the moment, to sort of sandbox or containerize your Windows 32 applications.

And they're really hoping that developers will write definitions to allow behaviors. That's what they're doing, right? So they're pushing on that. But to be honest, like, you know, I haven't looked into it yet. But for me, it's all like, can we create some of these definitions to help with that or enforce them maybe because again i wouldn't know how to configure them i wouldn't know how to set them up you know as an it administrator and it just falls back into the standard thing was okay well that's great that's that's the leading bleeding one percent of our endpoints that have that capability

yeah i mean you could derive rules from that but i mean i guess a lot of that's going to move into the os though if it's if it's an enforced sandbox where they have to actually supply a manifest presumably the os will be enforcing that as well yeah yeah but how does that will be the interesting part yeah yeah and you need to take you need to take the stick approach to change developer behavior for the ecosystem like and you think back to why everyone hated windows vista which was user account control was introduced right and everyone before that was just change developer behavior for the ecosystem like and you think back to why everyone hated windows vista which was user account control was introduced right and everyone before that was just writing software assuming that they had administrative access to everything and vista changed that and

then everyone you know it was backed off a little bit in windows 7 but the real thing that made windows 7 more palatable was the majority of the software ecosystem had already adapted to running with lower privileges so and this is going to be the same with sandbox i thought vista was hard done by actually i i did not i thought for its time vista was pretty i agree you know i'm like one of the 10 people in the world who seems to think that yeah i agree yeah it was it there was a device compatibility nightmare yeah just because they had... Just time. Yeah, that's it. The internals were changed completely. Yeah. I mean, it's basically the... I mean, it is more or less the basis of

Windows now, right? Is Vista. So, yeah. Alright, guys. Let's wrap it up there. Always great to chat to you both. David, Daniel. Really interesting to get your perspective on that. Pleasure to chat to you both. Thanks, Patrick. Cheers, Patrick. That was Daniel Schell and David Cottingham from Airlock Digital there. Big thanks to them for

that. And you can find them at airlockdigital.com. And I recommend you check out their stuff. But that is it for me today. I do hope you enjoyed that conversation. I'll be back soon. Thanks for your company.