CyberWire Daily - You can add new features, just secure the old stuff first. [Research Saturday]

Episode Date: August 14, 2021

Guests Will Schroeder and Lee Christensen from SpecterOps join Dave to share the research they recently presented at Black Hat USA on the security of Microsoft's Active Directory Certificate Services. Their abstract: Microsoft’s Active Directory Public Key Infrastructure (PKI) implementation, known as Active Directory Certificate Services (AD CS), has largely flown under the radar of both the offensive and defensive communities. AD CS is widely deployed, and provides attackers opportunities for credential theft, machine persistence, domain escalation, and subtle domain persistence. We present relevant background on certificates in Active Directory, detail the abuse of AD CS through certificate theft and active malicious enrollments for user and machine persistence, discuss a set of common misconfigurations that can result in domain escalation, and explain a method for stealing a Certificate Authority’s private key in order to forge new user/machine “golden” certificates. By bringing light to the security implications of AD CS, we hope to raise awareness for both attackers and defenders alike of the security issues surrounding this complex, widely deployed, and often misunderstood system. The blog post and white paper can be found here: Certified Pre-Owned blog post Certified Pre-Owned white paper Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. data products platform comes in. With Domo, you can channel AI and data into innovative uses that deliver measurable impact. Secure AI agents connect, prepare, and automate your data workflows, helping you gain insights, receive alerts, and act with ease through guided apps tailored to your role. Data is hard. Domo is easy. Learn more at ai.domo.com. That's ai.domo.com. Hello, everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities,

solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us. A lot of times, people focus more on adding new features rather than securing and modernizing their older stuff or just completely removing older things that shouldn't be used anymore. That's Lee Christensen from SpectreOps. He and his colleague Will Schroeder are joining us today to discuss their research, Abusing Active Directory Certificate Services. And now, a message from our sponsor, Zscaler, the leader in cloud security. Enterprises have spent billions of dollars on firewalls and

VPNs, yet breaches continue to rise by an 18% year-over-year increase in ransomware attacks and a $75 million record payout in 2024. These traditional security tools expand your attack surface with public-facing IPs that are exploited by bad actors more easily than ever with AI tools. It's time to rethink your security. Zscaler Zero Trust plus AI stops attackers by hiding your attack surface, making apps and IPs invisible, eliminating lateral movement, connecting users only to specific apps, not the entire network, continuously verifying every request based on identity and context, Thank you. can't see. Protect your organization with Zscaler Zero Trust and AI. Learn more at zscaler.com

slash security. Active Directory, first of all, is it's a Microsoft technology and it's used primarily to set up, configure, and administer users and computers inside of a network. For example, users log in with their username and password. A computer somewhere has to store that, and in a lot of networks, that's an Active Directory. I'd say probably more than 90% of businesses use Active Directory underneath. And so it manages authentication and also controls access to things as well, and then allows administrators to configure computers.

And Active Directory Certificate Services is Microsoft's public key infrastructure. And that means a lot of things. It means that it does several different things. But for us, what's really interesting is it adds a new form of authentication. This is kind of like our area of research is authentication. And so rather than just using a password to log in,

you can use something that's called a certificate. And it's just like a special kind of file that the Active Directory certificate services can issue to a user or a computer. And if you have that certificate, you can then log into Active Directory using that certificate file. I see. So basically it's a way to establish machines on a network, you know, within an organization to say, we know who you are, you've verified you have this

legitimate certificate. And so, you know, we're good here. Let's go on with our communications. Exactly. Yeah. So in your research, you point out that there was a particular sentence that stood out to you in the Active Directory technical specification. I'll just read it here real quick. It says, in the case of DCs, the external authentication information that is used to validate the identity of the client making the bind request comes from the client certificate. How did that start you off down on this pathway? Will, can you fill us in there? It's actually, I'll kind of toss it to Lee, just because Lee was the one reading the specification and then came to me in early January and said, hey, Will, did you know that you can use this certificate stuff? And we just kind of started diving into it.

So he piqued my interest because we've both been researching Active Directory for a while. And I was like, oh yeah, I've been meaning to look into that certificate thing or that PKI thing in Active Directory. So that was kind of the spark in Lee's mind that drove both of us down this very large rabbit hole of Active Directory certificate services and everything that it entails. And what was it about that, Lee, that made you raise your eyebrows, that grabbed your attention? I think in particular it was, you've heard about logging on with

username and password. We've done a lot of research previously with Kerberos. We've heard about NTLM, but I'd never played around with this thing that says you can log on with certificates before. I was reading the section in the specification about LDAP authentication, and I'd just never actually tried that before. I wrote up a quick POC of authenticating to LDAP using a certificate, and I just shot that over to Will, and I was like,

hey, did you know you could do this? And that kind of got the ball rolling, and here we are now. And with a lot of the research that we end up doing, we don't set out ahead of time thinking we are going to exploit this thing or find these security issues necessarily. It starts from kind of a point of curiosity, kind of like Lee mentioned of, oh, this is kind of interesting.

Let's start diving into this. And then usually out of that process, we end up finding something interesting from a security context. Well, let's go down that path together. Can you walk me through, what did your exploration look like? And at what point did you start to realize that perhaps there were some security ramifications here?

Yeah, I would say, so as we started looking into this, we definitely built on top of, you know, a lot of other people's work. So Benjamin Delpy, the creator of Mimi Cats, he's done a lot of research into this previously. As we started diving into this before too, we noticed scattered throughout the internet, people had touched on Active Directory certificate services here and there, talking about different types of abuse, different types of escalation techniques.

It was kind of hidden in different quarters of the internet. Part of our research was looking at what people had already published and aggregating that, but also distilling it all in a way that makes sense from an offensive perspective or from a penetration testing and red teaming perspective. So consolidating that all together. And as we did that, we applied this to, we found several different categories of attacks and tradecraft that we could do.

So different forms of privilege escalation inside of Active Directory, as well as ways you can persist on a machine or on a computer, or different ways to steal credentials or maintain access to credentials inside of a network. As well as finishing off with, that was kind of near the beginning of the research, and like Lee said, we would start, we always try to do our due diligence and just try to research on what existing work

is kind of out there and build upon that and aggregate everything and kind of iterate on it. Near the end of the research, we also realized like, well, we had this kind of pipe dream for a long time of saying, if we were able to steal the private key for a certificate authority, could we then forge our own kind of golden certificates as well? So that was kind of the final cherry on top, right, I think, Lee, where we were able

to do this domain persistence and actually build tooling and prove that it was possible to forge these unrevocable certificates as any user in the domain. So is this a matter of, when you're going through that aggregation part, when you're seeing the work that others have done, I mean, was it a matter of lots of people had found little bits and pieces here and there

and had said, oh, this is interesting, but as far as you all knew, no one had really rolled it all together to get sort of that old saying of the whole being greater than the sum of the parts. That's true. Specifically, there is a lot of stuff that we cover in the paper from an attacker tradecraft standpoint. Like we had mentioned, we have user credential theft

without touching LSAS, we have machine persistence, domain escalation paths, and domain persistence. So most of the stuff that we had found online tended to focus on one or two specific domain escalation scenarios. We cover eight in the paper, and one of them is completely novel. The most severe one is definitely novel that Leah discovered.

But most of those have been covered to some degree, but people would just maybe touch on one of them, like one issue, one misconfiguration. What we started to realize is that if you don't bring all these pieces together into one place, if people run across this one specific misconfiguration, we title those ESC 1 through 8 in the paper just for shorthand, that if they fix that one thing, they might think they're secure.

But there's so many of these different escalation scenarios that the vast majority of environments we've run across have had at least one of them. So there's a lot of value in bringing them all together so people can holistically say, is my Active Directory Certificate Service installation misconfigured or not? Active Directory Certificate Service installation misconfigured or not? I'd also add on to that saying, as we saw some of this existing prior work, and as we started looking at real networks,

so we're consultants. Our company is a consulting company, and we perform penetration testing and red teaming. And in our role, we're helping expand, as technical architects, our role is to help expand our capabilities. And so as we were performing assessments, we were trying out this new technique to try and formalize our process around looking for it. And we realized, too, that it's a very complex system

and there's a lot of gotchas that come along with that. So maybe the public stuff had documented a scenario where it's vulnerable, but if these other three configurations are in place, it's not going to work. Or if it's configured this way, you have to do another thing. And so there are some added complexities that as we started applying this to real networks, we realized, oh, this isn't necessarily as simple as a lot of people portray it.

And so for a lot of these techniques, we state several requirements that have to be in place beforehand before these are actually vulnerable. be in place beforehand before these are actually vulnerable. Can you take us through, you mentioned that one of the things you discovered was novel and also severe. Can you walk us through that one as an example of the sorts of things that you all are describing here in your research? Yeah. So this is escalation vector eight in the white paper. And what it is, is Active Directory Certificate Services has a function called enrollment.

And so a client machine, so I'm a user, I can basically say to Active Directory Certificate Services, hey, I would like a certificate. And ADCS, Active Directory Certificate Services, will send me a certificate. And then I can Active Directory Certificate Services, will send me a certificate. And then I can use that certificate to authenticate. Well, that's fine. Usually the protocol where that occurs,

it runs over DCOM, so it's an RPC-based protocol, basically. And what we found is there's an additional server role that many environments have enabled. It's called the web enrollment web interface or something like that. But when that's installed, there's an HTTP endpoint that gets enabled on the ADCS server.

And this endpoint uses NTLM by default and doesn't use HTTPS. Why that's interesting to us is now we can perform NTLM relay to that endpoint and request a certificate. When you combine this with some other tradecraft that's out there, three years ago I discovered a technique called, we created a tool called Spool Sample. It's a great name, but the whole point of

the tool is it allows me to coerce a machine to authenticate to me. So I can basically say to, you know, like a domain controller, hey domain controller, please authenticate to my machine. And when that domain controller authenticates to me, I can use NTLM Relay and impersonate the domain controller and access the ADCS HTTP endpoint and request a certificate as the domain controller. And this has since been expanded on more recently. So if you've seen in the news recently, there's an attack called Petit Patam. It's basically another coerced authentication technique. You can use that. Instead of using Spool Sample, you can use that tool instead, and that'll coerce authentication. Then you can relay it to the ADCS HTTP endpoint, impersonate a domain controller, and get a certificate as the domain controller.

And once you have that, you can basically log on as a domain controller and do things like steal users' password hashes, and then log on as those users. Wow. How serious do you consider these findings to be in terms of, you know, when you balance it against the number of organizations who are using this and the ease at which people could take advantage of these vulnerabilities you all are describing here, you know, how big of a flare are we sending up here? I'd say, so for Escalation 8, I feel like that's a 8, I personally feel it's a fairly severe vulnerability. I don't think it's that trivial to exploit. There are some preconditions around it,

but by default, if you install Active Directory certificate services with that HTTP endpoint, your environment's vulnerable. And every environment that we've been into so far that has that HTTP endpoint enabled has been vulnerable. So I personally feel it's pretty severe. In regard to a lot of the other tradecraft and the credential theft and the persistence techniques that we talk about in the paper, those are important to be aware of. I wouldn't say they're quite as severe because it's the intended functionality of ADCS.

This is a legitimate technology that's there for a purpose. Like a lot of useful technologies, they can be used for good, but they can also be used by attackers to achieve their own purposes. So I wouldn't necessarily state everything is a severe vulnerability, but it is things that network defenders need to be aware of because they need to have detections in place for certificate abuse. And then they also need to add in steps to their incident response process

to account for certificate-based tradecraft. A good example of this is, certificates are valid by default for, it's either one to two years. Computers and user accounts typically will have a certificate on a machine. If an attacker compromises that machine for 30 seconds, the attacker could steal those certificates and then basically be able to log on as that user

for up to one or two years after the compromise. And that survives a password reset, that'll survive a system wipe, because it's a completely separate form of authentication. So it's not a password, and because it's not a password, even if the password gets reset, the certificate's still valid. And so incident responders and defenders need to take steps so that when machines compromise, they also look at Active Directory certificate, to revoke issued certificates to that

compromised machine. I see. So what are your recommendations there? I mean, there's a lot in this research and I recommend people check it out. There's really, you guys have covered a lot of ground here. But from a basic point of view, you know, what are the best steps to protect yourself if you're running these services? Oh, there's a lot. So in the white paper, and we actually have a short link for the white paper for any listeners that want to check it out. It's c-u-t-t dot l-y slash pre-owned, all one word. It's also on the Spectrof's website under the research and development section. We have 40 pages of defensive and preventative guidance, like in the white paper.

So the issue with this, or one of the problems, is that because there's so many aspects of the attacker tradecraft, that there's not just one simple fix or one thing. There's prevention guidance that we cover of ways to kind of secure and lock down some of these things. For example, locking down the NTLM reliable endpoint, that HTTP enrollment endpoint that Lee mentioned. Or things like treating certificate authority servers as tier zero assets and things like that. There's also detective guidance. So for things that you can't completely stop, how do you, for example,

investigate if somebody or a compromised user had requested a certificate maliciously? Someone had taken an over their account and stolen their credentials by doing a malicious enrollment. It's so expansive for the different things that we cover that we have prevent one through eight and detect one through eight or something like that as well, as well as architectural guidance for how to structure Active Directory certificate services. So it's pretty expansive with the things that we touch on. So I think just at a super

TLDR high level, it's treat these servers like they are tier zero assets, and then come up with playbooks for your organization to investigate certificate enrollments and things like this, as well as actively auditing your Active Directory Certificate Services environment to see if any of these misconfigurations are there for the escalation scenarios. And for that case, we have a public tool already called PSPKI Audit. It is on the GitHub slash GhostPack repository, that organization.

So that is a defensive-oriented tool that we released along with the research about 40 days ago, or a little over a month ago. And those will enumerate this escalation one through eight scenarios. And then there's also advice this escalation one through eight scenarios. And then there's also advice in the GitHub repo and the white paper on how to mitigate each scenario. And we are also releasing an offensive tool at Black Hat here coming up pretty soon called Certify. So we kind of did a self-embargo for the offensive tooling

to make sure that defenders could try to audit and try to tune everything up and get everything mitigated as best they could. So if you're running Active Directory certificate services, it's worth another look. Don't assume that what you've got is properly configured or secure. Yeah, we definitely agree with that statement because many of the environments that we've looked at, they're run by very smart people and by very good Active Directory architecture teams. But Active Directory Certificate Services and PKI is almost an entirely separate domain of knowledge. And some of these

misconfigurations can be very subtle. It can be one bit flipped, literally one bit flipped on one certificate template that can cause compromise of the entire environment or something like that. And again, a lot of these pieces can kind of stack together to where maybe even an Active Directory Certificate Services Administrator wasn't even aware of all the possible ways it could be misconfigured. So if you are running ADCS, we highly recommend auditing for the escalation scenarios at a minimum. But definitely check out the white paper

for the other attacker tradecraft, like credential theft and things like that. I always like to compare Active Directory certificate services to the old big boiler that you have running in the basement. It's been working fine, but it's been there for probably 20 years. And as long as it's working, the network's going to keep running,

but you should probably get it inspected because there's probably some things wrong with it. And just given that it's so old, it's very likely that there's been misconfigurations over the last 20 years. Like Will had mentioned, in 80 to 90% of the networks we've looked at since starting this research, they've been vulnerable from domain users, so low privileged, to domain admin, so complete takeover of Active Directory. So for us, we know based off of real world experience that this is misconfigured. So using PSPKI audit to help audit for these things,

organizations definitely should focus on doing that. I will say that one common question we get is, did you report this to Microsoft? We did. So we reported everything, all the tradecraft and everything, which we assumed probably wasn't going to be a fixable situation or wasn't going to be a scenario. They really made changes because most of the escalations and other types of things for this tradecraft tend to

be post-install misconfiguration. So it would be administrators like accidentally misconfiguring something in an environment. So those scenarios tend to not be fixed by Microsoft. But that one escalation that Lee mentioned with the relay to the web enrollment endpoint is pretty much out of the box, and we do believe it should have been fixed. We did report it to Microsoft several months ago, and they came back with a this finding is valid, but we are not going to fix it type of determination. So in case anyone was wondering,

we did try to do our due diligence and we reported everything before we even published anything. And because of that, and because we believe these misconfigurations are so widespread, that's why we kind of did the self embargo of releasing all the information and defensive tooling first and waiting 45 to 60 days to release the offensive tooling. I'd add on to that too, kind of a more general problem the industry as a whole has is just either deprecating old technologies

or modernizing older technologies. And this specific escalation technique, the relay to the HTTPN point, it abuses NTLM authentication. And that's like an old protocol. And our industry knows it's insecure. But despite that, it's been over 20 years now since we've been abusing it,

it still hasn't been deprecated. And there's been very little, I guess, coercion by Microsoft to get people off of using NTLM. They actually issued a KB article in response to the Active Directory Certificate Services abuse that's going on, in response to our paper and the Petit Patam attack that's going around right now. And they said, we've talked about,

the KB article basically said, we've mentioned this previously, you need to work on disabling NTLM. And that's been said repeatedly over time, but I think once Microsoft kind of takes that stance of they're going to disable NTLM, that's when people are actually going to be protected because that's how you're going to achieve

maximum coverage across the entire industry once they make the change rather than telling customers, hey, you have the burden of knowing that you're vulnerable and then also configuring it so that you're not vulnerable. We think it should be fixed out of the box. And more generally, NTLM itself should be deprecated in networks. It's kind of like your analogy that you used earlier

about the boiler in the basement. And it's as if the building were built around the boiler. And to get it out and replace it, you have to shut it down, cut it up into pieces, haul it out, get a new one in there. And boy, that's a lot of work when everything right now is just running fine, right? Exactly, yeah. just running fine, right? Exactly, yeah. Yep, and that's kind of going back to the whole deprecating and modernizing technology.

A lot of times, product vendors, it's not specific to Microsoft, but in general, people focus more on adding new features rather than securing and modernizing their older stuff or just completely removing older things that shouldn't be used anymore. The same thing goes with organizations themselves. When they're coming to secure a network,

like network defenders, they may try and get the latest and greatest product to help secure their network, but doing the basics of disabling NTLM and doing network segmentation isn't always prioritized as much. And those will have a lot bigger bang for the buck than maybe purchasing the latest and greatest security product.

Our thanks to Lee Christensen and Will Schroeder from SpecterOps. The research is titled Abusing Active Directory Certificate Services. We'll have a link in the show notes. businesses worldwide. ThreatLocker is a full suite of solutions designed to give you total control, stopping unauthorized applications, securing sensitive data, and ensuring your organization runs smoothly and compliant. The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Trey Hester, Elliot Peltzman, Puru Prakash, Justin Sabey, Tim Nodar, Joe Kerrigan,

Carol Terrio, Ben Yellen, Nick Vilecki, Gina Johnson, Bennett Moe, Chris Russell, John Petrick, Jennifer Ivan, Rick Howard, Peter Kilpie, and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.