CyberWire Daily - Defining the intruder’s dilemma. [CyberWire-X]

Episode Date: June 5, 2022

For this Cyberwire-X episode, we are talking about the failure of perimeter defense as an architecture where, since the 1990s when it was invented, the plan was to keep everything out. That model never really worked that well since we had to poke holes in the perimeter to allow employees, contractors, and partners to do legitimate business with us. Those same holes could be exploited by the bad guys, too. The question is, what are we doing instead? What is the security architecture, the strategy, and the tactics that we are all using today that is more secure than perimeter defense? In the first part of the show, Rick Howard, the CyberWire’s CSO, Chief Analyst, and Senior Fellow, talks with Jerry Archer, the Sallie Mae CSO and CyberWire Hash Table member, and, in the second half of the show, the CyberWire's podcast host Dave Bittner talks with Mike Ernst, episode sponsor ExtraHop’s Vice President of Sales Engineering, to discuss Software Defined Perimeter and intrusion kill chain prevention strategy. Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. Hey, everyone. Welcome to Cyber Wire X, a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the Chief Security Officer, Chief Analyst, and Senior Fellow at the CyberWire. And today's episode is titled, Defining the Intruder's Dilemma. And what we're talking about here is the failure of perimeter defense, an architecture that we invented back in the 1990s, but never really worked that well.

The idea was that we would build giant electronic fences around all of our digital assets designed to keep the bad guys out. But we had to poke holes in the perimeter to allow employees, contractors, and partners to do legitimate business with us. Those same holes could be exploited by those same bad guys we were trying to keep out. The question then is, what should we be doing instead? What are the strategies and tactics that are more secure than perimeter defense? One idea that is relatively new but catching on is called software-defined perimeter. And another idea has been around for a decade called intrusion kill chain prevention. In this episode, my colleague Dave Bittner and I

invited two guests to the CyberWire hash table to discuss the issues. Jerry Archer, the Sallie Mae Chief Security Officer, and Mike Ernst, ExtraHob's VP of Sales Engineering. A programming note, each CyberWireX special features two segments. In the first part of the show, we'll hear from industry experts on the topic at hand.

And in the second part, we will hear from our show's sponsor for their point of view. And since I brought it up, here's a word from today's sponsor, ExtraHop. When it comes to enterprise compromise, we all know it's not a matter of if, but when. Yet, 75% of security budgets go to preventing intrusion, and we're losing the battle. It's time for a new approach. It's time to defend the win. Visit ExtraHop at the upcoming RSA conference to learn how AI-based network intelligence from Extra Hop stops the advanced threats that are already inside your cloud, hybrid, and distributed environments.

Intrusion is inevitable, but breaches don't have to be. Stop by booth S1377 or visit extrahop.com slash cyberwire. That's extrahop.com slash cyberwire to learn more. And we thank ExtraHop for sponsoring our show. I'm joined by Jerry Archer, the Chief Security Officer at Sallie Mae. He's also the founder of the Security Advisor Alliance, a nonprofit group of CISOs focused on finding and encouraging the next generation of cybersecurity talent. And you've been coming to the CyberWire hash table since we started the conversation some two years ago. So, Jerry, thanks for coming back to the show. Wow. I'm getting to be an old hat, huh?

Yeah, when your bio is longer than the show, you've been around for too long, I think. Time to retire. So in a survey done by the Cloud Security Alliance back in 2020, only a quarter of the respondents even had heard about software-defined perimeter. And I know I didn't learn about it until I interviewed you last May for my own podcast, CSO Perspectives, when we were talking about identity management. So for our audience members who are not familiar,

can you take a swing at defining what software-defined perimeter is? Software-defined perimeter essentially sets up what amounts to a software perimeter around your entire environment. And what that means is that you need to be pre-authenticated in order for the environment to recognize you. So basically, the only thing that's exposed to the internet, if you will, is a single controller port that you have to present your credentials to. If your credentials are

accepted by the controller, then the controller will notify the environment of what you have access to and allow you to come inside. Other than that, the inside, if you will, will not even acknowledge that it exists. So you can't even ping it. It won't even respond to a ping. So I don't even like the name software-defined perimeter because with this model, we've completely abolished perimeter defense as we knew it back in the old days. It's completely gone. Instead of rocking up to the workload that you're trying to get access to and entering your user ID and password like we've been doing for 20, 30 years, this goes to another place, a different place, not even associated with a workload.

You log in there within the, what you called it, the security controller, verifies your identity and verifies the authorization of where you can get to. And if it likes all that, then it establishes the secure connection to the workload. And like you said, that hides everything else. The bad guys don't see anything. I really like that architecture. It's hard to attack what you can't see.

It's very true. It's very true. On a side note, I was talking to my old buddy, Steve Winterfeld. He's the Akamai Advisory CISO about how the phrase software-defined perimeter is a bit misleading. Here's Steve. So it's interesting you ask around a software-defined perimeter because we go back to the concept of a perimeter. And we don't have a perimeter around our data because our data isn't inside a castle or any of those things we like to think about. It is in so many different places,

in so many third parties, in different kinds of infrastructure. I really am not sure we should be using a location-based word. I want it to become more location agnostic, move to something more like a identity where a proxy, where it doesn't matter where the person is. We're not talking about branches. We're not talking about remote office work. We're simply talking about access. And Gartner places software-defined perimeter on their 2020 cloud security harp chart about halfway down the trough of disillusionment

and gave it about two to five years before it's ready for prime time. And when I talk to security leaders about these new ideas that come down the pipe, most are thinking about how to deploy them, and some even have pilot projects. But you, shall we say, were an early adopter at Sallie Mae. You have it fully deployed and have had it fully deployed for years. So can you walk us through that process and how you came to decide that was the way to go? When we first started, that's how we made the move to

the cloud. One of our guiding principles was to try and not use rocket science technology. We wanted to use standard sorts of things that could be employed, for sake of argument here, to create a rocket science solution without the rocket science technology, because that's hard to maintain. The software-defined perimeter had been around for quite some time. It had been pioneered in the intelligence community as a way of mining open-source data with anonymity. And it had been taken out of the community by a group called Vitter, and Vitter pioneered it in the commercial space. So when we purchased it, it was Vitter as a company.

And then it moved over to Verizon, and strangely enough, now Verizon is going to discontinue it. So now there's more companies out there than Vitter who are now offering, essentially, the software-defined perimeter in various products that they now have. I mean, we were very surprised when Verizon decided to eliminate it. I guess Verizon eliminated product from a security portfolio and went primarily to services. I did not know that. That's really interesting. Well, you're right that it came out of the government channels earlier.

I mean, the DOD worked on a project called the Jericho Forum, kind of outlined some of these best ideas, but they never implemented it. And they just kind of let it die off. And then when Google got hit by the Chinese government back in 2010 in Operation Aurora, they redesigned their internal network from the ground up using first principles of zero trust and software-defined perimeter for their own internal stuff. And then later on, they eventually released a commercial product called BeyondCore. When did you first install your software-defined perimeter for Sallie Mae? Now it's almost five years ago. And we did it in the initial security stack.

So that was part of the architectural design of our initial security stack was to have the software-defined perimeter mesh. So that's the first step to get inside the front door is to be pre-authenticated. So one of the things that we recognized as part of that was that identity and access management became paramount. And the amount of identity and access management you have to do is significantly greater because now you have to go out and define every user that wants it to your environment. I totally agree with this. Architecture, to me, is so far superior than the way we used to do it. Tell me if I got this wrong, though. It's basically a SaaS application that does all your identity and access management for you and then arranges the connections to the workloads that your employees and devices

need to get access to, right? It's basically a SaaS app. Well, we still use Active Directory. So once you get inside, you still have to go through Active Directory and Active Directory decides what you ultimately, the resources that you can get access to. So you still have to have that capability, right? You have to be identified with an AD in order to get access to resources. So what's the big lesson learned when you guys deployed it? What did you run into that you

weren't expecting that you had to figure out? Yeah, I don't think we had much of a problem at all. Now, we, again, I can tout Vitter because they no longer exist as a company, so I'm not pushing a vendor that anybody can go out and engage with. We had Vitter that helped us out significantly in deployment. And because they had a lot of experience doing it, we really didn't have any trouble at all. I mean, we were very clean and deep, very, very fast to get it up and going. And once it's up and running, now it's just a matter of identity and access management. So let me change gears a bit.

One of the cybersecurity myths that started to emerge in the mid-2000s was that attackers only needed to be right once, where defenders need to be right 100% of the time. And we know now that cyber adversaries have to string a series of actions across the intrusion kill chain in order to be successful. That if we can break a link anywhere in that attack chain, we can defeat the entire attack campaign. In fact, we've completely reversed that old myth. The attacker has to be 100% successful across his attack sequence. We only have to be successful once in defeating him. So Jerry, I know you guys subscribe to the Intrusion Kill Chain Strategy

at Sallie Mae. Are you guys a big user of the minor attack framework to get your intelligence or do you get your intelligence about attack sequences from other intelligent sources or is it kind of a mix it's it's a mix i mean minor is the attack kill chain is there very much a lot of the stuff we use the software as a service model so we use a lot of tools or services that have their own kill chain models that we use. But I would say one of the primary kill chains is also the MITRE ATT&CK model. On another side note, I asked Steve Winterfeld about how easy it is to use the MITRE ATT&CK framework. I really love the fact that you can walk all the way through the process.

So if you go out to a CISA or FBI alert, you go down into the reference material, you'll see MITRE ATT&CK APT link. And if you follow that link, it goes over to the MITRE page on APT 41. And then if you click on the ATT&CK navigator layers, it just pulls you right into that attack framework and highlights which ones if you're looking at initial access it tells you the two they're using if you look at execution it tells you the five uh different capabilities they're using and it's just so easy then to turn to your red team or your pen test team and say, we think this group is going to attack us. We want you to follow this kind of run book to do an attack against us to validate whether or not

we're secure against it. I think it's just a great resource. But we've now very much focused on meantime to remediate or repair versus mean time between failure. We've basically acquiesced to the notion that there's no such thing as perfect intrusion prevention. You can't stop people from getting in. And honestly, phishing now represents, I think I read a statistic about 97% of all the attacks are now phishing. I hate to say this out loud, but I can't fix stupid. Somebody's going to click on one, right? What we've now done is we've really focused on EDR and trying to stop malware right at its inception. Put me firmly in the stupid box because, you know, we have one of those services at the Cyber Wire where they send fake spam messages to see if you're smart enough to avoid them.

And I've only been at the cyber wire for a couple of years, but I've been caught at least three times. And I know better. If we can't help me not get caught, I'm sure we can't help the grandmas out there. So I agree that you're going to be penetrated at some point. What we're trying to do with the intrusion kill chain defense strategy, though, is prevent the adversaries from being successful.

Do you task your intelligence team specifically to go after known adversary group sequences? Like, you say, hey, go see if we see anybody from Panda Bear running around your networks. Is that how you guys do it? Yeah, we do. We do a lot of purple teaming, and we use automated tools to do that. The purple teaming is a big way to put in the attack scenarios and look for the indicators of compromise to make sure that we can detect the bad guys as they're coming in the door and trying to execute. And so we do that on a continuous basis. So that's the way we look at the problem. And then we array our defenses based upon the current situation in the world. The other side of the coin is you want to reduce mean time to repair

so that the impact is less and less and less. So you want to get mean time to repair to zero. Take it into the cyber world, it's exactly the same thing, right? All of our defenses, perimeter defenses, firewalls, and everything like that are designed to put MTBF longer and longer and longer. You stop people from getting in the front door. But inevitably, when somebody gets in the front door, now what you want to do is you want to stop them as quickly as possible, stop lateral movement, stop any ability to move forward. So EDR, XDR are sort of your paramounts in terms of intercepting that before anything can happen. So when you're looking at an adversary group like Sandworm or Panda Bear or any of the 150 to 200 adversary groups that are out there. And they have a number of things they have to do in their attack sequence.

Let's say it's 100. So that means your team has to know there's a threshold. Like if you see one or two of these things pop up in your network out of the 100, it might or might not be Panda Bear. But if you see 50 or 60 of them associated with PandaBear, you got PandaBear in your network and then your teams can go and remediate that, right? Oh, no. We're much earlier in the kill chain. We're right at the inception when they first go out for command and control. Our goal is the moment that malware executes in the environment and goes out and looks for a payload or looks for any kind of command and control, we want to stop it right there. So we're very much focused on an endpoint kind of detection capability.

Obviously, the higher up the stack we go, the more things we're looking for in the environment, on the networks, and all that sort of thing. But I mean, I would tell you that we're really focused on the idea of endpoint detection right away. In order to follow the intrusion kill chain strategy, you need much more than firewalls the intrusion kill chain strategy, you need much more than firewalls

and intrusion detection in your security stack. You need endpoint detection and response, maybe even network detection and response, or combining them into extended detection and response tools like XDR. That's how you're able to track these adversaries, right? Exactly. We start early.

We have multiple agents sitting on every single endpoint these days, and they're looking for all kinds of abnormal behaviors and indicators of compromise. We take two and a half billion events a day and shove it into a data lake. That's our XDR data lake. So we're looking for any indicators of compromise across the entire enterprise. So Jerry, as always, you're well ahead of the most of the security practitioners that I get to talk to on this show, especially in terms of software-defined perimeter and the intrusion kill chain prevention strategy. Any last words about lessons learned that you figured out while you were doing this

for the past 14 years that you could pass on to our listeners? We started our cloud journey five years ago. And I would tell you that the biggest single thing that one can do is nibble your way to success. People who sit down and define mega projects almost always fail. When we started, we put out a cloud environment that did almost nothing, but it demonstrated we could go to the cloud. And we added a little bit more and a little bit more and a little bit more. And that's how we got to where we are. So take small bites. Don't try to boil the ocean is what you're saying. Take small bites, demonstrate success and keep moving forward. Good stuff,

Jerry. Unfortunately, we're going to have to leave it there. That's Jerry Archer, the Chief Security Officer at Sallie Mae. Jerry, thanks for coming on the show. Anytime, Rick. Thanks a lot. Next up is Dave's conversation with Mike Ernst, the VP of Sales Engineering at ExtraHot. So today we are talking about this notion of the intruder's dilemma. Let's start off with some high-level stuff here. Can you give us a little bit of the lay of the land? Like where, in your estimation, we find ourselves when it comes to this sort of thing? Sure. So I feel like we've got to start with a bit of background,

thing? Sure. So I feel like we got to start with a bit of background, which is, you know, the defender's dilemma is super common, you know, all over the press industry analysts. And that's the adage that, you know, the attackers only need to be right once, you know, defenders need to be right 100% of the time. And therefore, you know, you need to invest in perimeter security defenses, walls, castles, moats, however you want to describe the keep the bad guy out. And even though that's your objective, you're kind of inherently screwed because of this asymmetry. Do you think that that is an accurate statement, the whole notion of that the defenders need to be right 100% of the time? Yes. Well, with the caveat, if your goal is to prevent any aspect of your environment from ever being compromised, yes. If that's the objective is we must prevent all compromises, then you do face this asymmetry and you do have to invest an inordinate

amount of money, time, and resources into preventing that compromise. Well, let's flip it on its head then. I mean, we're talking about the potential for there to be an intruder's dilemma as well. Can you walk us through that idea? Sure. So, you know, this is the post-compromise reality, right? Like, we're taking the perspective that someone is going to get in somewhere, somehow, you know, someone clicked a link they shouldn't, they open an email they shouldn't, you know, they plugged in a USB drive that they shouldn't. Something is going to happen somewhere where somebody's going to slip through your defenses, and then you have an intruder. That's the starting point for the intruder's dilemma. So the intruder has made their way in and now they have some decisions ahead of them. How does that usually play out? So what does

the intruder do? So he's got your laptop or my laptop or a phone, whatever resource he's obtained access to, that's his foothold, his toehold. He's got one. Everybody starts with one, right? You got in somewhere. So the first thing you need to do is you need to do reconnaissance of some sort. What else is around here? What's adjacent to me? What other systems, devices, what's out there? You're going to do some form of scanning reconnaissance. You know, network reconnaissance is usually the most common step. You know, practitioners are going to be familiar with NMAP. You know, that's probably the scanning tool of choice for blue teams and red teams alike. So you'll, from your beachhead, you know, survey the environment, get a lay for the land, and then plan your next step.

And how do they go about doing that while also keeping their head down and not drawing attention to themselves? I mean, in the end, you still need to scan the environment over the network. There's no avoiding that. But you could be a bit more crafty or stealthy. You could do it very slowly, very carefully. You can run an Nmap scan slow enough that it's almost impossible to detect the scanning activity. Let's say I'm only going to look at one IP an hour. That might take you months, so unlikely. But there's various knobs and dials that you have to kind of control how obvious is your scanning activity.

And so once you have established what's going on around you, you've kind of mapped out that environment, what's the next step? Well, the odds are almost certain that you want to gain access to more machines. You know, it's very unlikely that the first footprint that you got a hold of is your end game and has all the, whether you're looking for data, whether you're looking to do ransomware, whether you're just looking to be destructive, it's highly unlikely that that first entry point

has everything you need. So you need to move laterally, which means moving to another machine, different area, maybe get to a server, maybe get to a device that is more stealthy, like a printer, or there's all sorts of smart devices out there. Maybe there's some older network security devices that you can get. I mean, usually people think the hardest thing is to get in, and that's why we spend so much money to keep people out. But then once you're in, there's not as much defensive capability to prevent this lateral movement. movement. Is it fair to say that with each bit of movement, with each bit of activity, that the intruder has provided the defenders with an opportunity for detection? Well, that's the intruder's dilemma that we want to get to, which is each of these steps that

you're going through from initial access to end-game success, the attacker needs to be right 100% of the time. So you've kind of flipped the dilemma on its head. Now it's the defender just needs to see one of those things. So let's say they didn't see your reconnaissance. They didn't see your lateral movement. They didn't see your established persistence. But

they did see you try to exfiltrate data. That's it. You lost. Assuming they can then go back and piece together the threads and see what else you did. But now the burden is on the intruder to not get caught. So is part of the notion that perhaps we're paying too much attention to the perimeter defenses?

I would say yes, because, I mean, while it's not something you can ignore, because the more effective defenses you have, I mean, you know, there's different layers. I mean, there's people out there that are running scripts all the time. There's the not serious actor, there's the cyber criminal, and then there's, you know, the advanced nation state threat. So you need a perimeter because you need to make it harder for an intruder to get in. But I don't think enough thought or investment goes into the, well, what happens if I was compromised? Like, what do I do next? And what tooling and capabilities do I have to prevent that initial intrusion from becoming, you know, a complete breach?

prevent that initial intrusion from becoming a complete breach. So what are your recommendations in terms of the types of things that are available to folks to be able to monitor inside the proverbial castle walls, to have as many opportunities as possible for detection, but at the same time, not providing too much friction for the users who are inside? but at the same time, not providing too much friction for the users who are inside. Thanks for that softball setup there. I mean, we're kind of biased in this sense, but we think the network is the perfect resource to pick up these post-compromise activities because it is everywhere. It is passive.

It is convenient to monitor. It is convenient to monitor. And it does provide you the opportunity to detect all of these post-compromise activities. The reconnaissance we talked about uses the network. Making your next move laterally uses the network. If you need to establish persistence, some kind of beaconing, command and control, uses the network. control uses the network. Moving data around the environment uses the network. Trying to get into the active directory or some of the key systems, it all uses the network, like literally every step of an intruder's post-compromise playbook. Help me understand how an organization goes about dialing this in, you know, to not be hit with a lot of false alerts, to not be chasing their tails.

What is that initial process like? So this is where, you know, network security has gotten a lot better. And, you know, a bit of, I mean, people have been monitoring the network for security purposes since time immemorial, but the technology hasn't been there to do much beyond, you know, there's a lot of store the PCAPs, capture packets, warehouse them, you know,

use them from a forensic perspective, investigative perspective, but that's not real-time monitoring. And it's been hard, you know, then we've got NGFWs out there which have better data and better visibility into what's going on, but it's still just scratching the surface at how much intelligence you can actually get off the network. And this is where network detection and response comes into play. Advanced technology that can extract all of the Layer 7 payload, all of the transaction activity, feed it into sophisticated machine learning engines to give you

better, more actionable alerts that are reasonable. And I guess I missed a step there. The IDS was one step in that evolution of network security, but that was a signature-based matching engine that

quickly became an alert cannon and was not super useful for security teams. Well, help me understand, I mean, to what degree is this examining the data itself, you know, actually going and looking at packets versus flagging activities, you know, as you mentioned, exfiltration. Is it a blend of both or how is all that dialed in? I feel like different vendors will give you different answers. From our perspective, the best data is in the packets themselves. I mean, that has everything. So if you can analyze all of the packets in real time at scale and extract the relevant transactions and information on those transactions.

That should be the basis of your network analysis because that has the full context. There are other kind of higher level network statistic analyses like NetFlow or firewall logs. And if we use an example like that, that would say, you know, Dave accessed the database. Dave pulled this much gigabytes

from that database. That's helpful, but it would be way more helpful to see the exact select statement that you ran on that database. That tells us exactly what you did. And there's a lot of other activity that has been hard to see

because it's encrypted and it's not accessible outside of the network. Things like MSRPC calls, Kerberos and LDAP traffic. So we can see, I mean, MSRPC is basically everything you can do remotely

to a Windows machine. There's a lot of anodyne, run-of-the-mill activities there, but there's also the ability to create a new service or to schedule a task. And we can pick that up off the wire, but you have to be decrypting MSRPC traffic and then you have to understand the protocol sufficiently

to know like, aha, this was a task creation activity that could be potentially malicious versus a lot of other MSRPC calls that are always going to be of no risk or concern. That's new. So how do folks best start down this path? I mean, if this is something I'm interested to know if this is a good fit for my organization, what's the best way to start? Regardless of where you want to go in the NDR space, you have to understand, to do NDR, you need network packets, and you need to find

key choke points in your environment, either in the data center or the cloud, where you're going to be able to collect that traffic. And there's actually a whole industry, I think Gartner calls them the network packet broker space, dedicated to getting traffic off the network into products like ExtraHop for this kind of analysis. And you'll need to have an idea for where in your environment is optimal to obtain that traffic.

Yeah, yeah. What is it usually like on the other side for folks who, you know, after they have started down this pathway and they see it's up and running, they're starting to realize some of the benefits. What do you hear from them? I mean, we enjoy that first conversation because there's almost always something insightful or actionable that shows up almost immediately, like just once after we've been hooked up. It could be something small, something from a security hygiene perspective that they thought had been decommissioned or they thought had been fixed, but we're just reporting on what's

actually taking place on the network. So it's a pretty indisputable, I think is the word. I don't know. You got me thinking. I think indisputable source of truth. If it crossed the network and it showed up in ExtraHop, it happened. There's no real debate on the subject. And big companies have complex environments and frequently will leave a test system there or a system that was scheduled to be decommissioned or upgraded and it didn't happen and just nobody caught it until we saw on the wire that it's still there and still exhibiting something from a hygiene perspective that should have been cleaned up a long time ago. There's usually a quick win out of the gate and then it's a wow. I mean, think about this. This is all of the traffic in your network, up to hundreds of gigabits per second, all fed into appliance, metadata extracted, built into our model of the environment.

And then the machine learning starts to kick in. It learns your environment. So, you know, gone are the old days where you had to manually configure all these alerts and tune all these knobs and thresholds. You just let it do its thing. And then it starts to tell you, you know, hey, I saw something that this particular device or endpoint has never done, looks suspicious, you know, this warrants a further look, you know, down to some, you know, I mean, it's rare that we find a smoking gun, you know gun attack in progress at the moment we do it, but we find a lot of things that look like suspicious activity, unusual behaviors that customers get a kick out of

investigating. Even if it does turn out to be totally acceptable, they get a feel for how this works. And this is that post-compromise posture that we talked about, where now they're looking at things that are taking place on the inside. And should they be taking place? It's a different vantage point than the perimeter that they're used to. And that's a wrap. We'd like to thank Jerry Archer, the Chief Security Officer at Sallie Mae, and Mike Ernst, ExtraHobbs VP of Sales Engineering, for joining us. CyberWireX is a production of the CyberWire and is proudly produced in Maryland at the startup studios of DataTribe, where they are co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Iben.

Our executive editor is Peter Kilpie. And on behalf of my colleague, Dave Bittner, I am Rick Howard. Thanks for listening.