CyberWire Daily - What Log4Shell has taught us. [CyberWire-X]

Episode Date: February 20, 2022

If 2021 taught us anything, it’s that our supply chain–especially our technical supply chain–hangs in the balance of a very fragile system. The year came to a close with the announcement of the Log4j zero day. Talk about saving the best for last. On this episode of CyberWire-X, the CyberWire's Rick Howard speaks with Tom Quinn CISO at T. Rowe Price, about the topic. Show Sponsor ExtraHop’s Head of Product, Ted Driggs, joins the CyberWire's Dave Bittner to examine what Log4Shell tells us about the state of cyber defense going into 2022, and what enterprises can do to prepare. Through these conversations, we explore the challenges that enterprises had in patching the vulnerability, take a closer look at the advanced post-compromise threat activity spotted in the wild, and glean lessons that can be learned to build resilience against the next Log4j-style zero day. Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. Hey, everyone, and welcome to Cyber Wire X, a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the Chief Security Officer, Chief Analyst, and Senior Fellow at the CyberWire. And today's episode is titled, What Log4Shell Has Taught Us. Over the holiday break in 2021, the entire InfoSec community reacted to an international incident response around the newly announced Log4j zero-day exploit. And if 2021 taught us anything, it's that our supply chain, especially our technical supply chain, hangs in the balance on a very fragile system. invited two guests to the CyberWire hash table, Tom Quinn, the CISO at T. Rowe Price,

and Ted Driggs, the Director of Product Management at ExtraHop, to discuss what the Log4J vulnerability tells us about the state of cyber defense going into 2022 and what enterprises can do to prepare. A program note, each CyberWire X special features two segments. In the first part of the show, we will hear from industry experts on the topic at hand. And in the second part, we will hear from our show's sponsor for their point of view. And since I brought it up, here's a word from today's sponsor, ExtraHop.

With millions of dollars to earn, today's cyber attackers have grown, shall we say, advanced. Whether they're hiding malicious activity in encrypted channels, laying low under the guise of a trusted third party, or taking advantage of the latest CVE, they know they have the upper hand. In a world where attackers have the advantage, ExtraHop is on a mission to help you take it back. Learn more about how ExtraHop helps you defend your enterprise against advanced threats like supply chain compromises, ransomware, and APTs. Visit www.extrahop.com slash cyberwire. That's E-X-T-R-A, hop like a bunny, dot com slash cyberwire. And we thank ExtraHop for sponsoring our show. To start things off, I'm joined by Tom Quinn, the CSO of T. Rowe Price,

and a regular here at the CyberWire hash table. Hey, Tom, thanks for coming on the show. Great to have you invite me again. Thank you. We love having you on, Tom. So today we're talking about the industry's great crisis over the 2021 holiday break, our collective reaction to the Log4j vulnerability, and any lessons learned that we can use for future problems of this sort that are going to come up to us down the line. Now, the Log4j vulnerability is a form of a digital supply chain attack, and these third-party attack vectors seem to be having a moment right now. We had some pretty high-profile attacks last year, like SolarWinds and Ascelion,

and then we got hit by this open-source software library vulnerability, Log4j, at the end of the year. Can you walk us through how and when your organization first heard about the problem and what you did to reduce the risk? So we have a variety of threat intelligence capabilities at the firm, and we received notice that this Log4J vulnerability was something that required attention. Our threat intelligence team, as well as our vulnerability management team, assembled to evaluate the issue based on our own risk lens. And then we recognized that it was something that we needed to do.

And then we engaged our technology operations teams. For T-Row Price, our technology operations teams manage and operate the vulnerability management execution capability at the firm. We focus on the risk and then we work with them to execute. So they did their part of this, which was verify and validate the inventories that we have in place. And we have a variety of inventories of software, both commercial, open source, and others. Our inventory was verified, and then we went about the work of testing and then certainly deploying the capability. You mentioned something about risk mitigation. I think it's important. The timing of the year when this occurred was right around the holidays. And one of my jobs at the firm is to make these risk calls where we actually had to call back

people from their vacations to come in. Oh, that's horrible. I know everybody was doing it though. It was horrible. And you have to, I also think it's fortunate that we had some remote access practice over the past two years with COVID being full-time remote. So people quickly assembled, thankfully, onto the remote access capabilities that we had and then did the work that they needed to do. And then in very short order, we're working on the things that needed to be done. The other side of this, and this is also part of the threat intel and part of the vulnerability management side of the program that my team does as well is now you got to start measuring. You need to measure to make sure and validate that the inventories are correct, validate and measure, right? That the fix that was deployed actually had the intended effect.

So my team was on this rinse and repeat cycle of validating the work that needed to be done. And believe it or not, right, we even noticed situations where conflicting work was occurring, where a patch was deployed, it was tested, things seemed to be fine, and then we ran a scan a day or two later and noticed that what was accomplished was actually undone. So it's critical to check and verify. Well, let's unpack some of that stuff because you threw a lot of detail out there. So first, I want to know, Tom, was there a moment that you and your team decided the problem went

from, oh, this is serious to, oh my God, this is serious. Was there one of those kind of wake-up moments for you? There was. And like, you you know we evaluate significant vulnerabilities on a regular basis right you know that's the nature of the work that we do but i think as we started to unpick log4j and some of the details for it you know we recognized pretty quickly the ubiquitous nature of it and i mentioned before you know about open source inventories that we maintain at the firm for software that we have and commercial inventories and applications that we build. As the data started coming in, we recognized how impactful it was to us, not just the industry. And then I think the part B to that, which was we realized that was like,

oh my, this is a big deal. When we started recognizing and realizing that many of our colleagues, many of my peers, our vendors had the same issues that had to be done, this really expanded this into a much bigger potential. You mentioned that you had on hand already developed libraries of all the software you were using. Walk me through that. Is that something you guys built in-house? Is it in spreadsheets? Are you using some software to do that? Or how are you guys managing your software inventory, both commercial and open source stuff? There's a multifold effort that we have. So for, I'll call them traditional operating systems,

we have a view on the inventory that's there. We have a very robust and well-known third-party tool that helps us keep the inventory. In addition, we have vulnerability scanners that acts as a check and a balance to that, a reconciliation capability. So that keeps the, I'll call it traditional operating system inventories well understood. And then for our development community, we have a build train capability where you not only store your code and the associated libraries are there. And by the way, we also do static and dynamic testing within this build train that we have. Some people may call that a CICD pipeline. Others may call it something else. But we also have a gate capability for libraries coming in. So there is a product

that we have and a capability that we have so that we limit where you can get open source libraries and other libraries that we use for so that licensing reviews can be done, version control, version management can be done, and a whole host of administrative and management actions can be taken. That really accounts for the vast majority of the inventories that we have. And again, the scanning technologies that we have, whether vulnerability scanning our intrusion detection prevention, and other kinds of telemetry points that we get. Because when we see scans occur or we see traffic occurring to destinations that we didn't expect to have Log4J there. It's another indicator that something may be going on. So we triangulate between those.

And again, it's a couple of enterprise-grade capabilities for inventory and reconciliation. And then I think just lots of good judgment being applied as well, looking at the corners of our network for other things that may be going on. So the Log4j software is a bit of open source software written for the Linux distribution. When you were checking your inventories, was it as simple as, can we just search for this component in the library? Was it as simple as that for the tools that you were developing, or was it more complicated than that?

So that's certainly one of the things to do. The other thing was to actually look at some of the code bases that we had and look for those calls. You know, when Log4J was actually being called as a function. So there's a couple of places that we did that with. And even from a network traffic perspective, I'd mention too, not the inventory side, Rick, and I know you asked about inventories, but again, doing triangulation and looking at other areas where you can make inference that Log4J may be there. Because, you know, again, you may see indicators and network traffic, right, that that's going on as well. There's two problems here, right? going on as well.

There's two problems here, right? One is code that T. Rowe Price is writing themselves, and they may have downloaded and used the Log4j component, right? And so you should be able to look it up in your inventory for that. But for the network traffic, that's for all the other software that you're running, both commercial and open source, that is running that module also that you didn't know about that you weren't really paying attention to. So that's how you would discover it. It's certainly one of the ways. And I think the other thing that we relied on, right,

was our vendors or certainly our significant vendors to be doing the same thing. There were bulletins that were published by significant software vendors saying, you know, listen, you know, we have a log for J dependency. This is what you need to do. Here's the version that you need to download. Because in some cases, what a company may do with an open source library, it's not as easy as even if we wanted to,

like to pluck it out and put our version in because you could damage the product, right? The appliance or whatever else it may be. So you're right. In some cases, right, our inventory may have seen something, but in some cases we needed to wait for the vendors as well to come back. So it's critical to make sure that you have very robust inventories because that speeds up the cycle for action,

the cycle for knowledge, right, that you need to do. And that's key, knowing that a few things are going to be missed. And that trade-off is something that we recognize and why we invest so heavily in the asset inventories that we have so that we really can make quicker decisions. And it's less for us to be looking around for, right, in other corners of the network. In the early days of the crisis, you know, we were all saying, oh my God, the world's coming to an end. But pretty quickly, the community discovered that if they just blocked X data coming from

log4j traffic going out to the internet that you could reduce the risk almost to nothing is that what you guys decided to do too yeah we took the same approach uh you know one doing the inventory work and you have to you just have to maintain the code base your software base and your inventories at a high level and then look at mitigating controls as well. And we did. So there is a variety of rules that we incorporated. And there were some rule modifications that we also made based on the traffic analysis that we were seeing. Oh, that you found on your own.

Oh, I get it. Yeah. Yeah. So because what we found in some cases was some of the rules were imperfect. And that's another reason why it's critical to make sure that you've got your own capability to evaluate what's happening on in your network. Because believe it or not, the bad actors are smart too. What? Tell me. They recognize what the rules are. They're being given out right to everyone else. What? Tell me. rule in general, right? If we have high-fidelity asset inventories, we can deal with the 20%

looking around. And I think it's the same thing for these rule sets. If they can account for 80% or 90% or whatever it may be and be effective, it allows us to focus in on areas where they may be less effective for a whole variety of reasons. Well, you keep mentioning asset inventories. The idea of an asset inventory has been around for decades. I mean, even when we were young fellows doing this, Tom, we were talking about that. But that's not the same thing as kind of a newer idea that's getting some traction here recently, which is software bill of material. Am I right that those two things are different or are they pretty much the same idea with different names? I think the concepts are related and I think that they're similar, but I liken it to

in the old days when people used to say, hey, you should only have access to the minimum amount of stuff that you need. And now that's morphing into things, concepts like zero trust. So I think, into things, concepts like zero trust. So I think, Rick, my view on this is it is similar. It's not the same. And it's the level of granularity, I think, that's being asked of things is different. And I also think it's a recognition that the way that software is being consumed, being used, being manufactured and alike. It's not just in your data center.

It's not just on your server per se, right? There's a lot of dependencies, right? Supply chain dependencies. So I think that phrase makes sense. And I think what that phrase allows one to do is really to have a much more holistic view about what you know where your boundary of control thought needs to be so that you can have a view on the work that's needed to to round that out that's my understanding of it too is like where asset inventories are kind of big blocks of software like you said we know about microsoft's operating system. We know about the Linux operating system. Where the SBOMs will help us down the line, because they're not here

yet, but where they're going to help us is when big commercial vendors used open source software and all those nested components that we don't see just on the normal day-to-day operations. We don't know that Microsoft used Log4j unless we see the traffic. So I think SBOMs will give us, like you said, that granularity. And I think, too, in some cases, there may not be an understanding that a vendor actually has a dependency on some of these, call it small companies maybe or maybe big companies. They're just pulling down components, right, that they may need or they think they may need to get something done. But anyway, that concept of billed materials, I think, is important. The amount of rigor that's going to be needed around that, I think, is quite a bit. But the tools that are out there, I think, have improved.

I won't talk about specific companies, but there are companies that are providing, I'll call it dependency analysis capabilities. They're pretty darn mature, leveraging cloud compute capabilities, cloud storage capabilities, analytics and the like. The modernization of thought and the modernization of capability is allowing people to have much more view about what's going on. And I even think, too, for some of the cloud-native companies, they've built a lot of their systems around high-fidelity view of components that they're using. So I think the industry's ability to do that or the desire to do that is catching up with some of the engineering and architecture and technology improvements over the past 20 years. So make a prediction for me.

How long before SBOMs are best practice and common practice for just general purpose network defenders? I think ubiquitous capabilities in that place or expectations is a long way off. My guess is that, say, the next three to five years, you'll have certain parts of either the stack or certain applications may have very high fidelity capabilities. I think for high-risk transactions and high-risk systems and high assurance needs, I think you'll see that first. And then I think that will just become, that will trickle down. And then over time, I think you're going to see that

in more and more news. Good stuff, Tom. But we're going to have to leave it there. This has been Tom Quinn, the CISO for T. Rowe Price and subject matter expert here at the Cyber Wire Hash Table. Tom, once again, thanks for coming on the showISO for T. Rowe Price and subject matter expert here at the Cyber Wire Hash Table. Tom, once again, thanks for coming on the show.

You're welcome. Thank you for having me. Next up is Dave Bittner's conversation with Ted Driggs, the Director of Product Management at ExtraHop, our show's sponsor. And before ExtraHop, Ted was a product manager for Windows at Microsoft, and he is a regular on tech security podcasts like Risky Business, Security Weekly, and DM Radio. In his free time, you can expect to find Ted on the side of a mountain zipping through powder or hiking up a rock. Here's Dave. I'd love to start out just by getting some insights from you. You know, I recall wrapping up the end of last year, I think for a lot of us as we were getting ready to hopefully take a break going into the holidays, and we started to hear word that there was this thing, Log4J, and indeed it was real, and indeed it seemed serious. We went through that sort of cycle of that sort of news coming through the ecosystem here.

I'm curious, what was it like for you as the reality of Log4J set in for you and your team? Great question. So first, there was a little bit of, here we go again, right? It's another December, but this time, instead of Russian state-aligned actors, we're dealing with kids playing Minecraft who have broken the internet. It was followed by the realization that unlike Sunburst, which was a concerted effort with a particular objective, a particular point of ingress, this was the other end of the worst case scenario. This was a vulnerability that was incredibly widely deployed library. So you were no longer looking for a particular program that was behaving erratically. You were looking for any program that could be behaving erratically. And the nature of the vulnerability being passed layer through layer through layer made it very difficult to even predict how deep past the perimeter you would need to go to really protect yourself.

This wasn't something where you would send a malicious request and immediately compromise the targeted server. That string might be passed through multiple tiers before it eventually was logged by Log4j and triggered the lookup or the remote code execution vulnerability. So from an NDR perspective, we started by asking, are we ourselves vulnerable? And then asking what challenges our customers would be facing with detection. And after getting repros up, we pretty quickly began to push out detections and to provide as much guidance as we could to our customers on the subject. Well, and where we stand today, you know, several weeks out from

the initial information, where do we find ourselves? Where do things stand at this moment? Exploit attempts remain rampant and probably will forever. The customers that we've worked with have begun to patch or to attempt to disable the impacted features. I mean, we heard multiple like this Envar is going to work. No way that doesn't. There's a new vulnerability with the threat context. The initial shockwave seems to have passed, but the long tail on this is going to be a problem for years to come. And so how does that inform the things that we do going forward? And I'm thinking of this from two perspectives. I mean, we have the famous unknown unknowns. We don't know what other vulnerabilities like Log4J may be out there, but we certainly want to protect ourselves as much as possible against those possibilities.

But then also going forward with the information we have here against that specific problem. Right. So at the beginning of COVID, a lot of organizations went quote-unquote boundaryless, which was a nice way of saying they hastily finished a VPN deployment and hoped that everything would be fine and shipped it because their entire workforce was coming from home. And then at the end of 2020, supply chain attacks like Sunburst blew another hole in this idea of the boundary or the perimeter. And Log4j is the third leg of that, where components and applications that you yourself might have built can now be

suborned to trigger remote code execution, to trigger C2 behaviors, and to establish that foothold inside your environment. We really need to continue to think that initial compromise is an inevitability now. There are too many ways, whether it is this or whether it's the next vulnerability in some widely used library,

that will enable an attacker to get a beachhead somewhere in your environment. And it may or may not be one hop from the perimeter. So there are now multiple avenues of ingress into the environment. In addition to your traditional phishing and credential theft, supply chain attacks, attacks against network infrastructure that's now exposed to the internet, especially after the shift to remote work. All of these are things that are making the possibility of initial compromise almost a certainty for every organization. So we shouldn't be defeatist. We shouldn't give up and just wait for attackers to get that foothold. But we do need to be thinking beyond that initial point into how we identify, contain,

and eradicate post-compromise. We've seen a lot of good efforts here from multiple parts of the industry. We've seen the work on SBOM, Cyclone DX working to make a lot of commercial SCA stuff available in the public domain. We're seeing better and better tools around detecting post-compromise behavior. More and more vendors are recognizing that perimeter or endpoint protections by themselves are insufficient and are building alliances, partnerships, or products to go after that. And we're continuing to see the SaaS providers who are running a growing portion of email

that's used for phishing and credential theft helping to lock that down. There's a lot of people who seem to sort of dogmatically believe that one of these is going to be the path out of this, and that's incorrect. We're going to have to keep doing all of these forever. You know, it strikes me that when we talk about a lot of these defenses, I mean, even the, you know, what people refer to as basic hygiene, we talk about it in kind of an aspirational way where you need to do the, you need to take care of the basics. And yet so many organizations have trouble keeping up with the basics. And for good reason, you know, patching is hard. It's hard to, all these things

are hard to do while you're in business. You know, a friend who jokes about it's like trying to change the oil while the engine is running. And how do you advise organizations to do the best that they can, given the reality that this stuff is not as easy as some people make it sound? That is a great question. So this is one place where Xtrop comes at this from a different angle than some might. Because we are a network detection response tool, what happens inside the endpoint is actually less relevant to us. So whether you patched it or didn't, we're going to still monitor it and look for evidence of exploit attempts and exploit success or anomalous post-compromise behaviors. So that's where we feel like these tools complement each other well. Having tools in the endpoint, having vulnerability management tools, having SCA tools, those help you reduce the risk that you will need to do a more expensive containment and eradication

after an attacker has gained a foothold, but you can't rely on those tools being perfect and expect to be successful. So what is the best practice then? How do people dial it in, in terms of the variety of tools that they have, using that limited amount of time, resources, money, all those things, turning those dials. So what's important to come back to is that security is ultimately everyone's job, right? The SOC team is not going to be able to make patches to critical applications during the Q4 change freeze and hope to be successful at having zero business impact.

So they need the participation of app owners to make those updates. And we did see a lot of vendors very quickly rally to distribute updates of their products to remove vulnerable versions of Log4J. The second piece is that the SOC needs to be integrated into an organization's overall cybersecurity strategy, right? It's not possible to treat security as this bolt-on sidecar that there's this CISO and there are some people who show up from time to time and tell you that you can't do something. Security is an ally in maintaining the availability and reliability

of the business, and it needs to be integrated into those conversations from the outset. You mitigate the time crunch by spreading the load of this across multiple parties. App owners who have that understanding know that they need to update their applications as soon as it's practical, while SOC teams have the tools available to them to protect those applications until such a patch can be made without jeopardizing the business. That's the best answer, but it's often not the easy answer. It's often not the realistic answer. the best answer, but it's often not the easy answer. It's often not the realistic answer. In those situations, it's a matter of combining defenses at the perimeter, on the endpoint,

and east-west on the network, and trying to identify using vulnerability management which systems are exposed to this and limiting their access to limit the damage that they could do if they are attacked while still unpatched. Looking ahead towards 2022 and beyond, are you optimistic that we're in a good place to take on these challenges? No, but we're getting better. So the work being done by the NTIA on SBOM is going to be hugely important for setting a standard for the entire industry around distribution of what's in this software, right? Part of the Log4j nightmare was that initial response of, I don't even know which things I have in my environment that use this.

I'm going to give a shout out here to the work being done by Cyclone DX in particular in automating that SBOM build process so that for a variety of common languages and platforms, it is able to recursively figure out those dependencies and auto-generate a build artifact to ensure that the thing that is distributed matches the actually built software. One piece that Xdrop feels is a necessary addition to SBOM is something we've been calling behavior transparency. So knowing what's in a piece of software is crucial for addressing vulnerabilities. However, knowing what behaviors a piece of software might perform in an environment

is crucial to identifying when it has been the victim of a conscious supply chain attack. We've been talking a bunch about Log4j as evidence of supply chain attack, right? We've been talking a bunch about log4j as evidence of supply chain vulnerability, which is certainly true, but the counterpart to supply chain vulnerability is a deliberate supply chain attack. In that situation, it's likely that the bill of materials won't show anything amiss because the attacker likely concealed their involvement in there or they've hidden themselves in some DLL that appears benign to an outside observer. So we think that both parts of that are going to be necessary, but the fact that the industry is recognizing and moving to address supply chain problems through collective action rather than through trying to have a number of unilaterally

imposed solutions is really promising. And that's a wrap. We'd like to thank Tom Quinn, the CISO of T. Rowe Price, and ExtraHop's Ted Dix for joining us. CyberWire X is a production of the CyberWire and is proudly produced in Maryland at the startup studios of Datatron, where they are co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Iben. Our executive editor is Peter Kilpie. And on behalf of my colleague, Dave Bittner, I'm Rick Howard.

Thanks for listening.