PurePerformance - Log4Shell Explained: How it works and how to protect your systems with Asad Ali

Episode Date: January 4, 2022

Log4Shell was an unwelcome early Christmas present for many IT teams around the globe. Asad Ali, Senior Director Dynatrace Sales Engineering, was involved starting December 9th – helping organizations around the world to react to the new vulnerability threat.In our discussion we learn how the vulnerability works technically, how runtime AppSec vulnerability detection eliminates false/positives and how teams around the globe are preparing to protect their software supply chain for future vulnerabilities.To learn more about Log4Shell check out Dynatrace’s Log4Shell Resource Center including educational blogs as well as Asad’s webinar on Detecting and Remediating Log4Shell with Dynatrace webinar.Show LinksAsad Ali on Linkedin:https://www.linkedin.com/in/alikingdom/Dynatrace's Log4Shell Resource Centerhttps://www.dynatrace.com/resource-center/log4j-vulnerabilityDetecting and Remediating Log4Shell with Dynatracehttps://info.dynatrace.com/global-all-wc-dynatrace-for-log4shell-18339-od-fulfillment.html

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. My name is Brian Wilson and as always I have my right hand man, my left hand man, my center man, wherever the picture aligns. Andy Grabner, happy new year Andy. Happy new year, that's right, it's the first recording in the new year. Yeah. Wow. It's the first show of the new year. We're back. Thank you, everybody, for your patience. Yeah, it's day three of 2022.

I think so far we have not discovered any new variant of the COVID virus. No new security vulnerability. But who knows? Yeah, who knows? And hopefully, I know a lot of people did not get much downtime because of said security vulnerability. See, sometimes I can't even talk. But for those who have been working tirelessly, we thank you for your efforts. that, you know, I always go back to the AT&T iPhone release when it first came out.

And for people like you and me, Andy, when nobody knew what performance was, when that finally hit and it was hugely public, we finally got to say, hey, look, everybody, number one, this is what we do. But it also made companies look up and realize, hey, performance is important because this is really embarrassing if that was us. I think we have our iPhone moment for security. Because, yes, there's been security incidents in the past and credit card breaches and all that, but still hasn't seemed to be top of mind.

And they've been semi-controlled or isolated. This recent one was just massive. And hopefully for everybody clamoring for security, they're like, yay, finally, although it's unfortunate that this happened, finally people are going to start paying attention to security. Hey, and instead of us talking about and making up stories and trying to explain what this whole thing's about, let's actually invite our guest, Asad. Asad, welcome to the show.

How are you? I'm doing very well. Thank you. Thank you for the invitation. Asad, you have been at the front lines, I would so say, at least from a Dynatrace perspective, just before Christmas, as Lock4Shell kind of was presented to us, to the world. Before we go into the topic, can you quickly, for the listeners that have never heard about you, just quickly explain what you do at Dynatrace and also a little bit of a background

and then let's jump into log4shell sure i've been with the database now for 12 years and i run a specialized sales engineering team within dynatrace and few of us in my team are part of the swat team for the application security offering that dynatrace has. So for the last one year, we have been involved with a lot of customer discussions showcasing our application security offering, enabling it in customers that are interested in looking at it. And as you know, on December 9th, when Lock First Shell, when it hit, everybody was clamoring to figure out whether they were affected or not.

So as Brian said at the call, very busy 10, 15 days for the SWOT team in direct ways that had been helping with the customers and so on. So, and I think this is actually a fascinating thing. Security vulnerabilities, there's obviously different types of vulnerabilities, I guess. But I think the biggest question for most of our customers is it just says, are we actually impacted by this?

Meaning, are we using that piece of software, that particular version of Log4j. And it's also all these systems that are using it also exposed to the internet. And that's why I think answering these questions was very critical for us. Now, can you, before we go into a little bit how AppSec works at Dynatrace, can you give us first a little background on the vulnerability? I know there's a lot of stuff written out there about how Log4j works and the vulnerabilities, but in a nutshell, can you, for maybe somebody that has never, you know, kind of dig deeper

into the actual vulnerability, explain how the vulnerability works and why it was important to fix it as fast as possible? Absolutely. So Log4J is one of the primary laboratories used in Java for logging. So even if I, when I used to first start writing Java a long time back, Log4J was there and it's still there today, widely used. Now, there are different versions of log4j,

it's an open source library. Now, the way it works is that typically as a developer, I write a lot of things into my log file so I can use the log files to look at them later on to figure out whether there are any issues going on in my code. It's also used for debugging. I just want to use it to traverse my code

to make sure that I'm following the right path. But the problem with log4j library is that it puts a lot of trust on the content that is written to the log file. So what I mean by that is it doesn't really check what kind of data is written to the log file. So if I say, hello world, and write it to the log, it will write hello world to the log file.

It will not check whether I have any special characters, whether I have anything malicious in my log entry at all. It will just do what I'm asking it to do. So if you're writing information, generic debug information and other things, it works fine. Now, what was identified in the log for shell vulnerability

is that I can pass special characters in the log entry. So, for example, if I'm running a Tomcat server on my machine, I say log the entry $pwd, pwd is the working directory in Linux. Because it has a special $ sign in the beginning of it, first of all, log4j doesn't care what I'm writing, but that dollar sign gets converted into a Unix command. It becomes a Linux command of putting it into the working directory. So instead of showing $pwd as a string,

what I see in my log file is the working directory where my Tomcat is running. So what happens is that the log function is very kind of realized, does not check for the content and just trust the content, used to trust the content very much. Now, with this one, what happens, I can as a malicious attacker, I can send data to your web application, which in turn may not check for what I'm sending in HTTP parameters or header. And if I see that you're writing whatever I'm passing to the log file, I can

capitalize on that and try to run command on your host machine data specific to your home machine. And that way, I can get information about your environment. But what also happens is that Java also has a, what's the right word, JNDI, which is Java Naming and Discovery Interface. Using that, I can make a call from my machine or Java running on my machine to somebody outside. So what happened with Log4j is instead of saying, hello world, if a malicious attack that I can send a special string to my web request,

you as a developer can get that data and you try to write whatever queries, parameters, or headers that I'm passing to you, you write to the log file. The log is written, the log data is, the data is written to the log file. The log is written, the log data is, the data is written to the log file, but before it is written, I can pass JNDI kind of request to your web request,

which will force the log for J to make a call to outside world and download stuff from there. So basically think about this way. If you're running a wonderful log for shell vulnerabilities in your Java code, I can download like Bitcoin mining on your host machine and start running that process on your host machine. So it is that severe that somebody can capitalize and can use it maliciously to really download stuff on my host machine without my knowledge.

So can I just, I wanted to ask a question because I'm confused as to the way that the logs would work. I mean, maybe I'm a novice thinking here, but I would have assumed that all the log would do is like pass the literal string as a string, but it sounds like it's being parsed and processed as being written instead of just dumping the data. Was that always the intention of these logging functions?

And was that something that was useful for developers to be able to like, was there a good use case that, that it was being parsed that developers can then do something like include a directory, the working directory in the log without having to specify it via the code? Or was this just something nobody thought of in advance that, hey, it's being parsed, maybe it can be exploited? No, so nobody thought about that. So it was, if I'm a developer and if I really want to write my working directory into log file,

I would use Java programming language to get the current working directory, take that little string, and then my intention would be to write that string to the log file and not use the special characters. But I got to say, to counter that with my developer background, I feel like this is a very convenient feature because then i can just write in some magic characters and all of a sudden

my log gets formatted in a way that i like it so i think i'm not sure what the initial intention was with it but if i knew about this capability i maybe as a developer that doesn't have any bad intentions, I would love to use that feature. But obviously now we know that with power comes responsibility and also with power comes the power of the potential of abuse as well. And I think this is exactly what happened here. But Asad, let me ask you one question that means this will only work if i can access let's say a website and i pass in parameters on the url and i know that this particular backend service which is a tomcat

or whatever java application so whatever java application is actually taking my input and writing it out to the log and And this is obviously not by default. I mean, unless there are some certain settings in Tomcat that by default will log out every single request that comes in, that might be the case. But in order to exploit that vulnerability, I also need to know that my input string will actually be written to log4j.

Yes, you're absolutely right. So if I, you're absolutely right. So if you have a backend Tomcat server and I, as a malicious attacker, send some arbitrary string or a malicious string to your Tomcat and you don't do anything with it, you don't write it to the log file, nothing happens. But as we know, though, with your scenario, Andy, I don't think the attackers are saying, well, do I know if this site works or this one doesn't? They're just going to brute force it here. Just going to try it, of course.

As many people and not care where it hits and just hope to get one out of 15,000 hits or something, right? Yeah. And for me, and again, correct me if i'm wrong i'm a little naive here but doesn't this sound like a little sql injection vulnerability as well it's very similar right with sql injection exactly the thing was that you know via the url you're pass you're passing in some sql comments and you know the application is then taking this one-to-one and passing it to your database and in this case it's just the same thing parameters are getting passed to log4j and if you are lucky and that application server runs log4j with that vulnerability turned on or with that version that has it on by default

then you might be lucky and actually get stuff executed on the target system yeah exactly like in SQL injection especially if you're not using prepared statement, then the SQL injection becomes a very powerful attack point because I can delete your database. I can grab data from your database. So very similar. Exactly. Now, if we look back at the timeline from Log4Shell,

you said December 9th was when it was initially detected. Can you quickly run us through now at the time of the recording? It is January 3rd. What were some of the other significant time points in the last three weeks? Yeah, so what happened is that late December 9th and early December 10th is when this was discovered on GitHub. And then the open source databases updated their databases to talk to me, to talk about the criticality of this issue. There's other companies like Snyk, they updated their database on december 10th and dynatrace because we use sneak we updated our database less than five minutes so what happened

is that it was it was called zero um zero vulnerability issue where the issue was found without a solution for it and then 2.14 version of block4j was the third version and anything below that was vulnerable to this particular vulnerability in in java 2.15 came out to solve this issue. And then as I looked at it, 2.15 came out and then a few days later, it was mentioned that 2.15 also has vulnerabilities. And then the recommendation came is that you should update to 2.16 version of lock4j. And as we approached Christmas time, 2. became available because 216 has some other issues, like has some other vulnerabilities in it.

And last week, 217.1 version came out that is supposed to take care of all these issues. So as we interacted with customers during that time, we are seeing that customers, there were three CVEs that came out, one for 2015, another for 2016, another for 2017. And we are seeing how our customers are tracking it, making sure that they are all passed upon all that. So that's the kind of the rough timeline from when it started to where we are right now. And obviously, these new vulnerabilities were found because obviously the whole world was watching,

and everybody was like, I'm sure pretty much every security expert was taking even a closer look at Log4j. And therefore, more vulnerabilities were found in such a short time, I guess. That was the reason. And then thanks to all these databases that are out there, you mentioned Snyk, and I think there's probably some other public databases with all the listed security vulnerabilities got updated and then populated to all the other different tool vendors like Dynatrace. And we were able to then tell our customers exactly which of their

software runs exactly these particular vulnerable versions. Exactly. And also the other thing was unfortunate timing wise was that if you look at it, October, November, December are typically the three months where retail customers and banking customers freeze their code. They do not make any changes because of the holiday season. And this time, they had to take that rule because the severity of the problem was so big that they had to... Even just upgrading the lock for jail library is a big change.

So that's another thing that we saw in our customer base no that's actually go on eddie no i never thought about actually exactly the timing aspect of it you're right because i the whole show holiday shop holiday shopping season and everything leading up to the end of the year is classically a freeze time at least in let's say certain industries where they have certain regulations or where organizations are still using, let's say, I don't want to say it in a bad way, but maybe delivery models and processes that don't allow a more flexible way of delivery.

And I mean, we've been talking a lot about continuous delivery, about DevOps, about breaking things into smaller pieces so we can independently update software on the fly and mitigating the risk or minimizing the risk of a bad update. And I guess all of these organizations that are either in highly regulated environments where they have no other chance or are still, let's say, have not adopted the new principles of, you know, continuous progressive delivery, they are now suffering twice because it's really hard to update these things. Yeah. That's exactly where I was going to go with that.

I was thinking not only is this a wake-up call for security, but a wake-up call for pipeline, you know, for a delivery pipeline, especially if you have those older models. I mean, if you're in the, if you're, and I hadn't thought about that regulated scenario, Andy, because when I was thinking about this idea, I was thinking, well, yeah, anyone who's still on the older deployment, or especially, you know, monolithic applications where the slightest, you know, change, you know, hey, we're going to update the version of JVM. I remember back working in my old days,

anytime there was a software update, it was all hands on deck, no code, no code can go in. We're just going to change the package all hands on, you know, and it was a huge thing and it's still a huge thing for a lot of companies right now. So hopefully for the people who have the ability to be more flexible, this might give them the little kick they need to do that. Cause I'm sure a lot of people were suffering under a lot of pain to make these updates and make sure everything was, especially if you're talking about 3 reload, 215, 216, 217, 217.1. I mean, holy crap. What a nightmare to have to go through.

Yeah, and especially from the outside, it may not look that big of a deal, right? You're just updating one library. But updating a library always means certain things, you know, change and break potentially. Also, if you think as performance engineers, right, if a new version maybe has new features or different features on by default, change memory behavior, change performance behavior, all of a sudden your whole application behaves differently than before because you also never had the chance to really test it out

because you had to react so fast to the problem in production. Therefore, taking maybe shortcuts or at least a direct route to production. So definitely not a fun time. Think about your case of using the advanced features. Like if it did do $PWD because you wanted to take that shortcut, well, maybe that's been deactivated now. So exactly. Best case scenario, you get $PWD in your log.

Worst case scenario does including that now choke up the log processing. Like, does that have any other does that have a performance impact? I guess I thought I wanted to ask. I'll go on this idea. Yeah. Well, I think so. Just to add to that, a couple of things is that if you have updated your library to 2.14 before this problem occurred, then you're already keeping your library up to date

and then you move faster to move to 2.15, 2.16, 2.17, and 2.17.1. But we also saw a lot of customers still using 210, 211 version of Lofa J. So going from 11 to 15 is a big jump, right? But you have to react. Another thing that we realized is that if you own your code, like a lot of customers said, let me look at my POM file, my Gradle file,

let me upgrade them quickly or automate that so that it downloads the latest version. You can do that. But what about applications, that third-party application that you don't have the source code for, right? They become challenging. Hey, I wanted to ask, I haven't been living in this space, and I didn't see anything hit the news, but did anybody get hit really hard by this exploit? Or was this a known exploit that we didn't really see anybody get hit with? You know, not that I'm looking for a horror story with this, you know, because I don't wish that upon anybody with this.

But did we did anybody get severely hurt by this that we know of? One thing that I've been reading a lot is there were thousands and thousands of attacks happening. At least people were saying there were half a million attacks that they noticed over time, over a period of six hours and all that. So there were definitely a lot of people out there who were trying to capitalize on this thing when there were a lot of attacks that were going on. So the question is, did anybody steal data?

Did anybody was able to do that? That is not out yet. I think, I mean, I'm not a hacker, but if I would do it, I wouldn't do anything right now. I just put the Trojan horse in there and then I wait until everything comes down a little bit and then I hit. So for any hackers out there listening, take some advice from Andy. Take some advice from Andy. Hold your horses.

Strike when I tell you. Because everybody is now looking very closely what's happening right but if you could put in a little malicious code and maybe some background process that looks like you know like nothing bad because i'm doing a whole lot and maybe just keep it running because you never know and then eventually i feel laughing that's right this reminds me of of you know i don't want to go down there but like i I remember after the first attack on the World Trade Center under Clinton, some expert was on TV and saying, well, a small plane is not going to do. You'd have to really use like, you know, full-size passenger jets. And I was like, and then a few years later, what happened? So it's just funny that like as a quote unquote pundit, you're putting the idea there.

It's good stuff. Good stuff, Eddie. quote-unquote pundit you're putting the idea there it's good stuff good stuff eddie hey um first of all asad thank you for for giving us some background on on block for shell um i want to now switch a little bit to the the detection mechanisms uh we i mean obviously our listeners know we all work for dynatrace and we have an appsec. Can you tell us a little bit about why our approach in Dynatrace is unique to specifically detect this vulnerability, but not only detecting the vulnerability, but I think also, and you made it clear in your webinar that you did

and in your blog post. By the way, we will provide the links in the summary of this podcast. But can you explain to us why Dynat Anand Choice approach is unique to the detection, but also telling our customers what are the critical vulnerabilities that they have? Because I think this is the big thing. Yeah. So what that is, is that the beauty of the offering is that our customers that are already using that as one agent, that using the One Agent,

they are getting all the observability and all the APM data in Dynatest already. The same One Agent also has the application security capability in it. And as we saw this one become visible in the open, customers were trying to enable this feature and literally we had to enable a license file for them. And the reason it worked so smoothly for us is because the one agent is already in place in the customer's JVMs.

And what we do is that that is one agent looks at runtime, any library that is getting loaded into memory. And as you know, and it had in Java, library get loaded into memory only when they are used. So the fact that we are looking at runtime and real-time. That made it very easy for our customers to see exactly which JVMs are loading what versions of our LockerJ. And the other thing that we do, which is unique to us in this marketplace, is that our one agent also automatically creates the whole dependency map. It knows exactly who is calling whom at the process level, at the host level, at the service level.

So what we were able to do with our customers was we not only showed them which JVMs were loading the vulnerable libraries, but we also showed them the selected set of JVMs that were exposed to the outside world directly. So that made it very unique. Also, we can also see whether any of the JVM that was loading the library, is it making a call to our database directly or not, something

we call as sensitive data. Because if I am a malicious attacker, my other intention is not only to run any code, but to steal data from you. So having that one agent, the customer did not have to do anything. We just dropped the license file, and it lit up their application security UI and.

So if I understand this correctly, first of all, our unique approach of runtime detection allows us to really see which JVM is loading exactly the vulnerable versions. Which ones are vulnerable, we get from the security databases like from SNUC and others. But the second thing is, if you look at a large enterprise, there's so many Java processes running around.

I'm pretty sure even on my laptop, I have many Java processes running that I don't even know that are Java processes. But the question is, out of these tens and thousands of JVMs, which are the ones I need to focus on? Because I have limited time to really address the problem. So do I focus on all of them? Or do I focus on those that are actually

exposed to the internet? Or, and this is the second thing that I didn't know, is that we also look into those where there is a connection to the database because once you allow a tech into that JVM, it's getting easier to then get the data from the database and that could be confidential data or whatever there is to it. So that is really unique. And the one agent, once rolled out, just uses its injection mechanism to also do the security vulnerability detection. It means no additional software needs to be installed. It's just, as you said, turning on the flag on our end, the licensing, and that's it.

Yeah, exactly. And also the other thing was, and this was a legitimate concern from the customer saying that if you drop this license file, will it create overhead for us? Will it slow down? And as I said, the H1 agent is already doing the work for observability and APM. So there's no additional overhead. And because of the criticality of this nature, customer, first of all, wanted to go and look at production environment. And because of our assurance, as well as we had already had customers that were running that as one agent with application security on in production, we had big airlines turning this on,

retailers turning this on in production during the christmas holidays and there was no impact at all that's cool yeah and i wanted to go back to your example earlier andy because i think i'm understanding this better now with you know you talked about the data access you know connecting to a database versus if it's somewhere in the middle so back earlier andy you mentioned the ideas if oh if i know they're using um the log4j I know they're using a specific header, I might be able to craft a custom message to it. But then there was the counterpoint we came up with of I'm just going to blanket attack, right? So the more likely scenario of the blanket attack really fits in here because if you're trying to access data, let's say you try to figure out a way to get SQL injection into the log4j with some other way.

Well, that would mean that next tier below that Tomcat instance would have to be the database. If it's not the database, it's going to do nothing. And the only way you can get it to do something would be as if you knew what the code was running on the backend to write an API call to the next tier, which would then pull something, but then it probably still wouldn't get back to it.

So it does make it like, you know, I wouldn't say impossible. You'd have to have somebody on the inside to exploit that. But to your point, that knowledge of whether or not it's directly accessing the data is a huge differentiator for that way. Because, you know, what hack is going to be something customized to a specific environment unless there's some, you know, conspiratorial, you know, attack specifically being done by, you know, Tom Cruise and his Mission Impossible crew to save the world, which is a whole different scenario. Yeah. I mean, one vector of attack, obviously, I think, Asad, you mentioned this in the very beginning is downloading, let's say, Bitcoin miners or stuff like that, right? I mean, it's one thing is stealing data. one other thing is stealing compute right resource power

i mean that's obviously that makes a lot of sense or you could think of uh that they are injecting you know bots that will then just i don't know send out whatever bots do right send out emails denial of service attacks they become part of the pot their bot network without people knowing it and then the the requests from these bots would come from trusted networks potentially yeah and that's obviously very dangerous well um asad is today's january 3rd as time of the recording uh do you have any uh latest information on what's the current situation is the the latest version of log4j safe and sound? Or is there still anything we can expect to come? Yeah, the 2.17.1 is the latest and the most stable version out there.

And I have not seen any update about it being also in one level. So it looks like 2.17.1. And also, I think this is something that time will also tell how well it is well tested and all that and also what happened is that we saw in our customer basis till christmas everybody was trying to update and they updated to whatever version they could and again as you as you said the processes that were exposed to the outside world, they kept more active eye on those JVM processes. And I'm seeing that now the

customers are back from vacation, that they will not only upgrade to 2.17.1, but it will be a lot more holistic approach they will take to make sure that all the processes are up to date and to the latest version. Now, I know we had, you know, with lock for shell, there's this vulnerability where it's processing. I'm sure everyone else now is looking at log processors to see if anything is going on. And I guess we haven't heard anything about any other vulnerabilities, but I suppose there's always the potential and hopefully we won't hear about any more, but has there been any, you know, do we know if things like, you know,.NET, Go, all these others have been, have vetted their, come out and have other process vendors vetted their logging system

to come out and say, yeah, we're clear, we're good? I definitely looked at Log4Net because the name is similar and of course for.NET, and nothing has come out for Log4Net. I think that is definitely much more safer. Nothing I have not heard from on the Golang stuff also anything whenever they did. But one thing that it was interesting as the team engaged with customers and enabled this feature in their production environment as well as non-production

environment so all the eyes were on the lock for shell vulnerabilities but when we enable the license for a lot of customers we saw some other critical libraries that they were using which these libraries and this one of these have been for two, three years now in certain cases. So it became important, and I think the message was clear that, yes, right now the focus is on lock for shell, but the most important thing is we need to keep the customer who got the impression and the most important thing is we need to keep the customer who got the impression and got the right idea that we need to keep, that they need to keep their environment up to date

and all different libraries that they use, not only for lock for shell. Lock for shell, I believe, was a catalyst to bring this awareness. And we saw same level of criticalities in other Apache libraries at customer size during holiday season and we brought it to their attention and they made note of that. So I think the message was clear and I think actually I was very pleased that we could partner with our customers to really help them out at this critical time and also help them prioritize which one to attack first. Another thing that we saw, I heard a lot from the customer is that a lot of customers have a lot of different tools, which is great to have.

It's good to have a gamut of tools, especially on the security side. One of the challenges that customers, some of the customers were facing was they were getting using different tools and they were getting different reports. Some of the tools were telling you what log4j files are sitting on your file system. But what that created was a lot of noise where they found log4j sitting in a backup directory

or a temp directory or a test directory. And you know that I, as a developer, as an admin, sometimes I leave things on the file system, but it doesn't mean that those libraries are being used. So by using multiple of tools, including Dynaprace, they were able to filter all the noise. And they are using all set of tools to really clean up their environment. But having a filter view and focusing on the GVM data more vulnerable at runtime,

they were able to kind of stop the bleeding or stop, put the bandaid on it. And I, first of all, thanks again for that reminder, because I, you're right. stop the bleeding or stop, put the bandaid on it immediately. And I, first of all, thanks again for that reminder because I, you're right, there's different approaches obviously to deal with security and just scanning the file system potentially generates a lot of false positives that then people waste their time. And if they, especially in times when they don't have time.

The other thing that I want to highlight here is just also pointing people to some information that we have out there on the YouTube channel and on the blogs around shifting security checks also left. I mean, it's very important, obviously, in production because these vulnerabilities get detected on the fly on software that's been out there for a while. And this is why you need to have this information. Production is critical, but it's also very important to prevent any security vulnerabilities

by including security checks as part of a delivery pipeline. And this is where we are also working with our customers on shifting not only what we call quality gates or SLO validation as part of our cloud automation and captain strategy to our customers in the pipelines, but also including security checks as part of the delivery pipeline. And because then if a developer makes, let's say, the knowing or unknowing mistake of including a vulnerable library,

your pipeline can automatically say, stop, don't push this into production because this has a known vulnerability. So check this information. Yeah, that's a great point, Andy, because other thing that happens with Dynaprace is because we are doing runtime and real time, we are able to catch it as the library gets loaded. One of the things that I, even not nothing related with lock for shell, but over the

last year, one year that I've engaged with the customers, one of the things that they find is tough to do on the CD pipeline is to have like a static code analyzing tool that looks for the lab data bundling your source code. They like to make it part of CD pipeline, but those processes take six to eight hours to scan through your code. Ouch. But for us, because we're able to, so again, I'm not saying those tools are not valuable.

They're absolutely valuable because you want to scan static also. But as part of CD pipeline, if you can do it runtime, real time, and get answers quickly, you can stop your build immediately instead of waiting for six to eight hours yeah imagine imagine what that would do to what was what was amazon's pipeline every what like three seconds or something that that that blows that out

of the water if you're doing that um i mean yeah as you said it's valuable and there's trade-offs but um, yeah. You know, Asad, you mentioned an important point earlier, and I wanted to just bring this back to the regular person, right? Because a lot of what we talked about today so far has been about enterprise systems or commerce systems, whatever stuff that you're engaging with. But you mentioned the idea of people are running old libraries that have been updated years ago. And so it's important to keep on top of the latest and, you know, like it or not, this is why we all need to continually update our own personal laptop operating systems and keep updating our phones and our phone apps. Because yes, we know we had the issue in the past where Apple was bricking our phones because they were trying to sell more.

But the real important piece of it, and hopefully they seem to have gotten away from that now pretty well, but it's really about getting the latest security patches. Half the time when you look at a release notes, most of the times it's like 90% security patches. So just for the average person, keep updating your stuff. I used to have arguments with people like, no, you need antivirus on your PC. Oh, why? I'm like, you need it, you know? And it's like, you need to update your phones

because if these exploits are being, maybe you have something running on one of your devices that has a similar exploit, especially if your phone is connected to your corporate system, right? That's another attack factor. So it's just always important to keep updating and keep running the latest.

And I think that's why that's so important when you mentioned, you know, not just scanning for log for shell, but looking at all those other libraries that are running in there and saying, oh, yeah, once we get past this emergency, we really need to tackle these other ones. And it's nice that the Dynatrace view gives you that view of the exposure, what's publicly accessible, what's data accessible and all that. Yeah, and I think also there is a little bit of interesting timing point of view because we have a lot of customers, as you know, who are moving from the on-premise monolithic application to microservices in the cloud.

And so a lot of them are saying that I'm going to freeze my existing monolithic. I don't want to make any changes to that. All the new functionalities are going to go into the cloud. But when something like Lock4Shell comes around, you have no choice but to create your Lock4J library, even for the monolithic application, because that's still running your business. It's still generating money for you.

So valid point that you raised, Brian, there. But a lot of customers are also choosing not to upgrade because they are already in a migration path towards the cloud. So it's interesting timing here. And also that means the saying, never change your running system is also basically blown away because you potentially have to change it and it's not under your control why

you have to change it. Especially for a very long time, I as a developer would like to use open source library because it just makes my life easier but comes with a little bit of onus

to make sure that you are on the latest and the greatest version. Hey, Asad, thank you so much for the insights and for being there. I know you have been involved from hour zero, pretty much. I saw you were getting out these blogs and the webinars and helping customers. And then I also talked with other colleagues within Dynatrace. They've all been very busy. So has the whole world. It doesn't go just for Dynatrace, but a lot of people around the world

that keep us secure. Thank you so much. I would love to have you back in, let's say, a couple of weeks to see what's new. I think that's always interesting. I think, Brian, we should have security definitely as a standard topic on our show because it is important, as important as performance. Exactly, exactly. And big shout out to, you know, not only you, Asad, but Daniel Carr and the entire security team. I know you've all been slammed,

but also just even the developers who've been working on this thing without uh without the knowledge that it would get such a spotlight um right so kudos to everybody out there from uh from our teams who've been working on that and suffering through all this uh thank you assad for being on it's been amazing any any final thoughts before we wrap up no first all, thank you for having me on your podcast. Another thing is I'm keeping my eyes open. I think, again, just keeping an eye on

keeping your libraries up to date is now becoming the default mantra now. And I hope that helps solve a lot of other issues besides lock for shell. Awesome. Well, thank you for being on and happy new year to everybody.

If you have any topics, ideas that you want to contribute to the show, you can tweet us at pure underscore DT or send an email at pureperformance at dynatrace.com. Thanks as always for listening. We're glad to be back. And hopefully, you know, right now we have two other good shows coming up after this. And Asad, we look forward to having you back on for the latest update. And thanks, everybody.

Bye-bye. Thanks, guys.