PurePerformance - How to protect continuous software delivery against supply chain attacks with Michael Plank

Episode Date: February 22, 2021

Software security is about securing websites against malicious attacks or using firewalls to prevent hackers entering your enterprise network. While this is part of software security there is much more that needs to be done – especially as more organizations are developing critical software it is important to protect the whole software delivery lifecycle from any malicious attacks along the supply chain.In this episode we have Michael Plank, Technical Product Manager at Dynatrace, talk about his latest blog post titled How Dynatrace protects its software development and delivery life cycle against supply chain attacks. We learn about attack vectors from development workstation until production deployment. He covers the strategies ranging from static to dynamic code analysis, vulnerability detection or code signatures. Tune in and learn that building secure software is more than ensuring your users have hard to crack passwords!https://www.dynatrace.com/news/blog/how-dynatrace-protects-its-software-development-and-delivery-life-cycle-against-supply-chain-attacks/

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. My name is Brian Wilson and as always, I always say as always, my co-host Andy Grabner. Andy Grabner, how are you doing today? I am exceptionally well because I just finished recording three breakouts for Perform, which is now actually in the past. So, and I hope it went well, right? Giving away the mechanics behind the curtain.

Always, Andy. That's all right. That's all right, yours. Everything is an illusion, everybody. Everything you see, it's all an illusion, including live. When I was growing up, we had live via satellite. That was the big technology when you'd have somebody halfway around the world and they'd have a big banner on the top that said live via satellite. And there'd be probably about a five second delay or so between when a person finished talking. So we should try

doing that. No, we don't want to do delays today. But it is really interesting, right? Because you are over in Austria, and we're talking as if we're just right next to each other. Exactly. We have our guest who is, I imagine, in Austria, although he did reference German probably because of the language, but I imagine he's in Austria as well. Yes, I am. So we have this, oh, you see he's speaking before he's introduced too. He doesn't know these little subtleties. He's a newbie. So it's just amazing. It's amazing what technology can do. It's really more

amazing how people abuse technology and try to exploit it. Are you alluding into some type of topic from today? I'm doing a really poor impersonation of you in your great segues. See, Andy, to our guests who I won't name yet, I'll just let you know, Andy is fantastic at summarizing, but also taking whatever stupid banter we're doing right here

and somehow transitioning that into the topic. And this was my poor attempt at being more like andy we should all try to be like andy so andy why don't you take over my failed attempts here no it was was a great attempt and thanks for um for making it easy for me so as you said right software we've been talking about software performance for almost five years now in this podcast yep and uh what's the best performance if the software is not secure? Right.

What if, right? I mean, who cares about software, the best one in the world, if you don't want to install it because it's a potential security risk. Now, when I think about security, I always think of, can I break into a system by,

I don't know, doing an attack on logging in, like with passwords. I'm sure there's different types of attacks where you can break into a system by, I don't know, doing an attack on logging in with passwords. I'm sure there's different types of attacks where you can break into a system. But on December, let me look, on December 16th, I saw a notification in my emails about a new blog post. And the title is How Dynatrace Protects Its Software Development and Delivery Lifecycle Against Supply Chain Attacks. And this triggered me to reach out to Michi Planck, who is our guest today, because I really wanted to know more about security in the delivery lifecycle and what is this whole thing about supply chain attacks.

So this is why I want to now officially give Michi or Michael or Mike the word. First of all, introduce himself and then we will slowly segue into the topic. But hi, Michi. Hey, hey, Andy. And hey, everybody. Nice to be here on the show. Let me briefly introduce myself. My name is Michael Plank.

I'm with Dynatrace for seven years now. And I am a technical product manager, part of the chief information security office at Dynatrace. And I'm personally responsible for the Dynatrace platform security. And that's why I have written this blog post on a very actual topic, on a very hot topic. And I'm happy to talk to you about this today on how we at Dynastrace do things. But maybe I also have a few recommendations to the listeners to dive into.

Yeah. Now, Michi, unfortunately, this is a podcast so that means people won't be able to see us however if people want to know how the milky really looks like there's a video on youtube and if you google for diamond trace ufo you are kind of the main character in that really cool you are you are the scientist in the beginning like close to the austrian alps or somewhere in the middle of the austrian alps and it's like so folks if you're listening if you want to see uh how he how he looks like and how how great he is and as an actor uh just google for dynatrix ufo youtube and you'll find the video

hey andy before we continue i have to, when did you start calling it a UFO? It used to always be UFO. UFO, UFO, yeah. You mean when I started calling it UFO? Yeah, yeah, yeah. I believe you used to. I know a lot of other, you know, my Austrian colleagues called it an UFO.

It's probably when we kind of launched the UFO, UFO. I lived in the States and people were looking strange at me when I said UFO. And so I figured out that it's a UFO. Well, congratulations on. Anyway. Is UFO something different in English? I don't know, but the reason why we insist it's not UFO is because it's UFO.

It's U.F.O. or period because it stands for unidentified. It's an abbreviation. Now, actually, I know we're totally off topic here. I actually really kind of loved the word UFO because I'm like, oh, that's awesome calling it UFO instead of UFO. It's novel for me. But it was also kind of funny because it's just like every time,

and this goes back way back, Andy, when we started talking about accents and you were like, well, you have an accent to me, hearing words like aluminium and words being said in different pronunciations in different countries speaking them. So UFO was a fun one. Anyhow, yes. I have to add one more thing on the UFO story.

Because when I first presented the UFO story in a Spanish-speaking country, and I said UFO, UFO, they had no fucking clue what I was talking about. Because it's a different abbreviation for unidentified flight object. And I was like, UFO, UFO, what? What do they call it? Object flight unidentified? No, I need to look it up. But actually Gabi reminded me about,

she explained it to me once. Unfortunately, Duolingo doesn't include UFO as one of the words in the basic version of learning Spanish. And I need to point out, that was the first time Andy Grabner cursed in the podcast. Did I? Yeah. It's amazing. Now we're going to have to do an explicit rating on that this is great so if any children are listening we

apologize uh but if any children are listening welcome and it's awesome that you're listening and it's okay that he curses he's austrian they you know they speak their it's different country anyway security is a hot topic anyway. No, seriously, it is. That's why I'm really excited about this. It's not just because people are starting to pay more attention to security within their organizations, but it's hitting the news more and more. I think between all the hackings that are going on, there are some large software companies that are getting exploited. So this is, as you mentioned, a very, very timely topic.

So why don't we get off the UFO and get back to security? Yeah. So Michi, do you want to give us a little overview of what you're doing and how at Dynatrace now we are securing our delivery pipelines? And also, what do you really mean with supply chain attacks? Because for me, this was the first time I heard it in the context of software engineering. Yeah, I think it would make sense to start at the very beginning and really briefly explain with maybe a scenario what a supply chain attack really is

and why it is so scary. Cool. So let me describe this worst case scenario. Let's say you have a software provider, A, who produces software that needs to be installed on customer's infrastructure. And now you have company X, Y, and C who install this software package from company A. Now, the thing is that in a supply chain attack, you would actually not compromise

the X, Y, and Z directly, but you would start hacking and compromising the software that software company A produces and provides by, for example, implanting or injecting malicious code to that software. And now this software gets rolled out to all the customers and now XYZ maybe do an update of the software and now they run the malicious software on their own infrastructure that, for example, exfiltrates all the data to the adversary. And that is such a scary scenario because for company X, Y, and Z, the software consumers, they actually didn't do anything wrong, right?

They didn't have any type of security related. They didn't do anything wrong, right? They didn't have any type of security related. They didn't do any mistake. It was the software package that they're using internally, which got breached. And now all their data got lost. So, and this is a worst case scenario. Well, basically for all parties involved, right? But maybe it's even more worse for software provider A, which is responsible for a huge, let's say,

hundreds of thousands of customers getting compromised because their software was hacked. And yeah, that's a very hot topic for every basically software provider who creates software which needs to be installed on infrastructure. And we at Dynastrace Office also provide such software and that's why we have picked it up. And not only after the recent news,

but we track this as the number one worst case scenario on our top business risks for years now. And it's not something new that we're looking at. And therefore we already had a pretty good answer that I could also provide in my blog post. With the recent news, I assume you're referencing to the SolarWinds story, which is, as you said, unfortunate because that's really a business deal breaker in the end, right? If you know this company unfortunately had this type of breach and now so many thousand customers of them are infected with this malicious software, that's really tough. But I would assume, so I mean, this is an example that we all know. It's recently in the news and it's really a piece of software that you install. But I guess it goes further right it

goes down to maybe libraries i don't know third party software that you just use but also i guess libraries right because i'm if i'm including libraries and i'm downloading something from docker hub or from i don't know github somewhere is that also the same kind of challenge? Yes, absolutely. As you say, third-party libraries, open-source libraries, it's everything what you consume from any third party, right? Is it a whole software package or is it just a library? In the end, as you say, everything is kind of the same risk. With everything third-party that you use, you basically have some kind of new risks that you introduce on the other hand you can obviously not do everything on your

own so um that's a huge challenge how to to balance um risks versus doing things on your own um yep now when you and unfortunately i've never been in that position where I have to purchase enterprise software and then going through, I guess, a vetting list. Because I assume if you buy software in the enterprise space, you do security checks, right? Upon the first installation. as the world is moving towards continuous updates, how can you actually ensure that every time the software and the AI, like in your case, is pushing a new update, that you get included into that update process and then enforce your own security policies? Is this something that the world is thinking about,

how we do this? Yeah, I mean, you can only do it. So no, I've never seen anybody who would do that on every release, right? To run any kind of security due diligence or security evaluation. I mean, we're also releasing major updates every two weeks and updates like every day. So no chance basically in today's world to do a security evaluation of each and every update. So what companies who are very security aware and have a lot of security obligations towards their compliance rules and such, they typically do an annual review of their vendors and basically start from scratch every year with every third-party vendor they have and start the security due diligence

from scratch every time and see if they were changed and such. But this is really more the things you do on paper to if something happens, you can say, yep, we did the reviews, but the real risk is... And also this recent case shows that you can have like every single security certification out there, which SolarWinds I think had to even, you know, go down to be able to install your software on on u.s governmental infrastructure so if if if you get if you get hacked seriously none of these certifications does say anything right yeah yeah it must make it sound hopeless and i bring that up in a way like you know is it that

we haven't caught up? Is it that the model has to change? You know, just even thinking of the software delivery model had to change. Is there some concept of a security review model that needs to change or something? Or do you think we just don't know yet at this point? Good question. There's a thinking face going on on mic. Yeah.

Just for our listeners. If you have a thought, Michi, then let us know. Otherwise, I have a question. Yeah, no, I don't have a thought right now. So here's my question. And this goes, hopefully, not in a discussion, in a quote-unquote religious discussion, but open source versus closed source.

So are we safer with open source versus closed source? Because with open source, everybody can do whatever they want. And if nobody's catching it, then obviously that, I mean, a lot of people can infiltrate very popular libraries if you don't have the right gates in place. So do you think this is a problem or the other way around would be, and this may encourage a lot of enterprise software companies to open source more of their software is saying, hey, we put our source code out there. Everybody can see what we're doing.

Everybody can run their checks against the open source code. So we have nothing to hide. So I was actually wondering if maybe either going towards open source could be a potential solution to the problem or opening up towards open source could potentially be more challenging. And therefore we need to rethink the strategy. I hope it's the first one going towards open source, but just want to see what your opinion is on that. Yeah, that's a very, very good discussion point because yes, this also has two sides.

So on the one hand, everyone thinks, yeah, open source software is definitely more secure because everyone can see the source code, right? That is totally true. In big software projects where you have millions and millions of lines of code, I mean, who has actually done the security review of all that code, right?

I mean, yes, you would maybe have a better control on pull requests or all the changes that are being done. But I think if we would open source Dynatrace today with I don't know how many millions of lines of code, you would need to start looking for, you know, through so many lines of code that it doesn't make it better to just say, yes, it's open source. Everyone can do a security review on that.

You're again back to, you need to make sure that you have security tools in place, which for example, do static analysis on your source code to check for common vulnerabilities, because that is also what the attackers typically do, right? They use the tools to look for, I don't know, like credentials in the source code, but then they would also look for common vulnerabilities in the code, which might be exploited. So all these things you would also typically do in closed source projects.

And on the other hand, you do have a bigger risk on open source if you don't have a very thorough code review process in place. It's so much simpler to get malicious code into the code base. So you have to have definitely a way better review process around that, which you might not need on that thorough basis on closed source. So, yeah, both sides. Yeah. Now, Brian, do you want to say something? No, no.

My brain is just spinning with thoughts. Nothing that I can verbalize or bring up at this point. Continue, Andy, please. So then you mentioned static analysis, right? That's one option. Is this what we are doing? I mean, can you give us a little insights on how, and this is the base of your blog post,

which by the way, we will link to in the podcast proceedings. What are we doing? How are we protecting ourselves? Yeah. So there are many places that you need to look at from a CICD perspective. And it always starts from code, right? So you would start off by having a look at, you know, how is code produced? Where is it stored?

You know, who has access and all that kind of stuff. So before you even think about any security tool, you need to make sure that you have the access rights and the basics on your code repository in place, right? That's the first thing. And then you, from, I think the, so that reminds me of one question, which I received from a bank who also did a security review of Dynatrace. And this guy, the security guy asked me, so how do you make sure that nobody checks in malicious code

into your code base, right? And that you really need to think about this. It sounds like such a stupid question, but it's a very good question. So if you, for example, don't have mandatory pull request or mandatory code reviews enabled on your repository, I mean, how could you ever say, yeah, everyone's good and nobody, you know, it's just not possible. It is. So this is where you would definitely make sure that you have, you know, the 4i principle for every code change that is production-related in place,

where Git and pull requests are a great option for that, obviously. And then, once you thought about how the code is produced, how it gets reviewed, only then you get into the further stages of the pipeline, right? So do you want me to go through some of the additional pipeline steps now? I think so, because I think this is very interesting. Also, from a delivery perspective, I just had a couple of breakouts at Perform on continuous delivery. And we have typically focused on automated testing, automated performance evaluation, automated deployments. And we have not so much talked about automating security,

even though really Higgs, he did a session on DevSecOps, showing our customers how to include the new security capabilities of the Dynamics RISC-1 agent as part of continuous delivery. But I know there's more possible, and that's why I think it will be very interesting to hear from you. No, absolutely. So let's start from, okay, once the code is checked in, what can be done afterwards?

So I mentioned static code analysis. That is basically where we only look at the source code. You have tools which try to find flaws in the code just from the text files. You can use open source tools for that. So SonarQube, for example, is something and find box and find security box are additional plugins that we are using to identify static flaws in the code. And you can also obviously extend those, if you're using SonarQube security rules available for that technology, so we built our own.

And then, obviously, the thing you need to make sure is that the static code analysis tool that you're using immediately informs your developers on something got found and it has to be fixed. I want to also talk about that maybe later a bit on putting tools in place which do fine things, but nobody taking care of those, right? But let's not go into this right now.

So on the static side, one other aspect that we have been focusing on very much during the last couple of to us a lot of times. Also on GitHub repositories, we saw it a lot of times that things like API tokens are getting committed to the code base, which is really a pain because these API keys can... It's like a password. It's like username plus password in one single character sequence.

So API tokens, for example, should be able to be detected by static scanners and should be even, you know, prevented from even being committed to the code base. So it doesn't get found. And you also have this problem with Git history that you even have to remove it from the Git history to hack us and also security researchers

to find secrets also in the Git history, not only just in the checkout. So that is what I can definitely also advise everybody to put controls in place which find secrets. One thing that we improved on the product side was we were using an API token format in Dynatrace, which was not easy, which was actually not at all capable of being found by a regular expression.

So if you think about providing any type of API token or any type of access token mechanism, make sure that you use a format that can be detected, which gives you as a developer the chance to find these things easily, but it also gives the adversaries and attackers

out there an easier way of finding those types of tokens. But still, I think it's better to be prepared and put security controls in place to find and then also remediate those secret findings. So I think that's pretty much it on the static side of things.

And then next, you would build and compile your software. I guess the next step in the pipeline would be to look at the third-party libraries that you're using. So this is what is needed by the build. And that's one of the next big sources of risks that you have in your delivery pipeline cycle. So most obviously, every software uses

a ton of open source libraries. So how can you make sure that you're using the ones without vulnerabilities? And there you have to use any kind of third-party vulnerability scanning tool, or obviously, Dynastrace AppSec, which helps you with that type of problem.

And these types of tools give you a list of vulnerabilities, these third-party known vulnerabilities, right? Yeah, I was going to ask about that there. These are the known ones, right? These are the known ones by the community and they are so-called CVEs. And yeah, that's a very big aspect also and a big time-consuming part actually for our

security and product security team to deal with those third-party vulnerabilities because there is no library out there which does not have any security vulnerabilities. And you always have to patch them. You always have to, okay, now a new one is getting introduced. You again have to patch it. And obviously, it's a good thing to patch, even though it's also time-consuming and sometimes not even necessary because you're maybe not even using the vulnerable code path.

But our customers are also very much aware of this type of security risk because there were big data breaches because of old, outdated, vulnerable third-party libraries. So yeah, I guess that's another step in the pipeline

to have a view on security risks in third-party components. Cool. What about the pipeline itself? I mean, if you use Jenkins, if you use whatever other delivery tool for building and for deploying,

isn't that a perfect attack with a malicious code that you inject, like a malicious plugin of Jenkins, for instance? I mean, I don't want to bash on Jenkins, but it just comes to mind because it's used so heavily. Yeah, no, absolutely. And I mean, here on the Jenkins side, you would again say, okay, these are open source tools. These are open source plugins. And we, as we talked about open source previously, we very much hope that, you know, this code gets looked at by many, many, not only engineers, but also security researchers really looking for vulnerabilities in those libraries and then file the CVEs for those. So I think for Jenkins, but still, I mean, this is definitely a risk. And with every plugin that you install for Jenkins, you need to make sure that this is not something that looks you know fishy now in your blog post i'm just browsing

through it because it's really thank you so much for this because it it gives such a great overview uh also very nice graphics so that people actually understand like me you know where in the delivery cycle in which type of system you need to be aware of of security, potential security breaches. Now, you mentioned databases of known vulnerabilities. So that means there's people out there or organizations out there that keep track of all of this. And then I see also a section further down in your blog post talking about independent audits of security control, which I think is also really interesting. There's FedRAMP in there, which I know we just received a certification for. Can you talk a little bit

about this? Because this is at least an area for me that I'm completely unfamiliar with? So these agencies or these audits? Sure. So we do two types of things. The one is so-called SOC 2 Type 2 certification, which is more on a company level. So they look at the processes and the things that we tell them that we do, they review. Like we say, they're asking,

how is code being reviewed? And they actually look at these processes if they are really in place like mandatory code reviews. They look at things like what type of change management process you do have in place, like do you use any type of JIRA or something. And so this, the SOC 2 certification is rather looking at the security controls on a company level and more on a process level. Where the other thing that we do on an annual basis,

so we do this recertification on an annual basis on SOC 2. The other thing that we do on an annual basis is our application security penetration test, which is also being performed by an independent external security firm. And we rotate that company also every year, which is really looking at the security of your software. And this is, for me, you know, we have a lot of these security controls

in the software delivery pipeline in place, but how effective are they really? You know, how much vulnerabilities do they actually catch but how effective are they really? How much vulnerabilities do they actually catch and how efficient are they? And this is where I think the annual application security pen test is also a very good indicator of, is the stuff that we put in place

over the last year really working? Because you basically see it in the number of findings they get, right? The number of vulnerabilities these companies find in your application. And for us, it really, over the years, it decreased. The number of findings decreased, where on the other hand, the scope of our product actually increased over time, right? So I think this is, at least for us, a very good sign that we are on a good track on putting those security controls in place

that these independent companies don't find really much, but still features are getting added every day. And also we increased the days these companies test every year. So it's not like they have just one day. No, we extend budget and days of testing with every year. And that is, I think, a very good check in time with a snapshot in time on an annual basis to say, how much do they find this year?

And obviously, this is not all what we do in terms of testing. It's obviously good to put your own pen testing guys in place or things like bug bounty programs. But still, all these controls that you might want to add to your secure software delivery lifecycle, I think they just all end up in an independent firm finding less.

That's really cool and i assume these companies are obviously paid by the number of findings so they have a good interest in finding a lot of things for you yeah kind of um i mean the in general the they uh they're paid on on a daily basis. Okay. But, you know, it is their interest to find things. It's, you know, you don't pay thousands of dollars for something where you say,

we didn't find anything. So they... And we, in every year's pen test, we do work closer together with them and also give them guidance and everything they need to really dig into the product really deep. And we always had, you know, it's not something you give them and it's done.

But during the testing period, it's back and forth between you and the security company. And I have to say, you have to pick the right vendors for this, obviously. But the ones that we picked, we always had a very good um feeling that they were really trying hard yeah yeah i was gonna say you can go into a rabbit hole of conspiracy and here in paranoia because i imagine there's gonna be some bad actors out

there who'll be like oh yeah we'll give you an all clear every year you use us every year we'll just say everything's good. This way people, you know, so when you talk about the supply chain, it's even about which companies you're doing to review your supply. It just goes down and down and down and down and down. And, you know, that's, you know, earlier I said my mind was just racing in all these different places and it just keeps on going because yeah you know you watch these movies where it's these you know humongous world like let's let's say um hydra right i just

got these conspiracies and everything going on to take over the world and you have to have a top notch agency like shield to be able to stop this all but that's obviously the fictional world but in a way it's real look at what happened with solar winds right um they they had sign off they had all this stuff how do you know and there's no it's so hard to keep on top of everything you have to kind of trust that the system will do as best it can but i think you know we're always one step behind and you just got to hope that you're not the one who gets caught you know with just the unfortunate luck to have had the the one wrong thing missed you know it's crazy but you could do your best and i think we sounds like we're doing our best we're doing everything we possibly can but it always

just sounds like it's not enough, you know, but I don't know if there's ever going to be enough unless we just, you know, the best way to get your kids not to eat candy is have no candy. Best way to have no software vulnerability is to have no software. Right. But anyhow.

Yeah, that's true. We will definitely always be one step behind. And, you know, a lot of big companies also say, you know, if they, if they're getting attacked by a nation-state

agency, it's almost impossible to defend against it. But still, I think that's the beauty of security and the huge challenge we face every day in security. We are never done. We have to look at the things that are happening out there and then immediately respond to what's happening and see if we have the right controls in place

to be safe against a new kind of attack. So yeah, that's what's never going to make it boring for us here in security. Yeah, that's cool. And I think this is true for security. This is true for other areas too. I think we will never find, and Brian and I are very big into what we all are, but performance engineering is something that we grew up with kind of. And so we also will never find the perfect software

that runs perfectly from a performance perspective. So it will never get boring there. And it's always a tool for security. That reminds me, Andy, too, you know, in the performance side, the big lament was always people don't take performance seriously enough and it's going to take down your company. You're going to have an outage, you know. in the performance side, the big lament was always people don't take performance seriously enough and it's going to take down your company. You're going to have an outage,

you know, and you can make even a stronger case for security, obviously. And both fields have struggled historically with getting the attention they need. And fortunately, it seems like performance in the last maybe five years really, really got a lot of attention and is finally becoming a first-class citizen in IT considerations.

But security still struggles, right? I mean, we hear about these disastrous things which may finally start pushing things over, but it's who wants to block everything down, who wants to lock everything out. It's a similar security and performance are sort of like brothers in this fight. We just happen to get a leg up on security. But I think, to be honest, I feel like security at this point is a little more important.

Yeah. Now, Michi, you mentioned it a little bit, but at Dynatrace we've announced a while ago that we also go with our product with our platform into the app security space with our one agent technology being able to find vulnerabilities are you using dynatrace on dynatrace are we using our own product? Yes, absolutely. No, yes, we are. And for us, it's very exciting because obviously everything is monitored with Dynatrace already. And now we can,

we just enabled the new functionality and we immediately got all the benefits of receiving all the vulnerability information and such from all the environments. It's actually really cool to get rid of all the static type of scans that we had in place for third parties. And also be able to finally get a real view, not only on development environments, where we typically do the scanning to have everything as soon as possible in the delivery cycle, but also have a real view on what is going on on the production side of things.

Where are we there on vulnerabilities? Maybe production is a few versions behind, so you might have another vulnerability here than you would see in development. And with AppSec, that's going to be a way more easier now for us to deal with. Perfect. So folks that are listening, this was a little plug for the new Dynatrace security capabilities in case you have not seen it or heard about it.

We have just recently announced that extending the one agent technology to scan for security vulnerabilities and i'm sure that's just the start but uh it's great to hear now michi is there anything else that you may kind of concluding to this is there anything else you want to make sure the audience understands kind of like like what are the best, what else do people need to know? What have you learned that you say, I would have wished that this is something I knew when I started with this job? Yeah, so one thing I definitely wanted to mention is code signing

and what that is, right? And then I may want to conclude with giving a little bit of guidance for everyone who also wants to look a bit deeper and maybe has not many security people who can look into such a supply chain attack scenario. But on the code signing thing, I think this is

where software companies really have to have a good understanding of what that is and how it is so much helpful to prevent such a supply chain attack. And this is basically, to really keep it simple,

you basically sign your software with a digital key that only you possess, right? You as a company. For us, we have those private keys, they're called, stored on physical devices that you would even have to, you know, steal from the data center to get your hands on these private keys. So you would actually sign your software with that keys you only have. And then this basically, you know, it's like a signature on a piece of paper where you say, okay, this is what I produced. And then you hand it over to your customers. And there, obviously, you have to have something in place

which verifies this signature, right? And having such code signing technique in place basically makes sure that the package you're producing within your CI, CD pipeline cannot be manipulated somewhere in between, right? Because you may upload your package to S3 here and other place there, and then your customers need to download the update

or software package. But code signing really gives the customer then the opportunity to verify that this package really has been signed by Dynatrace or by the software company. And this is

what definitely should be. And on the other hand if you install software on your Linux service and figure out there is no way for you as a system administrator or DevOps engineer to verify that this package was not manipulated

then you should ask yourself administrator or DevOps engineer to verify that this package was not manipulated, then you should ask yourself, is this really the right package I'm installing here? How do I verify? And yeah, so that would be just a recommendation here, not only for software providers, but also consumers. Very cool. just a recommendation here not not only for software providers but also consumers very cool and now i finally understand why when i download the one agent there is this third step that i typically skip because the first one is download the agent the third step is install so i always do one and three and the second step is verify that signature and now i understand

why this is so important. Exactly. And likewise, Andy, I always skip over that or tell people, yeah, you don't have to do that, though. I think now I could be like, no, do that, please. Okay, so co-signing. Anything else, Michi? Yeah, so the one thing I wanted to

explain a bit is how CICD or how development teams might want to tackle this thing of, okay, how do we now make sure that we don't fall victim to a supply chain attack. I mean, the software vendors who, not as a software consumer, but the vendors. And what we typically do here is like a four-step kind of workshop where the first step of this process would be to just paint the big picture of your CI-CD pipeline and all the building blocks it is made out of. And these are the actual points of interest where a malicious code could be injected or where something could be manipulated.

Then in a second step, you need some creativity. So make sure you have some creative engineers and maybe also some reformed security part of that workshop where you then brainstorm for attack vectors. So you think about, okay, here's the code repository. Here is our binary repository. And then you brainstorm the hell out of it on, how could that potentially be attacked?

And this is where we saw engineers go very, very crazy even and think of things that not even security people might think about. So that's always very funny and very good to have come out with a good list of attacks that you could also then be afraid of. And then once you have this list of attack vectors, then the third step is to rate these attack vectors in terms of severity, right? So, okay, this doesn't sound too bad

because we already have this and this in place. But, oh my God, if this attack, if this type of attack would happen, this would be a serious problem. So you put any kind of rating in place and then it doesn't have to be super fancy, but once you kind of picked your and rated these attack vectors

and figured out which ones are the most severe ones, then you go into the final step and figure out what are the measures and security controls you can put in place to minimize the risk of these attack vectors. And you might come up with, okay, we need MFA here and we need this type of thing here

to make these risks completely go away or to make this attack vector not an attack vector anymore. And if you do this kind of four-step approach, I think you don't need the super security guru to figure out where the potential holes are in your CI-CD pipeline. It's the engineers themselves who can figure it out. And with that, I think everyone can make the CI-CD delivery pipeline more secure. Andy, do you know what that last bit sounds like to me?

You mean that I should say we need to get this into Captain? No. I was going to say it sounds like theoretical chaos engineering. Because it's not like they're actually carrying out the attacks, right? They're not actually going, but thinking it through, talking about it, throwing up the scenarios and all that, it's...

So, Michael, Andy and I have been doing several podcasts on chaos engineering lately, talking about him throwing up the scenarios and all that it's and some some michael andy and i've been doing several podcasts on chaos engineering lately and we kind of like the topic because it's sort of an extension of performance um but what you described there sounds like a more of a theoretical exercise is in chaos but from a security point of view like chaos security yeah yeah i'm like what if x happened like what you know in thinking out and figuring out what would be the the risks and giving those ratings shout out to chaos very good point brian and thank you so much because i may need to include this in my panel discussion with anna medina from gremlin that i have with her for before. So tying security into chaos engineering,

that's a really good point. And I was kind of curious if you would figure out a way to bring Captain up and you sort of did. Yeah. No, very cool. Yeah. Michi, it's a fantastic topic. And thank you so much for the blog post

that inspired this podcast. I'm pretty sure you have more cooking and that you will probably have many more blog posts coming. And not only you, you also directed me today and reminded me to talk with Barbara. And as I'm sure other folks

in the security team at Danash as well, we have to talk too. I have one last question for you. So where, if somebody gets started or thinks they need to do more on security, are there any people that they should be aware of, people they should follow, conference they should go? What is the best way to get started?

How did you, did you just come? I mean, I'm sure you also went to somebody or at least, you know, looked at what others are doing. Any references? It's really the term application security that you need to search for and, you know, look for on Twitter. It's really AppSec or DevSecOps. These are the key terms I would recommend to look out for.

And then really, you know, there are so many security conferences out there, but if you want to go into, and a lot of them are typically more around, you know, company and endpoint security type of things or network security, more this type of stuff. But you should really look out for application security related conferences and topics.

Hey, Brian, do you have any final thoughts before I try to give a very quick overview? Summary? No final thoughts that I can pull in except for the fact that it's overwhelming. Hats off to everyone in the security field because it's just getting harder and harder. And it just proves the fact that human beings are assholes there i curse not to andy it's really unfortunate that uh all this happens because it's like why can't people just not do this stuff it's because it can be explained it doesn't mean it should but you know money's the driving factor for everyone and it's it's quite unfortunate um but nothing i

can do individually except for hey my only thing would be personally get yourself secure passwords, people. Stop using that same password on everything. Actually, real quick question, Mike. I know it's not real. Michael, sorry. Or should I call you Mickey? I'm kidding. I'm kidding.

Any thoughts on password managers since you work in this whole security section? Password managers or trying to manage a whole bunch of individual ones? No, absolutely. Password manager is the only thing that I can recommend. And having a random generated password for every single website. And that can obviously then only be managed

with a password manager. So yes, I do use one as well. And I think that's the only choice we have right now to be secure on the password side. So Andy, are we going to summon the Summary for the first time in I don't know how long? Yeah, just quickly. I will keep it short.

So what I learned is, first of all, read that blog, How Dynatrace protects its software development and delivery lifecycle against supply chain attacks. We'll put the link into the podcast proceedings. Really great. But what I learned, because I always thought security is something like

what we should say, right? Sniffing passwords or like trying password attacks. But securing applications really starts on the developer machine. And from the first time on, before they commit code, how can we stop bad code changes from entering Git even, doing pre-commit checks on tokens, on any security vulnerabilities? Then you brought up some great examples on, obviously, static code analysis, analysis dynamic code analysis scanning for known vulnerabilities and then going further which again and just brightened up my mind is signing code and finally i know what this section in our dynatrace one agent and i'm pretty sure in the active gate installation is also all about so thank you for educating me on that. Other than that, I can just say thank you so much, Michi.

I know that you are not only the great scientist behind the UFO, but also one of the people that make Dynatrace secure. And I know it's a hard battle because it's a hard battle to win, but it's great that we have you and your team to make sure we do whatever we can. And improving our product as well, it helps our customers to do the same checks in their delivery pipeline.

Exactly. All right. Any last thoughts, Michael? Anything else where we're all set here? All right, we're getting a check now. All good. All right. So thank you, everybody, for listening today, as always. If you want to contact us, anything else or we're all set here all right we're getting all good all good all right so

thank you everybody for listening today as always um you want to contact us you can reach us at pure underscore dt on twitter or and send us an email at pure performance of diet trace.com and michael do you have any do you broadcast stuff share security thoughts do you have any social media you want to share? LinkedIn, Twitter, anything? Or do you keep that all private? Nothing worth mentioning. Okay.

Check out the blog. We'll have the link there. Check out security, everybody. Keep an eye on that. And if you attended Perform, thank you. I hope it was wonderful i guess that's it until our next episode and andy we're just counting down now we're getting really close to year five so thanks everyone again and thanks for joining us this week bye-bye Thank you.