PurePerformance - Why Cyber Defense is Hard: A Closer Look at the latest security research with Stefan Achleitner

Episode Date: May 22, 2023

Security comes with a price tag, such as additional wait time when going through checks at the airport or when inspection network packages at your firewall.To learn about current approaches to cyber defense and cyber deception we invited back Stefan Achleitner, Lead Researcher Cloud Native Security at Dynatrace. Tune in and learn why it is important to keep changing and using different passwords, why you should monitor all your servers, what zero day vulnerabilities are, the role of eBPF in security and why we have to minimize false positives alarms like the Hawaii Missile Alert! Some of the links we discussed during the podcast can be found here:Our previous episode: https://www.spreaker.com/user/pureperformance/don-t-look-away-from-the-next-cyber-secuHawaii false missile alert: https://en.wikipedia.org/wiki/2018_Hawaii_false_missile_alerteBPF on isitobservable: https://isitobservable.io/search?q=eBPFCheck if you've been compromised: https://haveibeenpwned.com/Stefan’s SolarWinds Article (German): https://intelligente-welt.de/so-funktionierte-der-angriff-auf-solarwinds/Stefan's Problem wiht Passwords Article (German): https://intelligente-welt.de/passwort-manager-und-andere-loesungen-fuers-passwort-chaos/

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. My name is Brian Wilson and as always I have with me my co-host Andy Grabner. Andy, how are you doing? I'm pretty good. I'm just curious if you had any more dreams recently. I'm glad you asked. I'm glad you asked. Yes. Did you have a dream? Yes, I had a dream that I was you.

No. How is that possible? Nightmare? I don't know, man. I mean, I woke up, I looked in the mirror, and I did one of those where you slap your face like in that movie, Home Alone. I was like, Oida! And I was like Oida and I was like oh I'm Andy Grabner and I'm so happy because the last episode was so

wonderful because Brian wasn't there I got to be the star and my family is so proud of me and this is the best thing ever and uh now oh no Oida O, it's I guess the only Austrian word I could really use. Oh no, it's, you know, now Brian's going to be back and he's going to ruin it with his dumb jokes. And everyone's going to have to listen to that stuff again and his dumb questions. I wish that guy was smarter. And then I woke up so glad that I wasn't you because, you know. It sounded like me with a Scottish accent. You know, I was almost, as I was practicing,

I was almost going to do a line where I'm like, oh, I'm Andy and now I'm turning half Scottish and half Irish and I don't know what I am anymore. Oh, yeah. Yeah. But anyway, yeah, I sometimes mix up my accents. But what are you going to do? I'm not a professional thing anyhow um and it was sort of a repeat dream well not really i guess there was no repeat in

there ah there's no repeat dream i was i was gonna try to segue andy but you know i'm terrible at segue so why don't you segue and and bring us to where everybody wants to go which is back to our guest and exactly exactly but i i think you made me smile at least and chuckle, and I'm pretty sure many of our listeners as well. And the dream is a repeating thing, right? And also we have a repeating guest. And I'm very happy to have Stefan Achleitner again with us. Stefan, we did an episode with you.

I had to look back just a couple of episodes ago. We talked about current threats in cybersecurity. So just folks, before I let Stefan speak, Stefan is the lead researcher in cloud native security at Dynatrace. And it's really great to have somebody like him on the team doing a lot of research in security, which is very important. And we have him back because we had so many new ideas that came out of the first podcast that we wanted to make sure uh we follow up stefan now we shut up hello thank you so much for being here again hopefully you didn't have any strange dreams yeah thanks it's great to be great to be back um i actually had a strange dream i was like uh thinking thinking

yesterday oh i'm i will talk with brian wilson tomorrow and i thought i'm going to a beach boys concert but i was just guessed at a podcast you know i i got one of my favorite jobs i have to say my second favorite job on the record since i'm here speaking for diana trace so my my second favorite job ever i got because of my name it was at a record store and literally the guy is like ah your name's brian wilson huh yeah it'll be funny you're working at a record store come on in so you know i i don't mind the brian wilson jokes so bring him on i'm a failed musician he's a successful musician and if I can be mistaken for him I'll take it I'm glad you thought of me

look at that instead of talking about failed musicians let's talk about people that work in cyber security and have not failed because they're actually inspiring people with a lot of things that are happening Stefan, first of all thank you so much not only for being back,

but you did an amazing job in preparation of this podcast to kind of walk through or kind of give us things you wanted to talk about. And I know we probably go out of order on the way you proposed it, because I want to start actually with the topic of performance, because this is dear to both Brian and my heart. I think Brian you brought it up the last time you said hey you know with all

these security checks that we need to now add into our delivery process whether it's already in the CICD where we need to add security checks, static scanning, dynamic scanning, all the way into production where we do dynamic scanning all the time. It's a big penalty on performance, most likely. And then the question is, what's the give and take? What's the best kind of balance between security and performance? And I was just wondering what your thoughts are. Brian, before I let Stefan speak,

anything else you wanted to add on this question? Yeah, well, just along those lines, for years now and years, performance improvements have been influenced by the Google effect. Everyone expects faster and faster, at least people who are aware. And as Andy was saying, I think a lot of people are becoming a lot more aware, not only of security, but just regular Internet users are becoming more aware and or maybe even scared about security being hacked, getting things stolen, man in the middle, whatever it might be, probably not even thinking about the specifics,

what can we expect or how should we change our viewpoint from the Google one, two second page load to what's acceptable if we're going to have peace of mind with security? We know there's stuff in the pipeline, but also just if you're doing live scanning if you're you know there's there's got to be at some point some trade-off are we starting to see any of that trade-off is it really starting to impact performance um and is this something that we think people will accept yeah that's a good point. When we think about security traditional antivirus system,

there's always things going on in the background. And so you have some kind of performance impact. For example, if you think about a traditional firewall, which is there to inspect network packets, you can kind of imagine it like a border control, right? You look at the packet and wave it through. You look at the next one, wave it through. Of course, this is all happening in like nanoseconds, basically. But still, there is like a little bit of a slowdown.

And what we do in research, for example, is to find the right balance here, to find the right balance of optimizing the performance, but giving you the best value for security. So short answer, there is an impact on performance, and we will, I think, never be able to make this go down to zero, but we are able to optimize that to a very high level. Stefano, when you talk about you doing some research, can you elaborate a little bit on that? Is this research research and i'm making maybe a completely stupid statement like uh with your border control example right you

could say hey uh well i think at the airport security right they i think there's like a mandatory uh i guess you know more strict check for every tenth person that goes to just to make sure that you know nothing is kind of you know people people cannot just get through is this something where the research goes into maybe let's say less strict control for a certain amount of packages and then more strict for let's say every 10th package is well what what type of research can we can envision here yeah definitely research is security research security research, especially when you talk about network security, there's always multiple levels what you check. So maybe have a first

initial pre-filtering where you look for certain things. This is very fast. When you get some certain triggers, you maybe send it to the next stage. And then you do some deeper checks. Everyone who entered countries like the US, for example, you know you can sometimes be called into a secondary inspection. So you really can draw an analogy here. And when you especially think about network security, the research that is going into a product like a firewall here is, for example, that you maybe only look at the first few packets of a session. A network session can consist out of hundreds, thousands of packets. But you maybe want to make a decision or you have to make a decision at just the first two or three packets to be able to decide, is this session good or is it bad? Do I want to

block it? So I really only want to look at the minimum amount of data. And this is also true for processes, for example. I maybe want to inspect if a new process comes up. I want to inspect it for the few first seconds, what it is doing and then i basically uh let it go because i decided that's a benign process rather than a malicious process so yeah there's a lot of optimization research going into that you know i was going to say it's it's interesting these ideas of the packet like you know check the first ones and let the rest in right uh that almost sounds like a military strategy i think it was emily right some uh emily something a long time ago she did the whole thing

with the the whole spartans and romans i don't know if you remember that episode but you always see in in like um either war movies or action movies they let the first few people in to make them think it's okay and then they attack attack the rest, right? Um, which, which almost makes me think the inverse is one thing security people are going to have to stay on top of the never ending cat and mouse game of security versus the villains is if there is an approach, like we'll check the first few, the first few packets are good. We'll let the rest through and stop doing the deep check.

Well, that then just sets you up for someone to say, okay, well we'll fake the first few packets of being good and make the next batch bad, right? Not that I'm trying to give away anything, but it just highlights the complexity of the situation. And if you were to check every packet all the time right now. Let me take that back. The interesting part is that

the decisions that have to go into security are how much do we impact performance or just all of it by saying, do we check everything now all the time, or do we wait until that becomes a vulnerability before we take that step? We're aware of it, at least we know to look out for it. And if people start figuring out a vector for that, then we'll switch over to to that so there's a lot of decisions that have to be weighed it sounds like from that performance point of view where you're going to have to take some calculated risks for the sake of everything operating knowing that it might change later so yeah very very complex anyway andy i just needed to get there before It is. That actually brings me to some good ideas or to some good thoughts. It is a never-ending story.

It is a never-ending cat and mouse game, as you said. And adversaries, attackers certainly leverage that knowledge because they know you have to optimize for security to work. So you can only look at the first few packets. You can only wait a certain amount of time to inspect something. For example, I did a very detailed analysis on the SolarWinds attack a few years ago. What the SolarWinds attack, for example, did to bypass a lot of security checks, it didn't do anything for two weeks. It basically was quiet, right? It was a supply chain attack. It was deployed with a benign

software and it didn't do anything for two weeks. So that means basically every kind of sandbox where you deployed the SolarWinds, the infected SolarWinds software and nothing detected it because it was basically quiet and didn't do anything for two weeks. And who is waiting for two weeks to check if a program is benign or malicious, right? Nobody does. That's why also this attack was so successful, for example. So basically playing the nice guy for a long, long time and then you show your real face. So the continuous dynamic checking

is what would be needed for that then, as I guess a lesson not to say, okay, it didn't come on in the first day. It's clear. It's like it didn't come on the first day, but we'll do a quick check-in every day or whatever period it is to make sure nothing has popped up, which I'm sure, I mean, I don't know if security tools are doing that or not, but it sounds like that would be the lesson learned is, all right,

now we know we're going there. We have to do periodic checks. Exactly. yeah. I guess this is a great example of also why security is really hard. It's not

a problem you can easily solve also by just throwing a lot of hardware on the problem, I guess, because these things, I mean, obviously you have to do the trade-offs, as we said earlier, the trade-off between how much effort do you want to put in.

But security is not an easy-to-solve problem. And can you give us a little bit of more insights on why security is so hard, some additional vectors maybe? Because remember, we,, Brian and I, and I would say including most likely some of our listeners are new to that field and we would like to learn

and also have an argument like, why is security so hard? Yeah, you really need to have an attacker's state of mind when you think about security. Think about you want to break into a house, right? You want to just break in, steal something, whatever. You will probably go for the easiest way to get into the house. Maybe there's a window open somewhere.

Maybe you find the weakest store, whatever, right? You can really think about the same way in cyberspace. Attackers will find the weakest link in the chain, basically the weakest entry door into an application, into a system, into a network. This is from the perspective of the attacker. As a defender, you basically need to be prepared for everything. You need to make sure that every weakness,

every backdoor, every entry door for an attacker is closed. So that means you have to monitor for all kinds of vulnerabilities from the system side, from the network side. There are many attack vectors. And this is why defending or cyber defense is so hard and why security is so hard. Because you have to consider so many different aspects when it comes to security. I would like to give an example with passwords that everybody is familiar with, right? Everybody uses passwords.

And you always hear you should never use the same password again, right? You should never use it twice. And a lot of people might ask, well, but then I might have to remember 100 different passwords or whatever. It's impossible to do that if you don't use any tools like a password manager. But why is this so important? Let's assume you create an account on some not very reliable web shop where you order something once and then you forget about it and never use that again.

If this not very secure web shop where you order something once and then you forget about it and never use it again. If this not very secure web shop gets hacked and someone will steal your credentials, if you use the same thing everywhere in your Google account, LinkedIn account, of the biggest, really important accounts, someone could just reuse your password and your login credentials and then basically impersonate you, right? And get into your accounts, get into your systems. That's why you should not reuse a password. That's why you basically have to make sure that everywhere, every account that you have is safe against that and you should use a different password.

Or you can also think about, I have a server farm, right? Where I have 200 different servers with the same configuration. I could just monitor one of those servers because they're the same anyway. And that will give me insights if I have some weaknesses, some vulnerabilities. But then maybe every server serves a different application, serves some different customers. So you have different contexts of every server. And if you only check one, you might miss that someone already broke into another server. Or you can have zero-day vulnerabilities, right, in one server. Zero-day vulnerabilities is a weakness, a vulnerability that is not public yet, that is not known yet, that only a few attackers know. And if someone would exploit that

and you only learn about three days after that, but you never monitor a certain server or you never check a certain server for security or you only make random checks, not check everything, you could easily miss that. And so there is, again, your entry door for an attacker into a system. So that's why really security is so hard. Also what we discussed before

because you have so many different things, so many different aspects to consider when it comes to make a system really safe and an application really safe. I wanted to say as you were talking about the password, let's talk the podcast now, I have to change my passwords. But no, obviously I'm trying to not use password 1234 and not all over. But I really like the other example. I mean, both examples are great, but the why monitor all servers, Brian,

this reminds me a lot about the early days of observability when people were saying, hey, I have a server farm and we run the same code there anyway. Why do I need to install a monitoring tool on all of them? I just installed the one because I will find my performance hotspots. And guess what? It's not the case because if you have one user that is executing a transaction with a different i don't know let's say filter for a report and then then hits your app on a server that you're not monitoring and that goes crazy and blows up because of a of a problem then hey

you are you're not catching this right and even even if the same software runs, the transactions itself are different. Or even if your load balancer isn't balancing. Exactly. The same thing goes with security, right? And the other vector, you talked about zero day. If someone's hitting that one and you don't know about it yet,

but there's a lot of security software out there that'll be looking for people trying to attack a system. It might not be known vulnerabilities, but there are certain patterns to when people are probing. And if they happen to be probing one of the multiple servers that doesn't have it on there, you're not going to see it. We recently had a question about, can we do a similar thing with security and you know after thinking about it for a little bit for all the reasons you said there and more it's just like no you know first of all like everything you said there it makes absolute sense why you can't do that right or you shouldn't do that if you actually care about security

but then the other side of me was like people have to really just stop thinking about cutting corners with security. I'm sick and tired of getting emails or letters from companies that say, oh, your password has been breached. We'll sign you up for a year of password or identity monitoring with Experian or whoever it is. It's like, no, do this for real. And I forget who we were talking to before. We had another security conversation, Andy, where we're like, until there's a punishment for this, people aren't going to take it seriously. Until there's like severe penalties for these breaches, how are people going to, you know, take this stuff serious enough? The other interesting thing you said too is thinking like the criminal, right? And that reminds me of, of, of two things that people might, hopefully people can't

relate to thinking like a criminal. Hopefully all of our listeners are, you know, above the board. Um, but for anybody who has children, right? And even if you don't have children, you've seen the baby proofing stuff in target or wherever you are, right? And you see the things with the outlets, the, the door things. But when you have a brand new child, you have to, if you're standing up tall and looking at the house, what can they get into? You're going to miss a lot.

You literally have to sit on the floor and look around. Step one is just say, what can they get into? Because if you're standing up looking, you're going to miss a bunch of it. And it's that same idea of getting into the mindset. I even have a more complicated situation where I have a special needs daughter who will get into everything. And I can walk into a room and be like this, this, this, this, this, and this, and this. And most people are like, what are you talking about? I'm like, no, I know how her mind works.

She's going to get into that. But that's because we were forced into that situation. Most people aren't forced into the, how is a criminal going to think? That, I think, is the hardest part, because they're going to think in the most crazy, crazy, crazy ways, where you might look at security and say it's all buttoned up, and someone can come in and be like, oh, there's this huge gaping hole here that you didn't even think I could exploit, right? So, I know I'm ranting here, but as opposed to software development and performance, right, you're dealing with machines, you're dealing with code, you're dealing with physical limitations and capabilities of electronics and silicon. These are all finite types of things.

When you're going against security, you're going against wily human beings with bad intentions. So there's that additional factor that's not rational. That's not something you can say, I can measure this. And I think that's what makes security so tough. If you're computer minded, right? Yeah, exactly. It's all about, as you say, being creative, finding a creative way to break things. And this is really the the attackers mindset or yeah you you gave a very good example with little children or pets right yeah pets yeah hey then again maybe a stupid idea and but still i wanted to get your opinion what if we just turn the whole thing around and instead of really putting a lot of effort in making sure we only let the people through or we only block the people that really

do something malicious why not just uh create more alerts than necessary i mean maybe just like you know i don't know sometimes also you know, allow some false positives. What about this? You know, and so instead of really putting a lot of effort in, really only letting the good people, the good requests through, maybe sometimes just saying, hey, I'm not really sure, but from a probability perspective, it's time for another attack. And you don't look suspicious, but I still block you. Would that also be an option to address the whole thing?

Yeah, this is one of the big stories with false positives and insecurity to keep them low. And I would like to tell a very personal story here why false positives are so bad. I would like to take you back about five years, beginning of 2018. I was still living in the US back then. I was just finishing school and moving from Pennsylvania to California. And my wife at the time was moving to Hawaii because she had a two-year job there. And around that time, there was the North Korea crisis. It was all over the media. There were new rockets and everything.

And so one morning in January 2018, about 1.5 million people in Hawaii received the text message that there is an incoming missile. This is not a drill, seek shelter. My wife was one of those people. She called me immediately and said, hey, we got this text message. What should I do? I was like, okay, wow, like, what are you doing in this situation? I was asking her, can you go into a basement? Can you hide somewhere? And you're thinking, are these the last few minutes that I'm talking to my wife is World War III about to start.

About 20 minutes later, it was all over the news that this was a false positive. So there are actually push messages in the U.S. on your phone when something is happening. And so people were really panicking in Hawaii. There's also, if you Google that, you will find it. There's also a cool video on YouTube from the Jimmy Fallon show where Jim Carrey was actually also in Hawaii at the time and experienced the same. And he's also telling in a few minutes how he experienced that. And so this is a really, really prominent example how bad a false positive can end up, a false alarm, right?

And when it comes to security, to cybersecurity, if you think about a false positive, a false positive means that you block something that's actually good. If you think about in computer systems, if you block the email, for example, of a whole company by mistake, this is very bad.

But there have been incidents where network traffic, for example, was blocked. The processes of computer systems were blocked that brought a whole airport down that planes couldn't start or land or brought a whole hospital down where surgeries had to be postponed. These are really all true examples that happen through software updates

of security services that cause false positives. And so this is why it's so important and so bad when a false positive happens in security. So this is something we also try in security research to really minimize the amount of false positives that we have and be as accurate as possible, which is not always easy. And especially if you think about using AI, right? Chat GPT and things are all over the media now.

And I have been using it for a while now, just also playing around. And it really still makes a lot of mistakes, especially it gives a lot of false statements back. It is very useful for certain things, but it also makes still a lot of mistakes. And so that's why it will still take a while that we can fully rely on AI in security. At the moment, it's still mostly hybrid systems, more traditional approaches combined with AI. But so, yeah, those are some examples why it's really very important in cybersecurity to be accurate and to not make false positives. For me, I kind of assumed it was a stupid thing to suggest.

Yeah, but no, I really like that this turned into this explanation because especially your real examples of the missile attack and also hospitals, I think these are always things that stick with people, telling these stories with personal impacts.

And I remember when it was last year, I performed at our Dynatrace conference when I had Kelsey Hightower on stage. That was last, not this year, but last year. And he was telling the story when he was working for the organization that was responded, he helped them, an organization that was responded he helped them uh organization that was uh processing

credit card transactions and not only that but also uh food stamps not foods like digital food stamps right and then when their system got down that means you have real people that try to get food and then they stand in line and they swipe their food stamp card or food program card and then it gets declined but not because they don't have enough money on there anymore but because of a system failure. And I think this is a

very personal story that really stuck with a lot of people and I think that's why also Stefan, thanks for sharing your personal story. Hopefully it will

never happen again but it's a good story to tell and it will definitely bring awareness to the impact of false positives. I got three more on that, just to add, because I think they're interesting ones. So I had one similar to Stefan's.

Last week at Sales Kickoff, we got an alert on our phone from my daughter's school that they were on lockdown. And I was like, I ran out of the meeting to get in contact. I was like, oh my God, this is any get in contact I was like oh my god this is you know any parents worst nightmare especially in Colorado it happens enough out here you know and some there was there was a problem with the button that triggered the lockdowns so it was a total false one they didn't even like oh yeah it was just it was false like they try to play it off I'm like all those kids just went through this trauma of thinking they're on a lockdown all these parents just went through this trauma of thinking they're on a lockdown. All these parents just went through this, like similar to the whole missile thing.

Very, very devastating. But two lighter ones, right? You know, a lot of the other ones you were talking about were very severe, like the banking, the emails getting cut out. One just happened to me recently, which was a lot more minor, but is probably a lot more common to be business impactful on like a regular non-critical basis. I was going through trying to buy some new musical equipment and i've got a browser plugin that'll go through different

codes it finds for discounts right so it's going to try a bunch of different codes and then give you the best one well i guess the repeated reloading of that page with different codes triggered the security system to block me as if i was trying to hack so at that point i had basically i was like well i guess i'm not going to buy this. Or maybe I'll just, instead of waiting two days, because you know, I don't have the patience, I'll go buy it from the competitor without the 10% discount, you know? So they almost lost my business until I was like, oh, well, let me pull up my phone, go in with a new IP and finish the purchase. So just something as simple as, you know, I wasn't doing a hack. I wasn't attacking attacking. It was just the behavior they mistook as that, potentially.

Or maybe they want to block people trying to do codes. There's that one. But then when you said the thing about the food stamps, right? I just heard a thing the other day about, and this is a little bit beyond, you know, computer security, though. It does all tie back together. You know, on the credit cards, they have those chips now, right? And that's because those magnetic strips are very, very unsecure. There's no security on those magnetic strips at all.

At least in the United States, now for food stamps, it's called the SNAP program, they give you a credit card, which you can use for your food. But it's a non-chip credit card because they don't want to invest the money on that. So all the credit card skimmers can now go through, get the Snap People's card numbers, and they know at 10 o'clock in the morning on the first of every month, those cards get reloaded. So they have the numbers and exactly at 10, all the scammers just go through and run all those numbers and grab all the money because of the lack of security in that case, right?

They're allowing too much to go on. So, you know, similar things. On the flip side, there's such a complicated, I mean, again, I know we could talk for hours of this stuff, but there's so many different layers and some of it is just pure laziness and that's the stuff that pisses me off, right? When it's just the laziness.

It's like, we don't want to take the effort and or we don't care about the people impacted right yeah i mean in that case right if i guess somebody said hey if we can save 50 cents on every card multiplied by how many million cards we're issuing that's a certain amount and maybe but then allow scammers to steal all the money yeah i know but yeah because because not because nobody nobody thought this through. It's the same thing with, it's the same thing with what you said earlier. If you never get down on the floor

and see what the perspective is from the baby, you will never see what kind of holes you have. Stefan, I've been coming back to this previous story that Brian said with his attempts to try different codes, I assume attackers probably try different ways to get into a system, right? And then maybe it takes them a little while. I don't know. There's different stages of an attack, for lack of a better term.

How do you do this, actually? How do you trace? At which point do you find out this is actually an attack and when do you actually say, no, it's not an attack? Is there also some research that you can share with us? Yeah, very good point. It also fits in very well with what you said before, why we just not report every alarm.

And you are totally right. Attackers have to try. And most of the attacking attempts that you find on a security dashboard, on a firewall, whatever, are actually attempts from just usually automated tools that are trying. Is there vulnerability? Is there some backdoor? Is there something that I can hack into the system? usually automated tools that are trying is there vulnerability is there some backdoor is there something that i can hack into the system and mostly it's just it's just alarms that don't

do everything because your system is not is not vulnerable to but you basically need to find out what's the what's the what's the entry point in an attacker find and when can an attacker make the second step, right? Not only like the first initial, we are talking about reconnaissance or we call it reconnaissance when you try and determine if a system is vulnerable. We are interested to find the second step. interesting technology that is possible here is the concept of cyber deception, which means you are on purpose building in certain, let's call it traps. I'm sure everybody or most people have heard about honeypots. This is like one of the most basic forms of cyber deception, where you build in, where you have a vulnerable system that lures attackers to find a way into your system.

And you can do this on a much finer granular level. This is actually really something that we are looking at in our research. And this is so important, this concept, because it can actually give you a lot of threat intelligence, as we call it. So you really learn how attackers think and how they work and what they are interested in. For example, if you find that an attacker only looks for certain APIs, certain interfaces, you can learn and they are maybe getting into your traps. So they are accessing one of your traps and then maybe make the

second step. You can then say, okay, they are only interested in certain APIs on my system. So I want to give better security to my actual APIs that are there because I learned from observing those traps and

learning that attackers really go for them. I can then apply this on my actual resources that I want to protect. And there are a lot of concepts how deception can be done, what's an effective way. So we are, as I mentioned, we are doing active research on that. And this is a really very promising field to also help with this alarm fatigue, basically having too many meaningless alarms. So to really find out triggers that give you very high quality insights into what a tech is really trying to do on your system. So this can be very helpful for the defending side. I do hope, though.

I mean, I know you're doing research, and I assume a lot of the stuff that you're working on is getting published. But then if you publish how you are kind of doing cyber deception and it's been read then by the criminals, you need to maybe deceit the cyber criminals by putting out the wrong research results. So to put them onto the wrong track, I don't know. deceit the cybercriminals by putting out wrong research results. So to put them onto the wrong track, I don't know.

We need to deceive them, even with the publication. That's a good point, but also one very important thing that we have in security is no security by obscurity, meaning security that is just secure because nobody knows about how it works is

usually not really very safe and very good security. We really design our systems and design our solutions that they even work if attackers or if the public knows how they work. So this is really also one of the core concepts that we are following. And if you think about the open source model, Andy, right? You know, if everything is out there in open source, people can easily find whatever they want to attack it, which then, you know, inspires the community to be even that more diligent with it, right?

Because there's always been, especially when you start looking at government sector, can we use open source, can we not? and then is it more secure than you know private commercial software because everything's out in the open and you can directly see if there are anything there so i do agree with steph on there that like the more open you are about it exactly what you found you're telling the hackers you can't go this way and now they're going to have to spend how many more months to find a new way right which does buy you some time to try to think like them because it's not like tomorrow they'll come up with a new vector right they're going to have to spend just

as much time and effort to to find a new way in then or at least say great now we know that uh you know acme company who's using this new way is impenetrable because they did the latest stuff so we'll ignore them because they're good from that point of view where they're following the rules they're staying on top so we'll go after the people who are being sloppier so that you get to

that's the reward for staying on top is you buy yourself more time before the next attack but that's also you always have to assume you are going to get attacked, right? Or that you'll probably get breached at some point,

right? Exactly, yeah. Is there a concept of resilience in security? You know, in performance, we have resilience baked in, right?

We have redundancy, we have all these different switches and all. Is there any equivalent concept of resilience in security? Yes, there certainly is because cyberattacks typically work in stages.

You do reconnaissance and you find a vulnerability, you exploit that maybe first on the network, then on the system level. So there are really multiple levels how cyber attack works. This is also why this concept of multistage attack detection is very important that actually plays in from the cyber deception technology that I or approach that I mentioned before. And so when you have security, you typically have multiple layers of security, you have it on the network, you have it on your application, you have it on your operating system. And so it's, and this brings me back to false positives, it's sometimes better to not make a false positive, but let something through because another layer will catch it. And some types of attacks are maybe much easier to detect on operating system level than they are

on the network level, for example. And so yes, you always have multiple layers of security if you secure your system well and if you are not uh lazy to leave like certain layers out right so there's there's certainly just this concept of resilience yeah so it's not like in the hacker movies where they just get into one layer they can get into everything then right right yeah it's it's uh it's really funny some of the hacker movies, like they press three buttons and then they're in. It's not that easy. This also reminds me of a thing that time is an interesting factor as well, and especially now where we have very dynamic systems where you can spin up and down pots as you like maybe that's also a way of

defending where you say hey you know my pots are very short-lived and you're constantly recycling them not because you recycle them because they're failing from a performance perspective then it's kill them up but maybe you just recycle your pots all the time so that in case an attacker makes it into a pot it doesn't help him a whole lot, him or her, because the pod will be recycled anyway after 30 seconds and the new one will be spun up. And the same can be true for nodes in a Kubernetes cluster. So I think if you have an environment where you constantly recycle nodes and pods and everything,

then even if you make it into a pod, you have to be very fast to get from there to the next thing because basically the system around you is constantly being torn up, you know, built up and torn down again. Is that also a way of cyber defense? That could certainly be a way, yeah, that's definitely a concept that's followed somewhere, but it also always depends how deep an attacker is already in your system. If someone is in a pod in a Kubernetes cluster, they could certainly have the permissions to spin up other pods. So they could

basically spin up multiple pods where they're also in. And they could really leverage if there is, for example, a lack of security policies in a Kubernetes cluster. They could really leverage that and find out, hey, how much communication can I actually do in my Kubernetes clusters? What can I, in fact, right? Is it very easy to spread through all the pods? Because it is very easy, then this also doesn't help. And this is also something that we are researching, are researching, actively researching to find out what's a good balance here

to having a good security policy in a Kubernetes cluster while guaranteeing that everything really functions how it should function. And this is also not easy. This is not trivial because if you've ever tried making an optimal security configuration in the Kubernetes cluster, you will quickly find out that it's really not easy to find the optimal configuration. And so to support that with automated tools, for example, this is something that we are also looking into. So, yeah, it always depends how deep or how smart the attackers are.

And by the way, folks, if you listen to this and if you're interested in security and Kubernetes, we had an episode with Nico Meisenthal recently on how to hack into a cluster. I thought that was also really good based on one of his talks he's been doing at different conferences Stefan I have one more question for you and I know we are getting kind of close towards the end but I want to before I ask this question I wanted to make sure that you also got all of your stuff covered again

thank you for all of your preparation that you did to this talk I know it was very smooth as we were going from topic to this talk and i know it's it it was very smooth as we were going from topic to topic but really you made it very easy for us because of all of your prep work anything we haven't covered i think we have most of the things covered i'm just yeah looking through my notes wrong answer actually you should have said and there's so many more things but i think we need to have a third episode where we cover it so that would be the right way this is just for today this is just

in this in this little world in this in this little context but of course in the overall concept of of cyber security there's so much to talk it will be very fun for example to talk about some human factors at some point so yeah many more episodes to come, hopefully. Yeah, I hope so, too, because we need to educate people on security. And I think I still need to change all these passwords now. But last thing, because we touched Kubernetes, I was just at KubeCon in Amsterdam and kind of browsing through the halls of the expo area, I saw a lot of tools and a lot of commercial and also open source using eBPF, kind of like it's the big

slogan based on eBPF and also for security, not only for observability. I don't want to go too deep into this, but what are you seeing out there? Is eBPF something that is used for security? Is it what are the just maybe the high level view of what is possible with eBPF and maybe also where the limitations? And again, this could be another full episode to talk about eBPF in the context of security. Yeah, eBPF is definitely a big, will become big in security. I'm convinced of that.

And it is already used in certain security tools. Just very quick, ePPF is basically a way that you attach a custom program to the Linux kernel. And then it's basically triggered when certain events, like, for example, a system call is happening so you can observe how many tcp retransmissions a process is doing or how many files a process is opening so you can really collect very relevant security metrics very relevant security data and on also many different levels in a system. What is so important, as we already talked about, in security? So I'm very convinced that eBPF will be a big player in the security space. We're also looking in our research at some use cases for eBPF. And yeah, so I'm really not surprised that you saw a lot of tools at KubeCon

that already leverage EPPF. Yeah. So let's pick up this topic at a later time. And folks, if you're interested in this, I believe Henrik has already done some EPPF talks on his Is It Observable channel from an observability perspective. And he's also interviewed some of the two vendors actually on Is It Observable?

So isitobservable.io with Henrik Rexit, great content to learn more about what's happening in the observability space, especially as it comes to cloud native. Ah, man, I took a lot of notes. I really liked the beginning when you talked about the... I think it's always great to have analogies. The border control was a great one. I really want to learn...

If you have anything, Stefan, you said you did some initial research back then when SolarWinds came out, like the attack. If you have anything we can link to, please let us know. Please, folks, change your password. Use a password manager. Use a password manager.

Monitor all of your servers. Minimize false positives. I think that was a story that really kind of was stuck in my brain and I guess maybe I will dream about it

tonight as well and I will let you know the next time. And cyber deception, there was also an interesting concept. I liked it. Yeah, that would also be

an interesting topic to go into more detail. Yeah. Brian, what did you learn today? Well, really quickly on the password thing, I was using one very well-known

password manager and they got hacked. And let me tell you, switching password managers is real brutal.

There's no import, which is a good thing, I guess. But I think what I learned today,

especially since you ended the topic on eBPF, I remember several years ago we talked about EBPF, and I forget there was another tool for performance monitoring. And then we've talked about EBPF slightly in context of security before as well. conversation, I kept on thinking about the similarities between performance engineering, performance testing, and security, which I'm sure you can make the case for a lot of different practices. But since we have this background, it resonates there. And just talking about EBPF in context that both is making me double down on this final thought, for me at least. Whereas if you're doing both in performance and in security um the more you think about it the more you're aware of different vectors of different variables the more overwhelming it can become right yeah we'll set up a load test where we'll you know have 100 users come on well what about the ramp up are they coming in and out right

there's so many different variables and then do we have have cash, non-cash, right? Same thing as you start thinking more about security. Are they waiting two weeks, right? Are these different things, you know, are we catching it at the network layer, this layer? How do we test for these things? It's this never ending. The more you think about it, the deeper it goes. Thankless, really, really tough mental game to play. And it just you know i have to take my hat off to everyone working in security because also like performance if you're doing a good job in security nobody notices you don't get the accolades for good performance you don't get accolades for not getting hacked you only get you know the notice if it's not doing so So it's a very, in a lot of ways, a thankless job.

And I would love to see things turn around where people get accolades for how many, the company with the longest amount of months without getting hacked. Let's reward them, right? Let's reward the company with great performance. And on that vector too, a big thank you to everyone, at least at Dynatrace, for all their performance efforts

and all the security efforts, because they're all doing a great job. So we'll start the accolades there at home here. But yeah, there's a lot of similarities between it, and it's just never-ending hard. So thank you, Stefan, for helping us get our listeners to think about it, and thank you for the work you and your team and everyone else in security is doing yeah we really love what we are doing so it is rewarding but you're right like in many IT fields if everything works or everybody's everybody is expecting that everything works and they don't really see you or what's going on behind the screens, right?

All right. With this, I think we hit it with the timing that we set in mind because we have a hard stop here. Yep. And Stefan, thanks. It was great again. Lots of fun. We'll have you back. And then not with a Scottish accent

but with an Austrian accent maybe I'll have to come up with a completely different accent for you you know I grew up in New Jersey so I can do an Italian accent very well maybe I'll make you Italian yeah alright

thank you everyone see you next time hope you enjoyed the episode bye bye bye