PurePerformance - How not to get Kubernetes cluster hijacked with Nico Meisenzahl

Episode Date: January 30, 2023

Do you know that 53% of security related issues on Kubernetes are caused by misconfiguration? Me neither!To raise the awareness of how to protect your Kubernetes cluster and workloads from being hijacked we invited Nico Meisenzahl, Microsoft MVP and GitLab Hero, to walk us through a set of best practices that everyone in cloud native should know to contribute to a more secure cloud native environment. In our conversation we cover a lot of what Nico has shown in his recent talks at different container, cloud native and security related conferences.Make sure you check out the slides, github tutorials and recordings from Nico through those links:Nico’s Website: https://meisenzahl.org/Hijack a Kubernetes Cluster YouTube: https://www.youtube.com/watch?v=9wc34MozKokHijack a Kubernetes Cluster Slides: https://www.slideshare.net/nmeisenzahl/containerconf-2022-hijack-kubernetesHijack a Kubernetes Cluster GitHub Tutorial: https://github.com/nmeisenzahl/hijack-kubernetesConnect with him on LinkedIn: https://www.linkedin.com/in/nicomeisenzahl/Follow him on Twitter: https://twitter.com/nmeisenzahl If you want to hear more from Nico listen until the end and pick from one of the suggested topics

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance she died of a heroin overdose or maybe not a heroine i don't know how she died but famous very very famous person performed with lou reed uh performed with the velvet underground performed with you know at the factory andy warhol i'm just really amazed we've got this guest today i'm not going to go ahead and introduce we can still do some banter but i just got to get this out of the way because i'm just really really excited and you know i'm i'm really hoping to see how many people i stump with my my not too obscure if you're into music reference but uh you know very very pivotal and uh groundbreaking band she joined for an album um and it just amazes me yeah me too because i'm talking about no no no but i'm pretty sure i will learn something

once i figure out what you're talking about does our guest know any i don't know let's let's ask her or him or i don't know the reference doesn't make sense but actually let's let's bring the guest in nico meisenthal hi welcome to the show servus i would say servus to you if you would keep this in german we don't keep it in german because then we would confuse brian as he just confused me so nico welcome to the show and uh introduce yourself and let us know what your connection is with ua is 60 years like music from the 60s hey everyone to be honest i have no idea i figured as much yeah so yeah idea yeah Nico performed with a band called the velvet underground you might have heard Lou Reed no take a walk on the wild side was a very groundbreaking psychedelic garage rock band they were the influence Pink Floyd early before

that's started it was uh you know came out of andy warhol's system and junk and you know but the name reference of this artist nico nico yeah she uh well that's not her given name but that was her performer name and she joined them he basically warhol was really into nico and he said hey you're gonna do an album with the velvet underground he since he was their manager he it happen. It's one of their most famous albums with the Andy Warhol banana on the cover. So anyway, when I first saw the name,

I was thinking, you know, maybe, maybe somehow. Yeah. Nico's. We have a, we have a much,

you know, I think considering that the performer Nico is dead, I think the modern performer of Nico, and i'm not gonna even try your last name though probably gonna be a lot better conversationalist because i don't think nico would have much to say the other one right anyway this is the dumbest intro ever i think i think i get the award for the dumbest almost like and i think let's let's make sure we shut up and give Nico the chance not to introduce himself properly. Nico, who are you? I'm the other Nico. So Nico Meissenzahl.

I'm doing DevOps related topics in the cloud native, Kubernetes, Azure ecosystem. We are basically helping our clients and customers to get started in the cloud or to get faster, better, more secure. And if I look at your LinkedIn profile, so maybe people could have guessed from your name and your accent, you live in Germany, southern Germany in Bavaria.

And based on your LinkedIn profile, which we also link to in the description of the podcast, doing cloud-native Kubernetes and Azure at White Duck, you're doing a lot of talks as well. And I think this is also what kind of got the conversation started. Because the cloud-native space and the DevOps space is a huge space. And it obviously covers everything from dev to ops. We talk a lot about site reliability engineering. We talk a lot about chaos engineering. and it seems, you know, it obviously covers everything from dev to ops with, you know,

we talk a lot about site reliability engineering. We talk a lot about chaos engineering. All these topics fall into. The topic that interested me for today's talk is more focused around security, especially in the cloud native and the Kubernetes environment. And you had, I think you probably had different conferences where you talked about this, but it was called hijacking a Kubernetes cluster. And you had, I think you probably had different conferences where you talked about this, but it was called Hijacking a Kubernetes Cluster.

I walked through the latest presentation I have opened and will also link to your slideshare is from the ContainerConf 2022. And for me, the title itself, Hijacking a Kubernetes Cluster is cool. And I saw your YouTube recording and you actually went through and you went step by step into going into the cluster. For starters, Nico, have you seen kind of more security awareness now in the Kubernetes, in the cloud native space, especially since last year when we had lock, that's actually one and a half years ago almost, now 14 months. Log4J is on everybody's mind still kind of.

But have you seen any improvement in security awareness in our industry? Yeah, it's hard to say. So I would say in our cloud-native bubble, yes. So basically if you're in the bubble, I think more people are talking about security, DevSecOps and stuff like this. But in the real world, many, many customers, clients, companies are not really thinking about that. And this is also why I created this talk,

I think, one or two years ago, because I went to clients doing workshops and some new environments, so basically Kubernetes cluster, which I created some years ago, deployed something in there, no security in place at all, no policies, nothing. And then I thought, okay, maybe I should build somehow a talk showing how easy it can be. And to be honest, it's just some basic thing, though I'm not a security person or so, just doing also a little bit of security.

So it's nothing fancy, just basic stuff. And it can be pretty easy to get in the cluster. And yeah, mainly did the talk to raise some awareness because somehow it looks like it's still needed.

It's still needed, yeah. And I think this reminds me a little bit in Brian, we had so many sessions about our most favorite performance problem pattern, the M plus one query problem. And I think it often starts

with somebody's writing a new, a small app, right? And it becomes like a little tool and it works. And then all of a sudden this tool becomes popular and more people are using it. The data behind the scenes

becomes bigger. And all of a sudden the M plus one query problem really becomes a problem wasn't a problem before and it sounds like a little similar what you're saying somebody years ago started playing around with kubernetes and may and probably not with not somebody with operation experience uh security in mind right because you're just playing around with it and all of a sudden this kubernetes cluster from back then is now running in production. And it's not just running the app we built years ago, but now it runs critical business functions. And is that also what you see? Could this be related? Yeah, it's one part. On the other side,

it's just, I don't know, developer team or developer has a need. I need a Kubernetes cluster. Okay, let's go to a cloud provider and click two steps and you have a cluster and I don't know, it's called manage something. But it's somehow a little bit managed, but it's not fully managed, not fully secure. And you have to invest and you have to also know the ecosystem, of course, to know what to do. Yeah. So I think there are many many different scenarios um from not having operations starting small building up a demo environment and then it goes production as always um yeah different different pieces i could see that scenario quite easily you know yeah we all spin them up an experiment and then suddenly it catches on and yeah away you go and do you actually hit

the pause button say hey let's get some an expert in here to figure out how to set this up right before we continue probably not yeah and also what i think with the with with the sales offerings right as you said it's two clicks and you get a cluster and sometimes you forget even that you have clusters running around i mean typically you should not forget because at the end of the month you get a bill but if the bill is hard to figure out for what you actually pay you may still forget it and then months later all of a sudden somebody breaks into your kubernetes cluster that you've forgotten about and never patched and never updated but what i thought interesting and i want to point people to slide number four in case you're downloading the slides from his talk or from your talk you brought up based on your experience i guess as well

what you see out there i mean it's like it's a red redhead study or state of kubernetes security report but i guess you said earlier when you come to customers in the real world you see these things as well that study highlights uh the top security problems right why did they actually why do they actually exist and it starts with 53 are misconfiguration yeah that's great which is pretty easy in the kubernetes world so it's so complex you need to know the whole the configurations if you start from scratch basically starting with the kubernetes project many security settings are turned off by default. Also on the cloud providers, it's not that hard, but still you somehow get an easy access. Most or many security features are not turned off, turned on by default. And yeah, on the other end, you maybe misconfigure something, test something, forget to reconfigure it again because you don't

have infrastructure as code or something in place. looks like this isn't is a thing yeah yeah do you have for folks that are kind of now thinking oh man i have this kubernetes cluster that i started six months ago um do you have like like the the one or two things that people should look at in whether they have this particular feature turned on or whether it is it is whether they have security like enough security in the kubernetes cluster it it really really depends so of course you have the kubernetes level from my job the job and my daily business i would always recommend to get a managed offering it's not fully managed but it at least helps you and then it's it's really a layer of multiple things it's it's your application it's's the increase, it's the cluster itself, the container image you're using. So we have multiple stages.

And I would say if you implement easy best practices, which are easy to find in the internet, and at least pick some of them for every layer, you are at least better than nothing. So this is not rocket science. It's really about first step enable all those best practices, or at least most of them to get started. So it's really all the thing which I find really important to really start with those best practices and get some of quick wins.

Because otherwise you may be talking about two years about implementing security doing security project um after two years you have a big list of things you would like to do but still you didn't implement anything um so we start small start with the best practices and then iterate yes we all do in our daily efforts all other things as well yeah or should you yeah and talking about best practices uh you again referring to your slides to the talk you have listed some best practices uh in the slide to get the at the end um linking to you know kubernetes documentation like as you said like implementing a waf um these are things that you know as you just said it depends on the layer you have there's multiple layers in in a Kubernetes stack where you can tackle security, but that's great.

For example, starting with your applications, make sure that your application code is secure. Code scanning, stuff like this, it's not a Kubernetes-related issue, but it's basically the first step. Then making sure your container images are secure, you don't have any vulnerabilities, logs for shell, logs for text, and so on. I don't know, implementing Kubernetes policies, for example. So most of the time, Kubernetes policies are seen to implement guardrails in a cluster for the Kubernetes users, which is right. But on the other hand, it also works if somebody gets into your cluster, he also or she also had guardrails to say, okay, I cannot spin up a container with root access or with hostpass mounted or something else to make it

harder to get further within the cluster. Same with network policies. So many clusters have no network policies in place. Nobody in an on-premise world with a team set or something would say, hey, okay, shut down the firewalls, just starting and sending traffic around. Nobody would do it.

Why are we doing it in the Kubernetes cluster? Yeah. So earlier, and maybe I misunderstood what you were condoning before, earlier you had said something about, you know, start small,

don't make it a hugely involved process because then you'll never get anywhere. And I'm not saying I need you to commit to an answer and we're going to hold you to it. So don't take it that way. But, you know, my initial reaction would be to, oh, well, maybe that's a great time to go talk to your security person within the organization. It almost sounds like that could be a roadblock then to doing the innovation. And some of these steps that you mentioned here are some of those low-hanging fruit that you can take to just get ahead of that, get things started. And then as it starts taking off, then maybe start including that. But I think maybe your point was that there's a fine line when

if you go to the security department in your organization they're going to be like whoa whoa what is this we got to review this for like you know top to bottom soup to nuts and everything right so did i misinterpret that or is that where you were yeah no no no yeah i would say doing both of course talking to the security teams and so on but basically these are just default stuff. For example, we in our project just implement without asking somebody because it's just the default line. So basically, start iterating on the defaults and the best practices.

And on the other hand, of course, if you are an enterprise company, you have a security department, and there are also many small companies out there who do not have any kind of security team or something. Of course, those also need to talk to them and maybe also doing this with a shift left pattern.

So basically really talk to them in the beginning and not, I don't know, one week before going to production with the whole cluster. Nico, you were very upfront in the beginning, also in the slides where you say you are a lot of security experts, right? There's people out there that have been doing security for all their life. But I think you are obviously in the real. Do you see kind of what are the typical ways or the most common things that, common ways that attackers actually get into your Kubernetes cluster?

Is it because, you know, nobody did network policies? Is it because of a vulnerability like log4j? Is it because somebody didn't patch their Kubernetes cluster? Do you have any idea on what are the most common kind of reasons why people get kind of the door into your kubernetes cluster yeah so on the good side i never had a client or a potential client who had any issues with their kubernetes cluster security issues so fingers crossed but back to to the red-hat state Kubernetes security report, as we see, it looks like it's a misconfiguration of the images, of the running applications, and also 30% with security incident during runtime.

So basically really somebody getting into your cluster and then from there, you don't have any tool, any auditing was can help you to take that one and from there going further and going further can be just another container running in your pod in your kubernetes cluster can be any kind of data in your cloud provider or on-premises world but also can be getting access to resources compute resources outside of the kubernetes cluster so in my in my talk i'm doing a demo at the very end where basically get credentials from the underlying Kubernetes identity. In this case, it's Azure, but it also works with all other providers. And then basically using this credentials to spin up, let's say, a virtual machine and mine some Bitcoins completely

outside of the Kubernetes cluster itself, which is then getting really, really bad. And I guess extremely hard to trace, right? Because who would know how they, how an attacker, a hijacker got to these credentials, right? Maybe nobody thought about it. It's even possible through Kubernetes cluster and then they log into your your SaaS platform for your infrastructure, your cloud platform, and then all of a sudden spin a virtual machines and then run all sorts of workloads. I can really get back to the identity of the cluster, but then if you don't have any auditing tooling in your cluster, logging this, the information is gone.

And I think this is also, you know, the more, I'm also not at all a security expert, but obviously I think security, we all have to become more security aware in our industry these days. And now I'm more and more understand why, for instance, you know, when we are posting videos and we are posting blog posts with screenshots, we have a lot of great colleagues that always then highlight and, hey, you know, there is an internal name of a machine that you shouldn't have on this screenshot and i said and i always thought you know why it's internal but then if you think about it if if if one door somewhere else is open this door might allow somebody to leverage this information and then get to that machine and and i think that's this is part of the awareness that where i'm so glad that people like you are actually stepping up and and and i think that's this is part of the awareness that where i'm so glad that people like you are actually stepping up and say hey i might not be an expert but i learned something yeah and i think we need to bring our awareness to the to the community you know there's

there's a parallel there andy with uh with performance even right performance used to be in the world in the realm of performance people right right, who were performance engineers and all that. And as time moves on, developers start becoming a little bit more aware. They're not necessarily becoming experts in it, but they're like, maybe they'll look at their code and be like, oh, it looks like this is set up for an M plus one query. Probably shouldn't do that, right? And same thing with security. deep, deep stuff. But as developers and others are having to fix and remediate and encounter the security issues in the real world, they start becoming mindful of them. And it's that same pattern of now, you know, to look, oh, well, what happens? Do I have firewall on, as you said before?

Do I have in our pipeline, do we have a scan going on, right? Do we have some of these simple things that I know about now? I'm not going to be an expert necessarily i'm not going to know all the deep things but that mindfulness it's fun to see this pattern repeating in security as we all become more aware of it so it's and i think nico what you're saying is that you know you're not being a security expert you picked up on all this stuff and you've been embracing it and just being mindful and trying to spread it to others which is exactly what we all went through in the performance side right so pretty cool yeah and i think also in the kubernetes space it's so damn fast changing so if you i don't know just relying on kubernetes to to run your workload to run your applications and have basically a different job just you're just consuming kubernetes you can't

be up to date with all the changes and all the ecosystem around it um so it's easier for for us i don't know using kubernetes the whole day the whole week um but if you're just consuming it to run your applications and then need to be aware of the whole ecosystem it's it's really hard so i completely understand why there are companies out there who struggle. It's really hard. In your presentation, I also want to highlight, people always ask for tools, what type of tools can be used for certain things, whether it's about scanning of your containers,

scanning your code base, validating, I containers, scanning your code base, validating, I guess, whether your Kubernetes configuration is sane or not. You did a really great job also in the slides to highlight and link to all sorts of different tools that people can use. One of the things that I would like to get your opinion on, because for me, again again also not an expert in

everything uh they talk about s-bombs so softer bill of materials um I hear this more and more coming up in conversations um and you know I may ask a very basic question now but I still want to ask because not everybody knows about it so why are s bombs so important what what is the benefit of having s bombs and basically without s bombs you don't know your dependencies or not really know your pretend dependencies so maybe you have your dependencies somewhere in the list um package log chest for example but the main issue are not your direct uh dependencies but the dependencies of your dependencies and the dependencies of those dependencies. You are not aware of them. And with an SBOM software build of material, you basically get the whole list of all the dependencies and sub-dependencies.

And then basically, you have an inventory of all your dependencies. And this can help, but it then gets even better if you use this SBOM list and then check against the whole list for vulnerabilities and security issues and so on. program at least check um yeah i don't know for an issue in the corresponding github repo to see if there are fixes configuration changes you can implement directly um and so on um so basically without s-bomb um and yes it's pretty new this is also something which is not really out there uh in the let's say real world um without that you don't have any insights in your dependencies. So with this SBOM idea, this is the first I'm hearing of this, and it's quite intriguing. And Andy makes me think of the service flow map and all.

We're not talking about downstream services that we're talking about within your code, within your bit. What classes and packages are you importing? What third-party components are you importing? What do those ones then rely on? I remember a while back, Andy, we had something, some of our workshops required on some JSON parser that was an open-source third-party thing,

and sometimes you go to run and it's like, oh, that's not available right now. So we're not even trying to tell people at this point look at downstream services and see and work there because that's that's the other people's responsibility you're just saying within your own everything that you're importing everything that you're pulling in to run your bits okay yeah that makes it that makes a lot of sense so let's say you're using downstream and i'm like no no no no that's too much yeah so for example you're using log4j, and log4j has dependencies, and the dependencies

have dependencies, and with SBOM, you're getting the whole list. And just to kind of finish the SBOM topic, so software bill of materials, but that means there exists tools and repositories where I can

say, hey, the log4j version x, y, z, I need your bill of materials. And then I merge it with mine and then I get one list that is accurate for me. Or are these tools then that I need to run during build time that scan all of these libraries and then create on demand? Do you know how sophisticated like how good the s-bomb community or the repositories already are or the dictionaries whatever is out there yeah so basically you don't need to do it on the build time so you can do it afterwards

works for binaries works for container images and stuff like this um and most of the tools are just the binaries or you can easily integrate it in your pipeline, run it locally. Now, I think Docker Desktop also has an integration, you can do a Docker scan or something. So it's really easy to use. I'm not 100% sure. I think they all rely on a database, on a central database, at least for the vulnerabilities, of course, but also I think for the sub-dependencies, but I'm not 100% sure on that one. Might also rely on the different tools here. But yeah, I'm pretty sure they are also at 100%, but at least you have a good list, 95% more than without a S-bomb. Yeah. If I look through, I know we touched based on Kubernetes policies,

but if I look at it through slides, it's really great. I'm just scanning here and it sounds like I know what I'm like. I've done a great preparation for this, but actually I'm just following your lead here and some very simple things that you highlight here. For instance, Kubernetes policies, enforcing of, let's's say read-only file systems because if you are running an app and you know they don't need to write anything don't even give that container the privilege because if somebody breaks into that container then you don't give them right access to a file system where they can potentially do more damage yeah or at least it gets even harder to choose on so it's basically adding adding steps to make it harder for somebody who breaks it.

And yeah, read-only file system are one of them, because if you don't have them... So basically, in the demo I'm doing, I'm starting to download tools to make my life easier and stuff like this. Yeah, not possible if you have a read-only file system, for example. Yeah, that's a clear... Remember that. And folks, really watch it. When you were remodeling into the container and then you were downloading helper tools and all. And as you said, if you have this additional level of security, you will not be able to download anything, any of the helper tools.

Now you said you are in a situation where you are, you're helping a lot of your customers out there with practical advice with helping them with the Kubernetes and cloud native and DevOps journey. From a security perspective, besides what we've just talked about, is there anything else that since you may have started creating that presentation and given this talk multiple times, is there anything else that you learned that you say hey if i would do the talk all over again i would or if i would build a version two of this talk what else would i talk about yeah so basically um so i updated the talk over and over so basically i don't know i talked

about this multiple times but every time was a little bit updated um i currently working on a new updated version um and more focusing not on the hijacking part, but also more focusing on the branding part. Really doing a deep dive on the tooling, how to use the tooling, how to implement the tooling. So shifting from getting in the cluster to branding the stuff, which I hear in the slides, you're seeing it just touch point all the all the topics all the tooling but not doing a deep dive and this is um yeah what I will focus this year on too um and yeah also going into um yeah more the Kubernetes side so runtime protection as I mentioned the thing we talked

about earlier if somebody gets into your container or into your port in your Kubernetes cluster and then goes from there to your cloud provider, grab an identity. And so to really know this, getting alert or even just kill the process directly. So if somebody gets in your container, I don't know, 10 milliseconds later, the process gets killed. And with this, you're totally fine. So really focusing on the tooling out there and helping people to get started easily with that. I still like your approach though, right? It's like, what was the movie with Catch Me If You Can? The story of the guy who was kind of doing all of the fraud.

He pretended to be a pilot. It was Leonardo DiCaprio. He was playing the character, right? And in the end, in his life, he basically then became an advert, an advisor for, I guess, the FBI or somebody, right? Because if you are the best security expert is an ex thief. Kind of, if you think about it, I don't, that's exactly what you're doing. Your first figure out how can I break in?

And now you know all the ways in, and now you know how to close all of these loopholes or these doors. Yeah, we had also the idea to match those talks into one session, but then it's long, it would be two hours or longer, and really do, okay, getting in, blocking the request, getting in another direction, blocking the request. But yeah, never had the request, getting in another direction, blogging the request. But yeah, never had the option for that one so far.

Looking at your LinkedIn profile and some of the postings you did, I see you have been at ContainerConf 2022. Kubernetes is awesome. I also see, obviously, Container Security days, I guess. Is this where you're presenting next?

Yeah, yeah. It's coming up in early March and they're doing the next version of the talk we're talking about. Yeah. It's a great way. Where is this conference? In Hamburg. So I think it's March 8, I think. yeah. So it's basically from the container days, you went, which is every year in September, they're now doing a security version of it. One day went. Looking at your profile again, your official title right now is Head of DevOps Consulting

and Operations. DevOps is a topic that we've covered over the years extensively. We had Gene Kim on the call. We've talked about it? I don't recall ever talking about DevOps. No, no, what, what? It's because now they call it DevSecBizOpsTest. Sorry. Sorry.

That's fine. I'm also interested in one thing. I know in your slides, you also had like the infinity loop, but focusing now on DevOps and kind of automating things into the pipeline or into the delivery.

Could you recap a little bit, just high level, if you are advising to a development team or a DevOps team, where does security fit again from end to end? I know we covered it, but maybe an overview again, where would you strategically place security checks just to make sure we don't forget anything? Yeah, so basically everywhere. So I would really start in the coding phase, let's say developer implements a new feature, I don't know, fixes the bug or something, creates a pull request to, I don't know, spin up other validation pipelines to build it, run unit testing, et cetera. Why not directly also

run a static code analysis to see if the code is somehow fine? So really, shift left, get direct feedback. Do not block the developer, but provide direct feedback. So really, on the code level, and it's not related to Kubernetes or anything, really on the pull request phase, run static code analysis. Then, of course, if you deploy to the dev environment, you could also invest some time in dynamic code analysis, so really against the running instance, for example, running your S-bomb scanning,

getting your dependencies ready, checking your dependencies, and so on. And so on. Then deploying to Koei, running end-to-end integration tests, file security tests, and then going to operation. Of course, when you're operating the stuff also, and this relates on your platform, on your tooling, but then also things like if you go back to Kubernetes policies, network policies, container runtime security, stuff like this. If you're running on a cloud provider, they have security DevSecOps tooling enables those. Also here, we think it would be the first part where you enable security in the runtime while operation.

Also, there are many, many clients out there who don't do anything, just deploying the stuff. And if they, I don't know, deploy daily or multiple times an hour, they're getting updated dependencies because they're updating their code. But if they have a lifecycle of, I don't know, two months, they maybe do not update for two months. So it's really hard. And then, of course, going into the monitoring phase, also doing their security checks, validating their logs for any things, any requests that shouldn't be there.

And from there, starting in the planning phase, if you go the whole DevOps lifecycle again, and also in the planning phase, think about security. How can we better our security on the code level, on the platform level, and so on, and then starting over again. And maybe not only on the developer side for the actual application code, but also for infrastructure as code your docker files kubernetes manifest and stuff like this so not really limited to the application itself

yeah check all your code yeah i think you just had it brought up an interesting point because when we talk with a lot of our uh community our dynatrace community around runtime scanning, because that's what we do with our agent. And our talk track typically is, hey, you deploy in production and at the time of a production deployment, maybe the vulnerability is not yet known, but maybe a month later, right? Like Log4j was there for so long. And so that's why the continuous scanning is good.

But you also brought up an interesting point. It could be that you have done your S-BOM, your runtime, your container scanning at build time. But it takes you because of your process takes that long, a month or two until you actually release it into production. And if you don't do continuous scanning afterwards, then you don't know it you don't know it good point yeah you have to rescan basically um scan at build time rescan in the registry and also scan in your Kubernetes cluster to be aware if there's still something running in the cluster because you may have in your container runtime but maybe it's not deployed in your cluster yet or already upgraded again and it's an older version in the container registry left. So basically you need to scan in all phases all the time. And this is where all the runtime stuff really helps in the cluster to get

those awareness. And I think that bit with the container registry is an important bit, right? Because you're dependent on, you're going to check your code and you're going to check your package and you're going to make sure you're using all the right stuff. And then you're relying that the rest of the pipeline is up to date and is consistent and that everything you assume is going to make it forward is going to, but unless you build that and do that, you don't know. So if it's going through those checks at every point if somewhere it's like it's grabbing the base image from somewhere else or based on an older component at least that'll be caught at that point and i think a lot of i mean again that's more of an edge case but

again how do you know link to all of your material. Is this your kind of only security talk or do you have other talks as well that I may have missed? Or is this kind of like your security talk for people that want to maybe know or see more from you on security? Based on Kubernetes and Kubernetes security, this is the only one at the moment. Oh, wait, it isn't. So we had the talk I'm working at the moment

on is an updated version of also talk we did one time at least last year. So basically the parenting stuff. But it should be also on my slideshow. Let me check again. It's not it should be also on my slideshow let me check again um it's not it should be awesome it will be awesome it will be awesome um yeah here we go it oh it was all that container days but on the after a meetup but container days um last year which i did with my

colleague uh philip wells and this is basically the first version of the more focus on the prevent 10 days last year, which I did with my colleague Philip Wells. And this is basically the first version of the more focus on the prevent stuff side, which we will update this year and make it even better. I'm just looking through your slideshare. That's pretty cool. nmeisensahl. slideshare.net and again, link will be provided. You got a ton of presentations up there,

and especially if you look into some of the older ones, best and worst practices, IBM connections. Wow, you've been doing DB2 high variability. This long time, old world. Yeah, but that means, so your background is also on kind of the DBA's database. Is that something we can assume or not? So I started with, yeah, administrating in the IBM world. So IBM Domino, Lotus Node,

it's IBM WebSphere, DB2. Yeah, this is where I started. And then I went over to WebSphere, then WebSphere Liberty, Container World started. Then I started in the Kubernetes world, started on-premise, then moving over to the IBM Kubernetes service, I think it's called. Then I went into the cloud world moved over to

azure yeah i'm now i'm doing full-time azure uh cloud native stuff and uh we have moved from the ops administration side i would say in between so between developers and operations and helping both mm-hmm i'll try to help is lotus notes even still around it is uh so it it was lotus not then ibm notes and now it's hcl okay i had a friend way back way way back in the uh late 90s maybe i was still working in television and he was trying to get me into lotus notes and i was like looking at him like i got no idea what this is as soon as you said lotus notes my head just went somewhere and and i see looking at some of your older uh stuff uh kuba coming back to kubernetes you started with a kubernetes one-on-one talk, which is also always good for folks that are new. I guess this one is really outdated. No, but still, I think it's still, when I look at some of the things that you have in

there, you talk about node port, you talk about how pods through services get exposed to the outside world. I think there's some basic concepts that are still relevant. When we're talking about Kubernetes one-on-one, I also always want to do a quick shout out to our friend, Christian Heckelmann. It was a lot on Twitter because he did a great talk about how not to start with Kubernetes.

And that's always kind of my reference story on when people like some best practices on what to do and what not to do but yeah cool uh nico um i think for me this was highly educational right just uh i'm not i want to pretend to be a security expert but i think we all need to become better in in in security and also then kind of echoing what we learn to our community because everybody needs to become more aware. Looking further into the future, when we want to have you back as a guest, what other topics should we have in mind for you?

Because you obviously have a lot of real-world experience with your accounts. Any other topics that you want to throw into the pot so that we can pick it up later this year? Hey, yeah. So basically anything around infrastructure as code, Terraform, Kubernetes related stuff. I don't know how can I connect my Kubernetes with my cloud provider. So basically as a developer, for example, saying, hey, I would like to deploy my application, but also want to get a database directly. So like Crossplane and stuff like this.

You mean in the real world, there's something outside of Kubernetes where the stuff inside the Kubernetes has to connect with the outside world? Like a database? Yeah, to be honest, I'm forcing our customers to do so because I don't really like having self-managed databases instead of a cluster. So basically, we are trying to always use managed databases. And then you have basically, you need to talk outside. So this is one. Yeah, let me think. I'm starting in the sustainability world, part with Cloud Native though I'm

planning to provide a talk on that one this year. We have a talk related to Kubernetes is awesome, but so basically telling with real-world example when you shouldn't use Kubernetes, because there are many use cases where people use Kubernetes, but they shouldn't because they're basically getting all the complexities but not using the features of Kubernetes. Yeah. And maybe a little bit security-wise, using OpenID Connect in pipelines,

but not store any long-life secrets, but basically just using OpenID Connect to pipelines. Do not store any long-lived secrets, but basically just using OpenID Connect to grab a one-time token to connect to your cloud provider, for example, to deploy something or to connect to a Kubernetes cluster. These are topics you might see somewhere at conferences this year. Maybe, Brian, this is a good shout-out

for our listeners to say, what do you our listeners to say what do you want to hear what do you want to hear my vote would be for the uh people who are using kubernetes i shouldn't be yeah that's that's always my favorite is when people pick a technology that's just well what did we see the ecosystem versus the ecosystem right yeah exactly yeah so it's it's it's always it's always great to hear why it was a good choice or or even just why did you pick that if you can explain why you chose this and make it make sense then great because the best answer so far was uh because we need to be cool yeah that's that's probably the most common. To get people. So they have a Kubernetes

cluster just because they need to be cool. I remember this is now at least 10 years or maybe even longer ago when one of our customers came to us and they showed us their

new CRM system with high transaction volume. Not CRM, e-commerce system that they built with high transaction volume. And they decided to build it on SharePoint because SharePoint tables and views were so flexible, but obviously never designed for any high transactional volume. And we basically had to then tell them that the two or three years effort that they put in is all for nothing because they just made a wrong architectural decision in the beginning.

And I think the decision back then was also the new SharePoint and how easy it was to build new screens. But nobody ever thought about whether it's the right pick from an architectural perspective. Yeah. Wow.

Wow. It's almost like they're building a Lotus Notes. Yeah. All right, Brian, do you want to wrap it up? Yeah, sure. You know, one thing that fascinates me with all this is every time we hear about this kind of stuff, especially the Kubernetes topic,

it really reinforces the fact that when people think, oh, let's, let's move to Kubernetes. To me, Kubernetes is a software-based data center, right? And most people, I think, think of it as, oh, I'm going to something new for me to run my code on. Well, your data center. Right? And most people, I think, think of it as, oh, it's something for me to run my code on. Well, your data center is something for you to run your code on. But yeah, you do have a data center where you're running this stuff on. But now you're managing all your infrastructure. You're managing all your network. You're managing all of your code. You're managing everything all in one.

And it's forcing everybody to become so much more aware of all those other layers we used to have those i'll say nice in quotes but we used to have very walled off sections right infrastructure they just worried about the infrastructure developers never worried about the infrastructure and as you go into these platforms like kubernetes you can't have that approach anymore you know same thing now goes, obviously, with security. You have to be aware. What happens if somebody gets in? Are you exposed?

Can they get into your component and wreak havoc because you're not paying attention or thinking of the ecosystem? It's a much broader mindset that you have to have in these areas, which also makes it a little bit more difficult. But I think the icing on the cake here, or not the icing on the cake, the promise is that with really well-automated systems, if you really, really embrace the automation potentials in these systems, there's a lot of things that you would have to worry about that you don't anymore. Which JVM am I going to use? How much memory am I going to give it? You can automate these systems so that you don't think about these things.

You can start building in opinionated platforms into your systems. You can do it so that you can focus on your code, focus that you have these other things. And a lot of the other areas that you used to have to focus on are now just part of the system, which is the beauty of Kubernetes being a software infrastructure, if we want to look at it that way. So there are the tradeoffs. There are the ways to handle it. But it really requires that maturity to go into those systems and really start setting them up properly so that you can now focus on the more important things like your code and is my code secure and is my code performant. So it really gives you that ability, but it is a shift in

mindset and a shift in the way of doing things. Anyhow, that's what was going through my mind a lot as you were talking about this and making sure that so many more things that, let's say, the developer even has to be aware of. It's hard for a developer to have to do this stuff, right? And they shouldn't necessarily have to do this stuff, but that's the ecosystem. But again, if you could remove those additional components that they used to have to take care of, that frees them up to start looking at some of those other other things anyhow um nico thank you very very much i i hope we got everything in that you wanted to talk about uh if if you have any last minute shout out if you want to you know thank your puppy for or or whatever go ahead oh good yes thanks thanks for having me so um i think we rounded up everything

yeah thank you very much thanks to all of our listeners and uh yeah if anybody has any comments on what they want um nico to talk about next time oh i do want to say i took a quick look nico the the german artist i should say um did not die of heroin overdose. She died of a bicycle accident. But I think there was a period in there. So just be careful riding your bike, Nico. And wear a helmet. Will do. All right.

Thank you, everybody. Talk to you next time. Thank you. Thank you. Bye-bye.