PurePerformance - What makes GitOps Enterprise Ready with Christian Hernandez

Episode Date: March 11, 2024

Can you explain GitOps in simple terms? How does it fit into Continuous Integration (CI), Continuous Delivery and Continuous Deployment? And what are considerations when rolling out GitOps in an enterprise? To get answers to those questions we sat down with Christian Hernandez, Head of Community at Akuity, who has a fabulous analogy to explain GitOps that I am sure many of us will "borrow" from him. Christian also explains the ecosystem he works in such as ArgoCD, Kargo as well as OpenGitOps which aims to provide open-source standard and best practices to implementing GitOps.We closed the session with some advice around Application Dependency Management, External Secrets Operator and choosing the right Git Repo Structure.Here are some of the links we discussed:OpenGitOps: https://opengitops.dev/ArgoCD: https://argoproj.github.io/cd/Kargo: https://github.com/akuity/kargoArgoCon: https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/co-located-events/argocon/GitOpsCon: https://events.linuxfoundation.org/gitopscon-north-america/

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. My name is Brian Wilson and as always I have with me my special co-host Andy Grabner. Andy, how are you doing? Good, are we doing it the right way this way? We're not faking that we are the other personality? No AI fakes today, but I gotta say I had a strange dream the other day um just between recording i took a nap and i i had a dream so i was um i was down in uh in idaho springs in

in colorado it's on the way out to the mountains and i was in this gold mine it's called the argo gold mine right and uh i was i was around lost, and suddenly I see a bright light in the distance. And I could make out as it was approaching, it was a person with a minor helmet on. And it was you with a pickaxe chasing after me. And you were trying to murder me. I don't know if you've seen the movie, what was it? My Bloody Valentine, i think it might be called it was an old uh movie but like you were recreating that you were trying to kill me

murder me in this argo gold mine so fortunately i got out and was able you know i got up by waking up really um but that was the dream i had and you you were you were trying to kill me finally and i'm sure it it uh is very appealing to you to know Yeah, I'm actually more worried now after you tell me this dream that I'm hunting you in your dreams and I'm trying to kill you. I would never do that, obviously. So listeners, I'm not at all an aggressive person and I don't know how I make it

into the dreams of Brian like this. But interesting, because I did not know that there's an Argo Canyon. What did you call it? The gold mine. It's an old gold mine. that there's an Argo Canyon. What did you call it? The gold mine. It's an old gold mine. Yeah, it's Argo.

If you're ever driving out to the mountains, you pass it on the way and you can do tours of it. But yeah, it's called Argo. It sounds like we should. It's interesting enough. I think we should probably just pick up this topic now, but switch a little bit from gold mines to maybe GitOps. would that be a good segue both g words yeah yeah cool well maybe we should actually get somebody on the line who's been sitting there and waiting patiently until we're done with the dreams and i would like to introduce christian hernandez to the stage or to the microphone.

I think, Christian, thank you so much for being here. And as I said, sorry for listening to all of our jokes. Hopefully somebody finds it funny. But Christian, thanks for being here. Would you mind introducing yourself? Yeah, no, thank you for having me. So my name is Christian Hernandez. I am a, many things, but maintainer of OpenGitOps. I am a contributor to the Argo CD project.

I've been involved in Kubernetes for a while now. So I'm always in and around the open source space, especially around cloud native technologies. I'm currently working at Acuity as the head of community over there, community management. So for those of you who don't know, Acuity was a company started by the creators of the Argo project. So Argo and GitOps is very near and dear to my heart

and my day-to-day here as well. And I can attest though that Andy probably wouldn't attack you with a pickaxe. So just because I've known him for a while now and yeah, it's just a dream, I would say. Yeah. Hey, Christian, one thing that I noticed and maybe folks, if you listen to this, take a quick look at the screenshot we took

behind your chair in the far distance. I can see it looks like Bayern Munich at least as Bayern Munich and Ajax Amsterdam I guess so you are big into football or soccer as the friends in the US call it yeah I'm a big soccer fan big European soccer fan but soccer fan in general

so one of my many things I kind of wanted to create an Argo soccer scarf for KubeCon EU, but timing and budget didn't work out because I don't know if anyone knows you have to buy a lot of those

for an order to go through. So yeah, big soccer fan as well. That's kind of one of my hobbies that I do when I'm not doing tech stuff. Well, I hope you get the chance when you're in Paris for CubeCon to see maybe a game. I don't know. Paris Saint-Germain or what,

what will we watch? Yeah, PSG. Yeah, PSG. Really, really, really any,

any team that I can watch in Europe. I had the privilege to watch Barcelona play in the Champions League one time while I was working. So that's kind of some of the perks of traveling for work is that I get to drop by and see, at the time, one of the best teams that ever existed.

So that's also a nice perk. So hopefully, I'll get to see something in Paris. And folks, this is something, we just got off a prep call, the two of us for ArgoCon. So Argo is really big topic, right?

It's not just something that we're discussing here, just this hour that we have, but it's been, we two get on stage at ArgoCon, which is a co-located event at the upcoming CubeCon. We talk about bringing observability to Argo CD using Captain and OpenTelemetry. So folks, if you are interested in the topic, if you happen to be in Paris,

I believe this podcast will air just slightly before CubeCon, so find us on, I think it's Tuesday at ArgoCon in that week. And if you want to learn about observability and bringing observability to ArgoCD with Captain and OpenTelemetry, then join us.

But on the topic, maybe not everybody knows what ArgoCD actually is, what the Argo project is, what GitOps is, because today we want to talk about enterprise GitOps. Christian, for me, you are an expert in that space, a thought leader in that space.

For people that don't know GitOps and what Argo actually does, could you give us a quick summary? Yeah, so I'll give a quick summary. I'm part of the OpenGitOps sandbox project, which is really focusing on GitOps as a practice and as a principles. I'm going to kind of go over high level, but if anyone wants to know more, go to

opengitops.dev. There's this slew of information there where you can find out more information. So GitOps is I always like to equate it to a thermostat, right. And, um, thermostat versus, uh, something like on, on, on the, uh, like on the window, uh, air conditioning unit. Right. Um, and I, you know, one of those, you know, air conditioning units,

like when I was young and had my first apartment, it was really crappy, but not all of them are crappy. And I make that analogy because GitOps tries to differentiate itself from an event-based type of workflow to kind of like a reconciliation control loop type of workflow. And I think that's kind of the analogy I like to make is that an event-based workflow is kind of like me sitting there in my apartment being like, you know what? I am really hot. I'm going to stand up and turn on the air. You know what?

I'm really cold now. The air has been on for a while. I'm going to stand up and go turn off the air. That's an event-based, right? An event had to happen. I got warm. I got cold. That sort of thing, right?

Whereas a thermostat, you kind of set your desired state, right? An event had to happen. I got warm, I got cold, that sort of thing, right? Whereas the thermostat, you kind of set your desired state, right? I want it to be 70 degrees here, or for Andy, 20 Celsius. I want it to be, you know, a certain temperature in, you know, here in my office. And I basically set my desired state and the air conditioner will come on. Once it reaches a certain temperature, it'll turn off. But once it starts warming up again, it'll turn back on, right? Without me having to actually go and turn on the air and turn off the air. And so get-ups from a high level is a control loop. And it is the same way that you kind of manage your thermostat,

your air conditioning unit. You manage your system that way, right? So in cloud native, that system is Kubernetes, right? You kind of say, hey, I have this desired state of what I want my application to be, right? I have these, and leveraging Kubernetes' natural declarative view on how to manage the system.

So GitOps kind of goes up a layer and kind of instead of managing things, letting Kubernetes manage things at an individual level, in an individual domain, like replica sets or secrets or whatever, right? It has like its own, GitOps kind of manages that as a whole, meaning that like my application is made up of individual components, individual, you know, it may be like five deployments,

you know, four secrets, a couple of persistent volume claims, ingress controllers. But like, I want to see that as a whole. I want to control that as a whole. I want to control that as a whole. And GitOps really takes a holistic approach of, okay, Kubernetes is taking care of the individual files, but we need something to take care of the application deployment as a whole. And really, that's kind of where GitOps is, right? It is the thermostat,

but actually viewing the whole cluster state as a whole. So that is kind of where Argo CD came. Argo CD was developed in Edintuit, internally at Edintuit. They acquired a company called Applatix and those folks basically built... The original problem was, you know what, we have a bunch of developers,

and we want to on-ramp their application onto Kubernetes, but that's like a lot of training, right? We need some sort of interface in the middle. And using kind of the GitOps principles, they built something called Argo CD, where it's not only about deploying an application and making sure it's reconciled, but it's also information that developers want to use. Which I think what makes Argo CD so popular now is because it's an entire platform where you can actually go log into Argo CD and see, me as a developer, I can see my application and all its components, where it is, what's the status, is it synced, is it not in sync, I

can conduct the sync, I can conduct the diff, I can see high-level metric information about what is running on my cluster. And so Argo CD, the popularity of using kind of like those GitOps principles of keeping the state in a way that you want it to at all times, but also develop a tooling around it that kind of says, okay, hey, you know, I want to see my application as a whole. So that's kind of 10,000 foot view word vomit of kind of GitOps and Argo CD as a whole. I love that analogy. Thank you so much.

I will borrow it probably when I explain it. The manual events-based AC versus the thermostat. I want to just some clarifying questions. I know we always talk when we talk about Argo CD, we typically talk about application delivery, but Argo CD is not constrained to deploying applications, right? I mean, it's really, it allows you to apply any type of desired state and that is not constrained to, let's say, workloads or deployments or rollouts.

Because if we have tools like Crossplane we have other tools i'm sure where you can also declaratively define infrastructure components even outside of the kubernetes cluster if you think about you want to provision an external database or like a database service and you can provision this using using crossplane So just a clarifying question, Argo is not limited, quote unquote, on delivering just the applications, but also kind of the infrastructure and additional services around it. Yeah, and absolutely right.

Especially, like you said, the popular project Crossplane, I think is a really good example of that, where it extends the capabilities of Argo CD outside of just not just deploying application, but also deploying and managing infrastructure, deploying and managing policy, security policies as well. It's not just for delivering applications, but you can do other things as well. It opens up a whole... Projects like Crossplane, also things like

the cluster API operator. projects like Crossplane, also things like the Cluster API Operator. There's some Terraform Kubernetes controllers out there as well that help you manage things outside of just the Kubernetes ecosystem, but also other infrastructure components.

So now you're trying to this stuff I'm doing with the applications, I also want to do that with infrastructure components as well. And having these projects help bridge some of these capabilities. So now you have kind of like Argo B, you're like single pane of glass of deploying things.

So not just applications, just things in general, right? Things that you want. And also another clarifying question. So when you think enterprise scale, I mean, we often, when we do demos and we present concepts, we typically have a single Kubernetes cluster and we deploy our simple apps with one workload or maybe three workloads or five workloads. But what if you think bigger, if you have multiple Kubernetes clusters, because you may have dev environments, you have test environments,

you have multiple different production environments. From an architectural perspective, do you have a central control plane that is Argo and that is then managing all these target environments? What's the typical setup here if you actually go enterprise scale? Yeah, so typically people always start off with Argo in a hub and spoke type of deployments, right? So you have like a central Argo CD instance where it deploys to different target clusters.

Argo CD also has a feature-rich multi-tenant capabilities, things like RBAC, SSO logins, groups, those sorts of things, project namespaces. And so it kind of makes it like a multi-tenant system. But at some point, you're right. So it gets to a certain scale where having the hub and spoke model doesn't really work, right? It is meant to scale up and out really, really well. But typically, folks are, you know, kind of break away from having an Argo, central Argo CD system to

like many Argo CD systems, right? They'll have, you know, maybe like one for each environment, right? Like they have like an Argo CD for the dev environment, Argo CD for the production environment, you know, and each individual boundary that they may have from their organization. Another typical design that I see is kind of a hybrid model where they have kind of like a configuration Argo CD. So like the Argo CD per logical team, right? So like the infrastructure engineers, the platform engineers have their instances of Argo individual teams have their own

installations of Argo and kind of like take care of their own little little cycle as well and that's kind of the another thing that I see out there that typically gets done with, once you start getting to enterprise scale, right? You're starting to manage not only just like an Argo CD instance, you're managing many Argo CD instances, depending on how your organization is laid out. And that's kind of part of the problem we're trying to attack at Acuity.

Acuity has another model that we call the agent-based model, where instead of having a bunch of Argo CD instances, but also wanting to separate your workforce as well has an agent on each system that interacts with it. So you don't really have a single point of failure, that sort of thing. That design is also very, very popular. So there's a few things that happen once you get to an enterprise scale. Mostly end up being the fact that you're going to need some sort of hybrid or some sort of design where it makes logical sense to you, whether it be scale, whether it be organizational needs,

whether it be compliance as well. That also plays a big role in enterprise scale with Argo CD. First of all, thank you so much again for explaining the GitOps in general, the concepts, the reconciliation loop, Argo, very powerful. And I know there's other players in the market as well, also in the open source space. But at least from my perspective, I see Argo very prominent.

And I think that's also fair then to talk about this project. Now, enterprise GitOps, you just explained a little bit on some of the additional capabilities that you're expecting from enterprise when you roll this out on an enterprise level. What else is there when people considering moving towards GitOps and then rolling it out in an enterprise scope? Are there other things that people need to be aware of and not just think I just installed this open source project on all of my environments and

that's it? Any other considerations? What is enterprise GitOps, basically? That's my question. Yeah, so one of the things, and I think Andy, we were talking about this earlier, is that we were talking about how Argo is kind of like that takes the Unix approach of being a tool where it's like it does one thing, but it does very, very well. But there's other things needed that you need to kind of bolt on or kind of build tooling around to get the most out of your Argo City installation,

especially for the enterprise. One of the big things is, as you guys, is very near and dear to your guys' heart, is observability. Information needed about deployments, information needed about not only why something failed, but like why is something, you know, taking longer than it did yesterday? Or like what is going on in my environment when like the same deployment is rolling out slower today than it was yesterday? Or, you know, why is this, why is the disk all of a sudden getting full during a deployment? You know, that kind of information that you need, because we always talk about failure in IT,

in our world, for some reason, our heads are always thinking about what happens when something fails. But more often than not, we never get to the failure part. There's all these things that happen before the failure that are important, especially for your day-to-day. A failure is an event versus stuff that happens from day-to-day.

All the little nuances of delivering applications. Observability is very, very important. Metrics is very, very important. Especially in a GitHub important, especially in a GitHub space and especially in a cloud native space where now everything's spread out across not only

objects in a single cluster but many objects over many clusters over many geographical locations. There's just stuff everywhere. And so that is observability, one, is kind of like the number one thing that comes to mind with enterprise GitOps.

Another thing that is something that we've been working on at Acuity is the concept of, again, Argo CD does the deployment. It's really, really good at deploying applications and making sure it's rolled out. But there's a whole process that needs to happen before that deployment, right? And it's like that CI-CD process.

And in the GitOps world, a lot of people are shoving a lot of logic into their CI system just because it's just like a different world. They take what they're doing now, right? Whether it's Jenkins, I guess now people are moving over to GitHub Actions. But the same idea persists, or the same problem persists, is that you you have start having this bespoke scripts for your deployments for, you know, to deliver these artifacts to Argo CD where CI is actually meant to be more specialized. It does really, really well for building applications, for running tests, doing, you know, quality assurance, things like that.

But when it comes to bundle up deployable resources, that becomes a bespoke script. So this is a project that we started called Cargo at Acuity. And it's meant, it's specifically attacking the problem of continuous delivery. So it delivers the deployable artifacts to something like Argo CD. At Cargo, we're trying to build it to be really agnostic. But currently, the creators of Argo is going to make the Argo kind of like the first class citizen,

at least for the meantime. But bundling up, if you think about what it takes to deliver something in GitOps, it's a series of things. It is one or many image changes, one or many Helm repository updates, and one or many Git commits. And those can cascade. And so what we did with Cargo is we basically have taking CICD and helping out Argo a little bit more before it deploys the application. Cargo basically packaged things up and delivers those to you.

So if you think about it, Argo CD is continuous delivery. Cargo is, sorry, continuous deployment is Argo CD. Continuous delivery is Cargo. And so really with the GitOps, enterprise GitOps for me is kind of like these tooling that happens over and over again, right? So that layers on top of each other, like Argo CD depends on cargo, which then depends

on, you know, the CI system. And then cargo can then be used to, you know, check on like observability things, right? Like metrics and things like that in order to determine whether or not to, you know, consider something healthy or not. So there's many toolings. Observability is very important. There's also other toolings that can help. Some of the hard stuff people are doing now with CI, with GitOps.

A clarification. Cargo, you mentioned, is also an open source project? Yes, it's an open source project. It's an open source project, perfect. And then, you talked about Acuity, the company you work for.

How do you license, or how does this work? Yeah, so we're building a SaaS product on top of Cargo, Cargo and Argo CD. But we're open source first, open source company. So the Cargo itself is open source and can be used today. And that's how we're developing it. We're developing it first in the community. And once, you know, once we tested some of these theories out, right, some of the problem statements that I just mentioned,

it'll be built into our SaaS offering. So, you know, you'll have the ability to, you know, not only deploy your applications, but also then start tracking, you know, bundling up your deliverables in a sane manner, in a single pane of glass. I'm just looking at Cargo on my right side here, the GitHub page. And I think one of the things

that I think at least, it's my personal opinion, made Argo CD so successful is also that you had an appealing UI or at least you showed things to people what was happening and it seems Cargo is following that model. The first thing that pops into my eyes is a nice looking UI where you can see I think all these bundles and these bundles that you he then pushing over and then letting it be deployed by Argo CD. I also remember because we had Kelsey Hightower on a podcast a while ago

and then also I had the chance to meet him twice for a keynote we did together at Perform. And if I'm not mistaken, he was also big on giving this cargo a big shout out. If I remember this correctly. is also big on giving this cargo a big shout out. If I remember this correctly. We're trying to attack the problem holistically. There are tools out there

that do pieces of cargo, but nothing that does it holistically. And like you said, creators of the Argo project now are doing cargo. UI is very important. And so me, I come from a systems background. Everything's CLI.

Even in GitOps, I'm more comfortable in GitHub than I am clicking around. But it's important. I've managed developers. They love it. They want to, maybe they're not CLI junkies like I am. Maybe they just want to code and see their things.

They want to code. They don't want to mess around with deploying the scripts or anything like that. So it's something we're very excited about. We're always looking for feedback, by the way. It's still early enough. So those of you who are listening and want to give it a try, if you're in the Argo CD get-up space, want to try it out,

feedback is welcome. Now's the time to do it if you want to make changes. Hey, Christian, if I can quickly recap, just to make sure that I got it right. If you think about the whole end-to-end delivery process of software, we can break it up into three core kind of phases. The first, as you mentioned, is CI. It's actually building and testing that code, building it. And this is where you mentioned Jenkins, still very prominent out there. Some people, obviously, are moving to other tools, like you mentioned GitHub Actions, but basically building artifacts.

There is CI, continuous integration. The next thing would be continuous delivery, which means we are taking the artifacts and we're bundling it up and then we're delivering a package that then needs to be deployed. And this is where Argo CD comes in. So CI, build, delivery, package it up into a package and then deploying it in your target environment. Did I summarize this correctly? Great summary. I need you to do my CFPs now.

I just talk and you'll summarize, and I think it's better than chat GPT, I'll tell you. Andy's secret title is the Summerator. Yes, Summerator. It was really good. It also helps me to understand if I reflect on what I've just learned, just to kind of bounce it back

and make sure I understood this correctly. I think the other big piece, coming back to enterprise GitOps, I really like what he said because we have individual tools that do their job pretty well. But in an enterprise context, you have a lot of other things to take care of.

Security, policy enforcements, observability, you mentioned all of that. And so I think enterprise GitOps really takes the approach of GitOps, which you so nicely explained with your analogy of an AC versus a thermostat. So you are of GitOps, which you so nicely explained with your analogy of an AC versus a thermostat. So you are taking GitOps where you have a desired state that gets synced to the actual state, but there's a lot of things that have to be taken care of on an enterprise level

to really roll this out on an enterprise level. It's security, policy checks, and observability are a big part of this, so thank you for that. Because I'm sure many of our listeners are looking into Argo and see all these great tools, but then there's just more than one individual tool that can do CD.

This is why I think also platform engineering is seeing such a huge boost right now because as platform engineering teams, we want to build platforms that provide enterprise-grade support for developers so that they can deploy their apps in a secure way that's automatically observed

and that they are within the policies that they can move around. I think that's hugely important. I have a question because early in your introduction, you mentioned that you are part of the OpenGitOps kind of a group where you launched OpenGitOps. Did I remember this correctly? Yeah, I'm a maintainer.

I also helped bootstrap it initially yeah yeah can you then tell us what what what is the goal with open git ops because maybe that's that's something people would be interested in hearing more about yeah so the goal of open get ops was really mainly the initial goal was mainly put some definition around some of these practices. And, you know, GitOps at the time was very early on and very buzzwordy. So a few of us from the community, you know, the bulk being from the Argo CD community and for the Flux community,

got together and say, okay, like, what is GitOps, right? We didn't want what happened to like cloud to happen to GitOps, right? Because like cloud now is just like, what is GitOps? We didn't want what happened to cloud to happen to GitOps, right? Because cloud now is just like, what does cloud mean? Well, no one really defined it. I can plug in my hard drive to my Wi-Fi and that could be a cloud, I guess, right? There's really, you know, it turned into more of a marketing term, right?

And so we kind of wanted to put some definition around what GitOps is. And really, that's what OpenGitOps is. Just at the heart of it is like, okay, what does it mean to do GitOps? And again, if you go to opengitops.dev, there's four principles there. It's kind of laid out there with, you know, glossary

definitions and things like that. And so really, that was kind of like the initial like goal, right? The initial goal is to do that. And now is more about, and then it turned into like more about like education, right? And that's kind of some of the things I know that you've been involved with, Andy, about GitOpsCon and things like that. We do these types of events to kind of bring awareness. And now we're in the state of expanding and best practices. So expanding where you mentioned earlier about where does that demarcation of GitOps and infrastructure or GitOps infrastructure as code?

And it's like, well, we want to start moving more towards blading it over to like everything, right? GitOps, everything. You know, not just cloud native stuff, right? Just like basically, you know, go towards that goal. And also best practices. There's some folks that, you know, have done great things in the community. I know that we had talks from, for example, from Adobe and how they solved the problem and they figured this is the best practice.

If you are in an organization of our size and with our regulation, they have some best practices. And so that's kind of like Open, you know, open GitOps and the GitOps working group is mainly kind of just sharing best practices between like-minded people, right? We kind of wanted to have a place for GitOps practitioners, regardless of what you're using, right? We don't really care. Argo, Flux, you know, Spinnaker, whatever, right? You come and we talk over best practices and try to learn from each other.

I'm mainly involved in the awareness aspect of it. I was co-chair of and I'm still doing co-chairing of GitOpsCon. We're going to have GitOpsCon in April, co-located with open source summit um so it'll be there i think it's the monday april 15th um andy if you don't know for us that's tax day so

uh how fun tax day is um we're gonna have get ops con on tax day uh but it's co-located open source summit so if you're gonna be at open source summit drop by uh you know maybe a day early and um check out some of the talks we're gonna have at open get ops so um you know that's that's kind of a high level uh why the why and and you know and the what and what we're doing in uh in the open get offs group yeah where is it in seattle seattle washington and if you're waiting to the 15th to submit your taxes, come on. Yeah, exactly.

Exactly. Yeah. At that point, you're filing for an extension. Yeah. If you're really hard up.

Yeah. Hey, there's one thing you mentioned, I think, that also needs not a clarification, but probably just like to repeat,

GitOps is not limited to Kubernetes. I think that's a very big point, right? Yes, it's not limited to, and we wrote the principles with, we explicitly tried to not name either tools or things like Kubernetes or anything like that. We were trying to make it generic enough.

It doesn't have to be just Kubernetes. It stemmed from that world, but it definitely doesn't have to be just Kubernetes. And on Kubernetes, I guess one of the reasons why, obviously with Kubernetes, a new technology platform and a great way to then obviously introduce these concepts, but also, correct me if I'm wrong, I think the operator concept on Kubernetes itself was just perfect to build something like Argo with the reconciliation loop that allows you to, as an operator, say, hey, I

want to make sure that my temperature is always right. And if it's not right, I'm turning on the AC or turning it off. Or the Git repository, it looks different than what's on the cluster, and therefore I want to sync it again. So that's perfect for an operator. But it doesn't mean that you cannot implement something like this reconciliation loop outside of Kubernetes or use it inside of Kubernetes, just use the

control plane of Kubernetes to run your GitOps operator, but then still create resource sets outside of Kubernetes. We talked about cross-plane earlier, right? I can standard BC2 instances or whatever I want and define them in Git and then Argo is taking care

that these things are correctly applied or Argo and define them in Git. And then Argo is taking care that these things are correctly applied or Argo and then cross-plane. Yeah, no, for sure. Yeah, no, it does not have to be on a Kubernetes cluster. You get a lot by using Kubernetes, as you mentioned, right? Using it as a control plane, right?

Maybe you're not deploying applications to this Kubernetes cluster, but you're leveraging a lot what the Kubernetes cluster gives you with respect to a declarative infrastructure. Like you said, use it as a control plane to deploy your other, like you said, you can do spin-up EC2 instances and run a Terraform, apply to them, and you're using a

kubernetes cluster as a control plane so um so yes yes and right like you're gonna have like a mish uh mishmash of uh of a different tooling working together one thing that i also asked our previous guest uh still fresh on my mind because the recording happened two hours ago, is I asked her in the end, if you're interacting with the community manager, that means you're interacting with a lot of community members, and I'm pretty sure you are, I don't know, watching forums, Slack, and you're helping out. What's the top one thing that you say,

man, if I would wish everybody that starts fresh with git ops if they would know about it so that they don't make the mistake that everybody's doing or is that one thing if if you if if the world would know about one thing they would bug you less with the same question yeah how does is this is this something that you want to make sure that somebody that comes new to this topic, because maybe it's the first time they hear about GitOps and Argo today on the podcast, they say, hey, this is what I want you to know, because I don't want to get your question on the Slack a day after you started.

Yeah, so there's a few things that come to mind that I get a lot of questions on. A few things to keep in mind, right? So three of the top of my head. One is the biggest thing, at least in the Argo CD community, is application dependencies. So meaning that some of us, me and but like it's a very small subset of people that just relies on eventual consistency uh because i i like to uh deploy manage things that will eventually be okay as long as you wait long enough for everything to you know be be uh be ready right so um but that that being said, it's perfectly fine. And it's actually quite common to say things like, hey, I need this application to be up and running and healthy, even before I attempt to deploy another application. And so one of the big questions I get asked, and I think there's not a lot, it does give me an idea for content,

but there's not a lot of content around it is like how to handle application dependencies in Argo CD. That's very important. I would definitely research that heavily even before you attempt anything because there's different ways for whatever reason, the saying, there's different ways to slice that cat.

But to skin a cat, right? I really like slice the cat better. Slice the cat, yeah, yeah. For whatever reason. There's different ways to slice that cat, right? To skin that cat, that specific problem use case. So some of them works for other people other solutions

don't um so there's things like app of apps right application of other argosy applications there's application sets there's progressive syncs um there's just different ways of doing that and application health is very important how you define what's healthy right um you know what what is um you know for your application just because it's ready doesn't mean it just because it's healthy doesn't mean it's ready to receive traffic right there's like there might be a slew of other things that need to happen and so um application dependencies that's one thing i always say to research a lot there's i i gave you know the top three answers, what people usually do, but I definitely look into that

even before you try to attempt it because you save yourself so much headache. I always like to generalize my secrets, meaning that I am a big fan of the external secret operator only because I used to manage a big multi-tenant system and just because one team uses one secret, just because one team uses Vault doesn't mean another team is going to be

using Vault as well. They might be using the AWS key management system or some other or another team is using Doppler, whatever. the AWS key management system, right? Or some other, or like, you know, the team's using like Doppler, you know, whatever, right? Like they're using the external secret operator is very, very good because it's one provider for like a platform engineer that can have many endpoints, right?

So like I am just dealing with the external secret operator and how those secrets get in is the same. It doesn't matter if you're using Vault or if you're using AMKS or whatever the Amazon key management system is. It doesn't matter what the backend is. The management system is the same. So I always highly recommend

external secret operator. There's probably use cases for other things, but I really like the external secret operator as well. So those are the few things that just come into mind. Like, just like, you know, think about secrets, think about that. And also your Git repo structure. It's very, very tempting to put everything in one Git repo,

but more often than not, you're going to have two or three minimum repos that you have to manage. And that's just some of the pain that you have to deal with. I think as OCI becomes more and more prevalent, that pain will go away a little bit. But yeah, those are the things. Research. So like Andy says, you don't have to bother me.

If you're going to slice the cat, you're going to have to get a little bloody. Yeah, exactly. Exactly. Yeah. Just a quick comment on the three items. App dependencies. Remember, we were using Waves.

There's a feature of Waves where you can basically say which Waves certain apps should be installed if you use the app or web concept. Then the external secret operator actually reminds me of what we've built with Captain and the Captain metrics feature, where Captain allows you to bring in different metrics from your external observability tools. So you may have data in Prometheus, you may have data in Dynatrace, may have data in Datadog, in New Relic. Different teams maybe have different observability platforms, but as a Kubernetes

operator, you're just interested in giving me a particular metric, I don't care which tool. And so it's like the same thing with the Kept Metrics server. We are pulling in external data back into Kubernetes and provide it as Prometheus or through the Kubernetes Metrics API. That's good. And then last thing, the Git repo structure. Is there a best practices are

always, you know, it's a different thing to give best practices, but you said you end up with a couple of repos. Is it because you would have repos for the infrastructure or the core platform services that would be different than from, I don't know, your application repos? Or would you have multiple application repos?

Or how would that look like? Yeah, usually it's a combination of how your organization is laid out, your organizational structure. You have, or communication structure. I forget, I think it's Conway's Law, right? Is that what they say? But it does turn out kind of like what you were alluding to. It's like, okay, I have a repo for my infrastructure,

and even though I have Git branch protection rules and I have access to that, I want that separate from my application repo, only because you have a different lifecycle. You're going to be deploying apps

way more than you're going to be setting up infrastructure. The frequency at least, right? Normally speaking, right? Some folks build a new environment every time they deploy. That's actually cool, but I don't think we're all there yet. I think people just stand up still pets versus cattle, right? They still stand up pets.

At least for the time being. And so you'll have a repo just because of the different life cycles. So you'll end up having an infrastructure repo, kind of like an Argo-ish repo, right? Like components specific to Argo CD or whoever's managing

Argo CD, and then you'll have your application definitions. So you'll have, and that's really Argo CD, and then you'll have your application definitions. So you'll have, and that's really kind of like, I don't know if anyone either listening or you guys have heard an Intuit talk, that's kind of like how they set up their Git repositories. There's probably like three per application deployment just because there's one for infrastructure, one for Argo specific things,

and one for the actual application deployment itself. Okay. I think one thing we learned because we internally at Dimethrace also switched over to Argo CD and I just gave a talk with one of our platform engineering leads and we talked about this and he also told me in this preparation that we had different approaches in the beginning. We put everything into one repo, all the apps, and it grew and grew. And then one of the challenges we went into, because Argo needs to constantly sync

and fetch the Git repo. And if it's growing and growing, it's very big, you end up in challenges, right? So that's why splitting it into multiple repositories has multiple... There might be other reasons too. And...

Yeah, it's hard shoving all that information in cache, right, in memory. At some point it just grows to the point where it's just, there's not enough memory to do it. Like you get a lot of, like you said, performance issues as well. I thought about it from like a practicality process. Yeah, yeah, of course.

But you're right, like there's that aspect as well. You'll start running into memory issues and storage issues, especially, like you said, if everything's in one repo and it needs to refresh that cache, you're basically shoving potentially gigabytes worth of data in and out of memory every however many minutes. It's CPU overhead, it's memory overhead as well.

Yeah. I think we actually went into a situation where the sync cycle was so short that the previous sync cycle didn't finish yet. And I think we had issues there. Yeah. And that's also why we built some Dynatrace dashboards that actually pull in data from Argo because Argo is exposing a lot of Prometheus metrics. And also, we looked at logs. And so we created some dashboards

to help our engineers that are managing Argo to make sure that Argo itself runs efficiently, is configured efficiently, and is not running into problems or is causing problems on the depending systems like Git. Yeah. Cool. Christian, thank you so much for this.

From my perspective, I, again, learned a lot about enterprise GitOps, about equity, about cargo, about open GitOps. Is there anything we missed? Any final thoughts before we let you go? Probably a lot, but like you said, we only have a little time. I could speak for hours on the topic, but no, I think that's a good summary. I think at least for the time we have.

Awesome. Well, I just had one thought based on, you know, you mentioned holistic approach, right? And then naturally platform engineering came up. And I think that's a really important point, right? What we've seen from a lot of the cloud tool sets, Kubernetes and others, is that it ends up being a hodgepodge of different ideas and different people using them. And I think that holistic approach is something that's very, very important. In fact, we're seeing a lot of Kubernetes people trying to go to standardizations a little bit more nowadays.

And I think that's really where this idea of platform engineering becomes really, really important. Because if you have every team trying to pick their own tools or cobble together different pieces from different tools, it becomes a bit of a mess. Obviously, there are tools outside of the GitOps lifecycle, like you're talking about the keys, which observability platform. But when you're looking at that, you have to look at, just like everything else, you have to look at how is this going to handle our security? If we already have observability tools, is it going to be able to be compatible with them?

Is it going to work with our key stores? All these other kind of components. But if you have like what you're doing with Argo, much more of a holistic end-to-end piece of that, there's a lot less at risk. And that then really drives the importance of having the platform engineering, picking out these designs, picking out what's going to be what's going to work within that organization, instead of letting every single individual team pick and choose all their little individual components, except for the, you know, the things that are maybe feeding in. So I think that's a

very, very important piece and we've seen that for a lot of other technologies, that's where things start to fall apart when you're not making those organized decisions. So part of that planning, part of those what to know up front is look at your entire ecosystem, what tools you have, what tools you want to use, even which data you want to collect, right? Are you going to be able to collect that data? But I do want to ask one burning question that I think all of our listeners are dying to know at this point is, so you have Argo and then you said you have Cargo. Was Cargo named Cargo because you added a C in front of the A and made it rhyme with Argo?

Or is somebody at Argo a huge fan of Men at Work and they love their second album and decided they wanted to name it Cargo after the Men at Work second album? Well, I think it's a play on words with Argo. It's actually Cargo with a K. So for those who want to Google it because it's Kubernetes, right?

K, K, Cargo. It's definitely a play on words on the fancy. I was hoping there was a man in hats. Yeah, man in work. Man in work, yeah. Sorry. Anyway. All right.

Well, thank you very much again. This has been always enlightening. I was pretty quiet on this one because I don't work with this has been always enlightening i was pretty quiet on this one because i uh you know don't work with this stuff so much so i was more absorbing all the information um so you didn't have to hear any of my cheeky comments coming through um but i really appreciate it i hope yeah i hope all of our listeners appreciate it and thanks everyone for listening and really thank you christian for being on appreciate it thank you i appreciate it thank you, Christian, for being on. Appreciate it. Thank you. I appreciate it. Thank you. Bye-bye.