Software at Scale - Software at Scale 27 - Itiel Schwartz: CTO and Co-Founder, Komodor

Episode Date: July 15, 2021

Itiel Schwartz is the CTO and Co-Founder of Komodor, a Kubernetes troubleshooting platform. Previously, he was one of the founding engineers at Rookout.Apple Podcasts | Spotify | Google PodcastsWe discuss two major themes in this episode: the rise of Kubernetes as a popular orchestration platform, and the need for using an integrated service to understand and debug Kubernetes deployments.Highlights9:30 - When should a startup consider using a more heavy-duty system like Kubernetes, vs. managed platforms like AWS Fargate? What are the advantages of using Kubernetes over these platforms?18:00 - What are the new developments in the Kubernetes world? Why it may make sense in the future to run stateful services like databases on Kubernetes. Open Policy Agent25:00 - The motivation behind starting a Kubernetes focused company28:00 - What’s the biggest gap that Kubernetes users face while debugging their deployments? And how does Komodor help with that?39:00 - The surprising rise of Observability teams across smaller companies This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Welcome to Software at Scale, a podcast where we discuss the technical stories behind large software applications. I'm your host, Utsav Shah, and thank you for listening. Hey everyone, welcome to another episode of the Software at Scale podcast. Joining me here today is Etiel Schwartz, who is the CTO and co-founder of Komodoor, which is a Kubernetes troubleshooting platform. I want to get started with a little bit on your background and experience. So you were a software engineer for a while, and then you worked at another company that I'm familiar with, which is called Rookout. And I'm curious what got you interested in Kubernetes specifically, and maybe just in general, you

know, developer tooling and like debugging and making a platform like Commodore. Like what's your story? Yeah, sure. Sure. So let me do like my very fast, like origin story. I started working for eBay as a software and infrastructure engineer. So, you know, like working with massive scale and a very good system, like to be honest,

like eBay has its own backend and infrastructure and so on. And after a year there, I joined an Israeli startup and in Forte, it doesn't really matter. The interesting part about Forte is that they are very downtime sensitive and that means really great monitoring. And there I learned how important it is to monitor, how to monitor, what to track, and why to track.

So it was a very good school on how to troubleshoot and so on. And after a couple of years there, I joined Rookout, which is an Israeli startup that helps developers travel faster, basically with non-breaking breakpoints, as the first developer. And obviously, when you are the first developer, you can do a lot of bad choices, but then you need and expected to do them. So I was all in for Kubernetes because in my previous company, it was like a mesh of EC2 and Dockers and a lot of hybrid weird solution. I think now they are in Kubernetes as well.

But Kubernetes was so simple. I have a Docker, I can simply run it, and Kubernetes control plane will just make it work. So I pretty much fell in love with Kubernetes, and in Helm, I tried pitching in a couple of ideas or issues and so on to the community over the years. And as the first developer in a company, I saw one of my responsibilities was making sure

that both we are stable, we are up, and also the developers can develop easily on top of Kubernetes. So I had both the side of me of the more of like a normal, traditional DevOps, making sure the latency is low and the uptime is high, and everything just works. But the very neat thing about Kubernetes is that it allowed me and us to create a very simple workflow for deployment and CI-CD on top of Kubernetes. And this, in turn, gave our developers like us

a lot of ownership and full visibility on how the software is released, like we were CICD. So using Kubernetes, it was easier than ever allowing developers to develop, basically, and to be responsible and to enter their code. So yeah, I really like Kubernetes, and at that time, tools like CertManager, external DNS, and so on became popular as well.

And I remember it was so easy, making sure you have all the dev environment you want, each with its own external DNS and SSL certificate. And things just work. It's not easy, right? Making all of the different pieces work, it does take time, but with Kubernetes, it would probably take me months instead of days. Okay, so I think the two points which you're pushing on are developer productivity of developers who use a platform

like Kubernetes. Maybe you could explain to somebody who's just had experience with running servers on EC2, like running EC2 machines and deploying their application on that. What exactly is it about Kubernetes that makes life so much easier? What are some problems that somebody would face

when they're scaling up their application? How does a container orchestration system or something like Kubernetes help? Yeah, sure. I think the main point is it's really, really easy. I can't stress it enough. To deploy a Kubernetes application, let's say a backend Kubernetes application, I need a Kubernetes deployment YAML file, I don't know, like 10 lines of pretty straight YAML and a Docker file,

and that's pretty much it. I don't need to do anything else. I own 10 replicas, no problem. I changed the replica count on my YAML. In EC2, you need to have your init script and then your scale, your your group, your instance group, and then make sure everything works as expected, and then one of your instances die, and you didn't have tests.

And there are a lot of edge cases, but Kubernetes, the good and the bad thing about Kubernetes is it's so simple to get started. The bad thing about Kubernetes is it's so easy to get started because what usually happens is that for unexperienced people, they pretty much shoot themselves in the leg really fast because they have the platform that looks super simple, super easy to use.

I told you, all I need is 10 lines of YAML and everything just works. And they are like, okay, so let's do hundreds of microservices, and it will be easy, and Kubernetes will manage everything. And in the end of the day, I don't really think this is the case. I think it's easy to get started, but without someone strong or knowledgeable,

you are going to run into a lot of different issues. And it is not that easy. I wrote a blog post from my experience and from developers and Commodore customers that I talked with. The day two in Kubernetes, day two operation, I had my application and I moved it to Kubernetes. Now what?

It's very, very like it has, it's a very steep learning curve. Yeah, so what I was thinking of was it like, summarizing what you're saying, it's super easy to get started on something with Kubernetes and set up like replica sets and everything to deploy like a single application or a bunch of services. But then there's a lot of inherent complexity itself in running services, which is what bites you once your application becomes of a certain

scale and size. And that's the problem with Kubernetes in a sense that it's inherently so complex that even though it might show you a simplified interface it might fail in interesting ways yeah exactly and i think because it like hooks you up very fast and make everything looks really easy then when you understand that you are having issues, you're in the deep water now because you have your own production workload in Kubernetes. And I think it's true not only for Kubernetes. People are not that good on things, like a lot of people, mainly developers that didn't do a lot of infra,

are not that good in thinking about how i'm going to monitor my environment like i think it's true also like for ac2 and for like everything but because of the rapid speed of like developing in kubernetes then it's it it's much more like visible because everything moves so fast, but your monitoring, your learning curve, like for monitoring isn't that fast. Like after a while you are like, yeah, you have to install Datadog and Prometheus

and maybe Jaeger and maybe Istio to help me with the Jaeger and everything because you are like in a spiral, like you said, trying to gain understandability over your system and over its current status, but simply like fail because you are too deep. Does this make sense?

Yeah, I think that makes sense. Like you get some things out of the box where you don't get everything and then you basically make a patchwork set of things to end up like monitoring for tracing. All of that does not come like out of the box with Kubernetes. So you end up spending a lot of time

trying to debug and understand what's going on. Along with, you know, if you're running at scale and you have to start thinking about like network meshes and like things like envoi and stu and all these things and then all of these things together end up being super complicated at what point do you think a smaller company should think about kubernetes because like if i have like a 10 person startup with like three engineers do you think kubernetes is like the right solution for them to start off with or is it something that you have to think about after you have like certain number of services or how should you go thinking about that yeah it's a good question and when we started condor a year ago, like we had same choice. And I will say that like for us, the choice was quite obvious

because like we already know Kubernetes. So it's like cheating. So if anyone in the team already worked with Kubernetes, then I would say, yeah, definitely go with Kubernetes. Once you had like your basic learning step, the first learning step, it is easier. And yeah, even for a small startup, I would go with Kubernetes.

If no one really knows Kubernetes, I think it's a little bit more tricky. It depends what are your use cases. Are you going to have one monolith and that's it? If so, go on Fargate and forget about everything. I think Fargate gives a very nice solution like for I will say like simple application or application that doesn't need all of the Kubernetes like Network ingress like everything like Kubernetes has to offer daemon set, stateful set and so on so I would probably go with Fargate or like

something like the equivalent in AWS or in GCP and maybe if I'm very adventurous maybe I will go with Lambda or like serverless I'm a little bit more skeptical if I'm a 10%

team that no one really tried Lambda previously, if it would work out of the box. Because like Kubernetes, it looks easy, but then it becomes very complex very fast, and you're not sure where you are. So I think if I'm a 10% startup, no one had Kubernetes experience. I would simply go with Fargate and go to Kubernetes as I have more resources to put on that. And I will also go like monorepo and one monolith. And I'm not sure if all people will agree with me,

but I think it makes the most sense out of operational standpoint and also like developing speed. Yeah, I think as someone who is developing in a monorepo and using Fargate today, I certainly completely agree with you. And I love the point on, you know, first of all, it makes sense to just go with whatever your team is comfortable with using. So like if you have a team of Kubernetes experts who are going to be more productive in that setting, that's what they

should use. But by default, things like Fargate are good enough until you need a certain set of features. And that actually leads me to two more questions. So based on what you've seen in the evolution of things like EC2 Fargate, and you know, the similar ones in GCP, like Cloud Run, and I don't know how much you follow these maybe out of interest so you might not have an answer here and that's totally fine but like do you think these like managed solutions like Fargate are like catching up to systems like Kubernetes I know Amazon provides its own like EKS which is Kubernetes as a service but it's also providing Fargate and Fargate

is pretty good for your standard use case. So do you think you know they're adding more features and they're catching up to things like Kubernetes or there is a set of functionality in Kubernetes that you will never be able to get from like a managed platform? What do you think? Good question. So I can say from our customers that Kubernetes is by far more popular than Fargate. I think, I think like, you know, like Kubernetes, people go to Kubernetes because of the hype, right? Like a lot of people just choose Kubernetes because it became the standard fact, the standard defective. And I can't say the same for like fargate i think even forget are like trying

to catch up in the end of the day the kubernetes community is huge it's like both the like the platform itself like the project kubernetes and everything surrounding it i really can't see like fargate catching up like well But I might be mistaken. We can maybe speak in a couple of years and he will tell me, you know, you were wrong. And Fargate is like the next new thing. But I don't think people really like AWS that much that they want the vendor locking.

I do see a lot of Kubernetes cross clouds and so on. And it is a very good equalizer between clouds. If you ask me, I think Kubernetes is going to eat up maybe I'm mistaken here, but pretty much everything and became the uniformed cloud. And you'll have things like SQS, RDS and so on on top of Kubernetes instead of going to AWS and using like the managed services.

So if you ask me, I would put my money on Kubernetes and not Fargate. I think Lambda is like more interesting than like Fargate because it's more lightweight. It saves you a lot of money and it has its own elastically, elastically, I'm not sure how to say it. It is elastic. So maybe things like Lambda will catch up again, not that sure. I think projects like Vision

and Lambda on top of Kubernetes is super interesting. I'm really wondering if it's going to be the next big thing because I do see a usage for things like that. I love Kubernetes. I love Docker. If I could maybe do things that can scale very fast and do it like Lambdas on top of like kubernetes maybe it would be interesting and do you think like those projects are catching up like that so i i actually that's the thing right i think i don't have that much perspective on these things um but i think the

way you brought it up actually makes sense to me so there are a bunch of fears that you that are like not fears but concerns that someone might have on using a completely managed platform like fargate the first definitely is vendor lock-in like if tomorrow you're getting a large discount from gcp why it would be an immense infrastructure project in order to like go and decouple yourself from fargate to kubernetes so that's one thing that you might be thinking of right from the start but honestly the second thing you know there's so many things you get from kubernetes things like you know you like the way the thing that i can think of is auto scaling right like they provide auto scaling as like a binary

an open source service right off the bat so that's not something you have to think of and you don't have to couple yourself to amazon autoscaling, for example. So the point of the community makes total sense to me. And I think where it's interesting is actually those edge cases. Like if you're just running a web app that has to read and write from a database and return some results, probably any solution is fine with you. But as soon as you have to do something interesting, like some computation,

maybe you're doing like ML jobs nightly, which require like bursty capacity, anything like that is when you will start thinking, should I go with this managed platform that does not have as many documentation, that does not have as many users, where everything is closed source? Why should I

use that over this other managed platform that is open source and gets all of these things for free? So I think what I see is similar to what you see in the sense of if you're doing something very boring and you feel comfortable that Fargate will keep you going for the next five years or until your company dies. People choose that. But as soon as when they do something interesting, that's when they're like, you know what, it just makes sense for us to go with something like Kubernetes.

Yeah. Yes. And on that note, I think the answer to, you know, when you have to start thinking about Kubernetes is when you're doing something interesting, right? When you're doing something interesting right when you're doing something unique when you're doing something that needs like net like interesting networking stuff or even when you want to reduce your infrastructure costs right because i think kubernetes would be so much smarter at all of that versus like the high margin products that amazon is selling yeah there was just a i i just

wrote a read sorry an article about why k is super expensive, because people don't really like misuse it. So yeah, I interesting things coming up in the kubernetes world like i don't know anything about daemon sets and replica sets and all that um and what what what are some things that you get for free when you use kubernetes like auto scaling is one thing i can think of what else do you get and are there any like interesting things going on in the community that will like you know get shipped this year or next year or something like that i think there are like what are the the very good thing about kubernetes like it's like claim to fame and like the reason i think it's so popular it has a very a set of very clear apis what is an application what is a demon set what is a replica and it is so easy to write like wrappers around those things or like adding more capabilities so you get metrics for almost free you get login for almost free you can

use service smash like in like six lines you get service smash and envoy as a sidecar for each one of your containers and those things are super easy to use and you know like you can have your prometheus with hibana on top of like your kubernetes with i don't know like less a couple of hours like and that's it and i think helm as well also made kubernetes like basically you, it's like the distribution manager for Kubernetes. And it made, again, the ease of use of things. It is so easy, again, to get started. Like you want a Redis DB? No problem. You want an Elasticsearch? You got it.

Mongo? Yeah, no problem. Everything looks easy, again. I think it's not that easy for like advanced usage uh but i think it's very easy to get started i think the most interesting thing that will maybe happen in kubernetes and maybe it won't is our stateful set like not stateful set stateful set, like not stateful set, stateful applications are going to move to Kubernetes. Are we going to see like RDS on top of Kubernetes? Are we like, you know, you already have Postgres, but I don't know a lot of companies using Kubernetes as their main like database.

They are still using something external. And I think maybe not this year, maybe the year after or even after that, but I think this is the place that Kubernetes will shine or die. If stateful application could be inside the Kubernetes cluster, there are statefuls that help you do these things, but I think the maturity level is not that good for now. Another two trends, we see the rise of Argo CD, basically the best CD solution, I think. The best open source CD solution.

They are great. And they work seamlessly with Kubernetes. And the last one is maybe HPA. Not HPA, sorry. What's his name? One second.

The one with the helm. Open Policy Agent. OPA. Sorry about that so OPA that allows you to configure rules and policies on top of your Kubernetes cluster which is also super interesting okay what kind of rules and policies would you clarify that a bit yeah sure so you can think about something small don't allow me to have docker images that didn't came from a register like registry but you can have like more complex things don't allow traffic to go from i don't know from one place to another if you have a service make sure it has an owner or certain labels don't have open ports to the internet those kind of things

and again i see a very big spike in the popularity of opa interesting so you and i'm guessing this is again like open source plus the community will be able to share rules yeah exactly you don't have to rebuild all of this stuff on your own yeah exactly it's a cncf project like everything i said i think they're in the cncf okay including argo yeah argo is like officially in this yeah in the cncf okay and maybe one more question here is why what's the advantage of running stateful services in Kubernetes? What do you get out of that? So at the moment, you don't really get a lot other than having everything in the same place. But going forward, I think things like scale, like pricing,

and mainly cross-clouds will make it like... If all of the stateful application, your queue, your databases, everything is in Kubernetes, then you are not tied to AWS. You can leave tomorrow and go to Google and that's it. And you can also have like federated

like Kubernetes clusters that talks with one another. This isn't that popular from what I saw. Most people that I see have like cluster one in each cloud but i think it will be a trend maybe interesting yeah so it basically gives you all like today if you have everything in kubernetes you still have to think about migrating your data manually but in theory like kubernetes could just help you do that as well and then you're basically

completely decoupled from whatever hardware you're running. And that could, there would mean a lot, especially in a world where there's going to be more and more like data governance laws where you need to prove that all of your user data for a particular country is going to be only in a certain, in that country, for example. So that makes a lot of sense to me now.

Just the idea of like being super flexible about where and how you're storing your data. example. So that makes a lot of sense to me now. Um, just the, the idea of like being super flexible about where and how you're storing your data, if Kubernetes can manage that would be amazing. So clearly, you know, there are a lot of nice things of having Kubernetes, but there's also like a bunch of downsides. Um, and we spoke about this, you know, the complexity and just being able to debug. And clearly you have a story here because you've decided to start a company that is a troubleshooting platform for Kubernetes. So maybe I'd love to learn your story. Like why focus on Kubernetes specifically?

What was your motivation? Did you have like frustration as a user of Kubernetes at your previous company? And what made you so confident about starting a kubernetes related company yeah so i worked in lookout where we sold to other companies basically it's a company that sells to other developers so as my job i saw a lot of companies i did like uh pre-sales for a while like solution engineer and i saw a lot of companies and I think in Commodore, sorry, in Rook how things were from like it was okay. I had my own share of like frustrations but things were okay but when I went and talked with other people, other companies, webinar, you know, meetups and you see the pain.

I think the main thing that made us bet on Kubernetes is the fact that we bet about two things. One, Kubernetes causes certain things to become more severe. And B, production changes a lot today. A lot. Like dozens of deployments a day. And when an error occurs, lot today, a lot, like, you know, like dozens of deployments a day. And when an error occurs, like today, you don't really know where to look at, because you have so many microservices, and they change so rapidly. When you know, like, you are working up in like 2am or something like

that, and you try to solve an issue, you just trying doing all those kubectl commands and maybe you wake up the DevOps because you don't really know how to do and how to see changes and no one really track changes and so on. So you are lost. And what Commodore tried to solve is to be the one place that you need

in order to troubleshoot application Kubernetes. So because how we work, because A, we collect and save the history of all changes that happen in your system, both in Kubernetes and outside Kubernetes. And B, we simplify everything because we digest the data and basically think about a very easy,

simple timeline for each one of your services that includes when did it change, who changed it, what was included in the change and also the effects of this change because we also integrate with tools such as Datadog, Prometheus and Alert Management. So everything in one place. So I think it was like health my own experience and the other health my knowledge and experience uh seeing other people seeing other companies like i told you i really fell in love in kubernetes so i talked with uh in a lot of like cncf like meetups or like conferences when we had like physical conferences so it really taught me a lot about the state of most companies which is a little bit hopeless a lot of frustration

everyone told them that once they are going to be on kubernetes everything is going to be good but now they are on k, but things aren't that good. Like it's very hard to troubleshoot even more than before. And I think this is the place that like Commodore shines, like allowing both like expert and novice a very easy way to troubleshoot in Kubernetes. Okay. If you could tell me, you know, one killer feature when you demo that or when users see that,

that's when they want to buy the product. If you're comfortable with sharing this, what would that be? Like, what is the thing that, you know, hooks people on? Yeah, I think it's like the history of the deployments. The fact that we track, think about like a service

that is called Authenticator. I don't know, like it's a very important, a very important service in your system. Commodore tracks it health, like is it healthy, unhealthy, how many replicas and so on.

And if it's not healthy, why? But also the change and the very cool thing that we do about change is that we combine the infrastructure and the code into one single unit. So you can see that you had a change. This change included both like changes in replicas, in annotation, environment variable, but more than that, the image change. And inside this image change, Commodores know the commits, the

pull requests, the change as well. So in one very easy and intuitive UI you can see and go back in time and see everything and you can really see for our users you have a deployment and one minute later the service is down so it's not good right the service wasn't supposed to be down but it is but it's very easy to track who changed and what changed and so on and when I see and like open this I know this modal for our users there are just change shine and they are like we need this we need this like yesterday. Because it's so damn hard to understand what changed.

Okay. From my understanding, Kubernetes is mostly configured from Git commits, right? So what goes wrong? What's the difference between, you know, Commodore showing this on the UI versus being able to track your Git commits? And what is the augmentation there? Yeah, so GitOps is a nice concept,

but in the end of the day, a lot of things sometimes change. That's one. And the second thing is Git doesn't give you the state of the pods and deployments. And it's not really like... It also doesn't include changes that happen like from feature flags, configuration and so on so it lacks a lot of information

and it's not that easy like tracking down the history in GitHub because it is like the recipe that you wanted you know, you merge to production and you hope everything going to work and this is going to run in production but maybe you had an issue with the pipeline maybe all of the pod crashed why did it crash like what happened and like i think like githubs is a nice point of start but it's simply not enough like all of our users or like a lot of them use argo or flux or other githubs like mechanism but they are but they are running with the same

issues and problems interesting so and that makes sense to me there's so many things that you need like the actual non-repository state for example like an oom crashing your pod certainly that's not going to be in git and but but what would be good is to see you know you start a deployment with this commit and that caused the ooms and your pods to crash and that would be extremely helpful to someone yeah exactly so yeah you just describe like Commodore in a sentence. Cool. And can you tell me about the additional integration? Like why is the integration with, I saw that you have an integration with PagerDuty and like with Datadog. Why are those monitoring and like alerting integrations important?

Yeah. How does that help? Yeah, it's like, we have like two types of integration. One is like changes, lunch darkly and so on The other one are basically like alerting mechanism So it's a question of like cause and effect Commodore tries to help you to understand the connection between the two We don't only tell you that you have an alert or you have a change but we tell you that you have an alert or you have a change, but we tell you that you have an alert

and five minutes earlier, you had a change. And sometimes the change that you had is not in the same service as the alert. So because in Kubernetes, you have a lot of dependencies. So maybe microservice A is talking with microservice B and Microservice B just had an alert, but it didn't change. In the last two days, it didn't change. But Microservice A was just changed like a minute ago or two minutes ago. And Commodore used our knowledge of your system,

of the system topology in our advantage to give you insights more than like you had a deployment that crashed your system, but more of the line of you had a deployment and the application looks okay, but the surrounding services are crashing. So we try to give more insight than like the basic. Okay.

And Kubernetes provides enough information for you to figure out what the service topology is? Or are you doing something interesting with hooking on to traces or something? We do hook on to traces. Jaeger, Datadog, Kiali, New Relic, AppDynamic, I don't know. You name it, we integrate with it. And we take from kubernetes like the basic it doesn't really have like a very good like metadata on the system topology like kubernetes so you can you integrate with kubernetes to get deployment information you

integrate with traces to understand the topology and then you can say you know what like you deployed service a five minutes ago service a talks to service b and service b is having problems it might be related to service a that seems extremely useful um but it also seems like it would be hard to understand like it would be hard to implement something like that so um your system would have to be pretty smart to understand that but still it seems like it would be pretty useful. Yeah, I will say that we raise quite a lot of money. We are on the market only for a year. We raise a total of 25 million dollars just to make sure we work hard so you don't have to. So yeah, it is both interesting, fun,

and frustrating sometimes, you know, building the product because there are a lot of caveats, but I enjoy it and our team enjoy it. So what is something interesting you've learned along the journey that you did not expect when you first started building the product? Like you would have had a hypothesis,

an idea of this is what we should build. But then the more users you talk to, the product. Like, you would have had a hypothesis, an idea of this is what we should build. But then the more users you talk to, the more you realize, you know, there's this other thing which is pretty critical. Is there something like that that came up on the way? Yeah, I think that, like, it might be, like, surprising. But we were focusing really on changes, you know, at first, like, what change and the effect and data dog and like neuralic and so on but people are keep on asking us to help us like monitor the

overall like system health sometimes and basically commodore now allows you not only to see the changes but also the current health status of your pods. And we give insight if you're running, like if you have issues, we try to give you insight on what's the real issue and what's the real status of that pod. And when I started, I thought like everyone already have, I don't know, Datadog or Prometheus alerts, and it's like a solved issue.

But those tools are so complex sometimes that people are just logging into Commodore to understand what's the current status and the system status. Yeah, it seems like there's a lot of primitives given to you, but there's no inbuilt Kubernetes dashboard to show you. This is the best practice dashboard and you can modify it if you want,

but here is something that's provided to you. And it seems like, yeah, that seems like a real problem. Yeah, yeah, exactly, exactly. Okay, and in general, you are building tools for developers, right? And that's the market you're going for. What is something you've learned about, you know, the developer market in general like you are building tools for developers and that's the market you're going for what is something you've learned about you know the developer market in general like is there anything about you know who you're selling to or how developers react that you think was pretty

interesting you found throughout your journey yeah i think the most interesting interesting part is like observability teams which is like a new trend in a lot of organization there are like something between like a devops and i don't know and a developer and those are the teams that are building the monitoring tools for the other developers so it's a team that its only task is basically to help the the developers to troubleshoot faster i really enjoy talking with people on those teams. They have a great understanding of the system needs for the rest of the team. And it's something quite new.

We see it in elite companies, but I really enjoy talking with those kinds of people. And usually they're the ones pushing Commodore to the rest of the organization, which is also very nice for me. But we see how the developers react once they have Commodore and how it's like a weight is lifted from their heart because they can troubleshoot issues that otherwise they need the DevOps just next to them. Yeah. So that makes a lot of sense where you can have this one team or like, even like this tech lead of this team as someone who you can sell the product

to and that person or that team can introduce it to the rest of the organization. I'm pretty surprised that, you know, companies have such teams. Like I can totally imagine a company with a hundred or like thousand developers having such teams what is the smallest company says you've seen that actually has an observability team i think 50 50 wow yeah you know it's like developers or 50 50 developers developers yeah and you know like a team it's like two or three people but they're only responsible is making

sure the other developers are more productive and so it is quite cool yeah and maybe as as a final thought and question like what is the next thing that you're shipping that you're super excited about and you think customers want? Ah, good question. So we are currently on a quest on basically making sure that Commodore provides you with a very good insight on what's the real-life issue with unhealthy pods in your cluster so we are running a lot of like different heuristic in order to make sure that you're that once you have an issue

we help you to understand where to look at and where the problem probably lies it is super challenging we saw a lot of customers asking us, like, you guys are the Kubernetes expert. Can you help me, like, troubleshoot faster, basically, and give me some pointers, some tips, some insights? So I don't really believe in, like, things like AIOps or things like that. I think it's, like, it doesn't really work.

But I do believe in our harnessing our very good knowledge in kubernetes in favor of like helping to to understand what where to look at once you have a unhealthy pod yeah it seems like even like it seems like you could always charge like large, large consulting fees for being a Kubernetes expert, because that's what it sounds like. Yeah, I'll be completely honest with you. A year ago, we raised the money. We spoke with a lot of DevOps, just validate, or developers, or SRA, it doesn't really matter, to validate our hypothesis. And it was like very, very, like a lot of people told me like, leave the startup.

I will pay you like whatever you want. Just come and do a little bit of consulting here. But yeah. There's clearly a problem, right? Yeah, there is a problem. It's not a question. Like if there is a problem, if there clearly a problem, right? Yeah. There is a problem. It's not a question. Like, if there is a problem, if there is a problem,

yeah, you know, like, I try Commodore to be, like, the answer to this problem. Yeah. And in general, have you seen it getting easier to sell developer tools to companies or harder, like, to engineers? No, it's easier.

It's easier.'s easier like companies i think like companies understand developers are expensive devops are even more expensive and we go to a company and tell them we are going to save their developers and devops out of time like sometimes they don't believe us at first and then they try Commodore and then believe us but the money is not really the issue it's more like if your product does like what he says he's doing we're going like shut up and take my money and so yeah that that's super exciting to hear and yeah congrats on your raise and I think everything's going pretty well. It seems like y'all are solving a really hard problem for a lot of people.

So it seems like success is right there in front of you. And thank you again for being a guest on the show. I think this was a lot of fun. I certainly learned a lot about something that I have very little knowledge about. Sure. It was a pleasure to be in here.