The Pragmatic Engineer - How Kubernetes is Built with Kat Cosgrove

Episode Date: May 14, 2025

Supported by Our Partners•⁠ WorkOS — The modern identity platform for B2B SaaS.•⁠ Modal⁠ — The cloud platform for building AI applications.•⁠ Cortex⁠ — Your Portal to Engineering Excellence.—Kubernetes is the second-largest open-source project in the world. What does it actually do—and why is it so widely adopted?In this episode of The Pragmatic Engineer, I’m joined by Kat Cosgrove, who has led several Kubernetes releases. Kat has been contributing to Kubernetes for several years, and originally got involved with the project through K3s (the lightweight Kubernetes distribution).In our conversation, we discuss how Kubernetes is structured, how it scales, and how the project is managed to avoid contributor burnout.We also go deep into: • An overview of what Kubernetes is used for• A breakdown of Kubernetes architecture: components, pods, and kubelets• Why Google built Borg, and how it evolved into Kubernetes• The benefits of large-scale open source projects—for companies, contributors, and the broader ecosystem• The size and complexity of Kubernetes—and how it’s managed• How the project protects contributors with anti-burnout policies• The size and structure of the release team• What KEPs are and how they shape Kubernetes features• Kat’s views on GenAI, and why Kubernetes blocks using AI, at least for documentation• Where Kat would like to see AI tools improve developer workflows• Getting started as a contributor to Kubernetes—and the career and networking benefits that come with it• And much more!—Timestamps(00:00) Intro(02:02) An overview of Kubernetes and who it’s for (04:27) A quick glimpse at the architecture: Kubernetes components, pods, and cubelets(07:00) Containers vs. virtual machines (10:02) The origins of Kubernetes (12:30) Why Google built Borg, and why they made it an open source project(15:51) The benefits of open source projects (17:25) The size of Kubernetes(20:55) Cluster management solutions, including different Kubernetes services(21:48) Why people contribute to Kubernetes (25:47) The anti-burnout policies Kubernetes has in place (29:07) Why Kubernetes is so popular(33:34) Why documentation is a good place to get started contributing to an open-source project(35:15) The structure of the Kubernetes release team (40:55) How responsibilities shift as engineers grow into senior positions(44:37) Using a KEP to propose a new feature—and what’s next(48:20) Feature flags in Kubernetes (52:04) Why Kat thinks most GenAI tools are scams—and why Kubernetes blocks their use(55:04) The use cases Kat would like to have AI tools for(58:20) When to use Kubernetes (1:01:25) Getting started with Kubernetes (1:04:24) How contributing to an open source project is a good way to build your network(1:05:51) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:•⁠ Backstage: an open source developer portal•⁠ How Linux is built with Greg Kroah-Hartman•⁠ Software engineers leading projects•⁠ What TPMs do and what software engineers can learn from them•⁠ Engineering career paths at Big Tech and scaleups—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [e-mail]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

What is Kubernetes? Kubernetes is a tool that makes it easier to manage and scale an application that exists as a swarm of containers. So it will handle things for you like automatically scaling up resources, whether that's networking or storage or whatever based on demand. And it handles that for you without you having to think about it. You define upper limits for a particular type of resource and it'll go up to that as needed. It scales automatically. This is particularly useful if you're trying to keep costs down and keep availability high. The rise in complexity of applications built as microservices is what necessitated something like Kubernetes. You could do

all of this before the scaling of applications running as microservices in clusters or swarms. Before we had Kubernetes, it was just very difficult and it was very manual. Kubernetes is just an abstraction on something that we used to do manually. Kubernetes is the second largest open source project in the world after Linux. So what is it? Why is it so popular and how is it built? Today I sit down with Cass Cosgrove, who is the leader of the Kubernetes release team sub-project. Katz started off as a software engineer specializing in embedded Linux and got involved with Kubernetes by learning K3S, a minimized version of Kubernetes that is designed for smaller memory-limited embedded devices. Kat was a contributor to Kubernetes for years before becoming a lead on different parts of the project.

In our conversation, we cover the architecture of Kubernetes. and what Kubernetes pods are, why Kat thinks Kubernetes won and became the de facto infrastructure management solution, in part because of its strong focus on documentation, how to propose changes to Kubernetes via the KEP process, and how Kubernetes release happens in the span of 12 to 14 weeks, and many more topics. If you're interested in backing infrastructure or how well-organized large open source project operates, then this episode is for you. If you enjoy the show, please subscribe to the podcast on any podcast platform and on YouTube.

Kat, first of all, just welcome to the podcast. Thank you for having me. Unless you're a Kubernetes, well, an infrastructure person who actually deals with Kubernetes day and day out, most soft of developers, including myself, I've never dealt Kubernetes directly. However, so many of the systems that I deploy to, a lot of the APIs that run today, are probably on Kubernetes. I'm pretty sure, for example, use render. I'm assuming they're going to use Kubernetes.

at Uber, we use Kubernetes, and apparently they're one of the biggest deployments. What is Kubernetes? What is the relationship with Docker? Or if there is any? And as an average developer, when would you be exposed to it, if ever? So I think the average developer, especially if you're like a web developer or an application developer, you don't need to worry about what Kubernetes is. You're never going to have to touch it directly.

and it's a specialization. Being like a Kubernetes cluster admin or an SRE or something, that's a specialization. And I don't think that the average developer needs to know or care what Kubernetes is. You certainly don't need to know how to use it. You can know what it is, but you don't need to know how to use it. There's so much going on. It's complex.

It's notoriously difficult to learn. I don't think that there is a reason to dive into it. super deep as an average developer, unless you're trying to make a career switch into Kubernetes, which I couldn't blame you for because it's very lucrative. But Kubernetes is a tool that makes it easier to manage and scale an application that exists as a swarm of containers. So it will handle things for you, like automatically scaling up resources, whether that's networking or storage or whatever based on demand. And it handles that for you without you having to think about it. You define upper limits for a particular type of resource and it'll go up to that as needed.

It scales automatically. This is particularly useful if you're trying to keep costs down and keep availability high, for instance. And it looks, so this is direct from the Kubernetes documentation and this is what Kubernetes components look like. This is a very, very simple cluster. This box over here on the left is your control plane, and you aren't going to interact directly with this a ton. This is mostly automated. Over here on the right, where it says node, node, node,

those are all pods. And within these pods, run the Cublet, which is another little control thing. It's interacting with the Kubernetes API to stand up and down all of your containers. And all of your containers exist within here too. So if one goes down, like a pod fails for whatever reason, whether the code itself, the application crashes,

or it runs out of resources or something else weird happens, your deployment will stand it up again. automatically for you. You won't notice it. Your users won't notice it, probably. This episode was brought to you by WorkOS. If you're building a SaaS app, at some point your customers will start asking for enterprise features like Sammel authentication, skin provisioning, and fine-grade authorization. That's where WorkOS comes in, making it fast and painless to add enterprise features to your app. Their APIs are easy to understand, and you can ship quickly and get back to building other features. WorkOS also provides a free user management solution

called AuthKit for up to 1 million monthly active users. It's a drop in a replacement for Alt Zero and comes standard with useful features like domain verification, role-based access control, bot protection, and MFA. It's powered by Radix components, which means zero compromises in design. You get limited as customizations as well as modular templates designed for quick integrations. Today, hundreds of fast-growing startups are powered by WorkOS, including ones you probably know, like Cursor, Versal, and Perplexity. Check it out at Workoauts.com to learn more.

That is workOS.com. This episode is brought to by Modul, the cloud platform that makes AI development simple. Need GPUs without a headache. With Modul, just add one line of code to any Python function and boom. It's running in the cloud on your choice of CPU or GPU. And the best part, you only pay for what you use. With sub-second container starts and instant scaling to thousands of GPUs, it's no wonder companies like Suno, Ramp, and Substack already trust Model for their AI applications.

Getting an H-100 is just a PIPP install away. Go to modal.com slash pragmatit to get $30 in free credits every month. That is M-O-D-A-L.com slash pragmatic. And then so how does a pod and let's say a virtual machine connect? Or like in this case, when we say container, it's probably a virtual machine, right? Nope, there is a difference between a container and a virtual machine. All right. Containers are virtualizing the operating system and applications,

running within it, a virtual machine is larger and beefier, and it is likely also virtualizing hardware. So they are, like, they're different types of virtualization, and Kubernetes relies on many types of virtualization. Virtual memory is probably the most recent, like, I don't know, breakthrough in virtualization, I guess, and we crack that in, like, the 60s, but that's, that's what allow, like, things like containers to exist, virtualized memory. But yeah, a container is a type of virtualization.

A virtual machine is a type of virtualization. It's just much, much more freer. You can run Kubernetes inside of a VM. Like, there's nothing stopping you from doing that. But containers are not VMs themselves. They're smaller, more lightweight. They're easier to share around. They're much much longer.

And they're more configurable as well, right? Yeah. Yeah. Yeah. They're, you can just. Chuck one. Like the popularity of Docker is largely because of the shareability of Docker files and things like the Docker registry.

And now there's a million and one container registries. But shareability, the small compact size of containers is what makes them so attractive. It's why we generally, not always develop applications as microservices now rather than monoliths. That's the containers is what allow us to be. do that. And the rise in complexity of applications built as microservices is what necessitated something like Kubernetes. You could do all of this before the scaling of applications running as

microservices in clusters or storms before we had Kubernetes. It was just very difficult and it was very manual. Kubernetes is just an abstraction on something that we used to do manually. And now since we you're automating a lot of it with a very, very complex overlord like Kubernetes, we can scale our applications much larger, much faster. And we can build much more complex applications as much more complex chunks of microservices instead of having to throw a monolith in here or there. So this is like most advances in technology since the ENIAC machine in the 1950s, Kubernetes is just an abstraction layer on something that we used to do by hand that sucked,

that was hard to two by hand, and now we've automated it. And the trade-off there is, it's when you dig into it, it's confusing. It is confusing. Can we talk about the history? What happened? What was before Kubernetes? And as I understand, the roots are somewhere with Google, right? Because I think it's good to understand what is it that, you know, either cisadmins or other people

we're doing and then, you know, why, why we decide or why some engineers decided like, all right, let's actually build a better tool. And now the whole world just seems to have caught on saying, actually, yeah, we actually want more of this. Yeah. So Google has a history and a reputation of overengineering things, right? They, they do. So internal to Google, a long, long time ago, there was a tool called Borg. That is a Star Trek reference, right? You will be assimilated. It's still around. still around. And Borg was used internal to Google to manage their own clusters, their own,

their own microservices. And somebody decided one day, ooh, this is really good. I think we're on to something. We should open source this. And so they did, and they talked to the Linux Foundation who stood up the Cloud Native Computing Foundation as a directed fund of the LF. And Kubernetes was the first project, to the cloud native computing foundation.

That was in July, almost 11 years ago, and the first commit was by Joe Beta. A fun, fun thing on the name Kubernetes. It means helmsman in Greek, and a typical ship's wheel, our logo, has eight spokes, but ours has seven. As a reference to seven of nine from Star Trek, it's like a nod. to Borg, the original name. Yeah. Yeah.

But it took off pretty quickly. Before that, there were other, there are other ways to manage clusters, right? There's Mesosphere and there was Docker Swarm and a couple of others. Like, this was very clearly something that the market wanted, that developers needed, that SREs needed, that we didn't really have SRE as a term back then. Back then. We certainly do now. SRE is just like a super fancy SIS admin now.

But yeah, it was born out of Google solving their own problems and going, you know what? No, people need this. And they decided to ship it for free. And now we're the second largest open source projects in the world behind Linux, which we rely on. And as a fun history, I talked with Dave O'Connor, who was one of the first SREs inside of Google when it was internally called like that. but externally the term wasn't popular until much later. And he said this interesting story of how at the time when he joined Google, they already were managing thousands or tens of thousands of servers.

And the problems that they were facing is they went to like a hosting provider where they wanted to host their servers. And they told them how many they need to manage. And the hosting provider was used to manage maybe 100 at max and it was very manual. And then he said that they just needed to create the processes, the tools, etc. So is it fair that the understanding is that we had this explosion in both physical servers, then we had virtualization virtual servers, Google figured out how to do it. And then they just turned around and said like, okay, let's open it up.

Pretty much. Yeah, that's pretty much what happened. What? I mean, right now, I think it's a really positive thing on Google. But why do you think the motivation might have been for Google to, it is amazing that they did it? And the industry has greatly benefited from it. But they could have just said like, okay, let's just keep doing what we're doing with Borg and not bother too much. Because they still have Borg, right? They do still have Borg. As far as I know, they still have Borg.

I haven't asked a Google engineer about the state of Borg in a minute. They have. I recently asked. As far as I know, it's still a thing. Do you mean like why didn't they just like productize Borg and sell it? Either that or just why did they think this would benefit the industry and maybe also their business? sure, I mean, now that we have Google Cloud, like, I can see how, for example, you know, Kubernetes might have indirectly helped that. But I'm just thinking, like, just from a selfish perspective, this is a really unselfish thing from a corporation to do. I'm trying to understand that there's

usually a longer term motivation potentially. Or maybe it was just engineers being geeky and thinking this is really, really helpful. I know those engineers quite well, and they are massive dweeps, and I love them. But. Awesome. So, like, yeah, I totally get how people would think that open sourcing something as successful as Kubernetes may not be like a smart business decision. But look at the sheer amount of influence Google has gained over the cloud native space over that. They still have people, they still have employees sitting on the technical oversight committee for the CNCF. They have Linux Foundation governing board members.

They have CNCF governing board members. They have people, I don't think they have somebody sitting on, oh, no, they do have somebody currently sitting on the Kubernetes steering committee. They still have a degree of control. The way we govern ourselves as a project, no one company can have more than two representatives on the steering committee. So they can't rest control of the project from the people anymore, like any more than AWS or Microsoft or Red Hat can.

But it has given them a pretty extreme. degree of influence. I mean, we're here on this podcast talking about the fact that Google gave us Kubernetes. Google's directly responsible. That's fair. I'm trying to understand because I think it's great that this is happening. And I think the more we understand what the benefits are and why other companies potentially should follow. For example, Spotify did with backstage, their developer portal. And what they told me is they think it will make it cheaper on the long term for them to actually maintain themselves because now they're going to get other developers from other companies or independent engineers bringing the ideas and also standardization is is another thing.

Yeah, if your project is good, people will come contribute to it. I, we open source maintainers have mixed feelings about this, but it is a way to get free labor, right? Nobody pays me to maintain Kubernetes. It's not my job. I don't have a cloud provider. writing me a paycheck for running the Kubernetes releases. So we do do all of this work for free.

Another benefit is that open source software is inherently more secure than closed source proprietary software, right? That security angle is real hard to argue with. Kubernetes just addressed a nasty CVE very recently in Ingress EngineX. And that would be much slower to address on. proprietary software with a smaller team that can't work around the clock, right? You can't force your employees to work around the clock if it's a small group of people that are all in one time zone.

An open source projects, especially one as large as Kubernetes, we can respond around the clock. And how large is Kubernetes both in terms of the widespread in the industry, the complexity of the software and the team, as you refer to? Kubernetes, first I think I need to explain the different types of people involved in an open source project. Generally, you have users who are people who just consume your tool. You have contributors who are writing code or writing documentation or in the case of the release team doing some like project management type of role. But they don't have any control over the future or direction of the project and have like maybe limited permissions within repositories or orcs.

Maintainers have titles within a project. So Sigdocs technical lead, release team subproject lead are both titles. You exist in an owner's file and you have control over governance and policy decisions about the project that affect everybody. There are a couple dozen SIGs within Kubernetes, all with two to three chairs. and two to four technical leads, those people would all be considered maintainers, then sub-project owners exist below that before you get down to regular contributors. So there's probably, it's a large project.

And everybody in SIG leadership is a project manager functionally. Like we're all doing administrative work. We're all managing people. We're all affecting the direction of our various areas of the Kubernetes project. And there's probably 150, 200 people who would be classed as maintainers and then more than a thousand contributors a month. It is a month. Wow. It's massive. So about like a few dozen, let's say like 20 to 40 sig leaders, about 150, 200 maintainers and like a little pyramid almost. And then the contributors who are, I'm assuming contributors, some of them will be regular, some of them

We'll be just one off. Sometimes we get drive-by contributions like any other big project. And some people just stick around as contributors. And you go through what we call a contributor ladder. So you show up as a contributor at first. You work as a contributor for a while in whatever area interests you most. And if you stick around long enough and you want it, you will eventually become a maintainer. You'll end up leading a sub project or you'll end up as a technical lead or a Cig chair. That's what happened to me. I was on the Kubernetes release team for three years. And then I led a release myself, Kubernetes version 130 was my release. And after that, the SIG release chairs offered me the sub-project. And now I just keep running them and now I'm a maintainer. And because I had run

the docs sub-team for the Kubernetes release a couple of times, SIG docs asked me to be a technical lead. And that's just, that's how it works. Anybody can show up. Anybody can open an issue. Anybody can, um, open a poll request and dive in and help out. You don't even really have to be a Kubernetes expert to be useful. You just, you just have to know how software works or know how to write documentation or, um, know how to manage a team. And then so Kubernetes, widespread wise, as I understand, most of the large infrastructure, runs on Kubernetes, if it's large, it's the one that

makes the most sense. I'm not even sure there's a competitor that is open source. That is equally capable. Not that I would say is equally capable. There are certainly other ways to manage clusters. Docker Storm is still a thing. Mesos is still a thing.

But yeah, like your people are most commonly using some kind of managed Kubernetes service. like anything on AWS or GKEE, right? Or like Azure has their own service too. I'm pretty sure Oracle does. Red Hat certainly does. And then there are flavors of Kubernetes like VMware's Tansu, right?

And so given that this project is so successful, how does the funding work? I'm going to assume that some of the contributors, as you mentioned, are actually paid full-time working at certain companies. who have critical infrastructure running on Kubernetes, and these companies figure, well, it's cheaper for us to pay a contributor to ensure that our use cases are met. But for example, in your case and some of the other contributors, how does it make sense to spend a considerable amount of your time, which, you know, like we're talking about software engineers who are highly skilled spending here.

And there will be some goodwill, but what are other ways that the project remains as sustainable as it is. Unfortunately, for a lot of us, this is both our, this is a hobby, right? Like, I, I enjoy it. Many of my friends are other maintainers or other contributors. Like, some of my closest friends are also maintainers.

I flew to New York just to hang out with two of them a couple of weeks ago. No reason. Not work. Just hanging out with them. And that's, that's not too terribly uncommon. Like, think it's fun. Some people also do it as a career bump. This is something that the Kubernetes

project doesn't really love to admit, but we are a very, very large and very, very influential project. And having it on your resume can be a career bump, especially for people who are fresh out of college or very early in their career, which is why shadowing on the Kubernetes release team is such a popular entry point for contributors. We get hundreds of applications for the Kubernetes release team every cycle and about half of them are students. And that's, that's, that's pretty common too. Just just they want the, they want the career experience. That is encouraging to hear, by the way.

Yeah. It's awesome. It's a lot of students. It is so many students. And we do, we do get students onto the team. Obviously, I can't overload the release team with people who have no experience. But we, we try to fit a handful of them in every single cycle, which, which feels good. It's a good entry point for people's,

We get to watch them grow up through the project until ultimately they go get a job and leave and we never hear from them again, which is fine and doesn't make us feel bad because we really have more contributors on the release team side than we know what to do with. We cannot have them all on the team every cycle. But as for people who are paid for it, that's usually cloud vendors, actually. Like the big clouds pay a lot of people just to be contributors or just to be maintainers, either because they need something done or they want the influence or because they want the clout. Some people get hired by some company just because they have a maintainer title within the project. And that's it. It's purely clout.

And I don't object to that either, as long as somebody's getting paid to do this instead of doing it for free. But the CNCF doesn't pay us. The Linux Foundation doesn't pay us. The CNCF does pay our infrastructure costs. So, like, we're not starting a GoFundMe or putting money out of our own pockets to pay for our, like, testing or whatever. The CNCF pays that. That's part of what you get when you donate a project to the CNCF. They take on some of your infrastructure costs depending on where you are in the, like, sandbox incubating graduated scale.

But, yeah, most of us do this for fun. which is not the most healthy thing in the world and you have to set like really, really, really firm boundaries. Well, yeah, there's that thing that comes into mind. And also there's this famous XKCD comic where there's this big pile of rocks and there's a small thing which says someone's side project. So given how critical Kubernetes is becoming globally, it's a, I think it's a bit of a mixed feelings. On one end, I think it is cool to hear that it sounds like it is a very independent. independent project that doesn't really have any one vendor going over it. On the other, there is this downside of like, well, you know, like, yeah, it just just makes feelings.

It makes people, it makes me wins for small projects, but everybody starts as a small project. And however, like Kubernetes is so large and so influential and we have so much momentum. That XKCD comic is not going to become a problem. There is a, there is a queue of people. waiting in the wings who would love to replace me. And part of our, we as a project, our philosophy is that part of our job as maintainers is to be looking for somebody to mentor into replacing us. We have a lot of anti-burnout policies in place. Leading a release, like being the actual point person for a release, the person who ultimately

reports to me is a very stressful, very public, miserable job to do. It is 14 weeks of having to manage an un- 14 weeks, that's the release cycle. That's the release cycles. Yeah, like 14 to 16 weeks. It depends on when it is in the year. But 14 to 16 weeks is about how long it takes us. And it's 14 weeks of directly managing an unruly group of like 30 people

who are trying to manage interactions with potentially hundreds of contributors between them. And it sucks. And by the end of it, you are so tired. So we don't allow you to be involved in the release cycle after you lead one. You've got to take a cycle off. Also, SIG release's charter dictates that we will delay a release. We will push it back before we make our team work nights or weekends. We're never going to ask you to work more than you are comfortable working.

And we're very supportive of stepping back if you can't handle it. It doesn't affect your place in the project. You can come back the next cycle or whenever you're ready. We're very encouraging of taking breaks. So we don't really like to allow the burnout to build up to a point where we might topple over like that XKCD comic. Yeah. We're always replacing each other. So eventually I will replace one of my cigarettes.

This part is just very encouraging to hear. And this is something that I would assume that a lot of commercial companies, like a commercial setup, would, let me just say, hesitate in many cases, again, because of that thing. So I guess maybe we're seeing two parts of it. But this is just, it's the first time I'm actually hearing for such a large, such a critical project to actually have it in place, which is just music to my ears. We're unusually well structured for an open source project. We're unusually well structured for like an actual product within a company.

Actually, we are very, very well organized. by necessity because we're massive and we deal with so many people coming in and out constantly. We have no choice but to be super, super well organized. So why do you think Kubernetes, you've seen this story mostly from the inside and starting from the side of why do you think Kubernetes one so big? Why has it become the de facto choice for any, from any medium size, sometimes even small size infrastructure by the way. I've talked with linear and they chose Kubernetes very. early on. Apparently, they thought, you know, some people thought this would be like

overkill, but it actually turned out to be a great decision for them. Yeah. What do you think this project specifically won? What did Kubernetes do so well? I think we caught on early because of because of hype, right? Because it's the Google name brand behind us and the association with Docker, which was already very popular. It was a familiar company donating a project that relied on a tool many were already familiar with, and that got us initial hype cycle, right? But then I'm going to toot my own horn and say that the continued popularity of Kubernetes

is at least in part due to our truly exceptional documentation. If Kubernetes does something that you can touch as a user, as a cluster admin, we have documented it. Kubernetes uses something in the release cycle called a Kubernetes enhancement proposal, a KEP. This was inspired by Python's PEPs, right? And one of the things we require for a KEP to be considered complete and thus includeable in a particular release

is that if it has a user-facing change at all, even if it's just a feature flag, it must be documented or we do not allow it in the release. Today, as we're recording this, is actually the docs freeze for Kubernetes version 1.33. So today, a lot of cap owners will either merge documentation or I will revert their PRs. Wow. That sounds like tyranny.

Does it? Does it? You're never left digging through Kubernetes code to figure out like what this is. It sounds amazing as a user and as a developer. I'm cringing. You know, as a dev, like you really just don't want. There's so many excuses we make up for not writing documents that code will. document itself, if it's these documentation, it's not for rich and et cetera.

No, I'm really impressed. Yeah, I know the devs hate it. I know. But our priority isn't making the devs happy. It's building a usable, sustainable project that people want to keep using. That means we got to document that. I hated writing docs when I was an engineer, too. I did.

I've slung some truly horrendous code. that was entirely undocumented and I myself looking at it probably could not tell you what on earth it's doing so we can't have that in a project this large you know yeah and the project is this complex right so it sounds like you're managing to balance because what I understand is Kubernetes

is just to become more and more complex as there's it does you can tell more about it but there's going to be more more edge cases more hardware more set up more exotic this and that but when you when you have proper documentation at least that helps with it. If you don't have it, there's no chance. Yeah, there's absolutely no chance.

And so that has gotten us over the hump of Kubernetes is hard to use. It is hard to use. It's hard to learn. But it could be so much worse. And the availability of good documentation is, I give a lot of credit to that. Also, like, GKE has really great tutorials for learning Kubernetes. GKE is Google Kubernetes Engine, right? Is their host a solution?

Yeah, it's their host of. hosted solution. And they, they absolutely have the most, like, feature-rich Kubernetes hosted option by virtue of having had, like, several years of a head start on that. But yeah, there are, they're tutorials. They're just like, how to learn Kubernetes are actually quite good for, like, just getting your, dipping your toes in the water. And our own documentation, our own quick start guides. That's another benefit also of being an open source project. Like, just like an open source project, the software is more secure, our documentation can be better. We have a small army of people contributing to it and improving our documentation on a constant basis. So we

don't really tend to slip. I'm not sure if this applies, correct me if I'm not, but a lot of open source projects say, like, look, if you want to start contributing and you're not sure just yet, you can always just start with the documentation. Yes. Like it's an incredible, which is win-win. Yeah, docs are like a pretty typical entry point for people who are like nervous about contributing to open source for the first time, which is a fear I absolutely get. I was terrified the first time I opened a PR in a Kubernetes repo. It's nerve-wracking, especially if you're super early in your career, you know that you've got all of these people with way more experience than you like laser focused on what you just did. And like, did you apply a label incorrectly?

Are you going to get chewed out for that? But yeah, docs are an easier entry point for that. It feels like a little bit less pressure. And that's pretty common across open source projects, not just Kubernetes. The unique thing about Kubernetes and getting started is the release team. We will let absolutely anybody apply to be on the release team. And it's, you know, 20, 25 spots every cycle three times a year. So it's pretty competitive.

but we will let absolutely anybody apply. You don't even, you don't have to have contributed to Kubernetes at all before. Every cycle I've got to add two or three people to the org because they've never contributed before. This is their first time. But just emphasize, do I understand correctly? For example, we recently had an episode with Linux where in order for you to become a, well, to lead or release it, you will need to put in many, many years outstanding contributions

and so on. And there's no guarantee of doing so. Correct. But for other open source project, it is also common to have this kind of tenure-based thing because what you've said, I've not heard it. And any other projects say, like, we will actually open up the spots and you can apply. And taking people with little its own experience sounds like an approach. I've just not heard before.

Yeah. The application process we have and the way the release team works is unusual for an open source project. No, tell me more. So we talk about that a lot. I will absolutely talk about the structure of the release team. So the Kubernetes release team has two halves. There's the release team, which I manage and release engineering. Release engineering is doing the actual cutting of the releases. But the part of the team I manage is made up of several subteams and a release lead.

The subteams are communications, which is responsible for handling things like feature blogs. So that's working with cap owners or SIG leads to make sure that a particularly interesting feature that will make it into that release has a blog that goes out on the Kubernetes blog on a certain schedule. Oh, nice. Yeah, it's very cool. We usually have like between 8 to 12 every cycle that end up. And release comms does not write that blog. They're just responsible for managing that it happens. They're also responsible for managing interactions with the Cloud Native Computing Foundation to speak.

schedule the release webinar and with the media to handle the webinar. Yeah. We have a release webinar. We also have to manage the media to deal with press embargoes and schedule media interviews for the release lead and anybody else on the release team. This sounds like a publicly traded company doing its earnings call. It does, doesn't it? Yeah.

We also have released docs. I'm so impressed. Yeah. It's a lot. Release docs manages making sure that every one of those caps that has user-facing changes has documentation by its deadline. Enhancements manages all of those CEPs and make sure that they have met all of the requirements, the checkboxes and the cap, which is like has the lead opted in, is your code complete, is it tested? Did you get

a production readiness review? That kind of thing. And release signal, which is responsible for watching our CI signal boards and chasing down CIGs and cap owners to squash bugs to make sure that we don't have like a master blocking test failing right before we cut like beta two, right? They're also responsible for the go no go signal on whether or not we can cut a release. And this requires 20 to 30 people every single cycle. Five of them are leads and are selected. Then the rest are shadows that go through the application process and get pulled in. That is unusual. Most open source projects do not operate that way.

Most open source projects, I think, don't need to. That's a consequence of the fact that the project is so large and we have so many contributors. We couldn't manage this with a consistent group of like five to ten people without catastrophically burning them all out. We just simply couldn't do it. And even with the rotating cast, the release team can be a burnout factory because it's so much people management. It's so much project management. It's so much arguing with people and juggling, like maybe conflicting priorities between teams. So we don't like people to stay on the release team

for too long. We want to see you move up through the ranks a little, maybe lead a subteam, and then leave, find somewhere else in the project to go. Because if you stick around, you will burn out, right? I did not, obviously, which is why they gave me the bigger hat. It is very interesting for me to reflect on how much project management. and just how much work management is happening within Kubernetes, whereas when we did the deep dive with the Linux kernel, it was a bit of the opposite. But again, it was pretty clear there that in the case of Linux kernel,

a lot of this work that you're mentioning is happening at the actual releases, let's say, you know, like Red Hat and whoever picks it up. But I think it's just a good reminder of like this work for any large project. It's there. You could have shifted here and there, and you can play with it, but. It has to have. happen. Like, it would never, no, no project with more than, like, two people contributing to it is getting by without some kind of project management work. It's just that we don't have the title project manager. All maintainers are project managers to some degree. Like, we, we all have some

administrative or, like, governance policy position within the project. It's, it is project management. If you are a maintainer, we just, we just don't call them that. Yeah. So, like, the Linux kernel has them too. Like somebody is doing that work. They're just not, they're just not. They don't like that word, I think. That was my sense. Yeah, they don't.

I think a lot of, a lot of engineers are resistant to doing work that they don't, they don't consider real engineering, you know? Yeah, but we're kind of seeing this, by the way. Like now, you know, like software engineers, like we don't like to say that if we do product work, but now product engineer is becoming a thing, which is a software engineer.

who will talk with customers as well and who, and you know, same, same thing. I think there used to be a time, not these days, but maybe 10 or 15 years ago. Testing was, we had dedicated, you know,

testers or whatever. And then engineers wouldn't, engineers wouldn't or would object to do testing. But then, you know, we would write unit tests and then integration and autumn. And they're dismerged. So I think there's clearly a play with worse.

And what I'm personally seeing is, is a competent tech professional, May that be a software engineer or someone who was elsewhere. We'll just take on more and more responsibilities. A great product manager these days will know how, we'll understand code. They might not code by choice and not. Same, same thing. But even just talking with you, you've been a software engineer.

You've actually built like critical infrastructure. And now you're doing something where we're doing less of that. Yeah, I haven't written a line of code in months. I don't have a reason to. You could if you wanted to or if you needed it. I wanted to. Yeah, I felt like for fun sometimes. I'll do some hobby projects at home or whatever, but I don't have a reason to write code. I used to be an engineer. I can read and understand it just fine. I can write it if I need to. But it's not part of my job anymore. And like we, you see this within engineering at real companies, right? The more senior your engineering title is, the less code writing you're actually doing. A principal engineer should not have their fingers in the code nearly as much as somebody who's like an SD E2, right? Like, they don't. They're making architectural decisions about the product, or they're mentoring juniors. So, like, same deal within an open source project. I'm the open source project

equivalent of a principal engineer. I don't write code anymore. I make decisions about how we get this thing out the door. This episode is brought to you by cortex.io. Still tracking your services and production readiness in a spreadsheet. Real market service is named after TV show characters. You aren't alone. Being woken up at 3am for an incident and trying to figure out who owns what service, that's no fun. Cortex is the internal developer portal that solves service ownership and accelerates the path to engineering excellence. Within minutes, determine who owns each service with Cortex's AI service ownership model, even across thousands of repositories. Layer ownership means faster migrations, quicker resolutions to critical issues like Lockfordj and fewer adhere pings during incidents.

Cortex is trusted by leading engineering organizations like a firm, trip advisor, grammarly, and so-fi. Sol service ownership and unlock your team's full potential with Cortex. Visit Cortex.io-pragmatic to learn more. That is C-O-T-X.I-O-S-Pagmatic. Yeah, we used to have a joke internally within, I think, my team or my broader team back when I was, as you were up, it just went around. What's the difference between a junior or a junior? entry-level engineer, a senior engineer, and a staff engineer. It's like, you get a code review, the entry-level engineer comes in and says, oh, you know,

the order is not alphabetized as per our rules. Like, we should change the order of the variables. Senior engineering goes in and on the same code line leaves the comment. Actually, we should probably move this out into its own class because it should be encapsulated. The staff engineering comes in. Why are we even doing this? There's a service that is exactly this thing. You should be using that one.

Yeah, don't read. Don't reinvent the wheel. And, you know, the thing goes back to that the earlier career, you should, you should focus on the code for sure. But after a while, you need to see the bigger pictures. And I often saw some of the most impactful engineers on my team in a given, let's say, quarter if I had to use a business value, they did not by writing code, sometimes by deleting code or sometimes pointing out the work we should just avoid by, I don't know, like merging with something, just saying note, pushing back on product, et cetera, et cetera, et cetera.

Yeah, no one is to say, it's still, for sure. Yeah, but it's a baseline, right? Like, you need to know that, but there's a lot more afterwards. So in Kubernetes, if I'm a software or a contributor who has an idea, it's contributing to some code, what does that look in terms of release? How will it eventually get released? That's one part of my question. And the other part is, is does Kubernetes work?

work in a similar way to some other open source project where you kind of open up and you depend on what contributions are or do you have a roadmap saying, hey, we'd like to encourage people to build this functionality or solve for this problem? Yeah. So Kubernetes, like I mentioned earlier, has a process called a KEP. Kubernetes enhancement proposal. And if there is a feature that you want within Kubernetes, you go open a KEP. It's just an issue in the Kubernetes Enhancement Proposal or Enhancement's Repo, and there's a template for it. You go open your KEP and then a discussion will ensue with whatever SIG owns that. So if it is something within, I don't know, if it's a storage thing,

Sig Storage will own that KEP and they will come weigh in and we'll have a discussion about whether or not this is actually useful, whether or not this is viable, whether or not we have the resources to try to get this done right now, which is going to depend on other contributors looking at your proposal and going, ooh, yeah, I would like to work on that. And when you say discussion, is it email, is it async? Is it actually like video call? So because we're an open source project,

we prefer to do everything in public in a place where everybody can see it. So that discussion will happen on the GitHub issue or in Slack in a public channel. We very much discourage things from happening behind closed doors unless it's very sensitive, like extremely sensitive in a security type of way or it could like negatively impact somebody's career. So like code of conduct stuff is always handled behind closed doors. But discussions about features, that is all very public. GitHub issue or the Slack channel for, for instance, take storage.

If the decision is, yeah, we would like to do this, then. then work begins. Your Sig lead will need to opt in. It's just adding a label, Sig lead opt in to the KEP, and that tracks it for inclusion in a release, at which point you have a whole bunch of deadlines that you have to hit. Code freeze is the big one, obviously.

That's, is your code in? Then if it needs documentation or whatever, there's documentation freeze, there's test freeze. These deadlines all exist across the span of 14 weeks. And if you hit all of those marks, your feature is included in the release as an alpha at first. We do allow iterative stages. So you don't have to get the whole thing MVP out the door with your first alpha. You can have an alpha too.

Alpha features are off by default, but can be turned on with a feature flag. Once you meet certain requirements to graduate your feature to beta, you're expecting minimal architectural changes to the way it functions and interacts with the rest of Kubernetes, we'll graduate you to beta, and you are on by default, but can be turned off with a feature flag. Once it's very stable, very usable,

and you feel like you're ready to go GA, we graduate- In general availability, right? Yeah, yeah, we move it to GA in the next release, and it is on by default. and cannot be disabled. And then by that time, feature flag is removed, right? Yeah, there's no feature flag at that point.

And at any given point in time, roughly how many feature flags does Kubernetes have? Is it going to be in the hundreds or thousands? At minimum dozens, I can go look. Let me share my screen and I will show you all of the feature gates we have. I love it. This is what I love of an open source. You couldn't do this with commercial close.

software. No, not at all. So all of our feature gates are documented. This is usually all the documentation that an alpha requires because it's just like turning a thing off or on. You can see here what the default is, so whether it's functional, whether it's alpha or beta, when it was introduced and until when some of these are, you know, deprecated and will be removed in the future. Some of them haven't been graduated since then, but there are quite a lot. It doesn't look like hundreds, but it might be a hundred. It looks, wow. And so those are all features that are big enough to eventually,

some of them will graduate. Hopefully most of them will graduate eventually. Yep. And these are all graduated or deprecated and will be removed at some point. Yeah, so it's quite a lot. We talk about what the feature stages mean here, like really every, every aspect of this project is so well documented, even the way the release team runs, the way the project is governed is all documented. We don't keep anything as like tribal knowledge. We all, we write everything down so that there's no arguing about it.

But yeah, there's a ton of them. And also, you documenting this much, it's just gives us. me the idea that if anyone wants to get a, you know, a bit of a cheat of like how a well-run project this size runs for further inspiration. Yeah, you can just check it out. You can rip off the ideas that make sense. And also you can adjust it for for your size or for, again, you can experiment, right? I feel you always want to experiment, but getting inspiration of something that works is a good starting point. Yeah, please do. I mean, and just like, you know, not, not all products, not all applications, need to run on Kubernetes at first, that can very much be overkill

from an infrastructure standpoint. Not all open source projects need to be run like Kubernetes to begin with. There are some things that are just like good practice that you should ingrain from the very beginning. Like if it's a user facing change, it needs documentation. But you don't need like a structured release team the way we have it for your four person open source project that is releasing like one minor update a year. That's, that's unnecessary. And you've got like a couple dozen users. But for something the size of Kubernetes or close to the size of Kubernetes, yeah, that's super useful. Rip us off. Absolutely rip us off. That's part of why we document it all. And it's, exactly. This is beautiful like having this, this win. And also you don't really need to explain

it again and again. It's a, you can just point to the dogs. Yeah. The, the way every role on the release team has to be run is exhaustively documented. So every, single role has a handbook that goes week by week and tells you what to do, which is part of why our releases are so smooth. If something goes wrong, probably it's documented and we know how to solve it. Of course, every release, we get something new and exciting that goes wrong to document, but that's happening less and less over time as we run out at edge cases. So. Now, one thing I'd like to ask you, because I'm really interested, on how I was going, how do you think the Gen AI tools,

LLMs that more and more engineers are using to write code, help them write code, will impact how Kubernetes is developed. Because it's so extensively documented on one and I'm sure, you know, like these tools can ingest a lot of things. I wonder if they could be useful. Are you seeing them being useful in, let's say, using features

or for the release team to build your own tools to do some of your work? easier? Okay. So my personal opinion, I'm going to stress the word personal there, is that the overwhelming majority of Gen AI tools are a scam. They are a scam. They are designed to filch money out of VCs or the public or whoever else. The people fund the VCs, the liquidity partner or LPs. Yeah. The investors. I think most of them are a scam. I think a very small minority of them are genuinely useful, and the rest are just, it's a nothing burger. As a maintainer within SIG release, we have no use for generative AI or LLM tools. Most of the issues we are dealing with are people problems.

Gen AI and LLMs are not going to resolve that. We're people managers. That's what we're doing. So generative AI, not useful there. with my SIG docs hat on. We actually explicitly have a policy if you are not allowed to use tools like that. We caught somebody submitting a blog

that had been written with some kind of chat GPT tooling and we pulled it and told them not to do that again. We got a drive-by PR from a paid product that claimed to simplify writing documentation, maintaining documentation, and handling things like style. guide compliance. Kubernetes does have a documentation style guide. We got a drive by PR from them that stipulated, hey, if you accept this, we grant you a free license for this tool, as long as we can use

your logo. But here's the thing, it didn't work. It was making changes to a bunch of generated files that a human should never touch, and it wasn't actually enforcing style guide compliance. It made a bunch of mistakes. So even when the tool sounds useful, it is often not. Like, it doesn't actually do what it says it's doing on the 10. And also drive by PR in exchange for a logo on a paid product. That is that's really poor behavior. If you own a product and you're listening to this, do not do that. Please. We hate it so much because it creates more work for us. Because we have to have a discussion as leadership about what we're going to do with that. Like, you caused three meetings.

Please don't do that to me. However, where I do want to see it is in things like, that are just toil for us. So things like application of GitHub labels on issues or PRs. Like, I would like something to automatically, if the wrong language label gets added to a PR, I would like a tool that removes that label and applies the correct. linguistic label or the correct area label or it can't, it probably can't make assumptions about priority. But things like handling labeling is like, that's really annoying work that we have to do on just about every PR. If a new contributor doesn't understand how our labels work, they may not

apply any of them. And somebody like me has to go in and do it. So tooling like that would be useful, but we haven't, we haven't found somebody offering that to us. Yeah, just to add to or just add another perspective on the, the scam part or what I understand where it coming from, by the way, in terms of it is over, like, I think it's overhyped in so many ways. And I'm a little frustrated that we're, we're talking so much about what it could do one day instead of what it is happening. I'm very much focused on how we're using it today. An interesting input I got from John Osterhout, the author of philosophy of software design. He's a professor at Stanford.

And he has actually contributing to the Linux kernel. He's doing, he's implementing a protocol that one of his PhD students came up with. And he told me, I asked him if he uses Gen. AI much and he said he doesn't accept. He said it was really helpful for him to start contributing to Linux because he said it's so under-documentate. His criticism was that there's so little documentation that he actually turned to chat GPT to try to make sense of it and it helped him.

And his takeaway was, he said, I don't really want to popularize this, but, but I wish they would have added some more comments because then I could have done it without the tools. So I'm actually getting the sense of like Kubernetes is very well documented, which will obviously help people who are using these tools to, you know, to learn. I mean, you can probably just go to the source to be fair. You can, but like I totally get using something like chat GPT to explain. something complex to you. Like that, that is a good use of an LLM to me.

It's one of the most rights-produces with devs. No, that's great. I don't think you should be using it to generate a blog article or generate the documentation. You certainly shouldn't be using it to translate your docs. The Kubernetes documentation is localized into like a couple dozen different languages. And that's all done by humans, right? So I don't, I do not think that you should be using generative AI to do that. but using it to explain a complex topic or like look at some code and give you an idea of where

it does where it's going like what it does that's useful like that's that's a great way to use generative AI to me when do you think Kubernetes is a great choice to to use and when do you think it can actually be an overkill I get this a lot where especially small startups are thinking should we start with Kubernetes or will it just be such an infrastructure overkill that we shouldn't use it. So where would you draw that line or how do you see teams making this decision? I would say that you should think about how quickly do you think you're going to need to scale rapidly, like really rapidly. If you think that you're going to need to do that pretty abruptly, then I would use Kubernetes earlier because migrating over to it can be a pain.

What you need to really think about is not so much the is Kubernetes itself as an infrastructure tool overkill. It's can you afford the people you need to manage it correctly? A mismanaged Kubernetes cluster is a very dangerous thing. Kubernetes is not secure by default. You do require an expert. So my best piece of advice for running Kubernetes early is do not roll your own cluster. don't do that.

Do not do that. Use some kind of managed Kubernetes service that handles a lot of the work for you and then hire an SRE with experience with GKE or AKS or whatever tooling you're using. But just don't roll your own cluster. Kubernetes is easier to use than it was before.

I don't think it's as much overkill early as it used to be. Obviously, if you're, just running a blog or something, putting that on Kubernetes is ridiculous. My personal website does run on Kubernetes, but that's, it's a bit, right? Like, I don't need it to. Right? It's a bit. It's a pet project. It's a pet project. It's fun. It's easy for me to deploy something to a cluster. I can do that faster than I can do it with like EC2. Right. But that's because that's where my experience is. For most people, that's not true. So run it on some

kind of managed service. And if you think that there is a point where you are ever going to need to scale super fast, migrate to Kubernetes before that happens. So that may be handling. The addition there is, if he needs to scale super fast, you can always choose to just stay on a cloud provider and just pay massive amounts. You will be overcharge for it. So I guess Kubernetes makes sense when you actually want to control your costs a lot more,

right? Yeah, because it's scalable. it's not like it scales up and down right like your your costs aren't going to be consistent and that's that can be can make things a little bit difficult to predict but there are there are companies and tools that help you manage that aspect of running Kubernetes but it won't be consistently high at least because of that's up and down scalability but yeah just do it do it earlier rather than later for me but if it's just a blog if it's just a website or whatever, it's just a web application, you probably do not need Kubernetes if it's just a web app.

And so for those who are interested in getting started with Kubernetes for building their own things as a user, where would you recommend them to look? And what about for people who would be interested in dipping their toes and contributing? What's a good place to start? For starting to use it, I very much recommend our own documentation and GKE's tutorials. GKE's tutorials come with a little sandbox, so that you can actually do the thing, which is, yeah, very helpful. That's the way I prefer to learn personally. I need to be told how to do the thing

and then actually do it. That's not true for everybody. But that was a really, really great resource for me early on. And our own documentation has Quick Start guides for a variety of ways of getting going with Kubernetes. So I have always just used Minicube. There are many ways to get a cluster going, and it'll walk you through most of them.

If you would like to get started contributing to Kubernetes, you can come talk to me in the Kubernetes Slack, and I will be happy to help you find a place, but I will probably recommend that you start with documentation in an area that is of specific interest to you. So if you're particularly interested in storage or networking, I will hand you a storage or networking piece of documentation that I think needs updating.

If you like more of a people-managey thing, I will be happy to talk to you about the Kubernetes release team. However, again, we get hundreds of applications and there's like 25 spots. So I would not ride on that exclusively as a way to get involved in the project. But yeah, come talk to me on the Kubernetes Slack workspace, Kat Cosgrove, and I am more than happy to help you personally. Wonderful. And we've already talked, I think, about the professional benefits that

can easily come by being associated with a project at Kubernetes in terms of an active contributor or maintainer. Yeah, it's definitely a thing. Release team members put Kubernetes release team comms shadow on their resume. It's on people's LinkedIn. If you look at the number of people on LinkedIn who say they work at Kubernetes, myself included, mine says I work at Kubernetes. You do, right?

I do. Yeah. Yeah, I maintain the thing. So I do work there. But yeah, people put it on their resumes. It's popular, especially if you're a student about to graduate or you're very early in your career. It can definitely give you a resume boost. But what it also gives you is access to a very, very large network of people with significantly more experience than you who are generally quite friendly.

So it's a really good networking opportunity. We all hang out a lot. We all talk to each other a lot. we both right before CubeCon our big conference that we have twice a year there is a maintainer summit and we all get together and hang out and talk to each other for a whole day that's our own little miniature conference so there there is also the networking opportunity there too it's not just the resume fodder yeah and i like that you say networking because i feel it's people are starting to realize engineers are starting to realize with the job market becoming a bit more frosty that having a network

could be useful. And then there's a question of how do you get a network? And just my two cents is the best network I've ever had is it's not people who are kind of like, I know them with drive by. It's people who I actually worked with, like did something together. And, you know,

you can do it at your full-time job as just assuming you have one or on a project where you spend extended time and solve the problems, which project like Kubernetes or a different open source project. Yeah, for sure. Like if I've worked with you on the, on the Kubernetes project for a little while, I am happy to connect you with anybody else I know. Like if you're looking for a job at Docker or at AWS or Microsoft or whatever,

I know executives there by virtue of having done this for so long. And I'm happy to make connections. If I've worked with you for a while, I will not just like one PR is not enough. If that person has actually just, you know, good solid work. So I think it's just one more thing. if just don't do anything half-ass, just like, do a proper. You have a whole ass everything. Yeah.

I think the earlier people realize that the better. So to wrap up, I've just asked some rapid questions and I'll shoot and then you tell me how, like, what comes to mind? What is your favorite framework or programming language? Python, which I shouldn't say because I'm managing the Kubernetes project, which is written in Go, but it's Python. That was really fast. How come? It was not the first language I learned, but it was one of the earlier ones.

And I like that even if it's not good at everything, I can make it do just about anything. So that versatility is really valuable to me as a prototyping tool. So I will prototype something in Python. And if Python is not like the best tool in my toolbox for that particular application, I'll then go actually rebuild it and like go or whatever. is most applicable. I'm coming around to Python. I never really use Python.

I use other languages, C-sharp, JavaScript, etc. But you know what's getting me around to Python? It's now that I'm going to a lot of the getting started guides on, let's say, AI tools, etc. They usually have it in all languages, like from Python. Usually Python is the first one. And what I've noticed is Python is the least verbose. It is so easy to read.

And the easiest to read. And I just liked reading it. So now I just set up my thing. So I'm actually changing my mind on that slowly. I'm starting to see the pull. Yeah, you can almost just write pseudocode, and that's like very close to valid Python. Yeah. Which rules.

It's awesome. Yeah. You got to do something quick and dirty Python. What is a book that you would recommend? Well, so I don't read workbooks at all. I actually have a personal policy of not reading books for anything other. than fun. I only read for pleasure. And I read quite a lot. Right now, I'm rereading a Fire Upon the Deed by Werner Vinge, which is very weird, very, very, very strange hard science

fiction. It's one of my favorite books. I would absolutely recommend that if you like hard sci-fi. Well, Kat, thank you so much for going into the details of, I guess, talking through the surprisingly open nature of Kubernetes and very well-documented nature of it. Thank you. Happy to. I hope you enjoyed this deep dive into how Kubernetes is built, how it releases work, and all the organizational work that goes into keeping a project like this running. I found it particularly interesting to learn just how well everything in Kubernetes is documented, including how the project is organized and run.

If you ever find yourself inside a large project, feel free to steal ideas on how to organize things from Kubernetes. Thanks a lot for Kat for sharing to all these details. You can connect with her on the links listed in the show notes below. We previously did a deep dive into how Linux. was built and into a few other related topics. You can find all of these The Pragmatic Injured Deep Dives in the show notes, link below. If you've enjoyed the podcast, please do subscribe on your favorite podcast platform and on YouTube.

This helps more people discover the podcast and a special thanks if you leave a rating. Thanks and see you in the next one.