Software Misadventures - Behind designing Kubernetes' APIs | Brian Grant (Google)

Episode Date: July 30, 2024

As the original architect and API design lead of Kubernetes, Brian joins the show to chat about why "APIs are forever", the keys to evangelizing impactful projects, and being an Uber Tech at Google, and more. Segments: (00:03:01) Internship with Mark Ewing (00:07:10) “Mark and Brian's Excellent Environment” manual (00:11:58) Poker on VT100 terminals (00:14:46) Grad school and research (00:17:23) The value of studying computer science (00:21:07) Intuition and learning (00:24:06) Reflecting on career patterns (00:26:37) Hypergrowth and learning at Transmeta (00:28:37) Debugging at the atomic level (00:34:27) Evangelizing multithreading at Google (00:39:56) The humble beginnings of Borg and Kubernetes (00:47:10) The concept of inertia in system design (00:50:07) The genesis of Kubernetes (00:53:45) The open-source proposal (00:57:25) The Unified Compute Working Group (01:02:16) Designing the Kubernetes API (01:05:03) AIP.dev and API design conventions (01:08:02) The vision for a declarative model in Kubernetes (01:17:25) Kubernetes as a DIY platform (01:19:07) The evolution of Kubernetes (01:21:40) The complexity of building a platform (01:25:11) Style guides? (01:28:23) Gotchas in Kubernetes workload APIs (01:32:02) Understanding your thinking style (01:35:37) Reflections on Kubernetes design choices (01:44:08) The importance of getting it right the first time (01:48:13) Designing for flexibility (01:51:16) Collaboration and leadership (01:52:21) The role of an Uber tech lead at Google (01:56:33) “Giving away the Legos” (02:02:29) Picking the right person to hand off (02:06:41) Overcoming writer's block   Show Notes: API Design conventions: https://google.aip.dev/ Brian’s blog: https://medium.com/@bgrant0607   Stay in touch: 👋 Make Ronak’s day by leaving us a review and let us know who we should talk to next! [e-mail] Music: Vlad Gluschenko — Forest License: Creative Commons Attribution 3.0 Unported: https://creativecommons.org/licenses/by/3.0/deed.en

So much of software engineering, I feel like there's the whole like iterative aspect to it. But then when it comes to this, like APIs are forever. You only got one shot. Yeah, you really only have one shot. I mean, API versioning conceptually exists, but in practice, you know, probably we could have done without it. One thing which has been consistent throughout your career has been that you've proposed new projects and companies which have had significant impact, but obviously you can't foresee that impact. For example, you helped start Omega as one of the

research projects. You were deeply involved in Kubernetes. What are some of the ingredients in proposing a project and not only just proposing it, but actually seeing it materialize, convincing stakeholders and saying, yes, getting people behind this. What are the things that have worked for you? Well, I think a big factor is the environment. So back in the early days of Google is a very bottoms-up culture. And grad school is a very bottoms-up culture in some sense as well. So I think that's a really important part

is like, what is the culture of the organization? And do they support bottoms-up initiatives or are they much more top-down driven or customer-focused or revenue-down driven or customer focused or revenue driven and certainly something like a startup couldn't afford the luxury of just doing you know projects that might be useful right so i think the more scarce resources are or the more tightly coupled to demonstrating value or demonstrating revenue that just increases the the difficulty and the

hurdles of getting started, right? There's also a matter of trust with the leadership. You know, if you come in from the outside to the company or to the organization and you propose an idea and there's less trust in, you know, your judgment, then you have to make a much stronger case. You have to find business metrics, you know, which are reasonable things to ask for, but maybe more effort and take more time to actually, you know, put together all that data.

Or, you know, it may be very difficult to get concrete data or you may need to do experiments first in order to get the data. So there may be a chicken and egg scenario, things like that. So I think it really depends a lot on the environment more than any specific thing you do.

Because, you know, like if you can do whatever you want, then it kind of doesn't matter. You can just go do it. On the other hand, if there's a very bureaucratic process that requires, you know, a very large volume of data to go produce, it might be infeasible for you to actually go collect that data and do that project. Welcome to the Software Misadventures podcast.

We are your hosts, Ronak and Gwan. As engineers, we are interested in not just the technologies, but the people and the stories behind them. So on this show, we try to scratch our own edge by sitting down with engineers, founders, and investors to chat about their path, lessons they have learned, and of course, the misadventures along the way. Brian, so I thought maybe like a fun place to start is that you mentioned that you closely worked with Mark Ewing when you were an intern at IBM, who's the creator of Red Hat. Any like fun stories that you remember?

Yeah, fun story. So we both worked at IBM in 1991 before Linux existed. And the projects that we actually were hired to work on as interns never really got off the ground. It was a networking management product from IBM, but IBM had historically

its own networking technologies and its own networking products. And it wanted to get into kind of more internet oriented products, but it didn't have people with the expertise, you know, they were doing token ring and other things. And so I actually worked with Doug Comer at Purdue, like who wrote the book on TCP IP, literally, like I have a mug that says that. We had a book club, I think me and Ronik and a few other friends, because we didn't have a background in any of this. We, we came from, I think electrical engineering or like math. So then we literally had a book club for about like a couple of months,

like literally reading chapters from like the TCP IP people. There's like three volumes, right? I think we, we, we kind of, yeah, there's three volumes. So I like, I proofread the second volume or something like that so maybe i'm in the acknowledgement somewhere but so i was hired as the the nominal tcbip experts and it would have been an snmp based network management product and you know back in those days i was like implementing stuff from the rfcs and whatever but they were forming a new team and it just took them a while to decide what should the product be and things like that. They ended up licensing a product and white labeling it. So we actually never worked on it. So we had a whole summer to do whatever we wanted.

So for other reasons, which I won't go into, the local area network was very unstable. So we decided to make the RS6000 workstations be able to operate without the network. And back in those days, if NFS became unavailable and you try to access NFS, it would just, everything would hang. Everything would just freeze. So we tried to copy all the files locally. So we created this thing and we created like 80 pages of man pages of documentation where we would, and let me know if the connection to Red Hat sounds familiar, would synchronize files across the different workstations to maintain a set of tools that you could use locally. It would sync also email and groups, and we actually created a whole elaborate navigation

system using Emacs Lisp where you could navigate email and groups and all kinds of stuff. We wrote tens of thousands of lines of Emacs Lisp and shell scripts. It was a lot of fun. I learned a ton just doing that. But, you know,

the whole thing worked. And, you know, they probably threw it away as soon as we left because they didn't understand it. But after that, Mark was graduating a year earlier than I was. He was a year ahead. And he wanted to start a company. So at the time, he was dyeing his hair red. And the company was called, I don't know, the idea was called Z-something. I forget. dying his hair red. And the company was called, I don't know, the idea was called Z something,

I forget. But yeah, it had to do with this core idea of packaging, file packaging, I think. And I think that probably evolved into RPM later, some of the core ideas. But you know, that all predated Linux by a couple of years. So it didn't really all come together until a couple of years after that. But I decided to go to grad school instead. So there you go. That's pretty cool. By the way, can you show the front page of that manual again? Mark and Brian's Excellent Environment. Yeah, I hand wrote that because we didn't have a printer that could do that bigger fonts. If you look at it, it looks like type prints.

I forget what kind of printer it was, but it was not fancy. Um, so yeah, so we have, we, we did write a lot of documentation and try to make it full proof and all of that, but yeah, it was a lot of fun. So one thing which is pretty cool in this case is that not only did you build something very exciting, but you actually put together this entire manual or sort of man page. This is not something that's instinctive to a lot of interns. Let's just, or Let me preface that. I've worked with some interns. Many of them are excellent, but documentation is something that even it doesn't come as instinctively to, I would say, very seasoned software engineers.

But you were doing this as an intern at IBM. So what was the inspiration to do this? I don't know. You know, I have always been kind of self-directed. I've been fortunate to be in positions where I could be fairly self-directed in all of my projects and all of my work. And I kind of think about it end-to-end from the end-user perspective in terms of what would they be trying to do, what would they want to do, what would be their starting point, what would be the journey. And even before I knew about any UX critical user journeys or jobs to be done or anything like that, any kind of formal framework. So I don't know.

That part kind of came naturally to me. But in terms of figuring out something interesting to do and fleshing out the shape of it, the product side, the UX side, the documentation side, I think I've always been pretty good at that. So I wanted to... So we could leap ahead to Kubernetes, but on Kubernetes, I designed the information architecture

for the Kubernetes.io site. That's one of the things that I did. Oh, interesting. So, okay. That is something we definitely want to get into. I'm just going to take a note of that so that we don't forget. But before we jump on to Kubernetes, I wanted to rewind the clock a little further. Somewhere I read that you built a computer as a kid and that's how you got introduced to it. Is that right? It's almost right. So my father worked for the power company. So he was an electrical engineer, but, you know, for the power company, he repaired metering equipment. So he bought a kid's computer called a Kim-1.

You can Google it. There's, like, a Wikipedia page. It was a 6502-based computer. It had a little hexadecimal keypad and a little LED display. It would print out the hexadecimal digits a few at a time, and they would scroll across. I wire-wrapped the 4K memory board. So my father gave me the wire-wrap tool and said, here, you need to connect all these wires. So there's a lot of pins on a 4K memory board back in 1981, I want to say. And I helped wire up the memory board.

So once the computer was operational, there's no persistent storage. And there's no ROM with firmware or anything like that. So we would turn it on. And it would be a blank slate. But back in those days, there would be 6502 programs in assembly in Byte Magazine. So I'd type in the programs from Byte Magazine. I forget, like there's no assembler,

so I forget whether it had the machine code or we had to hand assemble it. But I would type those in. Games like Hunt the Wumpus i remember was my favorite and like words would scroll across the led display saying you feel a draft or something like that coming through the door and you have to decide which door to go through right like it's a maze that you have to figure out and draw on paper and can't really see so that was by far my my most favorite one i didn't do any kind of real programming on my own, just tweaked the programs really that were in there,

but I was like 10. So the next computer we got after that was an Apple II Plus, which is what my profile picture on most sites show these days, me typing away on the Apple II Plus. And, you know, there also Apple II was really, I mean, it really amazed me because we'd turn on the power and wow, there would be a prompt and there'd be a basic interpreter.

It blew my mind coming from the Kim 1. And it really amazed me because we'd turn on the power and, wow, there would be a prompt. And there would be a BASIC interpreter. It blew my mind coming from the Kim 1. And we had to buy the pieces of the computer progressively, like extra memory, the floppy drive. We didn't have a floppy drive initially. So, again, like I was typing in the programs. But BASIC was a big step up from hexadecimal machine code. For sure. So that was a lot easier to write programs. So I did write some programs for that, but, you know,

it's like you had to number all the lines and do go-tos. And then if you want to insert more, you have to renumber the lines if you didn't leave enough space in between and things like that. So that required a little bit of pre-planning. So I'd have to write them out on paper first to kind of figure out if I had enough line numbers space. And yeah, but but then you know I didn't do much really serious programming like I didn't start a company

like it didn't encourage me you can make money on any of this stuff or whatever and until high school there we had a program I took a couple of things one I took a typing class with actual typewriters and that actually helped a lot being a lot more efficient and i took a because it was that was considered a valuable job skill back then so you know our high school offered a typing class which was not totally normal but was useful and there was a programming class so my high school actually had a vax 11 750 i think and i got special access to come in an hour early and play with it so i did every day for a long time how do you get the special access by the way i talked to the instructor and i guess impressed him enough as like you know learning shell sort and it's like oh we don't do this until like the most advanced senior class or whatever it's like just give me the thing i'll just do that and yeah so a friend of mine a very

good friend of mine and i together we wrote a poker program because with terminal characters you could print out all the card symbols so that gave us an idea maybe we can build some kind of card game well let's do poker there's enough screen real estate because these are vt 11 um vt 100 terminals so you know they're not like high-res graphics or anything like that so we could have limited it's a little bit better than ascii art but it wasn't super fancy but it turned out we could highlight squares so we could make white cards and we could put the little you know club sign or spade sign heart heart sign on the cards, diamonds. So, yeah, so we did a poker game and we made it so you could play other players across different terminals. And also we created a, you could play the program.

So we created like a house player that didn't cheat. They would make semi-randomized choices and would actually sometimes win. So it became super popular amongst other students because we left instructions in the lab about how to run the program because there was a shared file system. So people would be playing that. So the instructor was a little bit annoyed by that, but just made a rule like don't play it during lab time. It's pretty cool that they let you run the program though. Yeah. Another popular thing back then was to leave a fake login screen on the computers to phish passwords for logging in.

That was, like, super easy. There's, like, no security. Because I had access an hour early, I could just go in and do that. I didn't actually ever do it for real, but I did make the program. For the record. For the record. For the record. But we did make the program. It's like, oh, record. For the record. But we did make the program.

It's like, oh, this is way too easy to do. How can we make sure this doesn't happen to us? So, you know, so we tried to figure out, like, how you could figure out if it's a fake login. It's mostly what we used it for. Wow. That's very impressive in terms of how you got into programming. So close to this, like, let's say, for example, going back to the internship with Mark Ewing. Now you decided to go to grad school,

Mark decided to start his company. And by this time you had enough experience with computers to know what the potential would be. And then on that time, you also saw a lot of, a few years after that, you saw a lot of these computer companies get started. What made you choose grad school as opposed to, let's say working at one of these companies

or even with Mark for that matter? Yeah, that's a good question. You know, I, surprisingly, computer science wasn't necessarily an obvious choice for me to go to college because I was like good at everything in high school. And they said, well, you're good at math, you should do engineering. And I was just going to go to the same college my dad went to. It was in Indiana. I grew up in Indiana, Purdue. And my dad said, hey, you know, they offer computer science. You like computers. You should do that.

So I was like, okay, that sounds like a good idea. I'll do that. That turned out to be a good choice. It just worked out well for me. But, you know, once I got into school, you know, like I mentioned, I was working with Doug Comer I hung out with grad students all the time like from my very first semester I was lucky to have a great set of teaching assistants in the intro programming class who would answer all kinds of

questions let me hang out in their office they taught me how to use Make and RCS, the version control at the time, a bunch of bash shell stuff. It was just born shell. And I learned a lot from them, but also for the very first semester, I just kind of expect grad school was kind of a normal thing. I hung out with a lot of grad students. I did research for professors. I implemented like SunRPC and XDR for another research project for another professor. The ACM chapter, student chapter, had all grad student leadership for the past, you know, preceding decades. Before that, I think I was the first undergrad president of that student chapter ever. Purdue had one of the oldest computer science departments in the country.

I think it started in 1962 or something like that. Oh, wow. So they had a pretty old. So I don't know. I just kind of became a thing I was just expecting to do at some point. And I did find that I actually like studying problems and studying the solutions. And that is something that has carried forward with me in terms of how I approach problems. and I did find that I actually like studying problems and studying the solutions. And that is something that has carried forward with me in terms of how I approach problems. I'm a bottom-up thinker, I've realized.

I like to study the problems, study the solutions, look at lots of examples. And from there, I infer, like, well, why didn't anybody ever try it this way? And then I could try it that way. So I wasn't planning on going in this direction, but I feel it's a suitable question. Again, if it's not a good question, feel free to tell us to move on. But this is something that

has come up a couple of times on this podcast that I'm just studying computer science at this point. Like going back to when you got started, you were typing in hexadecimal code on the computer to make it work. Then writing basic to, I would say, students today have it quite easy. They can write English and they get a computer program back. But there is a lot of benefit of studying the problems themselves

to develop that maturity of the problem space to come up with better solutions. But today you see a lot of people questioning, well, whether computer science is something that's worth studying given where LLMs are, for example. Do you have a take on that in terms of, is that still a domain that's worth getting into? Or is it like, well, AI is going to replace all the software engineers? I don't know that AI will replace all the software engineers. You know, I think it's too early to say where the bounds of what it can do will be

and how rapidly it will improve. Certainly, I was actually just reading Steve Yegi's really long blog post. Yes. I know Steve a little bit from Google. He's great. But yeah, I mean, definitely it has been my experience up until very recently that

there were a lot of weaknesses in terms of what I could do. You know, obviously that will improve how, like I said, how fast it's hard to say. I was working on some very specific code earlier today, and honestly, it didn't even occur to me to try AI on the code because it wasn't a large volume of code. It was more like tight, intricate code and not really algorithmically complex, but more kind of data structure type syntax kind of complex. And I think those kinds of things require a large amount of context. Like you may have to know, I don't know, thousands of type declarations. Or you may need to know general patterns across type declarations that are not explicitly specified

or things like that I think some of those things will remain to be challenging for some time but I always I have seen the analogies that maybe AI can replace the junior developers like in Steve's post I guess that's to some degree plausible certainly something I AI can replace the junior developers like in Steve's post.

I guess that's to some degree plausible. Certainly something I did as an undergrad was I wrote an assembler for an instruction set architecture. And I did that in a single sitting and it worked with no bugs. But that is something that's probably straightforward enough in AI. And it's just a large volume of relatively boilerplate code. I'm sure AI, once a context window is big enough, could do something like that relatively accurately. So if we can figure out how to leverage the AI

to accelerate those uninteresting, generate lots of boilerplate parts, because they seem good at interpolation and extrapolation from examples, then I think it can relieve a lot of the tedium of programming that some people dislike and leave the interesting parts. I've been pretty lucky in my career that I get to mostly focus on the interesting parts because once any project grew to more than a handful of people, I was spending more time

reviewing code than writing code or doing design work or doing some other part of the tech leadership role. So maybe more engineers could do that. And some says it's kind of a superpower. If you have a vision for what you want to build and it's a mere matter of coding, maybe the AI can do that coding for you. But you still have to have someone to envision what is the thing to build.

Do you have any advice for students who are probably just getting started, who are either about to graduate or maybe getting into school, for how to develop this kind of intuition when there's still AI which can help them with a lot of the boilerplate code that you said, but intuition is not something that they're going to get from that machine.

It's something that they still have to develop through experience. Go ahead. Yeah, I think different people learn in different ways. Not everybody is a bottom-up thinker in the way that I am. And I actually haven't seen data in terms of what categorizing different kinds of thinkers and how many of each kind there are, things like that. That would be interesting. Maybe the educational community has something along those lines, but, you know, certainly for the like lower, lower grades, high school and below,

you know, there's some issues and, you know, some people are more hands-on or Excel better more in kind of project scenarios where you're building something and other people can read from books, some people like video. So for me, I learn a lot just by looking at examples. So, you know, if AI were to generate a lot of code for me, and I could look at it and maybe try it in a playground, that would probably work well for me,

even though I didn't type the code in, right? Because I feel like I don't need to type the code in in order to extract value from it. One thing I do quite often is look at blog posts from other people to get an idea of how other people think about the system or the interface. Because I know not everybody thinks like me. So that helps me think, oh, you know, when they're presented with this, this is how they react to it. So I think there's some element for that. You know, it actually took me a long time to realize how I best learned. It was something I kind of did unconsciously for a long time. But yeah, it's hard to say.

Also, you know, I went through a certain journey to get where I am. And if I had gone a different path, would I still be able to, you know, have the same intuition? That's really hard to say. I grew up during the PC revolution and I actually read somewhere, I forget where, that a disproportionately large number of kids who grew up during the PC revolution

did a lot of this programming on their own while they were kids. It's just kind of a new, exciting thing. Not cool, certainly, but it was new and uh challenging um and uh yeah you know now the the environment is quite different probably no one would think of starting with hexadecimal machine code for whatever processor they're using but you know that understanding of how computers work really carried through with me through you know early part earlier parts of my career where I leveraged it quite heavily, like at Transmeta and Peekstream and in grad school and doing supercomputing and things like that.

And then also, even now, I feel like I just kind of have a pretty good intuition for how computers work, what the performance is, things like that. Like, I know how many instructions stuff will compile into. I would say I don't think many software engineers know that, including myself, how many instructions will it take for this thing to compile. So that is pretty cool. One thing, as you were describing how you learn, I was recognizing is it's a lot of self-awareness in terms of describing how you learn. I don't know if I can do that at least for myself.

I couldn't until the past year or so when I thought about it really hard. That's a reminder to a lot of us to think about it at least. I don't know if we'll come to a conclusion, but it's a good exercise regardless. Yeah, it is a good exercise. I actually went back and reflected through my whole career back from the beginning when I was programming with 6502 machine code. And I found that to be a useful exercise. I just wrote out a narrative of my entire career through the beginning to kind of see what are the common patterns? What did I like? What was I good at? What did I spend the most time on? I realized, wow, I spent a lot of time programming in c more time programming in c than any other all the other languages put together just because that's what

i learned at the beginning of college and then i did a lot of systems and compiler work in c and embedded systems and things like that in c then you know later c plus plus uh but then you later C++. But then by the time the Go project started around the time I started at Google, same year, and I was a compiler engineer, so I talked to Rob about it. But I knew from Java how long it took to build new language ecosystems. And I didn't think I wanted to be doing it for quite that long, so I opted not to do that. But, you know, it is. And it's impressive what Go has become today back from, you know, what it started. I guess, you know, I can measure back to when I started at Google 17 years ago.

But 20 years was my estimate. So it wasn't too far off. Yeah, for sure. What were some of the other takeaways from your reflection about your career well i did spend a lot of time on these lower level kind of systems and infrastructure supercomputers architecture compilers networking distributed systems threading like i've written multiple different threading all the context switching and everything yeah so the center gravity of my experience i would say was, was in that area. So when I was thinking about what I want to do, I was thinking, do I want to go back

to that kind of area of strength or do I want to go do something different or continue to kind of branch out in areas I've been focused on more recently? So that was just useful for me to just sort of see how I felt about different things going forward in my career. Since I was at the point, you know, the reason I was doing this reflection is because I was already thinking about leaving Google at the time and deciding what I wanted to do next.

I see. Were there like specific periods of like hyper growth, like in your career? And were there like any patterns to that? Hyper growth in terms of career trajectory or learning? Both, uh, like either or. Well, I learned a lot at Transmeta, I would say. And, you know, in terms of career growth, like position, I don't think I was there for five years. I don't think it was necessarily exceptional, but I did learn a ton as my first real job after grad school.

And it was my dream job in many ways because I mentioned that experience going from the Kim one to the Apple two where I would turn on the power and there would be a prompt. And I wondered how did that work? So even though I took architecture classes and digital logic and a bunch of computer science and engineering

classes in college, including building a computer-controlled car with Gene Kim. You can ask Gene about that if you have him on the show. We went to school together. I still feel like I didn't really get a deep, deep understanding of that until I went to Transmeta, which was a great experience being a software person at a relatively software-heavy hardware company in Silicon Valley. It was literally right across the street from Intel. I got the Silicon experience of Silicon Valley. And things like tape outs taking 12 weeks and bring up when the chips come back and nothing

works, how do you actually figure out what's wrong and make them work? It was also at a time when the industry was transitioning from 130 nanometer to 90 nanometer process, and the leakage skyrocketed. Intel's chip at the time was a Pentium 4, which was reaching the frequency ceiling of 4 gigahertz around that time, and it was just before the multi-core era. And also the industry was transitioning from aluminum wires to copper wires and starting

to outsource to a little fab called TSMC. And like all these changes were happening at the same time. And yeah, it was just a fantastic learning experience. When we started doing the 90 nanometer chip, I had a machine on my desk with a ball socket that I could just open and close. So I'd get a tray of chips, I'd slap in a chip, close the socket, I could change the frequency and the voltage and see where it broke. And then I would have to debug that with the circuit people and figure out why can't it add. So there would be certain combinations of bits in the inputs where you would get the wrong result.

Like, all the upper bits would be one where they should be zero. And it had to do with the way the dynamic logic wasn't fully charging within a clock cycle. But to work around that, what I did was rewrite the static compiler and the dynamic compiler. So Transmeta was a low, what we offered to the market was low power x86 chips. They were implemented as a custom VLIW

and we did dynamic binary translation from x86 to VLIW. So there was a dynamic compiler in there and an interpreter. And that was super awesome. All the sci-fi technology worked awesome. We had hardware transactional memory. And yeah, so by changing all the instructions, it turned out the memory unit contained an adder because you need to add to index into arrays. So I could actually replace

all the ALU ads with memory unit ads and it would be a little bit slower. But the trick was the memory unit didn't compute the condition codes. So if you wanted to do a subtract and then a conditional branch, it wouldn't work. So I had to compute the condition codes using bit logic. So if there was a branch following the app that the arithmetic operation fed into, then I'd have to generate the instructions to emulate the condition codes. But then once we did that, we could find the next bug because now adding would work and we could find the next bug and we could find the next bug and the next bug. So we could work around a certain number of bugs in software before the next tape

out because, you know, a full tape out would take a quarter. If you taped out some metal layers, maybe it would take a month. This is another cool thing in Silicon Valley is, and probably other places too by now, is you could actually send chips to a FIB, fixed ion beam machine, and it would push around atoms on the chip. What? Can you describe that a little more? Yeah, so let me look up, does this thing have chat fixed? Yeah, it has a chat on the bottom, right? Yeah, focused eye on beam. Check out the Wikipedia page.

You can think of it as kind of similar to a scanning electron microscope, but it pushes the atoms around instead of just looking at them. We also needed scanning microscope pictures to see all the copper wires that weren't formed properly. We had a big challenge with vias for a while. So vias are the wires that go between layers. Like there's alternating layers of metal and silicon. And there's little holes in the silicon to allow metal to drip through. But because of the viscosity and everything, like when the copper process was started, they either didn't go all the way through or they splatted out and shorted

things out. We looked at a lot of photos saying like, why aren't the V's working now? Why aren't the V's working now? But anyway, it's super cool. If you put dark gates that are disconnected onto the chip, you can either do a metal tape out to connect them in via wires in the upper layers, or you can, you know, just push atoms around. So. Is this like later on in your career, was pretty impactful because like, you know,

when you're debugging some issue in Kubernetes, you're like, you know, back in the days, I got to pull out a microscope, you know, to look at some of these things. So this is really like a walk in the days, I got to put out a microscope to look at some of these things. So this is really like a walk in the park. It's mostly the circuit folks who dealt with that sort of thing. We had issues like when the wires bent 90 degrees in one orientation, the optical process would make the wires fatter than in the other dimension. So then you get induction and all kinds of electrical effects. So I wasn't involved too much in that and in discussing what is the effect on the, at the instructions level, like in terms of, you know, what do we need to add extra latency between

the instructions because it was an explicitly, it was a statically scheduled VLIW. So I could just put more padding or I could use different registers from the register file. You know, there's a variety of things I could do to try to make something sort of mostly work or at least work enough better that we could find other problems. Now with Kubernetes, I think there's a different approach I applied, which is because I designed so much of the architecture, I found my understanding of the system was very stable over time, even without having looked at the code in detail or recently. Like I would just know how the components interacted

because I understood the overall system design. So, you know, there weren't a lot of changes to the system when I was actively working on it that would kind of disrupt my mental model. It's like, you know, I know how status works because I proposed that design. So, like, I know how this component should be interacting. So I can say, you know, did you do it this way?

Did you do it that way? You know, it sounds like it could be this problem if you didn't do it the way I expected. So, you know, a lot of the times with Kubernetes, I could debug things without looking at the code. That is pretty impressive. I want to get into some of the design aspects of Kubernetes too. But before we get there, one more step along the way.

So you mentioned you joined Google around 2007, 2008, if I get that right? 2007, yeah. And initially you worked on some of the multi-threading project, if I remember correctly. And then you got started on the Borg project. Yeah, so that's another example where I chose that project and identified all the subprojects and moved them forward and started a company-wide initiative to move the C++ code from single-threading to multithreading.

And wrote documentation and started an education program and evangelized it. So, you know, kind of similar to the manual in that sense. You know, so we brought in folks from the education part of Google and folks from like all different parts of Google to kind of evangelize that approach. Because Google started in 98. Also just before multi-core really took off,

you know, they started with cheaper single socket boards that they design or you know at least that's what they moved to very early so early on they started coding and this is before the c++ threading standard and stuff which also my group also was involved with getting that off the ground but you know so google did have its own internal thread primitive library but at the very beginning, the assumption was, well, threading is bad in Linux. There's no C++ threading standard. There's only one core anyway.

So just use a single threaded event loop. And that'll be more efficient. It'll get higher throughput, which is true. But it wasn't true by the time we had a course per machine. But by that time, Google had a massive amount of code already. Java did better because it had threading and concurrency built in. It had concurrency primitives, it had concurrency libraries that were pretty good. It had thread pools, it had mutexes, all that. So the Java code was in better shape in that regard.

Do you run into a lot of pushbacks during that evangelization process? Mostly around prioritization and urgency. Right. So like my team built a multi-threaded HTTP server and I wrote a little bit of the code in the core and web search eventually adopted it but like two years later but and by that time I'd already moved to board right so you know that actually that code is still used now I think yes so it was uh successful in that regard it It just took a while.

One thing which has been consistent throughout your career has been that you've proposed new projects at companies which have had significant impact. But obviously, you can't foresee that impact. For example, this project we just talked about is one of them. You helped start Omega as one of the research projects, too. You were deeply involved in Kubernetes. What are some of the ingredients in proposing a project and not only just proposing it, but actually seeing it materialize, convincing stakeholders and saying, yes, getting people behind this.

What are the things that have worked for you? Well, I think a big factor is the environment. So back in the early days at Google is a very bottoms up culture. And grad school is a very bottoms up culture. And grad school is a very, you know, bottoms up culture in some sense as well. You have to convince a professor or, you know, work together on an idea with the professor. But it's pretty open ended. And there's lots of different, you know, potentially viable projects that you can undertake.

So I think that's a really important part is like what is the culture of the organization and do they support bottoms-up initiatives or are they much top more top-down driven or customer focused or revenue driven but the environment in Google at the time was still such that pretty much anybody could start a project if it looked like it was useful you could get more resources and that was that was a good environment to be in. It is different now, I would say, in that regard and not for bad reasons or anything. And certainly something like a startup couldn't afford the luxury of just doing projects that might be useful. So I think the more scarce resources are or the more tightly coupled to demonstrating value or demonstrating revenue, that just increases the difficulty and the hurdles of getting started.

There's also a matter of trust with the leadership. If you come in from the outside to the company or to the organization and you propose an idea and there's less trust in your judgment, then you have to make a much stronger case. You have to find business metrics, you know, which are reasonable things to ask for, but maybe more effort and take more time to actually, you know, put together all that data. Or, you know, it may be very difficult to get concrete data, or you may need to do experiments first in order to get the data. So there may be a chicken and egg scenario, things like that. So I think it really depends a lot on the environment more than any specific

thing you do, because, you know, like if you can do whatever you want, then it kind of doesn't matter. You can just go do it. On the other hand, if there's a very bureaucratic process that requires, you know, a very large volume of data to go produce, it might be infeasible for you to actually go collect that data and do that project. So moving forward a little bit, obviously we've been touching on Kubernetes a little bit, but can you share a little more about how you got involved with the project in the first place? Yeah.

So, you know, I started on the Borg project as a follow-on to the threading project in 2000. I started in January, 2009, nominally to make the Borg control plane more multi-threaded, because that was kind of what I was doing at the time. Was that kind of a jumping off point for you to get into Borg as well? It was, yeah. The Borg cells, the clusters, were growing exponentially in core count, and the applications were not taking advantage of those extra cores at the same rate

because you know the number of cores per chip were doubling the number of sockets were also doubling at that time so the scale of the number of workloads running on the systems was and the number of users were growing exponentially effectively and so the team was kind of running to stay in place, effectively, you know, trying to keep up with the exponential growth. But, you know, there are some parts that have been parallelized, like the scheduling, computing, what's called the scoring function, computing the suitability of each machine for the workload. That was done in parallel. It's kind of an embarrassingly parallel problem,

that part of it. But many other parts were not. And so I actually joined the Borg team initially to do that. And the first thing we had to do is instrument the Borg master with the control plane with metrics, you know, kind of at the time Borgmon, which is what inspired Prometheus. There was only one metric when I started that was used in production. And it was called the machine ping period, which was the 80th percentile of the machine response time. But because it was single threaded,

it was sometimes processing machine responses and sometimes doing scheduling and sometimes replying to requests. So you could kind of infer that stuff, something was slow if that number got long, but you didn't really know what exactly. You know, there's two most common causes, but I've decided, look,

if I'm going to parallelize requests or something like that, I got to know how much time is spent on requests, what kind of requests are we getting, things like that. So I instrumented everything that I could possibly want information about, and we built monitoring dashboards. That actually turned out to be super useful in production for understanding what was going on, but also just useful in understanding the system. So I also just looked at, you know, what are all these clients doing? Why is it getting so many RPC calls? And it turned out that a very large number of clients were just polling continuously for information.

And this was because there was no change notification system. There was no watch API. There was no watch API, right? API. More than 99% of all requests were read requests pulling for information from these higher level controllers that were trying to do batch admission control or cron scheduling or whatever it is that they were doing. Map reduced jobs, checking for whether the job is finished, things like that. That is one of the first things I observed. So I went and just decided, well, all I need to do is parallelize the read request and that'll be good enough. And it was for the most part. But also, I learned that maybe there should be a change notification mechanism. For the service discovery system,

which is called BNS, the Borg naming system, those, the IP, the host IPs and ports dynamically allocated for all of the Borg tasks were pushed to Chubby, which is a key value store, Paxos-based key value

store that inspired Zookeeper and then later the Raft implementations at CDN console. So that was one of the first uses that I'm aware of, abusing a key-value store like that to do service discovery. Which is pretty common these days, by the way. Like today, that is like the status quo. That paper was published in 2006, so I think it's been common ever since then.

Yeah, but other information was published in 2006. I think it's been common ever since then. Yeah, but other information was not pushed there. So one of the core ideas in Omega was that we would have that kind of a store as the center so everything could be a controller. Because Borg had kind of the core that implemented jobs, tasks, alerts, machines, was this totally different kind of synchronous state machine checkpointing model. And everything else was a controller

that was asynchronously polling. So I said, well, why not just make everything asynchronously pull? That's a way more extensible model. So one thing which is interesting in this case you mentioned, when you were looking at Borg, there was only one metric use introduction. Now, for people who think about Google, they think about, well, Google's infrastructure is extremely reliable.

Obviously, every metric you can possibly imagine would be instrumented. There's this amazing Google SRE book that talks about how to monitor services in production. But this is back in like 2007, 2008 era. Was this pretty common at the time? You know, I'm not really sure about other systems. I think the user-facing systems had more components. Like the board control plane was just one component, really. The core was just one component, right?

And like everything was built into a monolith. And the team was relatively small at the time. I think the team size has grown by like 20x since then. So, you know, certainly it's not common now. Back then, you know, I think probably not. I think it's just probably because this was, Borg was started in the beginning of 2004, end of 2003. So, you know, it had its 20th anniversary this year.

Double the maze than Kubernetes. Yeah. Yeah. And you know, at the time it was pretty early and sort of everything was on fire. Like the clusters were growing exponentially and all the workloads were

getting migrated to board. And, you know, so I think, you know, just other things were the SLA as an internal system, it didn't need to provide five nines or even four nines of availability at that time. It was kind of like, everybody has to be multi-homed,omed you know clusters were taken down due to power maintenance on the bus ducts and and the generators on a regular basis so you know it's like ah your service shouldn't be in plus two anyway it's like we'll just if the cluster's down that's fine but borg actually operated on a principle of inertia everything just keeps running even if the control plane is down so you know the control plane could go down for an hour and everything would keep running and it would slowly degrade as machines died or rebooted or whatever but if you just stop all

the automation that's the cause of most of the churn it's like just you know shut off everything that's rebooting machines that degrades much more slowly than that. This way of describing, like in this case, how Borg worked, I think Kubernetes falls in a similar category. Again, it depends on how you deploy the workloads on it. But inertia, I forget who had mentioned this to me, but they said something on the lines of objects in motion stay in motion, objects at rest stay at rest. And when someone described it, I was like, oh, wow,

this is such a nice way to describe the property of the system. That is exactly it, yes. So you started the Omega. So, yeah, you wanted to come back to, this is a long tangent on how did I get involved in Kubernetes. So after doing the Omega R&D for a few years, a few things were happening. But one of the most relevant

things that was happening was cloud was becoming a higher priority. It was a relatively small nascent product. But it was on the road to GA.

It actually reached GA, I think, December 2013. So during 2013, Google was already thinking about how to accelerate the cloud effort, how to start ramping it up. The preceding years to that, we're kind of more R&D-like too,

like we had to change the way we operated our data centers, for example, because it wouldn't be acceptable to just take a whole data center offline and say, yeah example, because it wouldn't be acceptable to just take a whole data center offline and say, yeah, your VMs aren't going to run this week. Is that okay? We are used to that, but we had to actually change things like that, put in extra bus ducts and make sure that power would never go out, things like that. But by this time, it was considered to be ready. So they already

started thinking about how can we apply more of our internal infrastructure expertise to cloud? How can we accelerate cloud? So Joe Bita and I started chatting. I mean, we had met with Compute Engine as a board customer. But in terms of working more closely together, we started chatting mid-2013. And I think we may have even started reporting to the same director in Mountain View, because Joe was already looking on to do to the next things, like Compute Engine was off to the races.

He was looking for something new. And that's when we started discussing concepts like labels. So we actually created the labels, proposed the labels concept based on a bunch of my observations for Borg that he thought would work well in Cloud also. So that was the origin of labels in both Google Cloud

and in Kubernetes there. And then in August, something I talked about on Twitter and in the Medium post was this thing called the Unified Compute Working Group. Yeah. And that was started by a couple of directors to bring together a group of folks

from cloud and from Borg to discuss, you know, if we were going to build a cloud product that was more like Borg, more like something google could use what would that look like because app engine which had existed for a few years already uh platform as a service was considered to be too restrictive to run most google workloads it ran internal i.t workloads like meme Gmail don't run on it, right?

And VMs were considered, Google had never used VMs. Like that was basically considered a non-starter. They're too expensive, too opaque, too hard to manage, which I agree with all those things. So, you know, something that's container-like made a lot of sense. And Joe had discovered Docker, which I think was announced in 2013 sometimes. And it was pretty clear to us that we could see how our lower-level resource management C-group containers, which we added to the kernel, would fit into that higher-level user experience of containers that we never bothered to build because we did things like we would roll out standard base images to all the machines

and then create a true root on the machine for the Borg task to run into. So we thought, you know, the file system is not a problem because it's like decided, like we control the file system, right? And it's not like we're running arbitrary third-party applications because they have to be instrumented with Google's authentication system and Google's monitoring system and Google's logging system and everything. It's sort of this pervasive NIH effect. We could reuse libraries, but reusing whole components was really hard,

even to things like the build system and whatever. So what Docker did there was really interesting in that it solved pain points that we had, but we didn't know really what to do about or how important they were. How do you reproduce the production environment in development? It's like, well, everybody can just launch a board job. That's not a big problem. At low Like, you know, we had at low priority, you had free quota, you can just go do it. I mean, Google at the time had hard problems, not these easy problems.

So a lot of the things were just not looked at. Yeah, but, you know, but Docker had a really interesting user experience and that was before the Docker file. So, you know know you could start a container app apt-get install some stuff and and snapshot the image and it actually is really easily to iteratively kind of build up your image and then reuse it um so that was that was super interesting uh so you know we immediately all agreed okay it should be Borg-like. I knew from

my Omega and Borg experience that I didn't want it to be too Borg-like. I wanted it to be more Omega-like. And in the end, Kubernetes was really more like open source Omega than open source Borg. It's like Omega v2. And the initial assumption was that it would be a cloud product like Compute Engine and App Engine. So there's initial proposals, you know, through the summer and fall were kind of in that shape, I would say. And I started working on the API design. And I've shared some, like, screenshots from it and stuff like that as well. But that basic, you know, object model shape,

taking the Omega scheduling units, those became pods, the controller concept, decoupled functionality like auto-scaling would be a separate resource, things like that. That was taking shape pretty nicely. So in December, I presented that kind of API design, initial API design, and Brendan developed a prototype for seven. So the open source proposal, I think came in, in December and that was,

that definitely shifted the conversation at least, you know, amongst all the engineers who thought it was a great idea. Like, yeah, we don't want to be Hadooped. That's Hadoop is a verb, right? Or, you know, like the Chubby Zookeeper is a good example. You know, we didn't open source Chubby and pragmatically we can't really open source Chubby. So, you know, the open source community built something that is similar, you know, takes

a bunch of the ideas that we published, but not something we could run internally. So if we wanted a cloud product, that would be one way to do it. Because GCE... So remember, December is when GCE reached general availability. So it was not widely used at that time. So the question is, how do we gain market share? And I think we talked about this in the documentary also is, so we have many, many, the more we discussed it,

the more reasons we came up with for why open source was a great idea. Like, you know, whether it's disrupting the cloud status quo or shifting everybody from VMs to containers. Like that was a big part that Eric Brewer talked about at the 10th anniversary event. That video is on YouTube. I know Eric did feel strongly. And we thought, you know, containers, yeah, we understand containers.

We know, you know, everything we need in the ecosystem to make containers awesome. And that's part of why I got involved in cncf as well to build that initial product portfolio like you know prometheus like oh yeah we need something like that um and uh yeah leading up to 1.0 i wrote about wrote about this in the 1.0 blog post um is when we integrated prometheus metrics uh that ended up being a very synergistic relationship between the two projects. I think the simple exported variables for Go didn't quite do it for us for various reasons. I think the issue I linked to in my blog post discusses some of the rationale for that. But yeah, that worked great.

It was created by some ex-Googlers. And getting it into the CNCF worked very well for both projects. So anyway, in terms of shifting folks to containers, that was really, and to Google as a consequence, was kind of the big pitch. We have credibility in that space. We know what to do with it. We can kind of blaze the trail.

We can create the de facto standard. You know, there was a lot of concern that Mesos, which a lot of the larger companies were using, could be an option, or, you know, or that Docker or someone else, there were dozens of projects at the time, would create the de facto standard, or that Amazon would just create the proprietary standard,

like S3, you know, and then everybody would have to copy it. Right? So if we started with our own proprietary standard and Amazon did, you know, we might eventually have to launch a second product that matches their standard, which Amazon had to do to match ours. People keep playing catch up. So there are a bunch of interesting things in this journey of how Kubernetes came to be.

One thing that you mentioned that there was this unified compute working group. Typically, when a lot of these working groups come along, I want to say less than half of them are successful. I've seen a few of them, not very many, but that's been the general observation. It's mostly because too many cooks in the kitchen problem in some cases, in other cases, because it's very easy to get on a white board and brainstorm ideas, but you don't actually get to a place where you have something tangible to work with, which you can take forward. But in this specific case, all of you came to, I want to say, in the ballpark of what Kubernetes is today, or at least had the general idea, the direction was similar.

What made this working group get to, like, what made it functional, in simple words? Well, I'd say the whole working group, a lot of people were involved at the beginning. When it came down to actually figuring out what to build and building something, a much smaller group kind of self-selected and started working together. both on the control plane and on the node side. Like Tim Hawken, Don Chen, Eric Toon, Victor Marmal, Rohit Nagal, Vishkanan, they all came from the node side. And three of them went to go work on libcontainer with Docker. And the rest were working on Kubernetes directly. Also, they were building things like C-Advisor and Heapster,

which we were using for telemetry. So we'd all been working together for a while, for years up to that time, to varying degrees. And we got our working relationship with the cloud side, I would say, came together over the course of, you know, roughly a year, I guess, because Joe and I started talking in the summer of 2013, and then we launched open source in June 10th of 2014. Yeah. It was a really fast run up to open source, by the way. Yeah.

And really from the prototype in December, which I think the initial prototype was in memory and in Java, and it was VLA and Brendan and some combination of folks. We wrote it in Go with etcd and Docker and Salt. And we decided to just start working on that. And additional people came on board. It so happened that Daniel Smith came off another project in maybe, I want to say February or March, sometime in that time frame of 2014. Also at the beginning of 2014,

the organization, the cloud and the internal infrastructure organizations were officially smushed together. And yeah, I mean we started working on the code base together. Joe introduced us to the cloud networking folks so we could figure out, you know, is it possible to implement IP per pod and how to do that with the networking routes. In parallel, we were doing a bunch of due diligence on how we would implement containers with Docker. Like, hey, Docker doesn't expose network namespaces. How are we going to do that stuff? So kind of doing due diligence and validating our ideas at that stage. But we still didn't have approval up until the beginning, sometime mid-May maybe.

So you were building towards it, but you didn't have the approval. We were building towards it. So this is why Joe gave a presentation at Glucon in May and presented container VMs, I think it was called. And it was running this simple Python agent that Tim wrote that had the initial container manifest. And that open sourced that simple thing as dipping a way to get something out. We didn't need approval for that, really. And yeah, I mean, eventually we got the green light and we open sourced what we had. I wrote about this in the blog post. We just open sourced what we had and we had to rename task to pod and there was no real control plane in the beginning and so we had to go change it to actually kind of reflect the design i had in mind and that took six months ish after we

open sourced it um but uh the community was awesome the community was great i think clayton coleman from red Red Hat at the time started committing within like a week after talking to him at DockerCon. That's pretty impressive. So yeah, it grew really fast. So in a way, like the real release was 1.2 and a lot of the architectural pieces

and the API aspects haven't changed since, or like you said, fundamentally the way system works hasn't significantly changed since then. A lot of the features have been added over time. But that brings me to this part, like you were the architect plus the API design lead from Google, and I would say the Kubernetes API, as someone had described, I think Kelsey,

or I forget who, had described as like the declarative API to build infrastructure on top of. It's kind of like that. It's not just an API to, it's not just a container orchestrator anymore. People use it for various things like crossplane, for example. And this API is something that in a way forces one to do things right. Which like, for example, level triggered versus edge triggered, for instance. What were the motivations that led you to coming up with a design like this? Well, it's really those lessons learned from Borg and Omega.

And so, for example, another thing I observed from Borg is that I had this ad hoc RPC interface. So the RPCs, it didn't really have a firm concept of resource. It's not a REST API. It's a RPC API. And task was not a first class resource. Job was the first class resource. And that was an array of in tasks. And it had some task operations, but task was not a first class thing you could create

on its own so that ad hocness um made even even with the small number of concepts that borg had made it really hard to learn how to build a client and understand how to build a client which is something to build a client, which is something that I had to do. And as I was trying to collect more telemetry and figuring out how to instrument the clients and figuring out how to upgrade clients and do all kinds of things, it just was obvious to me that it was super painful and making it more consistent would be useful. Also, in the configuration tool, it didn't have a first-class plug-in mechanism the way Terraform did, but it was otherwise kind of Terraform-esque.

But it had a lot... And that was developed by Marcel, who's doing Qling now. I think around 2005-ish, maybe. It also had a lot of ad hoc things, like in order to diff the desired state in version control with the live state, there were a bunch of special cases

that had to be hand-coded. It's like, well, you know, this thing is going to change and this representation of the value is going to change and this other thing, you actually have to compare with this different field instead. And it was a total mess because it just grew organically, right? It was an afterthought. And I see a lot of that in Terraform providers as well.

So if you've seen AIP.dev, we've been developing a body of rules to try to so AIP 129 we just added recently shortly before I left to try to tell people don't follow these practices that will cause you problems down the road

for folks who might not know AIP.dev I've looked at it, I highly recommend folks check it out. But can you briefly describe what it is? It is a set of API design conventions. It's fairly well-structured. It was created by Google and is used by a bunch of other companies now that have ex-Googlers or who otherwise use gRPC. But it's about how to create resource-oriented APIs with gRPC and Proto

and follow some best practices and facilitate automation. For example, you can specify field behaviors like this field is immutable or you can specify this argument to the RPC is the resource and have some standard resource properties and things like that. In general, my opinion is the more standardization, the better. So it reduces flexibility, but it also reduces design decisions and makes it much easier to integrate clients. So I'd say the AIPs as they are didn't start with that end goal. I kind of pivoted them in

that direction. But yeah, it's what Google now uses as the basis for its Cloud API standards. But the older APIs don't follow all the rules. So there's that. But yeah, so anyway, consistency. So I also knew that because of all these other systems built on top of Borg, that in the controller model would all be more peers instead of layers, we needed an extensible concept space.

So in Borg, what happened is there were some places in the job request where you could stuff extra information. And these higher level components would just use that to stuff their own information. So they wouldn't have to use yet another storage system to store their own state. They could just like squirrel it away in the jobs, in the job to Borg and go look for that information there. So, you know, I knew that we really wanted to be able to, you know, even if it were in code in the beginning, which it was in Kubernetes, that we could add additional resource types. More like cloud, right?

Like cloud expects that you'll keep adding more services and more resource types. So if you kind of build a system with that in mind from the beginning, then you try to create some structure around that pattern. And it wasn't fully extensible, but even Red Hat was extending it from the beginning with their own types. And they were doing it in code instead of in CRDs or something like that. And that worked for us in the beginning. They added deployment config and project and some other APIs along those lines. So that gave us an early test that, yes, the model was extensible, at least from a code perspective and a pattern perspective. In terms of the declarative

model, I knew I wanted something way simpler than BoardConfig because the BoardConfig tool was fairly complex. It did these ad hoc translations and ad hoc diffs and some ad hoc orchestration. So like the replication controller, when we decided to create the deployment API, that was actually modeled after Red Hat's OpenShift's deployment config. But I knew that that was a pattern that I wanted to do to push that orchestration server side so that it wouldn't have to be in every client. Because it was already getting painful between the Google's command line tool, gcloud, and the Kubernetes command line tool, kubecontrol or before that it was kubeconfig and before that it was cloudconfig.

And a UI, if we wanted to have a UI and do a rolling update. You can see this in AWS where I think CloudFormation had some rolling update functionality for VMs, but if you used any other surface, you couldn't access that. So that was important to sort of have a position that we don't want complicated orchestration on the client side as much as possible. We just want to create the resources and have controllers do the orchestration.

That's what they should be good at. Also, I wanted this kind of simple declarative model. So that's where I started fleshing out the ideas for that. And first of all, bulk operations, I want to keep control to not just be one command, one resource, but I wanted the commands to be able to apply to an arbitrary number of nouns. So create dash F a directory, you know, delete dash F a directory or a list of resources. And that really was kind of a building block towards apply.

Issue 1702 is where I discussed the origins of the model there. And I wanted the desired state in Kubernetes to be not just what a human would apply from version control, but would be the union of all sources of truth for the desired state. And that was kind of a controversial position, but I knew that there was no hard line between what you would want to automate and what you would want to express declaratively in version control necessarily. Horizontal autoscaling is the canonical example. The replica count becomes something that's controlled dynamically.

You could say, well, let's have a separate value that an autoscaler can control, and you would report back the actual value. Well, that's where you get into this custom diffing sort of logic where you have to know that, oh, there's this other value I need to look at and I need to go invoke this other operation to change that value. And it just becomes much more complicated. So I wanted a model where, whether it's a default value or set by mutating image control, although that came much later, or set by an asynchronous controller, or set by another command line tool, or whatever it is, I wanted to be able to merge all those together into a single declarative intent in the API server.

So that's where the three-way diff came in, where we would diff the previous desired state, the new desired state, and the live state in order to merge them together in a way that preserved all the intents. That was tricky to implement, but Sam Goads from Box helped figure it out. Like, well, what if you have arrays that are actually associative and the key is one of the attributes and members of the array. Like, okay, yes, that was an unfortunate design decision. But so... Scrimmaging merge patch was one. Yeah, so server-side apply made that a lot easier and made it easier for tools like Terraform and

Spinnaker to do apply. And I think that got back my principle of keep the client simpler and put the complexity on the server side. But you can see Terraform drew the line in a different place where it requires a lot heavier client-side orchestration. Because a lot of these cloud APIs are designed not with the intent that you're going to apply and have the controllers just figure it out, but you can create one thing as inputs as outputs. You extract the outputs, you plumb them to the inputs of your next call, you make the next call, and then the next call, and then the next call. So those kinds of APIs are designed with the intent that you're going to be writing a program that is calling a sequence of APIs, which is not how people use the cloud.

Most people do resource management through tools, through command line tools, config tools, UIs, automation tools. The number of people who are writing programs that do some hard-coded thing is relatively small from what I've seen. There are some, you know, more sophisticated people who have built their own autoscalers for some system that doesn't have autoscaling or, you know, to build some higher level system on top in some way or to orchestrate, you know, multiple different workloads or jobs or things like that. But really a lot of those things become systems in their own right and not just a single workload.

And then we get back to the point of now you're using tools again. So I think a big thing that I noticed is that there wasn't a lot of awareness of the difference between control or management plane and data plane. So like the data plane, yes, everybody will be calling the data plane from their applications. They'll be re-modifying writing data in databases and storage systems and messaging systems and AI, now products and whatever now. So those things, usually someone is writing code and they're writing it to do a very specific thing with a very specific service. So the RPC model is a good way to think about it, I think.

And calling A, then B, then C is a reasonable way to think about how the code will be written. But for resource management, it's just not. If you can think of what Terraform does, it does a get diff, and then based on the diff with the live state, it figures out what kind of updates you need to do. Do you need to create the resource, delete the resource,

patch the resource, call some special methods, update the resource? And I think that sort of mode of operation, whether it's done automatically or semi-automatically or by human, is much more how we use the cloud or Kubernetes. So, you know, the more you can facilitate those kinds of management interfaces, the bigger your ecosystem and community of users will be. So you, sorry, if I may, like you mentioned that you're like a bottom-up thinker and it's like super impressive that you're able to keep all these different constructs,

right, like how they kind of interact with each other and all these things like in your head. Like, how do you... I'm good at that. How does someone go about like... Like, that's to be like a skill though right i mean obviously like i don't know i um yeah i don't have a problem with it i have a problem explaining it but i don't have a problem doing it like. Like, did you always find that to be very easy,

or like over time, like through different projects? Damn, okay, just naturally talented. All right, that's a bit harder to replicate. Okay. Sorry, go on, you had to follow up. No, no, I'm just gonna sob in the corner at how little things I can keep in my own head. It's okay, continue on, please.

So this is pretty cool, like in terms of how you came to many of the core design decisions for Kubernetes. And this is looking at the models you'd seen with Borg, with Omega. And you're not just thinking about how the architecture works, meaning how all these systems talk to each other. Instead of layers, you have peers. But you're also thinking about the UX piece, which is the kubectl apply dash f, you pass a directory just should work. I'm happy that I think that I designed the output for kubectl get pods. I think it's still the same. Oh, it's still the same. It's still the same. I said, I want to know, you know, like, if the pods are running or not. So that has to be in the output. Yes.

Now, one thing that I've been using Kubernetes over the last few years, been working with it. And one thing that you actually wrote in one of your blogs, by the way, you mentioned your blogs a couple of times. I highly encourage folks to check it out. You have a series of posts on Medium. We'll link to all of them in our show notes. There's a lot to learn from in just how you describe the technical history of Kubernetes, the road to 1.0, and a series of posts on just, I would say, what people typically refer to as infrastructure as code. There is a lot of good learnings and a lot of context to learn from. We can do another podcast on that. Oh, for sure.

So having said that, if you think about Kubernetes, it's similar to what you described cloud as, which is it's more like Home Depot than Ikea, which is it's more DIY. It's deliberately cloud in a box. Exactly. But it brings the usability versus flexibility trade-off as well. Like, it's very easy to do kubectl delete dash f. At the same time, if you're running something in production, you don't want that to be very easy.

Like, there have been, I forget which company it was, maybe Shopify, who described that they did something like kubectl delete dash f and wiped out the infrastructure. Spotify accidentally deleted their clusters with Terraform. So when you make things very easy, some of these destructive

actions also happen. So I wanted to get your perspective on this. I've come to this opinion over time, not where I started with, where I was like, every developer using Kubernetes should be exposed to Kubernetes, should just have access to kubectl and the manifest. Over time, I've come to believe that, well, Kubernetes is the right platform for platform engineering teams to build platforms on top of,

make life easy for us with engineers but you see many companies go through this phase where they start with kubernetes being the interface and then slowly start abstracting it away um i've seen the opposite many times as well yes um is this how you anticipated k Kubernetes to be used in the first place? Well, one thing I did is I looked a lot at platform-as-a-surface offerings and the degree to which they were successful or not successful, as the case may be. So I actually wrote one of my blog posts, Kubernetes in Application Platforms, kind of links to a bunch of things that I had looked at a couple years into the project. But, you know, because we wanted to be flexible enough to run pretty much arbitrary types of workloads, like any kind of workload that we could imagine that Google would want to run, you know, whether it's continuously running services or batch jobs or CI jobs or machine learning workloads. I mean, that was really a thing back then.

But MapReduce, we wanted it to be modeled after infrastructure as a service, which was clearly the standard flexible compute layer that everybody used. And all the platform as a service offerings back pre-container were just built on infrastructure as a service. So we knew that if we built an infrastructure as a service like Layer, that we wouldn't build something that we'd have workloads that couldn't run on it. We also knew that not only could we capture a lot of workloads, but we could capture these layers built on top. So we actually went through a period in the project where we deliberately chased other frameworks to get them to run on Kubernetes, like Cloud Foundry. It took them a while to switch from Diego to Kubernetes. But, you know, we did talk to them. We talked to the

Slurm folks a long time ago, because I had interned at Lawrence Livermore National Lab. So I reached out to someone at Lawrence Livermore. And, you know, it's taken them a while as well. But other things, you know, we implemented ourselves or we worked with the community like Spark. We got Spark running on Kubernetes. We put a lot of effort into getting stateful workloads to run on Kubernetes, not just with stateful set, but also, you know, other things that would be needed around that, like backups and persistent volume management and et cetera, et cetera. Yeah, that was a fairly deliberate effort to say, look, we don't have to be platform as a service. I wrote this in the scope document and in the what is Kubernetes document

also. Kubernetes is deliberately not platform as a service. It is designed to run platform as a service on top. Because my experience looking at these systems is that as the platform becomes you want to make it more ergonomic and reduce the number of steps people

have to take. When it's very generic, you have to configure more things, do more steps. There's more of an impedance mismatch. It's very freeform. On the other hand, making it very specific makes certain things not supported. So there's this weird spectrum trade-off space or 2D plot where you kind of want to find the sweet spot between saves a lot of time, reduces complexity, and also has a big enough set of workloads that it's worth doing. And that's really hard.

Like you can look out in the industry and there are some folks who have found some good areas like that, like for sell and Netlify, right? Static web apps is big enough because there's so many web apps, so many web developers. But if you go to a smaller niche, is it big enough because there's so many web apps, so many web developers.

But if you go to a smaller niche, is there a big enough total addressable market in some sense to actually produce a platform product for that space? It's really hard because if you look at the combinatorial number of requirements, it's like, well, you may need to be internationalized and be able to represent everything in German, or you may need to have GDPR in Europe or some other compliance regime in Japan, or you may need to support streaming workloads. If you're all these HTTP request-driven workloads and you don't support streaming workloads, how do you build a chat app or something like that? Or if you don't support GPUs, how do you build a chat app or something like that?

Right. Or if you don't support GPUs, how do you do inference? Right. So there's all these choices you make that make it both easier to use and easier to build. But it cuts off, you know, most of the workloads that exist. So, you know, I think another factor in a lot of organizations is they don't want to have too many platforms. They always have multiple platforms due to acquisitions and organic evolution and, you know, the history of whatever changes they have made. But I see a lot of platform and consolidation efforts.

And there seems to be like inherent resistance to having multiple platforms so but i think you have if you want to streamline the user experience you have to have multiple platforms if you have multiple categories of workloads otherwise you end up with a one size fits nobody kind of platform and you can build you know some common underpinnings. And we did that with Kubernetes. You know, there's probably more common underpinnings you can build like, you know, cloud native build packs or whatever.

But, you know, the, so I think for a specific organization, if you say, well, we have like a 10 year contract with Splunk and, you know, we already chose Akamai for CDN. And they have some combination of technology choices they made, and that always ends up being a factor of how do you glue all those things together. It's like, we're using gRPC and GraphQL and Cloudflare or whatever. It's like, okay, why?

But they are. So it's really, really hard to build a product that accommodates that kind of combinatorial complexity but integrates it at the same time, I guess. So one thing that you mentioned in the blog post, and this is something you see on the Kubernetes website too, which where, like you said, Kubernetes doesn't say it's a platform as a service.

Yeah, I'm pretty sure the text I wrote is still on the site. Yeah, I think it's like, it's a portable, extensible, open source platform for managing containerized workloads and services. That sounds like me. Yes, I'm reading off of your post. So one thing which one of my colleagues recently said was, Kubernetes is like, see, it needs a style guide, which I completely agree with. And I think there are certain style guides if one squints a little harder, and they are in the form of like the design proposals you see this, sorry, in the community repo you see API conventions, spec and status working with objects,

declarative management of applications, how to use conditions, for example. But these style guides exist for platform teams who are building platforms on top of Kubernetes. Do you think there is a need for a style guide for PaaS itself, like platforms built on top of Kubernetes? Would community benefit from that? If Kubernetes took a stance on, like, here's what a PaaS on top of Kubernetes would look like. Well, I am kind of surprised that nobody well surprised and not surprised but kind of surprised that nobody's built a distro so i'm sorry i was actually interested in the glass cube announcement you know in terms of

like a package manager that would help you manage the components that are installed. On Kubernetes, I pitched the idea of an add-on manager for many times, but it never really got traction to actually go build it. But much like a virtual machine, there's common things that people need to install in their cluster, whether it's monitoring agents, logging agents, operators of various kinds, admission controllers, other CRDs, etc. So creating a way to manage those things and an opinionated set to start from. If you don't have

your own opinions, start with this set. And the monetization model I think is challenging, but enough users of Kubernetes have this challenge that I'm kind of surprised that they didn't band together in the community and come up with something like this. So I think that would be one starting point. It's just like, forget the paths. Like, just what is sort of a semi-standard way of running a cluster? What do you need in it? What should you install in it?

Then if you wanted assistance with certain workload patterns, that's where I think it gets into things like what CICD system are you using, right? So that's where I think you could come up with. I think there are projects that are trying to put some high-level guidelines. They're not very opinionated, but more of just explaining what is the shape of things that you need in your platform. I think that could go a little bit farther in that you could have a reference implementation maybe. Certainly, I would cover more the gotchas that I've seen over the years, like what things to abstract and what things to not abstract

because everybody will end up asking for those things or how to do abstraction is another thing that I have a lot of opinions about, which maybe we can talk about when we talk about infrastructure instead. But because I think almost everybody thinks about them in the wrong way, or at least in a way that's going to lead to predictable problems. So talking about some of the gotchas, can you share a few that you've seen people run into more often than others? Well, I think one is trying to hide some of the options in Kubernetes workload APIs, like the pod spec. So in Kubernetes, in deployment, stateful set, daemon set, DemonSet, jobs, etc., we deliberately decided

to expose the full pod spec and not hide anything because that would cut off some use cases. I've seen this from the very beginning. Like I mentioned Sam at Box very early, they

tried to abstract replication controller at that time with JSON on it, and it didn't work. Those fields of the API are there for reasons. So it all depends on how similar your workloads are. If you build everything with the same application framework and it runs the same way in the same footprint with the same kinds of integrations to application config and monitoring and logging and it's very cookie cutter, then you

can build a really narrow abstraction that just spits out everything for that cookie cutter scenario. On the other hand, if you are bringing in off-the-shelf components with Helm charts, those are going to be totally different. Kafka is a different beast than a Spring Boot app. That's a thing where my bottom-up thinking comes in, where I look like, well, what are the clusters of workloads that you need to run and build abstractions around those? You know, if there's just a singleton, then fine.

That's Kafka, just run Kafka, do whatever. You know, on the other hand, if you have significant enough clusters of things, then you might build an abstraction. But if it's just one or a few, a handful, it's probably not worth it. The ROI is just not there. The abstraction will get in your way.

Like maintaining and extending the abstraction will be more work than just, you know, doing it manually on each individual thing. One thing you often see is sometimes there is this tendency to consolidate platforms as well based on workloads. And like you said earlier, you end up with the least common denominator experience

for every single workload owner where none of them is happy instead of all of them being happy right so kubernetes is pervasive like it's run on public private clouds on-prem on the edge you have this uh the chick-fil-a example where they have been running it on the edge i don't know how many thousands of clusters it's it's in every target store. Exactly. Yeah. So that must be very satisfying to see, obviously, something that you built getting this sort of an option.

It is, yeah. I love all the use cases that I never thought about, like Retail Edge and being in spaceships and oil rigs and whatever and wherever. We did design it deliberately to scale small. Like one reason why there's just the API server, the controllers and etcd and not several stateful components like a database, a message broker, etc.

is because we wanted to scale down and people talk about Kubernetes the hard way, but I think Cloud Foundry is a lot harder to actually run. We intended it for at least local development or smaller footprint locations. But all the things people have done with it at the edge on Raspberry Pis, things like that has been super exciting and interesting to me. If I can. So I think the last time I used Kubernetes was like four years ago.

So being a Kubernetes noob, I'm trying to fill my questions quota for this podcast. So taking kind of a slight tangent, but something you kind of coming back to is you mentioned like you're a bottom-up thinker. And you also mentioned that you've only discovered this in recent years as well, right? I was curious, this is something that I think I talked to our product leader at one of my first companies, he's top-down versus the bottom-up, especially for engineers, in terms of, I think it affects a lot of things, right?

Like how you communicate, how to be more effective and all these things i guess like did you notice a big change when you discover that you are like when you confirm that you're like a bottom-ups uh thinker and then also it's like should all engineers actually go and figure out like whether they're top down or bottom up i think it is worth figuring out i think it would have been useful for me to have figured it out a long time ago i'm sure people i interacted with and I generated like lists, 25 pages of bullets or whatever, they could probably have told me a long time ago that I was a bottom-up thinker.

My reading lists or my list of platforms and things look like that. Like I think my list of platforms was like 16 pages of bullets. But yeah, I do think it was valuable to understand because then I just embraced it. But also I have to kind of fight it when I communicate to sort of distill down, realizing that not everybody is bottoms up and they're not going to get the message. If I say, you know, here's a slide that has 50 things on it. What's in common in those 50 things? It's like it shouldn't be a guessing game you know if you're interested i can have a backup slider put in the speaker notes but

uh whether it's been useful or not it's not been long enough for me to be able to say um but uh i feel already that i'm more effective where I, whether I actually am, I don't know. But yeah, I do think it's valuable to understand that. Because then you can catch yourself in the sort of anti-patterns or figure out when you're talking to someone with a different modality. And you need to kind of translate. You said there's been any signs in the past, but like, how did you go and confirm that? Like, I'm bottoms up. I started writing out, you know, projects that I really enjoyed working on and what I did and what was in common about those projects. And, you know, one of the

things that was in common was areas where I could pretty much have free reign to figure out what is the problem to solve, what is the solution. I could say the problem space and the solution space and kind of have control over the direction. So sort of a blend of product engineering and UX, I guess. Cool, sweet. Thanks. All right, I got my quote.

Please continue. I don't know if that counts as a newbie question. It's just, it's a good question. So a couple of questions I have on the side of Kubernetes, again, trying to capture some of your reflections on where it is today. Now, Kubernetes is used in all sorts of places, but it isn't perfect, as no system is. I'm curious to know, what are some of the things that, reflecting back on, when you look at the system today, you wish were different about it? So I thought about this question, and it's hard. I mean, there are things I know that, wow, I would totally do that differently. But it's a tricky balance. It's hard. I mean, there are things I know that, wow, I would totally do that differently.

But it's a tricky balance. So when we were building Kubernetes, it was not the first rodeo. Depending on how you count, I'd done supercomputing before. There's Borg. There's Omega. So we were able to get a lot of things right out of the gate without thinking too hard about it. But we also had to build it too fast, really fast. And our main goal was a container orchestrator, which is a term I resisted for a long time.

But we weren't building a control plane system that happened to do container orchestration. We were building a container orchestration system that happened to have a control plane, right? And we knew we wanted it to have a control plane and what shape that would look like and things like that. But it was not the main goal. It was kind of a necessary evil in some sense. And also, if you've read the system Bible by John Gall, complex systems have to start from simple systems, right? Like every successful project that I've worked on started with a simple

skeleton and that had the basic shape of what you wanted to build and then grew from there. And now people complain about Kubernetes being complicated and I understand where they're coming from. There's lots of resources. There are lots of attributes per resource. Cloud is no different. Like, is it harder to learn Kubernetes than to learn AWS, Azure, Google cloud and digital ocean and Oracle? I don't think so. I think it's actually easier to learn one thing than, you know, several other things that are equally complicated or even more complicated.

Like, what is the difference between IAM and AWS, Azure and GCP? Like, I don't even know the answer to that yet. I'm just learning. You know, but it started simpler and the early adopters really loved it. And that was also super gratifying. Like they immediately said, wow, this is awesome. It makes my life better in all these ways. I get it. It's cool. So having that core set of advocates who are kind of invested in the project. I mean, this is why we actually deliberately sought contributors to the project as well

is the IKEA effect of, you know, it's more like Home Depot than IKEA, but the IKEA effect is when you feel more attached to something because you put effort into building it. And that really worked. I mean, it was insane and almost unmanageable to have 80,000 all-time contributors. But, you know, a couple of years into the project, something like 12% of users had contributed. Right? Significant. Yeah, it super worked as a way to make people feel engaged with the project and with the community.

And, you know, we also learned lessons from, like, the Mesos community where the big users didn't open source their frameworks. And there's a lot of fragmentation in that community. So, you know, we were deliberately trying to build something different there, not just technically, but community wise. So if we had started way more complicated, like if we had launched Kubernetes with the complexity it has now, could anybody have used it? I don't think so like i actually doubt it it would be way too much to absorb all at once but we had this body of users and contributors that built up over time so you and they only you know had to learn new things as we added new features and they could explain it to other people and people could build on top of it and then extend those things so we had like a smaller kernel to grow from.

So it's hard to know like what decision if we had done differently, sort of like the butterfly effect, like what would have changed in that process of growth? So that's a really hard thing to try to guess. But, you know, in terms of things that, you know, there are inconsistencies like the associative list, strategic merge patch stuff. I totally would have optimized for machine editability from the beginning. Because if you look at Helm values files, I have a lot of complaints about Helm and the templating syntax and whatever.

But I understand why it is the way it is. So that's a whole different tangent. But the values files are just nested maps. And because they're just nested maps, they're super easy to merge. So you can have a base values map and you can just overlay an environment-specific value map on top of it

and override values with no fancy schema information. So having something like that that's structurally simple that you don't need to figure out, is this a discriminated union or an associative list or is it a set expressed as a list? And those are just different forms of the problem I mentioned in Borg where you have to like diff one field with some other field and things like that. So, you know, it is systematizable. Like we did figure out how to

go annotate the schemas with the information about what are the merge keys and things like that. But because of the number of different clients in the ecosystem, it would have been better to optimize for machine editability and mergeability, I think, as opposed to thinking about in terms of human authoring. I think there would have been other ways to address the concerns we had about human authoring, like understanding is that map key a value or a key in the structure? That was the big concern about why we used and why we chose lists over maps. But whatever, you know, I think that could have simplified a lot.

Is it worth doing a V2 though? No. Like one thing that happens is you have to get stuff right to the beginning or you can't change it because it's too expensive. It's not worth it. So, you know, there's a bunch of stuff. Like I wrote a V2 API issue. I think it's like 8210 or 8120 or something like that. But let me just check to see if I'm starting to forget the numbers after 11 years. It's still pretty impressive that you remember some of these numbers. And I see this show up in

your blog posts as well, where you have links to a lot of the issues uh on github which is quite cool and I think that is one specific thing I would encourage folks to check out like go on those tangents read through the issues because all of that history adds to one's understanding of why the system was built a certain way at least it has contributed to me knowing more about the system and using it more effectively. Yeah, so I did not remember that issue number. Oh, here it is. 81.90. All right. 81.90.

But yeah, so we never implemented any of those things because it just doesn't work. It's kind of sad because it's like a wart in the side for all history. But the ecosystem of stuff that's been built in everybody's configurations that are using the current API is just too large for it to be worth changing. I do think there's some smaller things that could totally be implemented, like garbage collection of config maps. I totally think we should. Oh, yes like garbage collection of config maps i totally think we should oh yes garbage collection config maps i'm so sad that the two proposals that we had to do that just did not get go forward but uh what else i mean we could have made if we were doing a control plane thought of it as a control plane project from the beginning as you know we could have made everything srd from the beginning but that was so much harder

to build like the initial third-party resource thing had to be totally scrapped and redesigned we had to build aggregated apis first as a way to decouple that api server and then it took years before it was ga and you know even now do we have binary encoding yet like I don't even know if we have binary encoding yet right so it was much harder to build in a very generic way so it was totally the right call to not require that from before we even knew if you know it would survive right like we didn't know if the project would be successful there's tons of competition there were dozens of other projects out there yeah um I kind of figured that LibSwarm wouldn't be the winner, but other than that, we really didn't know. I was reflecting on what you were saying with the, just like, yeah, you have to get it right in the beginning or otherwise. In so much of software engineering, I feel like there's the

whole iterative sort of agile aspect to it. But then when it comes to this, it's to your point. APIs are forever. You only got one shot. Yeah, you really only have one shot. I mean, API versioning conceptually exists. But in practice, probably we could have done without it. I like that line.

Oh, that's good. So one thing you mentioned earlier around keeping the surface area of Kubernetes small in a way, and I think that attracted a lot of users or made it easier for a lot of users to deploy it in many different places where you have a few core components, you get them running and you have a functioning cluster. On the other hand, for some of the bigger users, or so-called hyperscalers, it doesn't quite just work out of the box because the surface area is much larger than the boundary of a cluster. And this is true for places like Google, for example, where the number of board clusters you have is not just not just one for example uh similarly any company at any at a decent scale would not have just one kubernetes clusters so do you think there is room for

something like kubernetes or kubernetes in a different form um that could work for some of these hyperscalers too or uh like there have been projects around multi-cluster kubernetes for instance but that area is still very abstract and undefined and everyone is kind of just figuring it out themselves yeah i think there are a couple ways to look at it one is managing multiple fleets of clusters you know i do think that's the Federation work did not go the direction I was hoping. So what Borg does is every cell is independent. So the compute clusters are independent, but there's a mesh for service discovery and authentication that spans clusters, right? So I had suggested, I think, that, you know, federations should kind of focus on networking, naming, and auth areas to try to just start there.

And it kind of grew to the entire API service. Borg did have another kind of proxy for launching workloads on arbitrary B board clusters, but those were batch jobs. And it also just had, it didn't have the same problem as, you know, like emulating all the dozens of built-in types plus all CRDs and all the behaviors across multiple clusters. So we did talk about another use case. So I think, I guess the short answer is have in mind what your use case is. Like if your problem is retail edge, then the shape of your solution is probably different than if you are sharding a cluster for high availability within a huge data center. And if your workload is a continuously running service, it's probably

different than a batch job. So if you build a more specific... I guess another observation about higher level abstractions is they tend to be more valuable if they're not just kind of restricting you and generating a pattern of config or API calls, but they're actually actively managing the underlying systems for you. So one example I call out is Spot. Spot lets you run a workload at lower cost, and it manages the underlying VMs for you. If they vaporize or expire, it starts your workload on another one. So that is providing an active service that's

kind of complicated for you to build. It restricts what you can do, but it's providing value for what it does. So I always look at that sort of ROI of the abstraction layer. Like, how much is it doing for me versus how much is it restricting me? So I always look for it to do something non-trivial. If it's something I

could just write a config generator to do, then I probably don't need it. And it'd be better to not have it because it's operationally more complicated and more restrict what I can do. So yeah, with multi-cluster, it's definitely a lot. It's very dependent on the work case. One thing about the Kubernetes design is it was designed with specific technical and business goals in mind it wasn't designed to scale to 100,000 nodes

deliberately like I mean in the beginning it just literally said well it will run on 100 nodes right and we grew that to 1,000 100 you know I forget what the progression was but 1,000 5, to 1,000, 100, I forget what the progression was, but 1,000, 5,000, 15,000. But because we assumed it could run on a single node,

the whole system could run on a single node with the API server, the controllers, etcd, and the kubelets, and we weren't going to depend on infinite cloud storage or having PubSub available or some global control plane building blocks. That limited the design, but the design was really focused around flexibility and extensibility. And like I said, the ability to scale down more than the ability to scale up. So if we were designing to run a global service it would have been designed totally

differently like watches watches super expensive it was designed for the case where there is a relatively small number of rapidly changing entities that need to be continuously managed by controllers so they always need to have this date and memory right if that's not the scenario it doesn't make sense to use that model right like if you have a very large like if you're a shopping catalog if you have a very large inventory of stuff most of which nobody looks at most of the time or it doesn't change most of the time then you wouldn't want to keep all that memory and send constant updates for everything right so you wouldn't use that model right so it really depends on what you're designing for. So if you wanted to design

a system to use GPUs all around the globe to do batch inference or something like that, I would design it totally differently. So there are a lot of things people agree on when it comes to Kubernetes. One thing which there's not quite a lot of consensus on is how to manage manifests and the infrastructure as code space. We're going to park this section based on what we talked about earlier. We can do a round two when it'll be more appropriate to talk a little more about that. So is it cool with you? If we park this, I think it'll be more appropriate to talk later. Yes, thank you. Okay, cool. So having said that, moving on to some other aspects of just how you operate as an engineer.

So you were a distinguished engineer at Google. You've been an Uber tech lead on many projects. What are some of the practices that you've found to be helpful when working as an Uber tech lead and you're collaborating with a bunch of other senior engineers too. And I should also maybe ask for some folks who may not know what an Uber tech lead means, would you mind describing what that position is? Sure.

An Uber tech lead is, I actually found, you know, the term is out there. I don't think it's just use at Google. I'm not even sure that it started at Google, but Uber is like over in German. And, you know, it's basically a tech lead of tech leads. And, you know, so you may not be doing the day-to-day,

you know, hands-on coding, issue management, things like that, but operating at a higher level. Like a standard progression that I see at Google, a staff engineer at Google could be a tech lead of a team of maybe 10 or 12 people. And I think of them as in partnership with the engine manager and a product manager if there is one, and maybe a project manager if there is one. Shouldn't really be one at that scale. But as running the whole software project,

they know enough that they can decide, do you do Scrum or Agile? Or what are the practices that the team actually uses to stay on the same page? How do you do issue management? How do you do version control? How should you do CI? Some of these come from the broader organization

or you just start using some tools, but they're capable of like making or at least evaluating or implementing all those kinds of decisions about how do you run a software project, right? And they will often be doing some amount of the coding, doing design work, overseeing design work, doing code reviews,

making sure the project hits its milestones, like I said, in some cooperation with the engine manager and other folks. But if you just throw a software project at them, they can figure it out. They can ask questions of the product manager if it's not clear. They can engage UX, things like that. A level above that, like senior staff level, could be someone doing that with a bigger project that's like, you know, two, three, four, five times bigger.

Or, you know, probably the bigger the project is, we'll have more tech leads. They delegate different components of the effort too, right? Who will be doing those kind of day to day activities i mentioned so they have the ability to up level what they actually do you know maybe make sure that uh your director knows that you need more product team coverage or you know things that are kind of more meta level more more organizational level, as well as doing some of the critical design oversight or design work, thinking more about the strategy, working together with a product team in terms of the roadmap. And how quickly things can get done, where the risks are, whether a change in direction is needed, figure out when to burn down technical debt. Because a lot of times people in the trenches have trouble popping their heads up to reflect. I have that problem even now. And it's helpful to have someone who's not swamped by the day-to-day.

If you have external customers, you can also engage with customers and things like that. So I think the Uber tech lead role for a larger software effort can be pretty important to make sure all the pieces of the architecture fit together, that you have reasonable conventions across all the teams, and you're all hitting consistent expectations. You don't want to have one team thinking, oh, we only have to be three nines and some other team saying, what, we're totally doing five nines. You need, in some sense, a decider and clear messaging about those things. And if teams aren't communicating with each other or aren't sharing designs with each other, they may set practices and expectations for how that stuff is done. Like I said, depending on what the inch manager and the Uber tech lead carve out as their responsibilities. But a lot of, especially in the kind of cloud space, I mean, these are products built by engineers for engineers.

So a lot of decisions are super technical, even product decisions or UX decisions. And a lot of the customers are super technical. So it's useful to have senior technical people engaged with all those cross-functional roles and uber tech leads usually do a lot more of that in my experience you know because they are freed from some of the day-to-day responsibilities more you know some of them really like coding you know even up to the most senior senior levels i find um i've barely written any code at google like honestly um i did most of the coding in my career in the previous couple of decades. And then as time went on, I did it more at the early stages of projects

when we were figuring out what is the design, what is the architecture, what are the concepts involved or for prototyping or certain squirrely pieces like I'm really good at memory consistency model stuff. So I can explain the x86 consistency model to you, stuff like that. Like, you know, why do you need a memory barrier after read? You know, I can explain that to you. But otherwise, I don't feel like I really enjoy it.

Like I spent almost all of yesterday doing coding. It was awesome. But I don't feel like I really need it because I can, I have the bigger vision for kind of where I'm wanting to go. And I'm kind of comfortable. I read this article about giving up the Legos. I had to do that repeatedly on Kubernetes. You know, like build one part, hand it off.

Design one part, hand it off. You know, get a SIG started, hand it off. You know, create the doc site, hand it off. I just had to do that over and over and over again, almost monthly, I think, for a few years. And I'm pretty comfortable with that. So I think an Uber TL has to be, because there are more moving parts that they are responsible for

and that they're exposed to, they also have to be more comfortable with that. You know, it's like that somebody's going to take care of it, you know, as long as it's clear who's doing it and what the expectations are. So when working with other tech leads who you are giving away your Legos to, how do you set those clear expectations with them? And how do you effectively give away that Lego? I think, you know, it's really hard.

I think there was a good blog post about, that I didn't write, about senior engineers going into a new organization. I can dig up the link later if you're interested for notes or something. Oh, yes, please.

I found that to be true, even within the same company, right? So if you have a small project and you pick the right people and they can grow into leads and as the project grows, those people grow, that's like the best case scenario. Because then you have a set of people who understand the project inside and out, who work together for a really long time. That's totally the best case scenario. So Kubernetes was like that for many, many years and it was great. My previous startup was also like that. While the startup was around, we started with the right people. As we grew the team and grew the software, they became the leads of their own areas, but were able to work really well together.

If you go into a new organization, it's pretty hard because you have to get to know people. You have to get to know how the organization works. You have to be more explicit about things and kind of document. This is the procedure I expect, or you have to try to adapt what exists currently and then maybe pivot a little bit if it's not meeting your requirements and you have to explain why you're pivoting it and you have to communicate things over and over because you know people change all the time you know people, people leave, people join. Also, it is, you know, just hard to keep all these things in mind when you have all other kinds of time pressures and whatnot. So it does help to be organized, maybe more organized than I am.

I think at some level, you know, I try to write down my ideas and my thoughts, but it tends to be brain dumpy. And it takes me more effort to sort of turn it into something more structured. Like you've seen the Kubernetes API conventions, I think. It is kind of brain dumpy. I wrote that doc pre 1.0 and just sort of expanded it a little bit after that, but never turned it

into something like AIP.dev that is a lot more structured, right? Because I gave up that Lego and went on to something else, right? I would say there's still value in reading the entire context, by the way. Like when you read someone's brain dump, it gives you perspective into how they are arriving at the decision and you don't just see that last decision itself. So many times I find that to be quite helpful.

Yeah, I mean, it does take deliberate effort. Like for Kubernetes leading up to 1.0 in July, just weeks before we announced 1.0, I read through the pile of markdown docs that we had and realized it wasn't clear how to use Kubernetes. So I started an effort to write a user guide. So I actually wrote the outline, broke down each section. I wrote most of the sections, handed off a couple of them,

and that became the original user guide for how to use Kubernetes on the Kubernetes.io. So I think something like that, but for how to organize your team, like what do you expect in terms of design docs, creating the design doc templates. At a larger company like Google, a lot of this already exists and you more just have to kind of say, we're going to use this template and that template and we're going to do things this way.

And, you know, we're going to use the standard, all the standard tools and whatever. And if there's multiple frameworks, you got to pick one, that kind of thing. So, you know, documenting maybe some of the decisions are arbitrary, so there's not necessarily a why it's just like, we have to do something. These are available. Let's pick one that's good enough and move on. Uh, that's totally fair. It's a little bit unsatisfactory to people later, but yeah, when I joined the board team, like there was tons of stuff not documented, like, I asked what does board even do?'s like uh you know there's no documentation so it runs jobs yeah uh there was not a satisfactory explanation of that

so yeah i guess thinking about like what's going to happen when you when you leave or after you hand it off like what do you want to not be dropped through the cracks? Think about that and prepare for it. So like when I was preparing to leave, I thought about this exact thing and thought about, you know, who's going to take over? What do they need to know? What artifacts do they need access to? Is there stuff that's not written down that needs to be written down?

Things like that. So that took several months to do that. And zooming in on like when you were handing off like say a project or like a lego piece um as ronald was saying um say if you you know you bring it up and then like three four people both like all volunteers like oh yeah i would love to work on this you know on this project how do you like pick in terms of like what attributes do you look for in engineers that will make them like a good um person that will like take it forward i don't think i've ever had that problem like usually nobody wants to do it you just gotta throw i mean that that is a scenario

when companies are hiring right like so you have to choose someone uh so i think you know that's i hate hiring it's hard um if you've worked with someone before it's easier because you can think about well much like i did with myself uh you know what what have they been good at how do they responded to different situations with were there you know any areas where they need to improve or things that did not go well? Why did that not go well? Some of the things are situational. So you can kind of think, is it a good match?

Definitely one thing I've learned is that early stage and late stage are totally different. If you want to start from a blank sheet of paper, you need a pretty different person than someone who is going to try to make sure everything is just right. You just can't afford the time and also you need that spark of creativity that people who are comfortable with ambiguity like like i mentioned before you know if there's multiple ways of doing something and you're not even sure if people are going to care about your

software you know pick one that's kind of good enough and move on. It takes a different... Not everybody's comfortable doing that. And some people are very comfortable making product decisions or UX decisions or something that they view as sort of not their expertise or not their job. They may be worried that they... Or doing something that they've never done before

that's outside their area of expertise. Those kinds of things can slow down projects that are just getting started. Some people are just, it just doesn't bother them, but that's kind of good because at some point it becomes a pain point. Like once you have customers and they expect the product to be stable and you can't just break things all the time or do random stuff that you have to support forever. I think one thing is just being a good role model. Everybody saw me writing all these sections of the user guide. I didn't get pushback from anybody asked to write a section, not even a single person. So if you know, if you show people that you think it's important enough that you're going to spend your time on it, that seems to work. I think

something else that I did is look for examples in other products and other systems that people thought were good. Like, what do you think a good way of documenting help for doing certain tasks is? Like, can you think of a good example and where that was useful to you? It's like, oh, yeah, GitHub has a really good page for, like, how to get help on doing any random thing. It's really easy to search in their help page for that. Or, like, Stripe has a really good API example. You know, whenever I integrated with that,

I could always find the answer to my question. Or, you know, like Stack Overflow always has the answer for this system, right? So then you can say, okay, like, that's going find the answer to my question. Or Stack Overflow always has the answer for this system. So then you can say, OK, that's going to be important to our users. Can you help make that happen for that area? So if they can realize, oh, it's actually been useful to me, and there's an example, and you think that would apply in the situation, that gives them some agency over the solution. I see.

Another thing that I think is good is sort of brainstorming mode. And I think AI will even make this easier where it's like, so one thing I find about writing is it's important not to get too attached. Just like write a draft, whatever, just pound out, like do the brain dump

get the information out there if it's worth polishing go back and polish it after you can like cut whole cut whole sections redo parts of it write it a totally different way but once you've done it once it's not so intimidating and easier to edit than to do it from scratch again. So for me, at least, like, I don't know if everybody is like that, but for me, I find that super effective. It's just, you know, just don't be attached. Like I learned that when doing my PhD thesis. That's real, that's real.

That's like 250 pages, right? And to some degree, some of it is, you know, multiple previous documents bolted together, but, you know, some of it had to be written from scratch, right? And if you get writer's block, it's a pain. So if you always want to be making forward progress, just get into the mode where you can always feel like there's no reason why you shouldn't be able to make forward progress.

Like, I'm going to write about this, and you just write stuff about that, and maybe it will be good enough that you can just keep it. Or maybe you can ask ChatGP to fix it. But getting the content out there, at least for me being the bottoms-up thinker, helps me think, oh, why am I saying this first? This other thing needs to come first,

so I'll just reorder them. And I just like reshuffle everything and of course none of the text will flow but then i can just like go turn it into paragraphs nice yes and i think this is because this is that iterative refinement process works for me nice nice uh right i was just gonna say that this has come up a couple couple times uh in our some of our previous conversations with as well like recently we were talking with kyle kingsbury uh and he mentioned a come up a couple times in some of our previous conversations with as well, like recently we were talking with Kyle Kingsbury and he mentioned a similar style too

where it's like, just pew everything out like just write it down without judging what that looks like and then you can always go post-edit and that'll help you get to a final outcome instead of trying to make it perfect from the start Right, and for engineers, if you have the help with a tech writer who can do editing or, you know,

in the pre-chat GPT era, the number one thing the engineer needs to spew out is what is the thing? What are the knobs? What are the features? What are the details? And once they do, then the tech writer can come in and say, yeah, but a user is probably going to want to do this. And you didn't explain that. How do you do that? And then they can fill that in. And you can say, yeah, but this thing doesn't make sense to me. It's like, oh, actually, that's a bug. I meant something else, right? So someone else can, once they have enough to even know what it is you're talking about, they can kind of extract the additional details, the structure, resolve the ambiguity.

And that worked really well. So one thing I had to do with Kubernetes is campaign to get dedicated tech writers working on the Kubernetes open source project and not just product documentation. Because a lot of Kubernetes vendors just wrote their own product documentation. And I argued that, look, you know,

you can read what the engineers wrote, but it's not going to be good if nobody can use the open source project. And, you know, I could spend all my time rewriting it, but I don't think you want me doing that. So, yeah, so we were able to get a good doc writing team, multiple people, actually working on the Kubernetes open source.

And I actually had to go to other vendors in the space like IBM and get them to agree to put people on documentation. So that was a deliberate effort that I did through wearing my Kubernetes student committee hat and my CNCF TOC hat saying, look, you don't want to contribute. Software engineers, how about documentation?

That would be super useful. So that actually worked. Nice. Brian, this has been an amazing conversation. We'll definitely do a round two and chat a bunch more about UX, infrastructure score, and a whole bunch of other topics. Before we close this, is there anything else you would like to add? No, it's been a good long conversation already.

I think it's a good stopping place. Yeah, you had a great selection of topics. It was a lot of fun. Awesome. Thank you so much for joining, Brian. And thanks for taking the time. This was amazing. Yeah, it was great talking to you.

Hey, thank you so much for listening to the show. You can subscribe wherever you get your podcasts and learn more about us at softwaremisadventures.com. You can also write to us at hello at softwaremisadventures.com. We would love to hear from you until next time. Take care.