The Changelog: Software Development, Open Source - ANTHOLOGY — It's a Cloud Native world (Interview)

Episode Date: June 8, 2023

This is our last week of hallway track coverage at The Linux Foundation's Open Source Summit North America 2023 in Vancouver, Canada. Today’s anthology episode features: Jeffrey Sica (Developer Experience & Programs @ CNCF), Eddie Zaneski (Kubernetes SIG CLI), Yaron Schneider (Co-creator of Dapr and Founder and CTO at Diagrid). Special thanks to our friends at GitHub for sponsoring us to attend this conference as part of Maintainer Month.

Welcome back friends this week on the changelog we are back at the Linux Foundation's open source summit North America 2023 in Vancouver Canada we have another anthology for you this one is taking you to the cloud native world this is after all a cloud native world We're just all trying to operate in it. On today's show, we're talking to Jeffrey Sika. Jeffrey is part of the CNCF. He works on the developer experience and programs. Eddie Zaneski, he's on the Kubernetes SIG CLI team, helping to maintain the Kubernetes CLI. And also Yaron Schneider, co-creator of Dapper and founder and CTO at Diagrid. But it's a big show, so let's get to it.

A big thank you to our friends and partners at Fastly and Fly. This pod got to you fast because Fastly is fast globally. Check them out at Fastly.com. And our friends at Fly help us put our app in our database, close our users all over the world with no ops. Check them out at fly.io. I'm here with Tom Hu, dev advocate at Sentry on the CodeCov team. So Tom, tell me about Sentry's acquisition of CodeCov.

And in particular, how is this improving the Sentry platform? When I think about the acquisition, when I think about how does Sentry use CodeCov, or conversely, how does CodeCov use Sentry, I think of CodeCov and I think of the time of deploy. When you're a software developer, you have your lifecycle, you write your code, you test your code, you deploy, and then your code goes into production, and then you sort of fix bugs. And I sort of think of that split in time as like when you actually do a deploy. Now, where CodeCup is really useful is before deploy time. It's when you are developing your code. It's when you're saying, hey, like, I want to make sure this is going to

work. I want to make sure that I have as few bugs as possible. I want to make sure that I thought of all the errors and all the edge cases and whatnot. And Sentry is the flip side of that. It says, hey, what happens when you hit production, right? When you have a bug and you need to understand what's happening in that bug, you need to understand the context around it. You need to understand where it's happening, what the stack trace looks like, what other local variables exist at that time so that you can debug that and hopefully you don't see that error case again. When I think of like, oh, what can Sentry do with CodeCover? What can CodeCover do with Sentry? It's sort of taking that entire spectrum

of the developer lifecycle of, hey, what can we do to make sure that you ship the least buggy code that you can and when you do come to a bug that is unexpected, you can fix it as quickly as possible, right? Because as developers, we want to write good code. We want to make sure that people can use

the code that we've written. We want to make sure that they're happy with the product, they're happy with the software, and it works the way that we expect it to. If we can build a product, you know, the Century plus CodeCup thing to make sure that you are de-risking your code changes and de-risking your software,

then, you know, we've hopefully done to the developer community as service. So Tom, you say bring your tests and you'll handle the rest. Break it down for me. How does a team get started with CodeCov? What you bring to the table is testing and you bring your coverage reports. And what CodeCov does is we say, hey, give us your coverage reports, give us access to your code base so that we can overlay code coverage on top of reports, give us access to your code base, so that we can, you know, overlay code coverage on top of it, and give us access to your CICD.

And so with those things, what we do, and what CodeCov is really powerful at, is that it's not just, hey, like, this is your code coverage number. It's, hey, here's a code coverage number, and your viewer also knows, and other parts of your organization know as well. So it's not just you dealing with code coverage and saying, I don't really know what to do with this. Because we take your code coverage, we analyze it, and we throw it back to you into your developer workflow. And by developer workflow, I mean your pull request, your merge request. And we give it to you as a comment so that you can see, oh, great, this was my code coverage change. But not only do you see this

sort of information, but your reviewer also sees it. And they can tell, oh, great, this was my code coverage change. But not only do you see this sort of information, but your reviewer also sees it. And they can tell, oh, great, you've tested your code or you haven't tested your code. And we also give you a status check, which says, hey, you've met whatever your team's decision on what your code coverage should be, or you haven't met that goal, whatever it happens to be. And so CodeCov is particularly powerful

in making sure that code coverage is not just a thing that you're doing on your own island as a developer, but that your entire team can get involved with and can make decisions. Very cool. Thank you, Tom. So, hey, listeners, head to Sentry and check them out. Sentry.io and use our code changelog. So the cool thing is, is our listeners, you get the team plan for free for three months, not one month, not two months, three months. Yes. The team plan for free for three months. Not one month. Not two months. Three months. Yes.

The team plan for free for three months. Use the code changelog. Again, sentry.io. That's S-E-N-T-R-Y dot I-O. And use the code changelog. Also check out our friends over at CodeCov. That's CodeCov dot I-O. Like code coverage, but just shortened to CodeCove. CodeCove.io. Enjoy. So we're here with Jeefy, but it's really Jeffrey.

Yeah, full name is Jeffrey Sika, but pretty much everyone in the community calls me Jeefy. People even on emails say, hey, please talk to Jeefy. And it's probably like, okay, but why the heck is this person like J. Sika at Linux Foundation? It's like, no, everyone calls me Jeefy. How did you, did you give yourself this Jeefy name or is it, you know, I mean, it's your handle. It is my handle. Self-inflicted wound here? No, no, not even.

A buddy of mine at this point, like 25 years ago on ye old AOL Instant Messenger, misspelt my name once, stuck. Jeff, Jeef. And then the Y was just like, you know, Jeef is pretty harsh. Most people are like, oh, Jeefy, because that's kind of like more of a pet name and smooth to say. Fun to say. What's your favorite peanut butter? All right.

I was going to say, I thought your mom would have picked it. My mother called me Jiffy Jeff for my entire life. I knew it. And guess what? She's a choosy mom. All we bought was Jif. The changel entire life. I knew it. And guess what? She's a choosy mom. All we bought was Jiff. The changelog. Sponsored by. Jiffy.

Jiffy. So what do you do, Jeff? Recently, new title, shiny new title, head of projects at the CNCF. Okay. Most people, when they hear that, they go, wait, all of them? Yeah. And I'm like. I would think all of them.

Yeah. And I'm like... I would think all of them. Yeah. Pretty much. Honestly, what I'm really doing is I'm a community member first. I came up as a Kubernetes contributor, been around for a while. So I know a lot of people. I know a lot of the communities and open source projects around it. So I can go and talk with them and figure out like, hey, what do you need? How can we help better?

What can we do better to enable your project? New projects that are coming in. Hey, how can these projects potentially collaborate? Because like I'm an engineer first and then kind of, you know, schmoozy, try to be nice to everyone second. So it's kind of hard to define my job and, you know, a job description,

but it's really talk to projects, see what the CNCF is doing, make community happy. Gotcha. You take the requirements from the customers and you give them to the developers. That's right. I joined a foundation so I don't have to hear those words. What you do at Inateck is you take the specifications from the customers and you bring them down to the software engineers. Yes, yes, that's right. I recently watched Office Space, so I had to bring it in.

How many projects are there? 160, and right now as of whatever today is the 10th, May 10th, I think there's 12, so there is some number above like 7 or 8 that are currently getting voted on to be adopted into what's called the CNCF sandbox. Think of it like proof of concept projects, projects that don't necessarily have a large community and they're looking to build a community. They apply to the CNCF sandbox and then those get voted in. Yeah, I talk with my hands as well. I'm somewhat Italian. I like it. I'm down with that. I talk with my hands as well. I'm somewhat Italian.

I like it. I'm down with that. I talk with my hands too. I want to get super excited and I'm super excited right now. So you got sandbox, you got incubation, you got graduated. Oh yes. Okay. So you're not all over all projects, but you are over most projects. Let's talk to people in the CNCF and see which, no, I'm honestly, it's over all projects because I'm interacting with projects at every different level. It's just I don't want to say I'm in charge of all of them. That's not true at all. But I would say I communicate with all of them, and I'm trying to help the CNCF work with projects in a better way. Gotcha.

Give us an example. How does that play out for recent? Recently, when I joined and one of the things that I've been really pushing for is a lot of the processes to grant projects access to cloud resources that are like group cloud resources under the CNCF. Or we have license scanning software or services. We want to give those to the projects and then step out of the way. Hey, we don't want to be the bottleneck. But most of the way that we grant that access is still a manual process, even though all of these things have APIs. Well, gee, you know, you look at what Kubernetes has done with their community management, like creating a user group in Slack is memes aside, like laugh, laugh at home. You edit a YAML file. Oh, you're joining a GitHub group or you've become

like a SIG chair. You're editing a YAML file. And then once that file is committed, it's just GitOps all the way down. Like your access gets granted in GitHub. Your access gets granted in Slack, all of that. Why don't we do that for all of these services that the CNCF is hosting? Right now, it's still ClickOps. That was cool when the foundation was 10 projects or 15 projects. We're at 160. 160 projects? And we're not slowing down.

And 12 more being added. That's crazy. No, those are up for vote. Those are up for TOC vote. How many get rejected? I actually don't have that off the top of my head. I would be willing to guess sandbox

wise, it's probably 75% acceptance rate, but please do not hold me to that right now. 9 out of 12 are getting in. Don't. Hey, hey, hey. We're not naming names. What is the, I guess, motivation? Probably might be the best word, but what does the CNCF do in terms of like,

you got 160 projects. What's the long-term goal? Is it to be bigger than that? Like what service do you provide to the cloud native world? Like what is it that you all do or hope to do? This is going to be interesting because if you ask different people in the CNCF, you might get a different answer.

And there might be a canned response and I should know it. My answer is there is, aside from the couple stable patterns like Kubernetes and the way that it has an API and like declarative over imperative stuff, everything's stable right now. Like that pattern is established. What things and what problems when consuming that pattern need to be solved? A good example was, okay, so now we can create all of these containers and orchestrate them in a meaningful way, but now we have a giant distributed system. What do we do in order to monitor that thing? Well, Prometheus came out of that. So this is a long-winded way of saying we have this foundational technology.

At this point, we are accepting additional projects to help flesh out what cloud native actually means. And the definition itself is evolving. We have a whole bunch of WebAssembly projects. Well, why is that? Because at its core, WebAssembly is, I don't want to say just another container runtime because that would be bad, but it is another application runtime. You build it a different way. It has a very different look and feel than a container, but still that idea still fits into the pattern of cloud native. So that still solves a problem. So, geez, what would I do?

TLDR, we're accepting a bunch of projects because not all of the problems or questions have been answered in what cloud native is. So you're attempting to and in many ways succeeding in defining the foundation of cloud native. Yeah. And everything was originally built on Kubernetes because that's what, I guess, was the founding project

that really kicked off. So we come back from the Dan Con days. We were early CNCF days, Michigan, but we were there when it was just two or three projects, a very small CNCF, the original founding days. And as we see it grow and grow over time, it's a lot of great stuff happening for open source, but you know, you're on the inside, you see what's happening. You are in touch with all these projects. What is the mission? Like what is

the end game for CNCF? Jeez, three. Honestly, it's what is next? The definition of cloud native in a nutshell is really doing distributive computing in a repeatable way. I mean, that's my definition in my old noggin, right? But that doesn't mean always use Kubernetes. Sure, right now, hey, Kubernetes is, I mean, you look at all the stats, there's adoption still like up into the right, it's a hockey stick. That doesn't necessarily mean it's going to be the same thing or it's going to be the answer. So like, what is the end goal? We don't really have an end goal aside from if you are doing some sort of distributed computing, like trying to solve or consume or build distributed computing, distributed platforms,

how can we do it but make sure that how it's being done is in an open source way? Maybe Kubernetes, you know, goes by the wayside and something else comes up. Maybe there is some new, like, WebAssembly orchestration platform and then everyone starts adopting that. We want to make sure that that's still possible. Like, the reason why right now Kubernetes is like, I don't want to say flagship, but the big thing that everyone thinks of with the CNCF

is because of its popularity, not because the CNCF is saying everyone use Kubernetes. If something else just starts shooting up into the right, we also want to be there to help enable them and make sure that the lessons we learned from Kubernetes just, again, hockey sticking up, can be learned over here so they have an even better experience than Kubernetes had. And it had a lot of growing pains. So let's not have another project repeat that.

Do you all want all open source projects that support cloud native to be a part of the CNCF? Not necessarily. Well, that's probably not a good thing to say for me and my employer, but honestly, I think that would not... Part of the charter in the CNCF, specifically the TOC, is they are not kingmakers. The TOC, the Technical Oversight Committee, which is like elected positions, they're the ones that pick which projects get adopted, which projects aren't adopted. Like, they dictate who's in the CNCF, and then we, the staff, enable them, support, you know, do all that sort of thing. So, like, I'm coming at this as my opinion, man.

Yeah, well, you know, that's just like your opinion, man. Honestly, I tangented. Like, I already forgot the original question. We're all just over here. I can ask it again. Please. I will do Big Lebowski references for the whole podcast. That's a problem. These guys are trying to joke with me. I'm trying to ask the question here.

We're hoping you forget it so he doesn't have to answer it. I'm with you, but I'm just saying he's trying to dodge it. Let's keep going. I'm going to slow my face. Let's try again. I'm with you, but I'm just saying like, he's trying to dodge it. Let's keep going. I'm going to slow my face. He's like, uh. Let's try again. Sure.

So I'm curious if, because it seems like you've got a repeatable way to support projects. Yes. So it makes sense that if it's supporting cloud native and it's open source, you'd want it as part of your organization. I remember now. Yeah. So there's like, I will go back to, there's like my personal answer and then there's probably the party line. Can you's like my personal answer and then there's probably the party line.

Can you give us the personal answer? The personal answer is I don't think that would be healthy for the ecosystem. Just, again, the tangent of the TOC and the fact that they say there are no kingmakers, same thing. I also think that if all projects were in one foundation, that's probably not healthy for the ecosystem. Like, cloud-native does not mean it is a CNCF project. There are plenty of other cloud-native things that are not in the CNCF. Right. Like, there's Nomad.

HashiCorp has Nomad. That's a container orchestration platform. There's still a lot of work being put in around Nomad, but that's not... But they're an IPO company, though, so it makes sense why Nomad isn't there because that would be troublesome for their business. But Nomad is an open source project. There's a weight, though, to being a project in the CNCF. You have the CNCF landscape, right? So by nature, you want to communicate what is and isn't.

But at the same time, doesn't that give it a weight to a project that is? Well, landscape is a bad example because the CNCF landscape has projects that aren't CNCF adopted or CNCF projects. That's true. I'll give you that. I was actually thinking Nomad might actually be

on the landscape. I haven't looked. Let me give you this example. So we've been here for eight hours, ten hours. I've talked to two people who have said hi I'm X

and I'm with Project Y we're the CNCF and it's like there's a there's a trend there's a clout to that there's like

yeah right so aren't the TOC then I mean aren't they kind they kind of are king makers in that sense right like they kind of are kingmakers in that sense, right?

Because they're the ones who decide who's in, and everyone who says that they're in, now they're cooler than they used to be. They can leverage the brand equity of the CNCF. Right. But in that case, the TOC isn't picking one technology over another, at least with the sandbox. What's usually happening is they're judging maturity, whether it does fit, like whether

it is a cloud native thing or not. Like if my transcoding software or some other random project that has nothing to do with cloud computing gets submitted to the sandbox, which that happens, TOC doesn't want that. Like that's not the CNC.

It has to be inside the scope. There's a velvet rope. So my personal opinion is I doesn't want that. Like that's not the CNCF. Yeah, it makes sense that it has to be like inside the scope. So there's a velvet rope. So my personal, my personal opinion is I don't think that's healthy for the ecosystem. Sure. But that said, and I think the party line would be if you want to be supported in the ecosystem and have the name, like the namesake of the foundation behind you. Yeah. You probably want to join the CNCF. I also have feelings that sometimes some projects probably shouldn't have applied. But again, that's why my personal opinions and the TOC are the people that vote on it, not me. Your job is to support the ones that do make it in. Yep.

However, they need support. And honestly, projects that aren't in the CNCF but are in the landscape, I'm still around to support and talk to because, again, I don't think this is necessarily a bad thing to have projects outside. Also, projects outside looking in could potentially spawn other projects that do want to come in. Sure. Do you like this job? Yeah.

Best job I've ever had. And I'm not just saying that because of, you know. Because it's being recorded. Because it's being recorded and I'm standing in a Linux Foundation event. No. it's being recorded because it's being recorded and i'm standing in you know a linux foundation event no uh my not so brief but honestly short resume career i worked at the university of michigan for 16 years and then red hat for three and then i started here a year and a half ago so out of those not getting into like different departments the university, but out of those three areas or places, Linux Foundation, CNCF is the best.

And your path came through contributing to Kubernetes? Yep. I actually did a little bit of contribution back in ye old days. Like we're talking 2014 when it was just open sourced and still under a Google, like had to sign a Google CLA to contribute to it. Then my path at the university kind of took me away from it after probably a year.

And then I started contributing again in early 2018 and wound up becoming a SIG UI chair. So the Kubernetes dashboard that some people kind of dunk on, they were having leadership issues. They just needed someone that could kind of come in and do more PM work. And also, I had a background in front-end work. So I came in and just helped them out, wound up becoming a SIG chair for a few years. And then I stepped down after I mentored someone up.

It's a Cinderella story. Nah. It's a Cinderella story. It's a Cinderella story. So you say you like this job. What do you like most? What is your favorite thing that you get to do every day? I feel like this job actually has a real impact on people's lives. Uh, when I worked at the university of Michigan, one of the things I did was informatics and like directly impacting patient care. I loved that. Like I'm not saying patient care and open source are similar, but there is definitely, you know, that impact where I know that I have helped and like impacted other people's lives here. Similar to like being able to help someone's patient care just by supporting like a clinical app that I wrote that deals with their results.

Different, but same. Um, that just gives me warm, fuzzy feelings. Cause I don't know. I'm weird. No,

that's cool. Make the world a better place. Impact. Change lives. I was always taught to leave the world better than you found it. I'm one of those people that will make the bed in a hotel room when I'm leaving. I didn't know those people existed. They don't.

Okay, so I'm a psychopath apparently? It's the endowment effect. That's what this is. The endowment effect is that you don't wash your rental. Say what? You don't wash your rental car, for example. It's the endowment effect. If you own it, you think it's more valuable.

And when you don't own it, you think it's less valuable. That's why we don't wash our rental cars. Yeah, but he makes his bed in his hotel. I know. He's the anti-endowment effect. Okay. The anti-anti. I can never remember the social experiment or the dude that did this,

but do either of you know about the shopping cart? Like, I don't even know what to call it. No. There's someone decided that you can tell whether a person was not necessarily good or bad, but more focused on the whole versus the self based on what they do in a grocery store parking lot. Do they put their shopping cart back where they are supposed to put it or not? And then you can watch people. And if other people actually take the shopping cart like someone else's and put it away, it's like they're the people that actually want to make the world a better place. Yeah.

You know, in ye old days, supermarkets used to employ people that would walk your stuff out to you. Well, guess what? ATV still does it. Do they still do that? Well, sorry. I spoke too soon. They do it for some. What? Well, usually for senior citizens. Okay.

Pregnant people. I don't know about trendy people. No, pregnant. I said it. They're like, nice shoes. I'm going to walk your groceries out. Trendy people. Do you remember that back in the day?

Were you around back then? Yes, but I... Kind of a small town in southeast Michigan. That never really happened. They never did that? We always... If a senior citizen or someone that was... There was a position called bagger.

Wasn't it called bagger? Yes. I mean, they still had baggers where I'm at now. There's still baggers in my grocery store. Yeah, but the bagger would actually walk with you out to your car. Oh, no. And load the bags into your car, and then they'd take your cart, and they'd take it back. Yeah, now that's called whoever, you know, delivers something to your car when you mobile order it from Target or, like, PetSmart.

I do miss those days. There's something about that. I think you're onto something, Jared, because what you said you liked to bite your job and how you get to change lives is similar to this because every step of the way you get the support. You get to make the process, the experience a little easier, a little bit better. Yes, the CNCF is the think you've unified a diverse... Let's hypothesize. If the CNCF did never exist or was never formed, how would... If cloud-native was never termed, or even if it is termed,

it doesn't matter, how would the world be if there was no CNCF to tie it all together? That's actually tough to hypothesize. So one of the biggest benefits, like thinking at a super high level, is we're a neutral place for these large vendors to be able to collaborate and essentially make everything better for the consumer in a standardized way. Take that away and what do you have? You have... Proprietary.

Everything winds up being proprietary. No clarity. No focus on users. I mean, they'll focus on users so far as once they get you in there, you're locked in. Like major vendor lock-in. I think that's the biggest thing yeah the vendor the vendor lock-in would be horrible like i can't even imagine it yeah and i'm trying like i'm trying to remember back in like heroku php like 2008 2009 days of hosting web services. Everyone kind of had their own thing. But even then, it wasn't that bad. Yeah.

Like stuff made sense. But also, no one was really sticking around long enough to potentially have, I won't say a monopoly, but a lion's share to lock you in so it doesn't make sense to shift elsewhere. At that point, everything was VMs, right? Yep, exactly. Hey, look, I can spin up a VM on my box, make sure it works, and then, I mean, ship the whole thing.

It sucks, but doable. Sure. Jeff, I'm glad you talked to us, man. Dude, this is awesome. Thank you for sharing your story and the CSCF stuff. Shout out to Kara for dragging me away from the booth. Real quick, what's your favorite project and what's your least favorite project? Go. Absolutely not.

I refuse. This interview is over. Imagine me knocking over the microphone. Well, not the project, but the people. Tell us what's your favorite person and your least. I'm just messing. Oh, actually that I can at least tell you my favorite person. Okay. Uh, I had a coworker who was also a roommate who was also my best friend and he's my best man. He was the best man at my wedding. Uh, we worked at the university of

Michigan since the start. We both moved departments from a pathology over to advanced research computing. Wow. I went to Red Hat. He went to Google. So my best friend's Bob Killen. He lives down the street from me. That's cool. We are almost inseparable, except when I get to go to events and he doesn't. Trust me, if he was here, I would have been asking for another microphone

because we just would have done that. We do have one more if we need it. So Bob, come on. That's cool. Oh, are you going to KubeCon Chicago? I'll drag him over. phone because we just would have done that we do have one more if we need it so bob come on that's cool oh if are you going to keep con chicago i'll drag him over let's talk let's talk off mic i got ideas all right thanks jeff thank y'all what's up friends i'm here in the breaks with one of our sponsors raycast i'm here with thomas paul man the co-founder and ceo of raycast so thomas i I recently moved from Alfred to Raycast. I'm on the pro plan, loving the AI integrations and everything else helped me to be productive. Also helped you

launch the pro plan recently on ChangeLog News. That was awesome. But what I want to know is why you built Raycast in the first place. I think software as we experience is flawed and inefficient. And I know this is a pretty big and bold statement, but this is really where the idea from Raycast comes from. Because when you think about it, when you interact with a computer, you have like a certain action in your mind that you want to do.

To perform that, you need to translate that into clicks and keystrokes on a computer. And that isn't really intuitive. That's not how we used to work. When we crap something in the real world, we just crap it and do it. There is no communication or something like that necessary. But somehow we got used to that this is how software works. And we work around that. It kind of works, but I feel really it's an inefficient way to use a computer. So with Raycos, we re-envisioned that and we said, what is if I

could use all my tools in a single interface? They look the same, they behave the same. I'm super efficient at it. I just enter what I want to do. Everything is driven by the keyboard, which I'm used to as a developer. We basically started building exactly that and started with the basics of like, what if I could launch an application? That's an easy task, right? What if I could find a file that I'm looking for? Okay, that's nice. But at some point reached in the threshold where we said, oh, but now I need to create a Jira issue or see my assigned issues and change the status.

That is where it gets really interesting. It quickly became clear to us like, wow, okay, there is actually demand for that. That was really the start of Raycast where we felt this is something special. There are like so many people who want to be more productive. They want to have a great tool that they can use, but they're also willing to put in a bit of work of maybe integrating with their own services that we don't have support for now. Okay, cool. So one of the things that stood out to me for your homepage when kind of learning about Raycast and discovering what it can do. It says in big, bold letters on the homepage, supercharge your productivity. Why is that the leading statement for Raycast? Yeah, so for us, productivity is like, it's very hard to measure

if you look down for it. People say they can do something faster. People say they're more productive, but it's very hard to quantify so we thought hey we have a tool that generally just makes you more productive in many different ways so it super charges your productivity it brings us to the next level you like just can do things much faster than anybody else you can interact with your tools quicker you're basically like operating on a different level there's always a saying of a 10x developer, which can do things a lot faster, right? So it goes along those lines where

when you see people using a Mac with Raycast, they use a Mac differently to somebody that uses a Mac without Raycast. Okay, so if you're on a Mac and you want to be productive, you owe it to yourself to try Raycast. You can try it free. Almost everything they have is free. I mean, lots.

I mean, I told him, Thomas, you kind of give away too much for free. But hey, that's that's their choice, right? But if you want to be productive on a Mac, Raycast. If you're using a launcher, if you're using Spotlight or anything like it, Raycast will take you to a whole new level. I'm using it. I love it. I think you should check it out. Go to Raycast will take you to a whole new level. I'm using it. I love it.

I think you should check it out. Go to Raycast.com. Again, Raycast.com. So the Kubernetes API, that's what you work on, right? I work on CLI. CLI. Oh, the CLI. Okay, it's an abstraction of that, right? You're actually interfacing with the API. We're probably the biggest consumer of theI. Okay, it's an abstraction of that, right? You're actually interfacing with the API. We're probably the biggest consumer of the API.

Okay. What is that? How does it work? So are you familiar with how the Kubernetes project is broken up into special interest groups? School me, school me. Yeah, so we got SIGs. So basically every part of the Kubernetes code base. What does a SIG mean?

Special interest group. Okay. And so we got a SIG for API machinery. They own the API and the stuff that runs on the master nodes. So I work on SIG CLI, which is the SIG for the command line tooling. So KubeControl, Customize, KUI, which is a GUI framework for KubeControl, a couple other subprojects. So I've been working on that for four years now

and it's a lot of fun. Cube control, huh? Is that official? Well, you will notice throughout this talk, I say it many different ways on purpose. You just called me out early. You're a diplomat. If you say kubectl here in a bit, it's

on purpose. We're also going to say kubectl. Who says kubectl? People say kubectl? Tube Cuddle here in a bit, it's on purpose. We're also going to say Cub Ectle. Cub Ectle. Oh, gosh. Who says Cub Ectle? Well, if you want to hit all the variations. People say Cub Ectle? Yep. Is it for fun or is it for serious?

I've heard both ways. Wow. Why not, right? If you can interpret something 17 ways, why not be 18? It's true. I just think that maybe Kubernetes is so complex and intimidating that whenever we have people want to talk about it, we just bike shed the kubectl thing. What do you think? Sure.

I feel like you and I always end up right here talking about the kubectl. For sure. You can go to kubectl.info, and it's a recording of Tim Hawkin who originally wrote it saying how he pronounces it. I think we had Tim on the show back in the day. We talked to Tim forever ago basically. The godfather. Yeah, when it first became a thing.

Yeah. Nice. He was at Google then. Is he still at Google? He's still at Google. There you go. Good for you, Tim. Slay it.

What should we know about the CLI? What's important with its development team, the SIG? Maintaining it. Maintaining it. Yeah. Like what's important with its development team, the SIG, like how's it work, maintaining it? Yeah. So one of the hardest things we have to do is say no to people all day, right? I'm sure a lot of people have told you that. But everyone wants a short flag for everything.

Everyone wants a long flag for everything. A lot of flags. Everyone wants every feature as a flag or command. What's the language of the CLI? It's all go. It's all go. Cobra. I've been doing a lot of Bash scripting.

I'm like, you know, at some point I'm going to graduate from Bash to something else besides Bash, but it does a lot. Oh, yeah. Bash scripting is a lot of fun and it's pretty powerful, but I feel like if I keep going in this direction,

Go. Yeah. I mean, I feel like I'm learning Bash, right? Okay. I've never sat down to properly learn Bash, and you can do a lot with it. Yeah. And thank God for ChatGPT,

because I'm learning Bash left and right because of ChatGPT. It's somewhat esoteric in my history, but I think having GPT would make it super easy to accomplish a lot of things. It is. I mean, there's a lot you can... I mean, you can iterate quite a lot with it, which is a side tangent from crafting a CLI with Go.

Yeah, but even the looping and the conditionals inside the loops, there's weird times where you use the square brackets and you don't have to. And then there's the flags. There's like conditional flags inside of the loops and stuff. How many square brackets do you use? Yeah, multiple square brackets change things. It is esoteric, but powerful. Very powerful. And you don't have to.

And there. It's already there. And you use it just, when I say you, I'm talking about me, use it just infrequently enough that you always have to Google for the syntax. Oh, yeah. So, again, GPT is for the win on that one. Yeah, for sure. And on that note, I am very thankful because I've, like, because this isn't about chat GPT necessarily,

but I think it has flattened the world to, like, allow people who are,Curious or BashCurious or ScriptingCurious. CubeControl. CubeCuttle. What was the other one? CubeCTL, CubeControl, Cubectl.

It's kind of cool to say, actually. Cubectl. You know what you want. You can describe what you want, but you can't quite get there. But if you learn enough and then you can repeat yourself, you learn that stuff. This episode brought to you by OpenAI. That's right. There you go. How many flags does QBectl have? Oh, man, I can't tell you that.

We got a lot. We got a lot of subcommands. We got probably 20 subcommands, maybe more, and they all have lots and lots of flags. We basically have an entire framework just to add flags to the commands if they get instantiated. What's the biggest challenge that you said saying no, but maybe personally, maybe not as a team, but personally? You know, you've been on the project for four years.

We didn't exactly hear about how you got there or anything like that. But what are challenges maintaining a project of that high demand and use? Definitely contributors, right? We have a saying on Kubernetes, chop wood, carry water. Say again? Chop wood, carry water. Chop wood, carry water.

Kind of doing the unglamorous work that someone has to do. And we need people to just come do that, right? Triage issues, respond to open pull requests, review, and you know, one of the things I encourage lots of new people to do is you don't have to be a reviewer for the Kubernetes project to go and review pull requests, right? Just doing an initial

pass of being like, oh, this is probably a better way to write this if statement, so you don't have like three else's under it, you know, just like little things. So that's what I encourage a lot of new folks to do is just start reviewing code, just start responding

to issues. Yeah. Just comment on the issue. Yeah. Who's contributing to the CLI? Who's contributing to the CLI?

Is it the SIG team primarily or is it outside contribution? So I'm sure every SIG would say How risky is the code? Well, it's probably the part of the oldest code base of Kubernetes itself, right?

Because you build the API server and the node, and then you build the CLI at the same time to talk to everything. So we got a lot of dragons that are there and a lot of stuff we come across. So it's funny. People don't realize that Kubernetes is all JSON internally, right? You hear the Kubernetes and cloud native world complain about YAML. YAML, yeah. And Kubernetes doesn't know YAML internally. It's all JSON.

It's all JSON. That's news to me. So it goes JSON to YAML on the response. And then when it comes to the command line, we actually marshal it back to JSON. And then we have to go from JSON to figuring out what Go type we have, right? So if it's a pod or a node or something. And so that's a large chunk of the code that we maintain is just dealing with marshalling from format to format to format and then figuring out what Ghostruct we have at the end of the

day. Why don't you just go from YAML to Ghostruct? From YAML to Ghostruct. We could. That would just take one marshal out of the list. It would. It's working with the... The Go YAML world is kind of interesting. We could probably talk about that for a long time.

But we have a forked version of the Go YAML project. Gotcha. There's many different versions. The project bundles... But this one is yours. Yeah, the project bundles like three of them. One didn't preserve comments or something in your YAML.

When you're dealing with client-side YAML for users, you want to keep their comments around. That's one of the problems with JSON, right? It's like no comments. No comments, right? So you got three YAMLs in there? We got a couple versions

of the same library, yeah. We try to keep one, but YAMLs in there? We got a couple versions of the same library, yeah. We try to keep one, but YAML is a special case. Sure. You got to do what you got to do. I like YAML. It's not the worst. It's not as bad as people make out. No.

I'd rather write YAML than JSON. Yeah. Agreed, for the most part. I feel like you can shoot yourself in the foot more with YAML. Yes. And complex YAML is very complex. And complex YAML is very complex. But simple YAML is very simple. Yeah.

I'm not against it. JSON might be easier to read if it's prettier. Yeah. Potentially. It's more verbose. Yeah. You can see the indentations and the nesting a lot better than you might, I guess. So I guess you can see either of those pretty easily.

I like it in YAML because my editor can show me, like, the number of tab indents I have, right? Right. So it can, like, show me a one, two, three, and that's really nice to see. Yeah. So that's your biggest challenge is this marshalling around YAML? Contributors? No, contributors.

Contributors. Yeah. So people working on the project, I work with people from Google, Red Hat. We had someone from Shopify that fortunately just got laid off. Pours them out. A bunch of Googlers, Red Hatters. Don't pour your gin out. Don't pour your gin out, no.

Water. Pour your water out. Yeah. And then we have people who come by and they want to get involved in Kubernetes. And they're curious about things. And the CLI seems like a great entry point. Yeah.

As a project, we're still struggling with mentorship programs and onboarding. And one of the hard parts is maintainer burnout. Because we can, early on I was very happy to sit down with someone for hours and just walk them through stuff, answer every question, help them write their code.

And then they make their one contribution and then they disappear and don't come back. And you do that enough times and you're feeling really crispy. Yeah, it makes sense. Do you, like, do videos? Do you find ways to not repeat yourself in that way? So you can say, like, here's me telling you how to do these things and sit down with you, but maybe there's a video you could do or documentation. And that seems to be the easy, hey, why don't you just do documentation? But is there a way you can sort of like put down the wisdom, so to speak, from a mentorship perspective and succession planning? This is

something that's big for maintainer month is how can you, how can you operate with balance as a team, as an individual, and then also how can you plan for succession when it's necessary? It's definitely something we're working through with the project. We have tons of developer documentation, right? Probably too much that people don't read, right? It's overwhelming when you first come in. Getting your development environment set up, right? It's a's so many moving pieces, and container runtime really only works well on Linux, and most people aren't running Linux as their OS.

How dare them? Right? Linux. Linux for life. But it's something we're definitely trying to work towards. We want to make as much onboarding material as we can. We've had mentorship cohorts, but at the end of the day, it's very complex as a code base. And it's just old.

And there's so many, there's so much, we don't say tribal knowledge anymore. What do we say? Preconceived knowledge. Wisdom. Decisions that were made a while ago, right? And people come in headstrong,

really wanting to help out and contribute. And it's like, well, we tried that and here's why it didn't work six different times. And that is the hard part, is the context and the history. How do we communicate that to new people? Right. What's the process to become a contributor long-term?

Like you put this time into this person, you walked to their code base and they gave one contribution and never came back. What is the process to have a long-term contribution plan? Is there a term of service? We hear from OSPO's like, hey, come for a term of service. That means maybe a year, maybe six months, maybe it's three years. And then there's repetition in that. How do you all plan that out? Is there a form and function around that? Do you know Mike McQuaid?

Yeah. Yeah. So Mike McQuaid, he's the lead maintainer for Homebrew. And he's got a blog post that he wrote back in 2018 that's kind of resonated with me. It's don't mentor first-time contributors. Don't mentor second-time contributors. Mentor third-time contributors. And it's the idea that, like I explained, you get burnt out if you keep spending time on people who just don't come back. But if they've made two contributions and they've come back for the third, it's like, all right, cool.

You're in it. You've gone through the hard part, the weeds. We can grow you into a maintainer. Because that's the goal at the end of the day is to grow people into maintainers. We want as many as we can get. What brings somebody back three times to the Kubernetes CLI, for example? What is it that brings them back? Is it the Kubernetes CLI, for example? Like, what is it that brings them back? Is it because they have a vested interest? They're super curious,

they have funded time interest, their employer pays for it? Like, what are the attributes of a person who comes back again and again? I don't have a good answer. I really don't. It's people who want to get involved and contribute back. And some people might be encouraged to get involved in open source. Some people want to learn Go. They want to learn Kubernetes in general. Yeah, we see people come for all different reasons. Some people really just want to build their resume and just want to build up their GitHub stats and show them that they've contributed.

So yeah, it is hard to filter through and apply the right time to the right folks. So what do you think of this word, rewrite? Do you like that word? It's a word. It's part of the English language. Okay. Have you ever considered it with the CLI?

Like not throw one out and start new fresh, but start fresh alongside the one that exists. Oh, yes. The parallel. The old big rewrite. The parallel rewrite. Because, I mean, you got a lot of baggage, according to you. And that's perhaps scary, but maybe in an open source world, not so bad way of like,

instead of just like trying to bring this one up to snuff, you just maintain it, status quo, and rewrite the sucker. Yeah. So we have an initiative that we've been rewriting commands to our new pattern that's more concise and we got the options and the flags dangling off the command struct. In a Go world, it makes a lot of sense.

From scratch is an interesting one. The Kubernetes project in a whole, we are terrified of breaking users. So the example I like to give is I've been trying to get delete confirmation into the CLI for the longest time. When you delete a namespace in Kubernetes, you delete everything that was in that namespace. When you accidentally delete all namespaces in your cluster, you've wiped everything out and you're going to have a bad time.

And I could show you tons of GitHub issues where people say, why was it so easy for me to make this mistake? Why didn't it ask me, are you sure you want to blow everything away? And the reality is that we can't just start asking, are you sure you want to delete everything? Because your CI pipeline would break, right? We'd break everyone's build. People are updating their CI runs and they don't know what version of the client they're using. They don't really read the release notes. So that's just an example. I've been trying to get delete confirmation since I

started. Isn't that what Semver is for? Major release. We don't want to do a major release for the project. As far as we know, we can barely get people to upgrade the minor versions. But majors are easier because people get excited.

That's right. Is there something to learn from the way Linux is distributed? Like LTSs and versions? I mean, every time I do a new Ubuntu installation, it's 18, it's 22, it's 20. And I'm cool with that. There's an LTS. There's a spectrum of risk.

It's clear. Is that a possibility with the CLI? This is a crucial piece. It's like the centerpiece for Kubernetes for the most part, right? It's the main consumer of the API. It's definitely the first thing you reach for, right? There are two answers there. So the first

one, LTS is actually something we just started talking about again. So we were on KubeCon in Amsterdam like two weeks ago and Jeremy Rickard from Microsoft revived the talk around the working group for LTS. So we did it a couple years ago. We determined that it wasn't something we wanted to do or support at the time

or had the capability. So that just got revived two weeks ago. And then the other thing, KubeControl is versioned as part of the kubernetes project itself right so 120 I can't release a separate version of kubectl

right so we do have a proposal out that probably needs to get revived but that was something we wanted to do but then you get the problem of the compatibility and skew matrix what version of the client is supported by what version of the API server and useful software gets upgraded so if matrix. What version of the client is supported by what version of the API server?

Useful software gets upgraded. So if you... Here's one thing we learned from GitHub and a lot of other things out there where it's like permission to mess up, permission to do something different. If you can release a different version of it in parallel that has what everybody wants

and it fixes all the problems and maybe internally it's easier to develop and it's potentially easier to have contributors and easier to document like that has potential there's an opportunity for that useful software just to get upgraded because hey this is just so useful yeah everyone this person's using it that company is using it and it's it's sort of like a social norm to upgrade because it's just useful. Right. The rewriting thing would probably get, like, it probably would be impossible to get through, right? Because we, through any significant changes to the project, go through what we call the KEP process, the Kubernetes Enhancement Proposals.

And I could just see, like, opening a KEP for, rewrite kubectl, and just, like, no. It just gets closed, right? Yeah. What if you already did it? What if we already did it? That's what I was thinking. I mean, it's not stopping. First time contributor shows up, I rewrote this.

There's nothing stopping us or anyone from doing that. Yeah. The reality is we are changing the tires on a bus that's moving 1,000 miles an hour down the highway, right? Maybe it actually turns into more like a Yarn and NPM kind of situation where it's not you guys that rewrite it, but it's somebody else that comes alongside and says, well, we can write our own CLI against the Kubernetes API, and here's seven ways it's better. And, hey, who wants to use this?

I don't know if you can actually just side install that sucker and use maybe its kubectl with C-U-D-D-L-E or something. That's a conference now. Oh, it is? Dang it. In a perfect world, the kubectl wouldn't exist. Why is that?

Because you can think of it like SSH for a server. I don't want my developers SSHing to my server. I don't want my developers SSHing to my server. I don't want my developers pushing and making configuration changes to my production server. I want a trusted build entity that is applying these changes after they've been reviewed. So it's just like, it's kind of giving the developer

keys to the castle. Providing namespaces. I'd rather not have to give people the client in the first place. I think instead of building one from scratch, I'd rather not have to give people the client in the first place. So I think instead of building one from scratch, I'd love to see us get to a point where the GitOps tooling and all this other stuff is in a place where you don't need it in the first place. You can rewrite it in a different route.

Write something else. In the GitOps world, build that thing to make it obsolete. Yeah, that's fair. Then you can take a vacation. Yeah, I would love one of those. What I like about this podcast, though, is we look at things like Yarn and NPM. We look at, you know, we're not only in this cloud native specific world and, you know, sort of have tunnel vision. We sort of see outside of all of software, what was done here to solve that problem and what was wise about that choice that we can apply here?

That's what I love about the conversation I think we get to have is that Jared and I have the luxury and the privilege to speak at software at large, really. Plus, we get to bike shed things, but not actually be the person that has to go paint the bike shed. That's right. We can give you the idea of, hey, Eddie. We're like, Godspeed, bro. I told Eddie to rewrite the thing, and he just won't do it. I got a good one for you all, then.

So I also work on the build and test infrastructure for the project. And we're unique as a project in that we handle distribution of all of our own artifacts and binaries. And our artifacts aren't just binaries. They're containers and OCI images, right? So our CI bill is like $3 million a year. Google gives us $3 million of GCP credit. Shout out to them. Thank you, Tim. And I think it costs us like $250,000 a month for storage and network ingress and compute and egress. And we're working very hard to get that down, actually. Amazon just also gave us a $3 million donation

and we set up a registry proxy. Thank you, Amazon. And for a while, everyone was downloading from our container registry because you can't just mirror a container registry like you can mirror a Linux kernel, right? And so I think some work can probably be done on that space, but that's a problem that we deal with

that a lot of other projects don't deal with is we have to distribute and front the bill and host all this stuff ourselves. That's a big bill. That's a hard problem. $3 million just for CI. Have you tried R2? Free egress.

We are talking to Cloudflare for a bunch of different things. They would love that, I assume so, yeah. Yeah, hopefully they help us out. Yeah. We want to do caching, too, with Cloudflare, right? Or Fastly or someone. So shout out to them, please. We like them both.

We're expensive, but we're very expensive as an open source project to support. And crucial. It's a cloud-native world. Yeah. Just trying to operate in it. You probably know our audience to some degree. What else is left unsaid? What else should our audience know about crafting the CLI and interacting with potential contributors?

Maintainer hacks. Yeah. Maintainer hacks. Sure. Maintainer hacks. So my maintainer hack is that I triage new issues first. And people kind of, this is controversial

probably, a lot of people should, you should start with the oldest issues and triage them. We found that our newest issues are probably the most relevant, just because we get hundreds of issues a week opened on the project, right? And we have, the way the Kubernetes repo works is we have the main KK repo, the Kubernetes slash Kubernetes, right? And we have the way that Kubernetes repo works is we have the

main KK repo, the Kubernetes slash Kubernetes repo. And then we have staging repos. So, so kubectl is a staging repo. So we don't actually accept pull requests to kubectl as a repo. It has to be made to the main project in the staging directory and that gets replicated to our repo. So we track issues in both places and we take PRs in one. So we track issues in both places, and we take PRs in one. So we get issues all over the place, and I can barely keep up with the issues that are on my repo, let alone the main one. Yeah.

First in, last out. Yeah. So I start with the newest ones because they're usually the freshest and most relevant, and a lot of times we can just close them right off the bat because it's a support issue or something or a new flag and you're just like no or you're you're eight versions behind please upgrade and try again right or it's an issue that's like help i just deleted my whole namespace right yeah that one is really hard to sorry about that can i send you a bottle of gin or

commiserate with you so we do have plans for that. So we have been working on trying to get that in. What is your day like with issues? Like how many hours a day, either directly in issues or procrastinating, do you spend on issues? Wow, what a call out. Surprise, Kubernetes isn't my full-time job.

Okay. Oh, I thought it was. No, I do. I used to work on the EKS team at Amazon, so I would spend most of my days on Kubernetes. And now I do stuff with supply chain security and some other projects like Sigstore. It's an open SSF project. Yeah.

But yeah, so we have a bug triage once a month that we go through, where we'll go through as a group. And the idea behind this was that knowledge transfer, where we can talk through the history and the context that people don't have. And we invite lots of new people. So if you're listening and you want to get involved,

join us for our once-a-month bug scrub. We have biweekly SIG meetings. And we have from twice a week to every other week. I have a Kubernetes meeting every Wednesday. So it's bug triages once a month, and then our general SIG meeting is twice a month. I have a Kubernetes meeting every Wednesday. So it's bug triages once a month and then our general SIG meeting is twice a month. Gotcha, okay. And so join us for that.

It's github.com slash kubernetes slash community and then the SIG CLI folder right at the top. It has meetings. So it's all public agenda and it's all recorded. So 9 a.m. Pacific time. Cool. There you go. Well, thanks for talking to us, Eddie. Yeah, thanks for having me, y'all. It was a blast. Let. Pacific time. Cool. There you go. Well, thanks for talking to us, Eddie.

Yeah, thanks for having me, y'all. It was a blast. Let's play Zelda. Let's play Zelda. That was awesome, guys. Yeah, man. That was fun. This is a ChangeLog News Break.

Even legendary computer scientist Donald Knuth is playing with chat GPT. Inspired by a conversation he had with Stephen Wolfram, Knuth asked it 20 questions and wrote up his analysis of its responses. His questions are interesting, much more intentional than anything I come up with. He asks things like, does Donald Trump eat betel nuts? Write a sonnet that is also a haiku. What is the most beautiful algorithm? Stuff like that. He then provides the answers verbatim and his conclusions. Here's my favorite thing he has to say about it. Quote, I find it fascinating that novelists galore have written for decades about scenarios that might occur after a quote singularity in which super intelligent machines

exist. But as far as I know, not a single novelist has realized that such a singularity would almost surely be preceded by a world in which machines are 0.01% intelligent and in which millions of End quote. Despite this game of 20 questions, Knuth does not plan on continuing his generative AI research. He says he's going to spend his time developing concepts that are authentic and trustworthy. You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com slash news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com slash news to also receive our free companion email with even more developer news

worth your attention. Once again, that's changelog.com slash news. Where should we begin? Dapper. Dapper. Let's begin with Dapper. All right. Open source, CNCF, graduated. No, not yet. Not yet. Okay, sorry.

It's incubating. Incubating. Yeah. We will graduate at some point, but, you know, we're not rushing it. We want to make sure we get the most out of the CNCF incubating stage. We are doing lots of things in the CNCF, integrating with other projects. We want to make sure we have this core integration with all of the other CNCF projects before we graduate. Okay. So yesterday you said you started Dapper at Microsoft?

Microsoft, yes. That's correct. And you're working for them. Yep. And you built Dapper as an open source project? Correct. And then, well, first of all, what was it? And then tell that story. What was Dapper when you built it then, what was it? And then tell that story. What was Dapper when you built it then? And what happened next?

Yeah, so in 2018, I was at Microsoft, and I was working for the Azure CTO called Mark Rosenvich. That was an incubations team whose job was basically to look for bleeding-edge technologies and come up with innovative open-source technologies that could really give Microsoft a boost in the ecosystem. And yeah, I was mostly working on open source. I was contributing to Kubernetes, Terraform,

a bunch of other projects along that lines. And then I met someone called Mark Fussell, who today has became the co-founder of my company, Digrid. And we were looking into how can we improve the lives of application developers, not necessarily DevOps or infrastructure people, on top of Kubernetes in the cloud native space. Because the ratio between a DevOps engineer and an application developer is 10 to 1 in

favor of an application developer, and we call them the silent majority of cloud native. Because if you look at the CNCF ecosystem, most of it is around how do you do GitOps and Ops and security and supply chain and CICD. But there is no one out there that's really solving the problems of core distributed systems challenges. And this is why we came up with Dapper as this core tool that developers can use to focus on their business logic and not distribute the systems issues. Okay. A core tool so developers can focus on their business logic and not distributed systems problems, is that what you said? Yeah. What are the distributed systems problems?

Yeah. And how does Dapr deal with them? So for example, as a developer you have to make sure that your application is first of all secure and second of all reliable. And that usually translates into a lot of boilerplate code that you as a developer need to write on your own to basically make your application more secure wherever it's running. And Dapr will basically give you the security

and reliability features out of the box immediately. And then you have to write state. You have to manage state at scale. You might be writing to Redis or DynamoDB or Cassandra or Google Firebase. But if you have multiple services writing the same data all at once, you are probably going to want something

like first rate wins or last rate wins. And you're going to have to do Pub Sub and leader election and config management and secret management. And all of these infrastructure things really add up when all you want to do is focus on your business logic so that you can ship your feature out and get your next promotion, right? Right.

And so Dapp really gives developers these APIs that give them all these pub-sub event, async eventing paradigms and service-to-service invocation and stateful management paradigms. They can focus on what matters to most of them. So do you describe it as a framework or a toolkit? Yeah, I think a framework is a good definition of it. It's an API that you call, so it doesn't compile into your code. It's a sidecar architecture, so there's a process running next to your application. You talk to it via HTTP or gRPC, which makes Dapper really inclusive,

because if you're a developer coming from Python, Java, C Sharp, Rust, whatever language, as long as it can talk HTTP, it can talk to Dapper. Okay. And so there's a bunch of client libraries probably for different languages that talk to Dapper? Yeah, there are. They make the development experience nicer, but if you want to, you can just drop down to HEP and gRPC directly. Sure.

Yeah. All right. So I have my business logic and then it's calling over to Dapper and telling Dapper to store some data, give me some data. Yeah. Handle state that's for you, do PubSub between services, yeah. But then the nice stuff for ops people is that no matter where you're running,

you can basically tell Dapper to work with the infrastructure of choice for your team. So Dapper doesn't replace a state store or a PubSub or a configuration store. It actually has this component model concept where you can plug it in to work with whatever database or PubSub or SecretStore your cloud's running. So we have a hundred of these community contributed components

that we maintain. And as a DevOps person, you can say, hey, if I'm running Google Cloud, I'll have Dap or work against Firebase. Running on-prem, it'll work against Redis. And as a developer, you get really consistent API. So in a multi-cloud environment, you write your code once and you can basically configure Dapr to work against whatever infrastructure you're running. That sounds cool. Is there like a Dapr stack? Is there like a default set of these are the plugs that we recommend you plug in, but you

can plug in whatever you want? Yeah, you can basically plug in whatever you want. So that's a really good question. We have the concept of a pluggable component. So for example, if you are using Dapr to talk to some proprietary system that you can't contribute upstream back to Dapr, we have a way for you to write that plugin and run it on your own.

But we also have maturity levels. So we have alpha components, beta components, stable components, and we recommend people use stable components for production. Other than that, you're free to do whatever you want. Dapr will make sure that all the best practices are really encapsulated in the API calls for you. Huh. So how did Dapr escape Microsoft? Or how did you escape Microsoft with Dapr? Or was it an escape at all? Or maybe you just left?

Yeah. So Dapr was open sourced first in October 2019. It really picked up. We have a lot of end user adopters today from IBM to Microsoft to Alibaba Cloud, NVIDIA, NASA is running Dapr in outer space as we speak, by the way. I think that's the coolest use case of Dapr. It's got to feel good, right? Yeah. It's like the ultimate edge deployment, right? Yeah. Which is nice.

And so it really picked up. We saw a lot of community contribution. Then we decided that, you know, we're going to give it to our foundation because, you know, we want to really make sure that it grows and that we bring other vendors in, other companies. So it arrived in the CNCF.

We were, I think, the first or second project to make it straight into incubating. We skipped the sandbox phase because we already had a lot of end user adoption, a lot of contributions coming in. And yeah, since then, the project really took off. And at some point, VCs basically came up to me and were like, hey, you know what? How about you spin off Microsoft?

We think there's going to be a good business here. And I basically told all of them no. So I was like focused on my career at Microsoft. And Mark, my co-founder of Dapper and Digrid also, which is our company, was also busy having Dapper really take off the ground. And a year later, we were having a hallway conversation. We were like, look, we think Dapper can have a much broader future, and we have our own vision for distributed systems and where this can go. And this needs to happen outside of Microsoft. So yeah, we basically started Digrid.

We left Microsoft in very good terms. We're still very friendly with all the people there. Microsoft's doing an awesome job on the project. They're contributing to the project along with Alibaba and Intel. They're the main contributors who are on the Dapper Stream Committee alongside us, Digrid.

And yeah, it's been a fun ride. It's pretty cool to be able to start a project inside of Microsoft, work on it at Microsoft, for Microsoft, donate it, or I don't mean donate, it's not the right word. When you CNCF something, is it donated?

Yeah, it is donated. That is the right word. It's the right word, yeah. Okay, donate it to the CNCF and then start a company around it Yeah. Is it donated? Yeah, it is donated. That is the right word. That's the right word. Yeah. Okay, donate it to the CNCF and then start a company around it that builds on it or around it or for it after that as a startup. Managed.

It's a managed version of it. Yeah. Yeah. That's a beautiful world, man. That sounds, yeah. You were kind of saying no for a while. Yeah, for a long while I was so focused in building Dapper into Azure services, like Microsoft Managed Services.

They have a service that integrates Dapper, so that's what I was working on. And I was like, I always thought I would be an entrepreneur and start my own company at some point, but I didn't see it coming at that point in time. So I told the VCs, yeah, it's not for me right now.

But some of them persisted, and in the end, yeah, we took it and went. So what turned the no into the yes? Was it a deal you couldn't turn down from a VC, or was it your partner that was like, come on, let's do this? It was a combination of things. I think mostly we saw Dapper really take off, and we figured out, yes, there can be a business model, especially around helping enterprises operated on Kubernetes.

You know, Kubernetes is a complex piece of software to operate, and so we really saw the struggle of developers operating Dapp on top of Kubernetes, and we knew we had something to give there. This is not something we could have done with Microsoft. But also, ultimately, our vision is to come out with a distributed systems API platform that developers from serverless platforms and really platforms from all types of compute can leverage. So it's like serverless Dapper.

You can run it outside of Kubernetes. You can run it whatever you want. And to do that, it needs to be multi-cloud. And so that was another reason why we thought we'd leave Microsoft and start it with our own company. We really want to build our vision of distributed systems through the Dapri APIs. Okay.

What year was that when you started the Diagrid? It was January 2022. So a year ago plus and change. Yes. There's some nice logos here. You got IBM Research. This is for your company, Diagrid. IBM Research, Intel, Microsoft.

Hey, makes sense. You did that integration. Alibaba Cloud, Huawei, Bosch, Ignition Group, Tencent. I mean, these are like major enterprise players. Yeah, there are a lot of other players who have not come out as public adopters yet. Really, some of the biggest names in the industries. And what's fascinating about Dapper is that it was adopted by the tech side of enterprises before it was adopted by startups, for example. And you usually see it the other way around. You know, as a company

offering commercial products on top of Dapper, we're not complaining. That works out really well for us. That sounds great for you guys. Why do you think that was? Is it because it solves enterprise-scale problems? Yes, I think startups, what's most important to them is to make sure that they deliver on their business, which means they want their infrastructure to be as reliable as possible. So they're not as likely to take on new bets and new technologies. But enterprises, on the other hand, they have resources, and they look at new technologies as But enterprises, on the other hand, they have resources and they look at new technologies as a way to go to market faster,

reach good market faster, and really outpace your competition. So they're much more open to new tech. And I think also it coming from Microsoft really gave it like the enterprise stamp that made people feel really comfortable adopting it. Why is it important to have a managed version of Dapr? Yeah, so if you're running on Kubernetes, for example,

you need to manage Dapr yourself. And as a developer, you just talk to the Dapr APIs, it's easy. But as an ops team, it's really difficult to babysit the control plane. On Kubernetes, every type of technology that has a control plane that manages a data plane like a service mesh, Istio, Linkerd, Consul, Dapr is no different. It's really troublesome. It's a lot of cognitive overhead for infrastructure teams.

You need to upgrade, downgrade, do certificate renewals, you know, monitor, observe the infrastructure. So we basically do it for you and we take all of that pain away for you. And then the other product we're coming out with is serverless dapper. So using dapper outside of Kubernetes on whatever compute platform you want. Browser, Wasm, Edge, Google Cloud Run, AWS Lambda, whatever compute you're running on, you'll be able to use Dapr. Is it a problem of scale that makes you want to go managed?

Or is it, like if I'm a small team with, let's say a three node Kubernetes cluster, is managing Dapr, myself, my ops team, not a big deal, right? Yeah, if you're a small operation, then managing Dapapper yourself will probably be something that you should be able to do. It's once you get to much, much bigger. Huawei size, IBM Research size. Well, slightly smaller than that, too.

Like, we have really good end users for Diagrid, like Sharp Parameter, for example. They're a mid-sized company. They wrote their own application platform, and they replaced it with Dapper internally because they want to really replant them something that was standard. And they're a five-person development team, I think, and they're using our services to manage it because they're a small team.

They want to focus on their business logic. They want to focus on managing Dapper. So this also helps smaller teams. Yeah. Can you speak to the reluctant founder journey to some degree? Like you said, you eventually wanted to be an entrepreneur. You just wasn't sure when. And speak to the, I have this open source project. I incubated it or I am incubating it inside of CNCF. Why incubate or donate to the CNCF? Like what does that benefit the project? You speak to all those details. For those listeners out there thinking, I'm you. I'm a version of you at some point. I

may do something like this. Why did you take this route? Why was this donation make sense? And this whole route makes sense for your, I guess, your journey. Yeah. So we donated Dapper to CNCF while we were at Microsoft. And the reason, the main reason why we did that was to really gain new contributors. Dapper had a lot of contributors, but being vendor neutral is something that's really important. You know, if it's a project that spins out of Microsoft or AWS or Google,

and it remains under their proprietary, you know, licenses or control, then users of other clouds might not feel so much inclined to take a bet on it. Because they will go like, oh, it's a Microsoft thing or it's an AWS thing or it's a Google thing. But when you donate to CNCF, you get this vendor neutrality and you gain these new audiences of contributors who are coming in from every walk of life, every cloud platform or technology that contribute to your project. So your end users grow, your contributor audience grows, and people see

that this is really something that can adhere to many users from many cloud platforms. We didn't want it just to become an Azure thing. So primary benefit is vendor neutral. Yes. And new contributors because you're seen as a level playing field, no bias. Correct.

Right, no corporate overlord necessarily. Yeah. Okay. How has that benefited Diagrid? How has that benefited your companyrid? How has that benefited your company in terms of like commercializing this open source, your journey to get venture-backed funding? Like how has that helped in all ways the business angle of, has it been a lot easier, I suppose, to do this route? So, you know, there's a lot of commercial entities that back open source

projects that are not CNCF projects. You projects. I can name many. But I think the one major benefit of being in the CNCF was looking at the contributor growth since we joined, because Dapper picked up a lot of new contributors ever since we joined. And when you pick up new contributors, eventually it translates into end users, which translates into new business. So, yes, that makes commercializing it easier. You have to spend less time working on the open source project than you would have if it wasn't in CNCF because you get this awesome power of the open source contributions helping your project, where otherwise we would need to fund a really, really large team to work on open source. Right.

What's the license of Dapper itself? And is there anybody else who can do a diagram? Could like Jared and I be like, you know what? Hey, we're leaving here today and we're going to compete. Yes, you can definitely do that. Dapper is Apache 2. That's mandated by the CNCF. So all CNCF projects are under an Apache 2 license, which is very flexible in how you commercialize it.

You can do whatever you want. You can start your own service around it. Dapper and any other project in the CNCF. So you're competing on, I guess, your ability to do the managed service the best, right? So if somebody comes out and competes with you, they have the same Dapper core or whatever it might be. They can spin up a version of that. Now, it wouldn't be cool necessarily to do that, but they could. It's possible.

Yeah, definitely. And we welcome competition. Look what's happening with Argo. It's a CNCF project that picked up on a lot of traction. CICD side, there's multiple companies trying to commercialize it today. Microsoft's commercializing Dapper. I actually build

Dapper into a managed service, so I kind of in a way created some of my own future competition, which is pretty cool. You know, the Microsoft people are great and competition is good because it makes everyone better. But yes, we believe that in Diagrid, because Mark and I, my co-founder, created the Dapper project and we're core maintainers of the project. And we're also on the Dapper steering committee alongside Alibaba, Intel and Microsoft. Then we have a very good, you know, overview into the project and we have a very good overview into the project, and we have a

very good understanding of the technical aspects of it. But you didn't name yourself Dapper Inc. Yes, yes, we didn't, for two reasons. One is, well, a legal requirement. We can't because Dapper is under trademarks of CNCF, so that limits you. But even if it didn't have that limitation we still wouldn't do that because we don't want to tie the fate of our company to one single project. At some point, Diagrid will eclipse Dapper. Dapper is an amazing framework

helping a lot of developers out there today and we will be invested in it for as long as the company lives. That's a promise to anyone out there listening to this but we will also want to give, you know, our own take about distributed systems that might not necessarily have something to do with Dapr. Our core at Digrid is to make application developers

more successful, whatever they're doing. And Dapr is one way of doing it, and there may be others. And so, yeah, we named ourselves Digrid because that's an architectural term that helps buildings be built, you know, faster and more reliably. And that's what we want to do. We really want to enable architectural patterns for application developers to be better.

Is there a parallel to Dapr or a comparable that people may know about? Yeah, so Dapr is really polyglot in that you can talk to it from any language. I think if you look at individual programming languages, you'll find equivalents like Spring, for example, for Java, or Spring Cloud, right? So it's like a Java framework

that gives you all of these developer primitives. It's like Dapper for Java. And you have Micro for Go. Yeah, those are the immediate two that I can think of. That helps. So are there drawbacks to the polyglot style versus, I mean, I'm sure there are, but HTTP works pretty well. Yeah, it does.

I mean, like if you're writing a very extremely low latency application, Dapper might not be for you, right? Because you still have an extra network call. Right. And so like if you're writing a trading application and you need like, I don't know, microseconds of latency, if you're adding a trading application and you need, like, I don't

know, microseconds of latency, Dapr might not be a fit for you. But we do believe that, you know, in that terms, performance is good for, you know, 90% plus of use cases.

Another reason why Dapr might not be for you is if you need really, really specific features from, like, Kafka or AWS DynamoDB because Dapr is an abstraction layer

on top of this infrastructure. In many cases, it adds features that you don't find on top of these cloud services, which is really helpful. But in some cases, you won't find the feature that you're looking for. So if you need something really esoteric, Dapr might not be the best fit. Makes sense.

Lowest common denominator across what you're trying to do. Cool. Anything else? Future, is the project mature in terms of feature set, or is it like you got huge plans for Dapper? Yeah, we have huge plans. We've recently added workflows, which is really nice. So very workflow as code type of programming model

where you can tell your code, hey, sleep for 100 years and then kick off this process and it'll be reliable and secure. And we're adding cryptography APIs, blob streaming APIs, document store APIs, SQL APIs. There's a whole world of APIs getting added to Dapper. We have eight today and we're going strong on 12, I want to say, in the next year. Awesome.

Very cool. Thanks, Jaron. Thank you. Thanks for having me. Thank you. Jaron. Jaron. Jaron. My bad. Thanks, Jaron. My bad, Jeroen. Thank you. Thanks for having me. Thank you. Jeroen. Jeroen.

My bad. Thanks, Jeroen. My bad, Jeroen. Okay, that completes our transition to Apple. I mean, well, that's the wrong announcement. My bad. That completes our Open Source Summit North America 2023 in Vancouver, Canada coverage. Big, big, big thank you to our friends over at GitHub for sponsoring our efforts to get there and get all this awesome hallway track coverage and bring it back, cut it up and make it awesome and then share it with you because, well, we love you.

Okay, so if you want to give us some feedback on this episode the link is in the show notes but one more thing coming up next speaking of apple coverage we have our friday show coming up the next changelog and friends we have mike mcquade joining us for wwdc coverage WWDC coverage, Vision Pro, all the updates, everything. And then next week on this show, we're talking PassKeys. It's going to be such a fun conversation. If you want to know about PassKeys, come back next week. We've got something for you. But that's it.

This show is done. Big thank you to our friends over at Fastly, over at Fly, and also our friends at TypeSense. And of course, the infamous and also famous Brake Messer Cylinder, because those beats, well, they're banging. That's it. The show's done. We'll see you tomorrow. Bye.