The Changelog: Software Development, Open Source - Gleaming the KubeCon (Interview)

Episode Date: November 30, 2023

This week we're gleaming the KubeCon. Ok, some people say CubeCon, while others say KubeCon...we talk with Solomon Hykes about all things Dagger, Tammer Saleh and James McShane about going beyond cloud native with SuperOrbital, and Steve Francis and Spencer Smith about the state of Talos Linux and what they're working on at Sidero Labs.

this week on the change law we're gleaming the cube con did you see what i did there okay some people say cube con while others like me say KubeCon. I'm pretty sure I'm right. After all, it is called Kubernetes, not Kubernetes. So we have an anthology episode for you. That means we grab some of the best conversations from a conference and bring them back to you here on the pod. First up is Solomon Hikes talking about all things Dagger. And after that, we're talking about going beyond Cloud Native with Super Orbital with Tamer and James. And last, we talked with Steve Francis and Spencer Smith from Sedera Labs talking about Talos OS.

Also want to give a big thank you to KubeCon for bringing us out to this conference. It was awesome being there. Such a big conference. Wow, just so big. And of course, a massive thank you to our friends and our partners at Fastly and Fly. This podcast got you fast because Fastly, they are super fast all across the world. Check them out at fastly.com. And our friends at Fly will help you put your app in your database, close your users

with no ops. Check them out at fly.io. What's up, friends? This episode is brought to you by our friends at Neon. Serverless Postgres is exciting, and we're excited. And I'm here with Nikita Shamganov, co-founder and CEO of Neon. So, Nikita, one thing I'm a firm believer in is when you make a product, give them what they want. And one thing I know is developers want Postgres, they want it managed, and they want it serverless. So, you're on the front lines. Tell me what you're hearing from developers. What do you hear from developers about Postgres managed and being serverless?

So what we hear from developers is the first part resonates. Absolutely. They want Postgres. They want it managed. The serverless bit is 100% resonating with what people want. They sometimes are skeptical. Like, is my workload going to run well on your serverless offering? Are you going to charge me 10 times as much for serverless that I'm getting for provision? Those are like the skepticism that we're seeing, and

then people are trying and they see that the bill arriving at the end of the month, and like, well, this is strictly better. The other thing that is resonating incredibly well is participating in the software development lifecycle. What that means is you use databases in two modes. One mode is you're running your app and the other mode is you're building your app. And then you go and switch between the two all the time because you're deploying all the time. And there is a specific part when you just like

building out your application from zero to one, and then you push the application into production, and then they keep iterating on the application. What databases on Amazon, such as RDS and Aurora and other hyperscalers are pretty good at is running the app. They've been at it for a while. They learned how to be reliable over time. And they run massive fleets right now, like Aurora and RDS run massive fleets of databases. So they're pretty good at it. Now, they're not serverless, at least they're not serverless by default. Aurora has a serverless offering. It doesn't scale to zero. Neon does, but that's really the difference. But they have no say in the software development lifecycle. So when you think about what a modern deploy to production looks like, it's typically some sort of tie-in into GitHub, right? You're creating a branch,

and then you're developing your feature, and then you're sending a PR. And then that goes through a pipeline, and then you run GitHub actions, or you're running GitLab for CICD. And eventually, this whole thing drops into a deploy into production. So databases are terrible at this today. And Nian is charging full speed into participating in the software development lifecycle world. What that looks like is Nian supports branches. So that's the enabling feature. Git supports branches, Nian supports branches. Internally, because we built Nian, we built our own proprietary. And what I mean by proprietary is built in-house. You know, the technology is actually open source, but it's built in-house to support copy and write branching for the Postgres database.

And we run and manage that storage subsystem ourselves in the cloud. Anybody can read it. You know, it's all on GitHub under Neon Database repo, and it's quite popular. There are like over 10,000 stars on it and stuff like that. This is the enabling technology. It supports branches. The moment like over 10,000 stars on it and stuff like that. This is the enabling technology.

It supports branches. The moment it supports branches, it's trivial to take your production environment and clone it. And now you have a developer environment. And because it's serverless, you're not cloning something that costs you a lot of money. And imagining for a second that every developer cloned something that costs you a lot of money in a large team, that is unthinkable, right? Because you will have 100 copies of a very expensive production database. But because it is copy and write and compute is scalable, so now 100 copies that you're not using, you're only using them for development, they actually don't cost you that much. And so now you can arrive into the world where your database participates in the software development lifecycle. And every developer can have a copy of your production environment for their testing for their feature

development. We're getting a lot of feature requests, by the way, there, people want to merge this data, or at least schema back in into production, people want to mask PII data, people want to reset branches to a particular point in time of the parent branch or the production branch or the current point in time, like against the head of that branch. And we're super excited about this. We're super excited. We're super optimistic. All our top customers use branches every day.

I think it's what makes Neon modern. It turns a database into a URL and it turns that URL to a similar URL to that of GitHub. You can send this URL to a friend, you can branch it, you can create a preview environment, you can have dev test staging, and you live in this iterative mode of building applications. Okay, go to neon.tech to learn more and get started. Get on-demand scalability, bottomless storage, and data branching. One more time, that's neon.tech. we are at KubeCon

Cloud NativeCon in Chicago and this is our first ever in-person interview with all three of us Adam Jared myself yes and our guest Solomon And this is our first ever in-person interview with all three of us. Adam, Gerard, myself. Yes.

And our guest Solomon. Hello. Hey. Really? It's the first time? Yes, it is. In person, all three of us, first time ever. It's been a long time coming, right? It's been years.

Yeah. Literally years. But here we are. I have to say you all look fantastic in person. Thank you. Thank you. Zoom is great, but this is better, right? We're in three dimensions. We are. I have to say you all look fantastic in person. Thank you. Zoom is great but this is better right? We're in three dimensions. We are. Full-bodied. Yep. And smell. All of our senses

are engaged. I'm glad that everyone made it. So so glad. I have to be honest on Monday just as I was about to check in for my plane. Yeah wasn't sure I was going to make it. On time or at all? No, no, at all. Why not? At all. So I arrived at the check-in desk. I wanted to check in, and I didn't have an ESTA.

An ESTA is like an electronic visa for UK citizens and other nationalities to enter the US. I renewed my passport, and I still had a valid ESTA, but ESTA is linked to your passport. It's a little bit like the load balancer is healthy, and the appliances are healthy, but the two are not talking. Okay. So users are getting 502s all over. Right. So I had an hour. Luckily, I was early at the airport, so I had just enough time to apply for one.

But hey, this guy has two ESTAs. What is he doing with two ESTes and two valid passports so i had to go through extra security checks so it wasn't instant and i didn't know whether i'll be able to check in wow i know right always have a plan b always have a plan b yeah so the plan b was like take the later flight i took the middle one anyways but that was that was uh any any other fun that travel stories? That scene from your memoir is definitely getting in the movie. Right.

Okay. I have no excitement on the way here except for I'm a person who gets, I wouldn't say existential dread. I like the idea of doing things, but leading up to the actual thing, I dread every moment of it. And then when I get to the thing and do the thing, I'm loving it. Describe. So like, I didn't want to come.

Okay. I think none of us do. Like, our home is amazing. Why would you leave? But like, intellectually, I was like, yeah, let's go, that'd be great, blah, blah, blah. Let's do it, let's see Gerhard. And then as it approaches, I'm just like,

this was a terrible mistake. Why are we going? Specifically me, why am I going? And then I go through all the steps, and I just overcome the dread. And then I arrive, and then I'm like, oh, it makes total sense why I did this. And I'm glad that I did. And that's not just KubeCon. That's every event in my life. Wow.

Pretty much. Does that make you procrastinate preparing for this event? Oh, I've always procrastinated regardless of that. But yeah, I'm getting ready at the last second for sure. But I'm not late. Right. Just enough, like just enough procrastination has always been nice.

Everything worked out. Everybody's here. It's good. We're here. Even though, even though you couldn't find each other at the airport, right? Why is that? Yeah, Adam.

Why was that? I was at the wrong airport. Or let's just at the wrong airport. Let's just say a different airport. Right. I didn't know that Chicago had two airports. I thought it was just O'Hare. So did I.

But I was not at O'Hare. He was. I was not. We landed at about the same time. And so he got his bag and he's like, we'll all Uber over and grab you because I was at a different terminal. But I was not merely at a different terminal. I was at a completely different airport.

Thankfully, I shared my location, and he realized how far the blue dots were from each other, and all was resolved. But yeah, that was funny. That was a good one. Again, everyone's here. Everything worked out. We did it. We did it.

The hard part is done. Yeah, we're here. Cool. So KubeCon is about companies and CNCF projects announcing big things, doing amazing demos, you know, launching whatever features. And I think that's like part of the buzz, right? That's where people come here. I like to talk about cool things.

And at this KubeCon, Dagger had a booth and showed some demos for a feature that officially has not launched. What is the feature that has not launched yet? It was a top secret demo for 10,000 people. Yeah, we have this project underway called Project Zenith. It's a future release of Dagger. Not too far in the future, I hope, I think. And it adds the one feature that our whole community has been asking us for a year, ever since we made the previous major release, which was to support many languages.

That's immediately, actually, one of our most engaged users, Mark, yesterday, reminded me that the first time I saw that release, I immediately asked when is the next big thing coming. And it's reusable cross-language modules. So if you can write your CI pipeline in code, you're very happy. Solves all these problems for you. Runs locally. You can test it. You can collaborate on it with developers, et cetera.

And you can modularize it. You can say, oh, that function here, that's cool. That's my build. I optimize it. It's perfect. I want to share it with everyone else, or I want to reuse it in my next project. So if it's some weird YAML proprietary thing, you can't do it. If it's code in your favorite language, you can do it, but only within the confines of

your language. So if it's a Go function, you can share it with other Go programs. In a lot of software organizations, you've got multiple teams using different languages and the tools that go with it. So I'm here in my little Go team, and I've got the perfect build, and there's another team over there and they use Python, maybe they use Dagger also, because the platform team just kind of evangelized Dagger. So everyone's happily using Dagger in these silos, but they can't use each other's stuff.

And the problem is, the I in CI is for integration. So that's what you're supposed to connect it all together somehow. Right. And so that was really the next, you know, until that's possible, the full potential of the algorithm cannot be realized. And so that's, now we have it. So you can write your functions in Go with the perfect pipeline logic. And then someone else writes their functions in Python, and they can reuse each other's functions, and it works.

And so not only does it work, but after a few tries, I'll spare you the details, but we got a DX, a development experience that's really not only productive, but actually fun. And you find myself, even myself, just trying to sneak away from my other responsibilities to just spend 20 minutes just, I want to hack on this module idea that I have, you know. And it's just fun, and it's kind of spreading like wildfire in our community, you know,

which is a niche community. But yeah, that gut feeling that, oh, God, I want to play with that right now, you know, and the ability to get a quick win, that's really exciting. So even though the feature is not launched, we just wanted to share it because it just gets people excited. So, yeah, we're showing it at the booth. Have you seen any of the demos, Adam, Jared? No.

Nothing? Oh, wow. Okay. Well, after this, when we stop recording, maybe I can show you a few, even though it might have been set up. That would have been amazing. We only watch the demos where you might win $100 at the end. Oh, I see.

And the person compels you to come over and sit. No, you have to sit here now. I'm like, oh, I do? Yeah. Because you might win $100. I'm like, oh, I might? Sit right here. Literally, that happened to us. And I

felt bad for her, so I was like, okay, I'll sit there. And the funny thing was, I don't even, I honestly can't remember what the demo was. But she sat me four feet from the screen. And the guy is projecting for the audience. And I'm like, he's going to burn my retinas. They hold your. Pretty much. I got on my phone and just played on my phone for 10 minutes.

It was Portworx for sure. Don't name and shame him, man. Oh, sorry. No, no, no, no. Did you win at least? No. Oh, come on. I never win any of these things.

Okay. Did someone win? Someone won. Okay, so it was real. It was real. That's crazy. Yeah. This might sound bad, but it doesn't feel bad. Well,

maybe it does. It's kind of a carnival thing going on, like, or a fair in the expo hall. You guys are in your spot, so you probably don't see it as much because you're just at your booth. But, you know, there's a lot of people who are vying for attention. And these companies with lots of big budgets and flashiness and all these things. And, I mean, it's huge here. I've never been to an expo hall this size. I've never done, like, a Comic-Con or a, what's the big one, CES. Reinvent. Yeah, where these are, like, tens of thousands of people in expo halls.

So this is new to me. I've done, like, smaller events where there's booths, but it's not like a carnival or a fair. I mean, it's difficult to get people to pay attention to what you're doing, and so people pull out all the stops, such as giveaways, people on microphones literally calling you over, similar to how you would at a county fair where a guy wants you to smack. Step right up. Step right up, yeah, exactly. So that's what we've been experiencing. We haven't stopped.

Very interesting. Hey, you. You in the hat. Yeah, exactly. We would like that. I see you looking at me. Come sit down. Yeah. And I'll just give you stuff. Like one time we just slowed down because we saw Arun Gupta was part of a panel. Ospo. Ospo sit down thing. And we just kind of were walking by slowly like, hey, it's Arun. We know Arun. And then some lady walks up and she's like, you guys want some popcorn? No, stay right here. Stay right here. Here's some popcorn. You guys listen to this. Give him a ticket. Give him a ticket. Where's the ticket? And I'm like, cool. Thank you. And we start walking. She's like, you guys want some popcorn? I was like, stay right here, stay right here. Here's some popcorn. You guys listen to this. Give them a ticket, give them a ticket. Where's the ticket? And I'm like, cool, thank you.

And we start walking, and she's like, no, you want to stick around, because after this is the demo. And I'm like, oh, there is. I want to stick around. That's a long foreplay, right? Yeah. Ready to go, though.

So I have a bit of an allergic reaction to demos, but we'll watch yours, Gerhard. We'll watch yours. Sounds very interesting. My biggest frustration was, because in a conversation like that this guy was talking about I actually want to just live code it with him because that would be the best demo and actually it's and we can because it's it really you know I'm gonna stop selling overselling and I apologize I'm excited but you know and I just I was like let's do it now let's do your module now and I

opened my laptop and I just couldn't get Wi-Fi to work. And then I went to tethering and I couldn't get tethering to work. And I'm like, okay, well, maybe not today. But here's a video loop of a cool demo. You know, but yeah, you just want to go and code the thing with people. You need to basically approach people where they're at. So when I first started using Danker, I assumed that it's a replacement for CI. And I know many people still have that misconception.

They think, oh, big Dagger is going to replace my CI. And obviously, I've learned that Dagger is so much more. But for the people that still think like that, what would you say, Solomon? Does Dagger replace CI? It's funny because, well, you're right. We deal with that question all the time. So why would I replace Jenkins, GitHub Action, CircleCI? And the first thing we say is, you should not.

You should keep it, but you should modernize what's running on it, basically. And we're really piggybacking on a pattern that already exists, right? Because the teams, the software teams are not stupid. They know, okay, every line of proprietary configuration I put in this, you know, CI platform in the sky, it's only going to run there. And, you know, it's not real code, so eventually I'll run into limitations. So a lot of those teams, what they do is they try to keep the, I guess there's two schools of thought. There's the CI maximalist, just put everything in that YAML.

And the other school, the pragmatic school, I would call it, is, okay, put as little as you can get away with. And then have that CI run a shell script and make file something that can run in the CI environment, but also in the dev environment. So you can actually, at least part of these pipelines, you can run locally and test locally, et cetera. And so we're just taking that pattern and doubling down on it. We're streamlining it, modernizing it. And so in that pattern, you don't have to replace your CI platform. You just have to use it in a more reasonable way, as a runner, as a runner infrastructure. I do think that CI as a category,

this is the market of CI platforms. I think one effect of Dagger succeeding in the future, knock on wood, is that, but if somehow we fail to finish the job, someone else will, because it has, some things just have to happen. CI as a category, I think will just go away. It's no longer needed, really.

So these are two different things, right? We're not telling teams, get rid of your CI platform now. We're doing something more incremental, low friction, et cetera, but the end result of everyone doing that is eventually, eventually someone someone not us will kind of say the way do we need this now like i think that's the end result what would that day look like like what would be different from today well if you think of the big stages

you know of the life cycle of that code that you're trying to get out there into the world right you got development and then you got production and in the middle you have ci right but why because it's it's really it's an artifact of some technical limitations it's because it's all the pipelines you need to run you know build test you know lint whatever and end-to-end testing deploy a staging environment whatever it has to be it wasn't practical for a long time to run it embedded in your dev environment or embedded in your production deployment. But if you miniaturize the CI pipeline, if you can make it small enough that it can actually run in any dev environment, it can actually be embedded in any deployment pipeline, then

it becomes, CI just becomes a feature of development or production. It doesn't have to be its own standalone thing that you push to. It's weird if you think about it. We just got so used to it. What are the technical limitations? Is it the ability of your local dev environment to run, to do builds fast, to do tests fast? I mean, what has been holding dev back that's no longer? I mean, is it our M3 Macs that just make it not matter anymore? Is it software? I think it's a combination of things. One is containers have to exist and be ubiquitous because you need a way to make all that stuff

reproducible. Otherwise, it's not worth it. I think what I would call, there's a family of tech called DAG tech, you know, that just had to mature. The ability to execute tasks in parallel with, you know, Pensy's model as a graph. I mean, Make has been doing that forever, but it's been kind of doing it on its own.

But recently, in the last, what, 10 years, you've seen things like Bazel, Buck, Nix, you know, everyone's sort of independently exploring this model. And it's really matured. And one of the big benefits, honestly, is caching. So everything's got to be cached or cacheable for the thing to make sense. I think also just the amount of computing power that's available, I guess. I mean, we joke that the biggest, you know, we asked teams, what do you think in your organization, what's your number one provider of CI compute? You know, people are like, I guess, well,

we did the migration from Jenkins to GitHub Action, so I guess maybe Azure now. No, I guess it's still, you know, the original. But the answer usually is Apple. So we've got these beasts on every desk. That can run all that stuff just really well. And people don't realize it because there's this software blockage. Because you've got this platform over there and this guy, and you can't run it locally.

So no one's trying. But if you try, and you add caching, and you add containerization, you're like, wow, this is actually not as big as I thought it was. Why am I paying $5,000 a month to run this and it just completed in like three, four seconds on my Mac? Have you ever tested a MacBook to see how long you run it at 100% CPU utilization? You can run that thing at 205 degrees for about hours before it implodes. I know because I've done it.

You tested it or you imploded one? Well, I write all of our archives to 7Z, and that process, the algorithm to compress it, is almost half. And so that's where I test it. So I have a script that will loop through directories, and they're usually 5, 10 gigabytes, right? So it's got to compress by half multiple directories.

And sometimes I'll do them in batch. I'll just make a list of them. It could be 20. It could be like JS party, you know, whatever, whatever, moving it to archive. And I'm like, this thing will clock out 100% CPU utilization, and it'll be 205 degrees for as long as it takes. Could be hours.

And I'm like, lately, I'm just like, how far can I push this thing? And it just does not die. Like it just won't. So you've got so much to use there. All cores, 100% for an hour and a half. No melting. Nice.

So coming back to the heart of Dagger and what Dagger is, I think more importantly is who was Dagger built for? Who's a Dagger user? I think it's evolved a little bit over time, but the answer was right in front of us the whole time, I guess. We started by just interviewing as many software teams as we could you know and talking about their problems first in a very broad way and then gradually Narrowing down because you you apply pattern matching the same problems come up

So you know you you hone in on that problem, you know And so deployment came up a lot of course deployment in the broad sense, you know getting the app out there It was a very shocking realization for me that it was just so painful because i came out of docker thinking wow that was hard but now we've solved it you know i mean i'm exaggerating but okay surely things got better but i just witnessed you know a lot of frustration and pain and it was across every team size you know small huge didn't matter it was all a mess but so that affects everyone in the organization you know, small, huge, didn't matter. It was all a mess. But so that affects everyone in the organization, you know, application developer all the way to infrastructure. But I

guess there's this percent of a platform engineer, I guess. A lot of times it's aspirational. It's not the title or it's not the reality of the job, but it makes sense to describe. Someone should be in charge of the platform, you know. So. Someone should be in charge of the platform. Basically, someone should be in charge of the factory, running that supply chain, making sure it works. And so that's who Dagger is for, ultimately. And then they serve everybody else. So it's sort of a two-step process where we serve the application developer via the platform team supporting them. The platform team sometimes is just one of the devs who you know he said yes to fixing the build once and now everyone just assumes that's the person to ask you know the designated DevOps person so you know that could be that could be who

we're helping or he'd be you know 50 people full-time. But fundamentally, it's the same job. And how do you imagine this person using Dagger? So in the ideal world, this person, the right person has Dagger at their command. How do they use it for their day-to-day job? What does that look like? I think that person uses Dagger to push work to other people in the organization so that they don't become a bottleneck or they are a bottleneck so that they can stop being a bottleneck. You know, because one of the downsides of the thing not being Coat, the thing being the pipeline, you know, the factory, is that, well, it's not familiar to any of the development

teams, you know, so they don't want to touch it. And whether it's GitHub Actions or Jenkins or whatever the factory is, all the developers have to use it. They have this harness to use. But for them, it's like commuting to the big factory. They just go. Someone tells them

where to push the buttons and they can't wait to get out of there because it's not theirs. And if there's a missing tool for them to be productive, like there's this build tool I'm using, it's missing from the factory, there's an idea box, I guess. But they're not going to go and mess with it any more than they need to. But then who is? Well, the platform people who built the factory, now it's their job to do that. But they don't know, they're not familiar with the tool they're supposed to integrate.

And there's a sort of exponential scale problem where there's a matrix problem. Those teams we were talking about before, the Go team, Python team, there's an ML team now somewhere, and they've got a whole bunch of new tools. They're like, oh, here's a model, so can we just deploy that now?

The DevOps team usually is like, I don't know how any of this works. Now they have to go on the side quests. Like, okay, let's learn all these tools and figure out best practices. Meanwhile, they're holding up everything and maybe there's a pressure to ship fast. We're seeing that especially with these AI features now because management is like, we've got to ship the AI thing. And the ML team, we're like,

we've got the model. The model works. Look, this Jupyter notebook is perfect. What's taking so long? And so all this pressure goes on the platform people because they're holding things up. So I think the way they use it,

ideally, is to push work to those teams. Go see the AI team and say, okay, what's your favorite language, Python? Okay, here's a few examples. Here's some docs. Give me your build. Give me the functions you need me to integrate. And they do this with all the teams.

And then all of a sudden, 90% of their workload, the worst part, goes away over time. So they free up time to actually do their job, to strategically integrate that stuff. That's the ideal scenario. There's definitely something interesting there with the MLOps thing, which seems to be different enough that people are coining a new term for it. It's DevOps at the end of the day, but there's a significant enough difference in how these things go out than traditional software that there are now experts in this particular subcategory. Yeah.

And obviously, the AI gold rush is on. And so if you think about what the pickaxe is for the AI gold rush, well, you may think it's the models. It seems like maybe that's kind of true, but also in the long term, is it going to be false as those become commoditized? But maybe the ability to deploy the models, the tooling around actually taking that stuff from a Jupyter notebook into production in a way that's reproducible and not expensive and doesn't run your cloud bill up like insane is the kind of tools that people are going to need in order to find the gold, so to speak. Yeah, exactly. We're seeing the beginning of that. But what's interesting is MLOps today is 99% experiments and training. It's either ML research teams, training models,

fine-tuning models, whatever, or a gazillion people messing around because it's fun. No product in sight. Then you have the steepest funnel in the world where these, I don't know how many, if you go to these online communities, it's insane, the number of people in there. There's everything in there. There's a graphic artist that's been playing with generative AI for images, and they're in there messing with it. And so it's the graphic designer to webmaster to web designer to web developer pipeline all over again. That's just

one example. But then on the other end of that funnel, there's actual products, software products leveraging AI. And it's a trickle right now. It's just, you know, that slope is just incredibly steep. And along the way, all software engineering best practices basically fly out the window. You know, nobody knows how to ship the thing. People who know how to ship software don't understand what's going on here mostly.

And then people who do don't know how to ship software. So I really think it's gonna it's gonna end up being revolutionary in the impact the kinds of products you can build but from the stack point of view i think it's going to be a large but still incremental upgrade to the stack but at the end of the day it's still a stack you still got to ship it you still got to build into this thing it's still a software engineer's world. This is hopefully a very enjoyable question for all of us because we get to dream. And we can start with Jared, which is what is your dream for Dagger? Because we are using Dagger. What do you wish Dagger did?

And it doesn't have to be a timeline, but if you could wish for Dagger to do something for you in your app, what would you wish that it did give me a million dollars okay well let's start with a hundred dollars one million dollars so is that how we buy our oxide rack? Yes. Our first oxide rack. Is this it? All right. We use these two. Yes. So then we can run Dagger on it? Yeah, exactly. No, I don't know, man.

I don't really have dreams for Dagger. I don't dream Dagger at night like you do, Gerhard. But as an app developer and a business owner, I want all this stuff to disappear. Like the Cialis world that Solomon just talked about? Yeah, I want the Cialis world and probably the codeless world as well. I mean, so this is why I made this modules thing as interesting to me. As we transitioned into the Dagger, I'm like, wow, Gerhard's writing a lot of code. And I feel like-

I don't want to look at it. Yeah, I don't want to look at it. And it's all good code, and I haven't looked at it, and it's reasonable. How do you know if you haven't looked at it? No, I said I have. Oh you did? All right. Yeah, I judged your code. It looks all right. It looks nice. Thank you But I'm like surely there's like a happy path for like basic things that we do Yeah, that's probably part of the modules thing and you know reusable And so I would expect the amount of custom pipeline code in our code base to go down over time as we become less custom and as modules become... Because I just kind of want to say, I just go back to Git push Heroku master.

And I know we have that. We don't push to Heroku. We push to our master branch on GitHub and it deploys out. I'm happy with all that, but there was never any code, there's never any Heroku code in my repo unless I had to have a custom build pack, you know, back in the day. And that's because, or for me, I don't want to have any code unless we have to do something custom. And so I would dream that the amount of Go,

and I know Elixir support is coming or there? Yes, it's already there, yes. Ours is Go right now. But I would imagine that as Dagger matures and the ecosystem builds out, that our code goes to maybe zero, maybe not, maybe one. Yeah, that's a great point. So that's kind of what I would desire. Not because I don't like your code,

just because to me it seems like these are... The build pipeline and stuff for a typical 12-factor web app to me is completely uninteresting. Now we do some custom stuff that's cool, but I think for most people who just want to have their thing out there in the world, they would love for you guys to just abstract it as far away as you possibly could. Yeah, honestly, going back to the.cloud days when we were building a competitor to Heroku. We were in that business of making it all disappear. That's kind of the genesis of everything else. Our experience that, and the insights we got there, we just developed an opinion on what's

the right way to solve this problem. And then it's been a long journey, doing Docker and now Dagger. I think it's basically the same journey we're on still. And the reason Heroku didn't actually solve it, neither did.cloud, is that I think the dream at that point was the way to make it disappear for you is to have two people involved.

There's you and there's a Heroku, basically, two tiers. Basically we concluded that there's a missing layer there. It's got to be you, someone who owns your platform, and then someone under that who makes it possible to do that cost effectively. So you, your platform team, could be a team of one part-time. And then under that, the operating system, basically. And I think Heroku tried to do that with Buildpacks, realized, oh, it's never going to work. And we tried to do that with something more custom, container-based.

And then we were like, okay, even that's not enough. We've got to start over. And so the journey has been, our bets is that the only way to solve this properly at the scale of the whole world of software, which is huge. There's a lot of software factories out there, or they should be factories. And right now they're like workshops. You've got to build the foundation from the ground up, and I would say, okay, Docker was step one. You've got the very bottom foundation, and then if I go to you and say, just use Docker,

everything, you know, you have Docker, now figure it out. You're like, yeah, I like Taroku better, and then you add what we're doing now, you know, these programmable pipelines with reusable modules. And it's much more abstracted, but still you're like, yeah, but I don't want to write my own module. Even if it's a 30 line module, like why do I need to write code at all? And then, but if Gerhardt's happy writing that code, I'm happy, you know, and then eventually we get to the point where Gerhardt part-time can actually give you Heroku, but it's your Heroku. Right.

Because later you'll say, oh, but can't we have this custom thing? There's this ML thing that would be cool for processing our recordings. Now it's got transcripts and it's super efficient, whatever it is. And Heroku would be like- We're very far from that happening. Yeah. Heroku would be like, oh, that's on our roadmap for 2025. And Gerhardt's like, already on it. And then the next day he's there and it still feels like Heroku would be like, oh, that's on our roadmap for, you know, 2025. And Gerhard's like, already on it, you know.

And then the next day is there, and it still feels like Heroku. That's the dream, right? But we just had to, there's this whole layer cake that has to be built. And I just, the reason I'm excited and I can't stop shutting up about it is because I really think we're getting super close finally, you know. Yes, it is. And I'd love for this to be complete while I'm still alive. That would be nice. That is definitely something worth working towards because it is aspirational.

It is something that will improve everyone's lives. It doesn't matter whether you're an app developer, a business owner, a DevOps person. It doesn't really matter. It will basically raise the value line for everyone. And when the tide rises, all boats go up. So that's what we want, right? Everyone to have a better experience. And it's very foundational. And it doesn't matter whether you're using GitHub or GitLab or Azure pipelines. It really doesn't matter where you're coming from. Kubernetes, not Kubernetes, it's okay, right? We're all in this together and we all have problems to solve

and lives to get back to. So let's make it as painless as possible. By the way, I feel like that's something that's missing in the KubeCon world. I feel like this community is missing its connection to developers. You know, it's a whole lot of infrastructure stuff, and everybody knows it's needed. But at the end of the day, the part that's left unsaid is it's needed to ship something, and it's left unsaid because it's needed to ship something and it's left unsaid because it's implicit you know of course we know but also we don't really know how to really deliver it in a great way so there's a lot of there's still a wall and there's still a lot of

trust as we got this we're doing the you know it's going to scale and but it's you know i'm not seeing hordes of developers here just being so excited to be at KubeCon like I can't wait to see how you're going to ship my app next yay you know

it's just there's still a big gap you know as we prepare to wrap this up I have one last question for Solomon I know Christmas

is far away but people have to buy presents and prepare so people that are listening to this that are thinking, oh, what should I get Solomon for Christmas? Do you have a few items that you would like? And it can be, you know, whatever. It doesn't matter. It can be tech related if you want or not.

And if anyone else has anything to add, because by the way, I already got my Christmas presents. So I'm good. We were talking about that the other day but anyways back to Sullivan what's that what do you want for Christmas yeah that's that's like

you know I can give you a moment yeah I have family members who always ask me this and I never know what to answer but I love tea

like I mentioned before so you can get me tea that's always appreciated what tea do you like I like chai you were paying attention but I do love actually any any good tea I'm not for any good tea unless but it has to be tea without all the added flavors you know they add all these flowery or fruity favorite flavors

is this is this the kind of answer you were looking for and the answer is it has to be yours it's all good i don't need actual Christmas presents no no no It's all good. I don't mean actual Christmas presents. No, no, no. It's all good. It's something, you know, that's coming. People are thinking about. You can give me a hug for Christmas. I'll be happy. Hugs.

Can you say that one more time? Can you say a hug? A hug from you. A hug. A hug. Well, we don't have to wait until Christmas. We can do it right here, right now. He teed it up.

He's like, let me. It's all good. I've always wanted an ugly Christmas sweater. That's an American thing. I always thought it was cool. We don't really have that in France. Ugly Christmas t-shirt? It took me a while to understand that tradition, but now I get it.

I love it. I'm on board. What brought you around? What enlightened you? I don't know. I don't know. I guess. I have no idea.

Okay. Maybe I saw a movie and an actor I loved was wearing one and then that was it. Yes. I don't know. National Lampoon's Christmas Vacation. Well, with that thought, go and research it. Thank you, Solomon, for joining us. It's been a lot of fun.

I look forward to what we do next. Thank you. What's up, friends? We're working closely with.tech domains to feature startups that are participating in their startups.tech program right here on the changelog. And I've never seen anything like this from one of our advertisers, but it does make sense to me to show off not only what.tech domains offer, but also who is building on.tech domains. And I'm here with Simon Schillenbeck, founder of Handprint Tech. You can find them at handprint.tech. So Simon, tell me about Handprint. What do you guys do? Handprint Tech is a Singapore-based digital technology platform that connects companies to the most impactful non-profit organizations in the world. Our mission is to make it easy and more valuable for companies, regardless of what they do,

to make a positive impact in the world. So we fit within the sustainability space, but our approach is very different. Most of what sustainability, especially from the corporate side, has focused on over the last 30, 40 years is to do less bad, reduce their negative impact, reduce your footprint.

But handprint focuses on the creation of positive impact. Okay, that makes sense. So how do you help companies then? The way we help companies achieve this is firstly, by curating and digitizing the best NGO projects, the best charities, the best impact organizations in the world and improve their capacity for high quality reporting so that companies can have absolute trust that their money that is going to these organizations is well spent. Secondly, we built a variety of tools that enable companies to embed and automate the creation of positive impact through their normal business processes. So you can integrate handprint

technology in e-commerce, in bank card transactions, in advertising, in event registrations, in CRM systems, in whatever you can imagine, so that just engaging in your normal business practices can be linked in an automated fashion to the creation of a small positive impact, which is a handprint. And by doing so, we've been able to demonstrate that this can truly increase the company's capacity to instill loyalty, to facilitate engagement, and to stimulate growth. Creating a handprint can really align with your business objectives. And in doing so, companies can capture value from working with us and grow with the planet. Grow with the planet. Okay. That sounds good to me. Let's talk about.tech domains then. So I

see in your footer, your business name is officially Handprint Tech PTE Limited. What does that mean? What made you choose a dot tech domain the domain choice was kind of an interesting story so we're as you mentioned we're handling tech pte private limited so that's a singapore legal entity which is pt ltd we initially wanted to name our company something else namely the green company with three e's. And very luckily, the Singapore government turned that down because they already had another company called the Green Company with two E's. And they said, well, you can't have almost the same name. And we'd already started building our tech platform.

And we had named it Handprint because Handprint is the flip side of footprint. Footprint is all you do that creates a negative impact and destroys the world. And handprint is actually a scientific term launched by people at MIT to qualify and quantify the sum of positive impact that you're doing in the world. We started building that platform and then said, why not just name the company after it as well? It does have a good ring to it. And so we chose handprint.tech because we wanted to make it very clear that we are not a nonprofit. We are a technology company.

By having the handprint.tech name, it becomes a lot clearer that, okay, this is a tech organization that primarily is a tech company. The only difference is that our product is a positive impact in the world, but we do this through technology. Okay, make sure you check out we do this through technology. Okay.

Make sure you check out handprint at handprint.tech. Again, handprint.tech. And of course, go to startups.tech slash changelog if you want to have your startup that's building on a.tech domain featured on a show like this. Don't wait. Go to startups.tech slash changelog. Go to startups.tech slash changelog. Again, startups.tech slash changelog. So, Tamer and James from Super Orbital, welcome guys. Thank you. Thank you.

I said I only had one question, but then I thought of another one, so I got two questions. I'll start with the softball. What do you guys hope to get out of a KubeCon like this? You come to these events, what are you looking to do? What are you looking to accomplish, achieve, happen? What's the goal for these things? That's a great question. I don't know. I hope I don't sound too jaded to say that going to the talks, it's not my number one goal with coming to these events, right? I've been to a lot of

conferences and I've come to the decision that I can just learn what I need to off the internet. YouTube's amazing and blog posts are amazing. But what you can't get anywhere else is the real world conversations, that hallway track with the people who are actually using these technologies and really pushing the limits of them and doing things in anger. And you get to learn from where the brochure advertisements of how all these tools breaks down. Right. And especially for a company like us, that's extremely important. We learn a lot from our engagements, working with our clients, working on the technologies, but the more we can get from the general community and actually talking about real world situations is just invaluable. And so far, we've got a ton of that and it's been wonderful. I'll make one caveat to what Tamara said. I do love the hallway track and and that's really important for meeting new folks and building the network of people that are dealing with this, the challenges of synthesizing, you know, the stability of the open source projects and the long term health of those against the needsers of Cilium, and hearing their experience and sharing their successes

in building a product, an open source product that collaborates on the needs between these major companies that are supporting the products and the vast community that depends on them. And that's been my favorite talk at the conference so far was hearing from one of the Istio maintainers about really in depth about how they are maintaining the Istio ecosystem of controllers and how they're thinking about the next evolution of that.

So that was a really valuable talk for me. You were talking about that one even amongst us beforehand, how much you got out of that talk. Yeah, absolutely. You get a few talks at the end of the week where it's like, I'm going to go back with a notebook and a piece of paper and really learn from this. You know, in person you listen, but you need to go back and really think through the things that they're saying when it comes down to those highlight talks that you get out of the week. So for someone listening to this, what is the title they should be searching on YouTube when these talks come out? Yeah, the talk is Lessons Learned in Building Controllers from John Howard at Google.

So he was talking about the Istio ecosystem. And so that, yeah, Lessons Learned for Building Controllers was the one. James teaches our programming Kubernetes workshop. And so for him, this is ideal. He gets to incorporate this and everything and, you know, figure out where we've been telling people the wrong thing. I feel like there's a missing service out there.

Maybe this is like a curator, but, you know, all these talks are going to go online. And now every conference is putting the talks online. And I'm with you. I don't attend the talks, not because I'm not interested, but I'm more interested in what's going on in the hallway, in the expo hall.

A lot of times we have a booth, so we're recording conversations. And it's just to be there with the people. But there are talks that I would like to go see, but I don't. And at the end, you're like, I'll go watch them later. But really what you need is someone like James to say, you've got to see this talk. Yeah, you need like a Yelp-like rating review system. You kind of do. You almost need like, or even just if we could do each other a service and like blog after you go to a kubecon or uh all things

open or any of these events and be like look if you're going to watch three talks i went to all of them you know or you can't go to all of them here but you know i attended talks for two or three straight days and these are my favorites then we could help each other like find the good ones without having to, or even just having that FOMO of like, I don't even know what was good at this particular event. And one of the things that's useful to do as well is connecting with folks ahead of time that you search through the conference schedule and like, I knew John was talking, but there are a couple of folks as well that I reached out to ahead of time and had some

conversation. And I think, you know, when you can combine that hallway track with the real, you know, you put all those pieces together, that's when you can get, you know, Hey, conference talks can sometimes have a little bit more shine than the reality. And so you get, you get behind the curtain a little bit. And we had a great conversation with a couple of folks yesterday who were speaking today. And when you're able to really get to that depth of, where did you fail along the way? And oftentimes, those are the things you learn the most from as well. I just got done saying that I don't like going to the talks. And for the most part, it's true. But that is the one useful thing, is when you know somebody's going to be talking

about a technology you're interested in, and you're going to the talk pretty much just so that you can come up to them afterwards and say like, yeah, yeah, yeah. OK, that's great. Thank you so much. Wonderful talk. But let's get real. When did it break down? This is three months on from that. Like, how has it been actually shining since?

Right. I think the people that bring various perspectives, whether you're preparing for a talk, whether you're preparing for a conversation, recorded, not recorded, doesn't really matter. It's like those different perspectives, they all come together, and there's something for everyone. Even if you're into bubble machines,

you will find them here, and they're fun, right? Hopefully it's not just that. It would be a bad place to come just for that but that is happening it's it's it's a bit of everything for everyone there is and here's like this diversity that kind of stimulates the brain maybe overstimulates it pacing yourself is really important knowing what you enjoy is really important if you try and do it all not only you will not manage it you'll get very frustrated and really burnt out i mean in a few days you days, there's parties, there's everything. You were supposed to have a second question, weren't you? Can I ask my one question now?

You can go for it. I preempted myself. You can go for it. All right. This one requires some setup. Okay. Oh, my gosh. Is this a whole gig?

This is a bit. So you've been on ShipIt, was it once or twice, Tamer? Twice. Twice, okay. But who's counting? Yeah. And I'll say that after your first appearance on Ship It, not because of that appearance, but afterwards,

we had a retro, and you asked me, like, what's good about Ship It, what could be better, any suggestions, and what did I say? Do you remember what I said? No. Okay. I said, I want to hear more from Tamer Selah. I said, you should have him back on the show.

Okay? Yes. Remember that? Yeah, I remember that. That's pretty much my only feedback. I was like, no, this is the compliment. No, no, that's right. I'm sorry it was a compliment.

I enjoyed listening to you talk. I thought it was a great episode. Was it your birthday the first time or the second time? I think we wished you happy birthday. It was the second time, right? Oh, okay. That was, I remember that. It's basically a small bribe.

All right. Yeah. So there's the compliment. Now, okay. The carrot? That was the carrot. Okay, great. I enjoyed listening to you.

Now, we do clips. You probably know about this because I emailed you to write a blog post. Yeah, yeah. So you know where this is headed. So we do clips, you know, promoter shows, pretty standard stuff. And I put together a clip from one of the two episodes. I'm not sure which one. Put it on YouTube. And it's been one of the most commented on clips that we've had since.

It's no such thing as bad publicity. Yeah. And so the title of the clip, which I picked the title, wasn't you, but it's NixOS is interesting but has fatal flaws. And you mentioned something about Nix's fatal flaw in the video. It's a 60-second video of a much larger conversation. It wasn't even your point. You were making some point about science fiction novels and foundation.

How wrong. You had returned to reading foundation. And it was, not quintessential, it was the naivety of what they thought the future might be. Just a different path. Exactly. Which I just talked about that recently on Back to the Future too. They totally missed smartphones. Hoverboards would have been cool, but the ones that we got weren't cool. It could have been. Yeah, it could have been. Alternative future. But when I watch it with my kids, they're like, this movie's lame. They were way wrong. And I'm like, yeah, but when I watched it, I didn't know they were going to be wrong. Anyway, so that was

your point. NixOS wasn't your point, but you mentioned that and I picked the title. And I'm like, yeah, but when I watched it, I didn't know they were going to be wrong. Anyway, so that was your point. NixOS wasn't your point, but you mentioned that, and I picked the title, and then I wanted to read you some of these comments. All right. Is it a fair? That's what you want to know. Are the comments like a fair? It's some sort of fair. All right.

I want my 61 seconds back. I thought it was over 60. I wish I could do that for you. Such a vacuous video. I love the part where they never mention what the fatal flaws were. This video will age like milk when Nyx takes over the world. Tether Sale has no idea what he's talking about. To be fair, that last quote you got from somewhere else,

that's a pretty common quote on the Internet. Reply here if you find the fatal flaw. All right. Anyways, this guy is daft. Just play along. Very good. Very good. So, I mean, the one question that I came prepared with is,

what are NixOS's fatal flaws? Because I like to let these people know. All right. Okay. Yeah. No, totally fair question. Totally fair question. I think that that title might have been a bit exaggerated.

NixOS had one fatal flaw, which is the usability of Nix. And I've never talked to a single Nix advocate. And by the way, I love, I really do love the passion in the Knicks community. They created that, right? They created those comments. Totally. Because that 60 second clip was, there wasn't a lot of content there, right? And there's just a lot of inflammatory. I don't know why you picked that segment. Anyways. I know how to clip stuff. He did his job. No, but what I was saying was,

Nix does have that fatal flaw of a really horrible, like learning curve user experience, right? And even talking to, for example, some of the people inside Shopify. Shopify was touted as a place that was going to use Nix holistically throughout their entire developer experience. And they tried and they put a good effort into it. But I've talked to a lot of the engineers that said, no, it was too hard to understand, especially for our new engineers. And it just didn't work, right? That being said, I know there's a lot of initiatives to fix Nix's usability, and that's great. I want to see that happen because I personally

am actually very excited about some of the aspects of Nix, like what it opens up. NixOS, sure, but just Nix as a package manager in general is just very interesting. It's a really cool technology, but also timing. I mean, I read some of those comments too. And for some reason, this message was lost. So what am I going to do? I'm going to say it again, right? Docker solved a lot of the problems that Nix is supposed to solve. There are ways to use Nix and Docker together. And a lot of the complaints I saw said, you don't understand Nix or he doesn't understand Docker. And I'd like to think I understand Docker. If I don't fully understand Nix, fair, but I did a lot of studying on it. Like I think I understand Nix too.

Docker is not just about running containers. It's not just LXC. Docker solves three different problems. Running your container in a secure multi-tenant fashion is definitely one of the problems Docker solves. It's the most obvious. Packaging your dependencies, so basically the first one was Docker run, Docker build. Packaging all of your dependencies into one unit of distribution

is another huge problem that it solves. The third is as just a package distribution system. So Docker Hub is the third, right? Docker Build, Docker Run, Docker Hub. Nix solves the last two. Nix solves the packaging, your application, and its dependencies better than Docker does, right? Too many people don't understand that if you run Docker Build twice, and you're not careful about your layer caches, you're not going to get the same result, right? I hate that about Docker. You remember Bosch. Bosch got that right. Nix is a better version of that, right? But still, Docker solves it for the masses. The masses don't care about that one little niggly part of Docker build, right? The mass is just like,

whatever, it's Docker, it's everywhere, right? And Docker Hub solves the distribution problem. I want to install Nginx, I just Docker run Nginx, done, right? Nix also solves the distribution problem, but Docker's got more momentum. So everybody has a Docker image. Not everybody's got a Nix, what is it? Flake. Is it a flake? That's what it's called. I thought flake was like an... It's an experimental thing. Nix package.

Now I'm showing my ignorance, right? Yeah, yeah, yeah. Nix package. Nix package, okay. Well, the flakes have had their own issues in the community. Yeah, they're there. Yeah, I mean, the community is split.

There's a couple of things, but you mentioned something really like the UX, right? The DX and the DX especially. Who doesn't understand Dockerfiles would be my question. Dockerfiles, I hate that format, but it's obvious, right? It's real obvious what's going on with it. But it's like, my apologies to Solomon, but every time I see a Dockerfile, I'm like, really?

I personally like YAML. We are fixing that. It's all good. We like YAML. It's like confession. Yeah, I'm like the only person. We are fixing that. It's all good. We get it. We like the YAML. It's like confession. Yeah, I'm like the only person who likes the YAML. I don't, I don't. Anyways, but my point is Docker does three things.

Nix does two. Nix does not solve the running in actual isolation, right? Nix solves the isolation of dependencies in a very good way, but it doesn't solve running in namespaces, in C groups, in security, and all of that. Nix doesn't solve that. So if you combine Nix and Docker, which there's that Nixery, which is a great project, that Docker registry that can make Docker images on the fly

based upon the Nix packages, that's a cool application of Nix. I still believe that Nix has a future in today's technology arena, but as an implementation detail, right? And I think there are some small teams that are basing everything on Nix, and they're having a lot of success with that, just because of the learning curve. I just don't think it scales to any larger environment. And people like Mitchell, they go all out. I love his blog posts, but man, they man they're hard I mean if you were to try and do that yourself you're like whoa really do I really want to do all this stuff I mean it's

great but I really want to put all this effort in some people do and that's okay but most people don't yeah and again that that should be fine too and I mean I hope this isn't offensive but to be frank not everybody's a Mitchell even if I had the time and energy, Mitchell was a genius coder. I mean, he really was. So Nix is a fantastic system if you can adopt it in an isolated, hermetic environment, right? But it's not for the masses. Docker's for the masses. I'll go comment that.

Tamer says go to hell. Tamer says docker is better than nix. Tamer or someone. That's the next episode of the flame war. There you go. No such thing as bad publicity. Just say superorbital next to my name. Tamer of Superorbital.

Yeah, yeah, yeah. Thank you for answering my one question. I'm all out of questions. Cool, great. Well, I have a few more. That's okay. Right. I do have to say that Docker, I personally don't like it,

and I don't like it from the Mac experience perspective. You've got to be on a Mac, so you have to run it on Linux. Otherwise, you get a terrible experience. Docker Desktop, I mean, I have it. Sorry to interrupt. Have you tried OrbStack yet? No, I haven't. What is OrbStack?

OrbStack's a replacement for Docker Desktop. It's one developer's doing this thing. It's not open source. So hang on. Okay, where is it coming from? Who's the company behind it or who's the developer behind it? It's one developer.

I don't actually know his name. Orb Stack. Orb Stack. News to me. It's great. It's super fast. And it launches whole VMs. Literally, I can launch a VM and get a shell in it in like a couple of seconds

because it uses the macOS native virtualization technology. Is it like a drop-in replacement kind of thing? Yeah. Oh, totally. Really? One-click install, drop-in replacement,

works faster and better than Docker. Does it work with the Docker CLI? Yes. Really? We didn't change any of our tools. It just worked. Orbstack. Wow.

Okay, so everyone, I'm amazed. Whoever that developer is, you can pay me later. Superorbital. That's right. I checked too. The Orbstack. me later. Superorbital. That's right. Back to the orb stack. Maybe it's Superorbital.

James, it was you all this time. You're the developer. My secret plan is revealed. Right now. Sorry, I interrupted you. No, no, no. That's worth interrupting for. That sounds really promising.

It was, for sure. I also, I've always been really bullish on the whole remote developer paradigm, right? The GitHub code spaces, Coder. We use it in our workshops, right? Oh, it's great for education, I'm sure. Yeah, yeah, it's wonderful for that. But I used it a long time for development, too. I ran an instance in Google. It's old school. I just terraformed it up, you know, packered it and all that. And it was a pet, right? I had to maintain

it and all of that. I didn't go so far as to make it something I could kill and recreate, but it was great. I ran a really complicated TMUX setup on it so that I had basically a persistent shell. So I could launch some big job and have it running on my shell and just close my laptop and go on the flight. And then later on the flight, open it up and type a few more commands and close the laptop again. So you're right. Especially Docker desktop experience on macOS is not great. I'm not going to go so far as to say I think this is actually happening, but I'm always hopeful that more companies will invest. One of our clients is Bloomberg, and I've talked to them, and they have a GitHub code spaces type thing

internal to them, and it's been really transformative. I think a lot of companies should investigate that a bit more. Okay. So what do you use in your workshops? What does that solution look like for the students? Oh my gosh, if you could see the horrendous things I have done to terraform the bash scripts that I've written to run our workstations. So our infrastructure for the workshops is we deploy for every squad. So when the students go into their breakout rooms and they're doing their labs, we pair them up because I'm ex-pivotal, so I believe in pair programming. But also just for learning, it's great to have someone who's advanced paired with somebody who's still learning or maybe put three people together if they're all kind of struggling

a bit and they can help each other out. So for that, we use VS Code Server on these cloud workstations. So they get a cloud version of VS Code that they go to in their browser, right? It's got a terminal built in and everything. And the beauty of that is that that workstation is now fully configured to talk to their Kubernetes cluster or whatever we're doing.

Kubernetes clusters, we run like real clusters in the clouds. We also have to provision that for every student, right? And we set it up so that each squad is provisioned and destroyed separately from the others so that we can easily like recreate a student's environment if something goes sideways. Although at this point we've been delivering them for so long that it's pretty polished.

Everything just kind of works. But even, you know, one of the pieces that I've really liked working with recently that we've added is this ability to, you know, when you're working with Kubernetes, you often want to eliminate some of the complexity of, hey, we're going to work with Istio. I'm going to install Istio for you to have that already in there. And so we have this ability to customize per lab environment and dynamically template the YAML files that each student is going to need so that they don't have to go in and fill, okay, I'm student five dot whatever.

Everything just works. We want it to be... Eliminate the undifferentiated stuff and really focus on the learning that they need to do so that everything that they're doing is focused on that topic rather than you know go type in hey i'm you know student five and find that you know find some of those things that don't don't matter as much and one of the frustrating things every once while a student or somebody new to super orbital will say like hey have you looked at this docker solution or this other platform that will run temporary workstations for you.

And they're just running inside a Docker container. But because of the stuff that we teach, I mean, we teach Docker itself, obviously, and you can do Docker and Docker, but it's ugly and it's confusing. Same reason we don't use like kind for Kubernetes. We want students to see what it should really look like talking to real VMs.

But like next week, I'm teaching containers demystified, which is Linux systems calls of how Docker actually works, basically Linux systems programming, right? Where we dig into all the namespaces and cgroups and we do a pivot route and all that kind of crazy stuff. And you just can't do that inside a container. So it means we have to go, ironically, we have to go old school and go back to Terraform and Packer and provisioning these machines in the cloud. And it's kind of

painful because of that, but we make it work. But I think having an environment that is a little bit more accessible is important so that we can say, yeah, you can go access this stuff and you can get hands-on with some things that you can get deeper in rather than just you know deeper than just kubectl apply we're trying to eliminate the toil for the students i mean the worst worst thing in the world is when you're taking a workshop and you're struggling through half an hour of kubectl doesn't work on my laptop right it's just what's the value there, right? Yeah, for sure. Well, a real world. That's actually funny you say that because I had similar experience when I was teaching web development years ago.

And I was trying to provide a happy path to learn the concepts of web development. And a lot of it was just like, let me make sure Postgres is running on your machine and all this other stuff, getting people set up. And I wanted to remove all of that so they could just do the focus but at a certain point you're also like well this is what the work is like yeah you know so you want but you want them to be able to learn it you know so it's it's a it's a balance there yeah no that's actually a really good point and it's the best counterpoint that a student once brought up to me about the fact that we have these fully provisioned and provided workstations for you is that it's like, well, I really wish we had set it up on my laptop so that now I have it set up.

I have all the right utilities downloaded and all that. And I ain't going to argue against that. I mean, it's a trade-off. Yeah, there it is. It's almost like there's maybe two two approaches in that you know if you want to do this asynchronously on your own time go ahead and do that because by the way the real world is going to be on your own maybe with someone else helping you but when you're here do your homework come here and then let's go because you can do that whenever and you will

figure it out we can't anticipate all the things that will happen right you downloaded the wrong binary you know you had a bad morning whatever it happens right the most silly mistakes are the We can't anticipate all the things that will happen, because you downloaded the wrong binary, you had a bad morning, whatever. It happens, right? The most silly mistakes are the ones which are the costiest ones, and you can't predict them. So even if you try to inject some artificial faults, like in your setup, and go and figure it out,

like slightly broken things. We do that in our war games. We'll break all the students' clusters, and they'll race to fix it. It's actually pretty fun. But yeah, we also, we let the students clusters and they'll race to fix it. It's actually pretty fun. But yeah, we also we let the students download like the students get an email afterwards with the home directories and the labs and all the binaries and stuff but So, you know, I can be like, oh you can do this yourself at home and they can but it's not the same

But yeah, it's a trade-off right? We we only have so much time with the students We want to make sure it's spent them actively learning. Right. What I would tell them is that I'm here to remove roadblocks for you. I mean, it depends on who you're teaching. I'm not sure who your students are, but these are like boot campers where it's like they're career switchers. Is this for me, right?

And so I want to get them to a point where they can answer that question. And I tell them, I'm here to remove roadblocks. That being said, in software development, there's going to be serious roadblocks and your ability or your willingness to persist through those is actually what's going to inform your success or failure. Because conceptually, I think most people can get there over time, but you have to be able to power through. And so you're kind of on training wheels with me here, because I can save you hours by saying,

oh, you missed a flag when you started the thing. Which is like, who wants to spend hours doing that? But when you hit the real world, if you're willing to figure that out and just toil, then you find success eventually. But some people just give up and they're done. And it's like, well, this career isn't for you. Yeah, and some people find that debugging rewarding. Sometimes when I'm in the zone, I find that debugging

rewarding. If I'm really approaching it from a systematic scientific mindset, Julie Evans just got done putting out a whole bunch of zines about debugging. And I loved them because it's like, yeah, that's how you do it, right? You approach it with the right mindset. But if you're just not in the zone, debugging is a slog. It's just like two days of just banging your head. Oh yeah. And the yak shave. I mean, you end up finding other problems along the way, and then you're so far away from what you started at.

And that's very unsatisfying work. But yeah, when you're doing it systematically and you find that one thing, especially when it's somebody else's fault. Yeah. Very rewarding. Oh yeah. I still remember like, it was you. It was 20 years ago. And I'm like,

it can't possibly be a bug in GCC. And it turned out totally a bug in GCC. That's a satisfying find. Yeah. No, I remember, you know, I think back to my early career days. And, you know, one of the first times I solved a problem like this that was persistent, you know, this sort of thing where they're having this problem and it was, you know, failing in one environment and, you know, people are all upset about it. And it's like, and you get down to, you know, the two days of debugging and figuring it out and, you know, finding out the one line that didn't get correctly updated. And there is a point of reward to that when you're really, when you've really dug into it. It's a moment that's really burned in my brain from over a decade ago now, right? Yeah. Yeah. So you've been doing these trainings for

how many years, the workshops and the trainings? Since 2017. Wow. That's a long time. Is there a lesson that took a long time for you to gather from these workshops? It was like a valuable one, something that you haven't realized year two year three but as you were gradually you know making more workshops more trainings they're like something that now you know that it's really valuable that other people maybe miss whether it's a student or maybe what these workshops are so something that is not obvious to begin with but then you realize we keep seeing this pattern you mean in like how how to make the workshop successful or do you mean from the attendees

like i attend this workshop and i have like a certain result after it i have there's an outcome for me and you have the students that you know they keep telling you the same thing or you keep hearing the same feedback in that the workshops are great in one way that maybe it wasn't obvious and they keep hearing the same feedback over and over again? Well, it did take us a while to find the right balance between in the weeds details versus big picture. So like I said before, when I'm creating the workshops, early on, I got into the flow of starting with the lab and starting with the tests even before that, right? Because especially for some of the more exploratory stuff where I was kind of learning as I was

building it, right? Which is really fun, by the way. The exploration I was doing would kind of, the notes for that ended up being the tests and I would you know go through it in this big bash script and eventually kind of build out what ended up being the lab one of the things we've been talking a lot about lately as we as we get into the student shoes a little bit more is the learning over the course of you know we have three to five day courses right and it's it's a different type of intensity from your regular working day. There's a lot of information packed into a really short period of time.

And so what I found to be really interesting from a student's perspective is trying to help them get in the right mindset of learning without getting overwhelmed by, oh, I have this concept and this concept and this concept I'm trying to put together. And so the thing that I've done in recognition of that is really find ways in our discussions at the end of chapters, at the end of our day, we always spend some time in conversation,

and kind of take a step back and try to boil things out a little bit. It's like, hey, we talked about a lot of things today. And we go through some questions, but really say, give them a little bit of a chance to breathe and think for a moment before jumping in with throwing a ton of content. I love going through, for myself, I love going through caps in the class. Because when you look through a Kubernetes enhancement proposal, you're really seeing the why behind the platform.

And so we spent a lot of time doing that, but you can't jump from lecture and lab and thinking about this stuff intensely to then, you know, spend another 30 minutes, like going into the most intense discussions, you know, reading Brian Grant comments from 2018, you have to work into that so that it's not overwhelming and so that you can so people can get the most out of the class from where they're at because some students are just learning some students are you know looking at open source projects and seeing how we can how they can apply the concepts that we talked about in that day and so you need to balance it and I think

from the student perspective you need to understand kind of where you are and how you can say, look, I'm going to take a step back right now and take a breather and gather for myself rather than having to engage. And where other students are, hey, this is the moment for me in my depth of understanding to get further in. Yeah, related to that, we are always questioning what's a bridge too far, right? And James has been working on a Go course, or actually a couple of Go courses now. And one of the things we debated in the beginning was, do we teach test-driven Go, right? Because for some students, that's just obvious. Like, yeah, that's how I want to learn. We do all of our development test-driven, but for some students who've never heard of test driven development, that's another paradigm shift that you have to teach. I think we, we, we, we decided to teach the courses with tests. Yeah. I'm looking at like,

wait, which way do we go? Development, but yes. Yeah. And, but we might end up deciding after this course actually launches and we give the first few maiden voyages, we're going to have to refine that, right? There's certainly going to be some work to explain TDD just enough for the people who don't know it, but not so much that you're boring the other people. It's that disparity of the people with lots of experience versus the people who are new to something right the wider that The class disparity is the more challenging it gets right and your students are all over the map on that It depends on the company that we're teaching right but Some of our cohorts have been all over the map and it can be really challenging. What we found, and maybe this is my bias, but I feel like

if you're at a conference and you're at a talk that's extremely deeply technical and maybe over your head, that's still more satisfying than if you're at a conference and you're at a talk that's maybe a little boring, but you're getting all the information. I think students would rather try to swim in the deep end. Exactly. I think students would rather... Try to swim in the deep end. Exactly. I would agree with that. So we err on the side of far too deep on our workshops.

That's a good one. That's a good one. So the reason why I ask that question is because I know that within us, X pivots this like, you know, refinement, the constant improvements, you know, the retros, we have them there for a reason. It's all like to reflect back and to keep improving all the time we can work out and the reason why i ask this is because you know if you approach it that way what you do over years compounds and where you end up it's like such a different place in so many ways and it has like the bet the the depth

and the breadth and everything and and you see it much better. You just expand everything. So I think the CNC of landscape and Kubernetes itself needs curation, right? It's almost like it's too big. So how does that even happen? How does the curation happen so that it's still relevant and it doesn't limit people to the choices that maybe some are good for them? But you can't, again, give them everything because that's too much, right?

And you're talking about an individual's journey, but what you're saying is equally true for companies, right? Companies at different stages, they should be using different sets of technology. Right. And what we actually see, I don't know if this makes sense, but this is what we actually see, is that the younger companies tend to use the more interesting technologies, the technologies that are newer and edgier. But they still have to worry about innovation point spending. So they'll still limit how many of those technologies they use. And the larger companies are the ones who are using the more traditional stuff that's

been in the CNCF forever. The interesting part about curation is understanding how many innovation points come with a specific technology. How many innovation points are the project of Linkerd versus the project of Istio? Oh, yeah. It's a part of the project. The Istio core team will talk about,

they understand that there's more complexity in Istio than Linkerd. And, they understand that there's more complexity in Istio than Linkerd, and part of that is by design. It's trying to solve a wider problem set. But sometimes we go into a company and they talk about what their current stack looks like, and to us, it's very obvious after that first conversation of like, oh man, you have exhausted your innovation point. You are doubled over of, you know, of your innovation account.

Credit cards are maxed out. They shouldn't be spending anymore. You've got credit card debt of innovation points that is going to rack up real quick. Are you a Pivotal Labs guy as well? No. Okay. Because it's sounding a lot like story points at this point. Do we need a Fibonacci scale of how many innovation points per project? No, no. Tamara may have been doctored

at me over the last few years. But I don't think, I'm not all the way there right now. But it's just a basic understanding of even just like a t-shirt sizing, right? Is like, what kind of complexity are you taking on with a project, right? State on Kubernetes, right? You have to understand if you're going down that path, you are making a necessary trade-off. And for some folks, that's reasonable and that's an investment they want to make. And for other companies, we have encouraged them, why don't you move, you know, you need to move away from this

because you've gotten in a situation that's not going to be tenable for the long term. This was a great conversation. We still have to wrap it up, unfortunately. We had a lot of fun. Thank you. Which is the key takeaway point for the people that... Nix, we can do Nix. We can do anything.

Innovation points. Actually, I think innovation points is... That's probably the key thread to everything we've been talking about, including Nix. There you go. Well, yeah. And learning about that within your organization. I think we can combine some of the things we're talking about is that you have to... You see the impact, you see the, you know, how your organization is growing along the way. And that's where we can really bring that continuous improvement is, you know, maybe you get really good at using your innovation points in certain areas and, and, and it's a journey, right? You don't, you're not going to get it right on the

first iteration. And the goal is to learn and grow there. And I hope that's, I feel like that's a takeaway. You know, if Tamara, you know, it's one of the models of our company. So I'm hoping that we can. Yeah, always be simplifying, right? Because you're going to need to spend more innovation points later. So you need to earn it back

by reducing the number of technologies you're using and reducing that complexity. So not improving. Not improving. Simplifying. Great. That's a great one. Reducing that complexity. So not improving. Not improving. Simplifying. Great. Great. That's a great one.

All right. Head on that. I think it's time to end. James, it's been an absolute pleasure. Thank you. Tamer, third time. They're only getting better. I have so much fun on these.

Yeah. Thank you. Thank you, everyone, for joining. See you next time. What's up, friends? AI continues to be integrated into every facet of our lives. And with the rise of AI, technology is starting to become more, I guess, kind of human in a way. And that's causing some mixed feelings amongst myself as well as other human beings who use it. And it's just literally so weird to specify humans.

It's just, it's real. But that's why I'm loving season three of the TraceRoute podcast. It's a podcast that explores the people who shape our digital world and how technology is changing society. In every episode of TraceRoute, expert technologists peel back the layers of the stack to reveal humanity in the hardware. Season three so far has explored our love-hate relationship with AI, and there's so much more to come. Get keyed into this podcast today.

Listen and subscribe to the new season of TraceRoute on Apple, Spotify, or wherever you get your podcasts. We are in Chicago at KubeCon 2023, talking with two people that I bet my production Kubernetes on, Steve and Spencer. Welcome. Thanks. Happy to be here. What is the best thing that happened to you so far at KubeCon? When I was walking here from the conference this morning, a guy just stopped me and said,

"'Oh, so there are lives, what do you do there?" And I was like, well, I'm CEO. And he's like, then he just went on a 10 minute spiel about how much he loves Telus operating system and how he's building a production cloud hosting environment on it. And he was using it at home for his home Kubernetes deployments

and how he likes the architecture. So it's just like that is pretty cool, just when people have such enthusiasm for the product and the architecture, and that's pretty common. Yeah, and I think, honestly, mine's kind of the same. I mean, I had someone come up to me and say, hey, you're from Sedera? I love Sedera Metal.

My whole team loves it. And so it has been interesting to see, I think, people actually recognize the logo and stuff now, which certainly wasn't the case, you know, a couple of KubeCons ago. And if people had heard of Talos Linux, they hadn't heard of Cedera Labs. And so seeing some of that change, I think has been really, really fun for sure. So I think, again, big fan of Talos. I'm using it myself. As I mentioned in the intro, I bet my production on it and it's been great. And there's a couple of

things there that people miss, right? Because they think, oh, it's just another operating system which has Kubernetes. We had one of those. But obviously this goes much further than that. So people that are familiar with Talos, they know what is inside of it. But the one core concept which I find fascinating is this controllers concept that you mentioned, Spencer, which isn't just in Kubernetes, it's also in the operating system. So how does that work? How do controllers work in the context of an operating system, which is not Kubernetes? Right. The project that specifically does the controller-based stuff for the operating system is called COSI, C-O-S-I. And that's the Common Operating System Interface is kind of what that stands for, if I'm remembering correctly. It's been a while since I

looked at the acronym. But yeah, so the way that works, I mean, if you think about Kubernetes controllers and the controller paradigm, it's really just, here's a set of inputs and here's a set of outputs, right? And the controller handles that, right? So you're looking at informers and things like that in Kubernetes to watch events. Because Talos is all API driven,

we have the ability to essentially do the same thing at the operating system level. Cozy as an idea, we have larger ambitions for because outside of Talos, we feel like any operating system should absolutely be API driven. Even if it's an Ubuntu system with systemd as the init system, all that stuff, you could easily use Cozy and extend it to basically do some of that on basically any Linux distro. But obviously, Talos is the reference implementation of that. But yeah, it's been interesting to switch focus a little bit. When you stop caring about the ins and outs of the operating system, you want it to be

controller-driven essentially. I mean, because you don't want to deal with that stuff. You want to say, you know, my IP changed, I need new certs or whatever, just do it. I don't care. Right. And that's kind of the beauty of that, that paradigm. Okay. So we'll skip past the no SSH part because it's also very cool, but it's like, there's not much to say about that other than you don't get an SSH into the operating system. Which causes some issues with some companies' security policies, because they're like, we have to be able to scan on SSH to make sure your SSH is secure, amongst other things. Like, well, we don't have SSH, so it's definitely secure,

but you can't scan that. Interesting. So, what would those companies do that need SSH to be able to scan, but they don't have SSH? What is the solution there? Some of them we've actually worked with their security software vendors like Palo Alto. And we've worked with them so that their software now recognizes Talos as a secure platform. Sometimes it's just a matter of convincing them. Okay. Yeah, these security agents are funny in general, right? Because I mean, even if you have to run it on the machine, because a lot of enterprises you do, of course,

but you've scheduled it basically as a pod in Kubernetes and as a daemon set on these nodes. And then it tries to like use a bunch of binaries on the host system to check in things. And you know, Talos has none of that. So it's like, you know, a lot of it is arguing like, look, what you guys are looking for just isn't possible. So, you know, youos has none of that. So it's like, you know, a lot of it is arguing like, look, what you guys are looking for just isn't possible. So, you know, you should approve it anyway.

Okay, okay. That makes sense, okay. So as I was mentioning, SSH, interesting. I didn't know about that. That's a very interesting segue. The other interesting, again, personally interesting, is the CubeSpan feature, which allows you to start like clustering.

It's like, how easy is that? Seriously, I mean, this is not marketing because I'm using it, right? And I've seen how easy it is in practice. And I was like, wow, this thing is great. Like WireGuard, all seamless. It's a little bit of that tail scale experience that maybe some users are familiar with

because they maybe have it on their phones and computers and whatnot. But it's a little bit like that and you get it like in Telo. So how does that work? Yeah, I mean, CubeSpan is exactly that, right? phones and computers and whatnot, but it's a little bit like that and you get it like in Telo. So how does that work? Yeah, I mean, CubeSpan is exactly that, right? It's WireGuard built directly into the operating system essentially.

So when you say, I want CubeSpan enabled, we kind of couple what's going on in the local Kubernetes cluster with our discovery. We have a discovery service too that your nodes, if you have it enabled, they'll reach out and say, hey, I'm part of this cluster and here's my info. Here's all the IPs I know about, right? And so that's how we orchestrate that wire yard mesh. And what's been super interesting about KubeSpan, I feel like when we first launched it, it didn't get the traction I think we kind of hoped for. But now, as it's kind of been baked in for a while and people have really started using it we've seen some really cool architectures come into play because of it for example we you know we have a

bunch of bare metal users and we've seen a big you know for us at least a big swing towards bare metal and edge right because that's where people are wanting oh yeah something like talus right yeah but yeah it's been interesting to see the architectures that come up with that because we've seen folks not want to you know you don't want to spend 20 grand on three control plane nodes that you know are boxes in your data center right so like cubespan lets you do things like i want my control plane in the cloud nearby and it's smaller vms and all of my bare metal is orchestrated with Cubespan and it just works. It's a pretty cool setup for sure. Okay. So it's been about 10 months since we've last spoken.

It was January 2023. 10, 11 months roughly. Depending on how you look at it, it can be either very short or very long. But I think in our world, it's a very long time. A lot of things are happening. What are some of the highlights that happened for Sudero Labs and Talos in the last 10 to 11 months? Yeah, lots. Probably the biggest one from a company point of view is the launch of Omni, which is our SaaS for managing Talos clusters. It's a SaaS for bare metal, which sounds kind of an oxymoron, but you bring your own servers. They can be bare metal. They can be edge servers. They

can be in the cloud. You boot off an image or an ISO or an AMI or what have you. And that's basically it. Again, using WireGuard built into the Talos kernel, those machines will register with the SAS UI control plane side and show up as unallocated nodes. And then you can just go through. You can use a UI to say, these are control planes, these are workers, go create me a cluster. You can template all that

and do it in an API driven way for if you're doing GitOps. And it just makes cluster creation declarative. It makes cluster upgrades, operating system upgrades, ACLs. It solves authentication problems that ties into your enterprise SAML or other identity

provider. So that got launched in end of February, early March, and that's done really well for us. We've got people running hundreds of clusters on it, and it's a great product. Okay. Kallos itself has added TPM encryption for secure boot, probably a variety of other things. Oh, Kube Prism, which is a great... Oh, yeah. Yeah.

Oh, yeah. Tell me. Okay, we'll get into Kube Prism in a minute. Oh, yeah. Oh, yeah. We have a story there. Okay, cool. What about you, Spencer?

Anything that... Yeah, no, I think Omni's the biggie for us, for sure. I mean, I think what I do at Sidera Labs is I do the operation. I kind of head up two parts, the operation side of things and then the customer success side of things. Right. So I've been part of running Omni for since we've launched. And that's been an interesting set of things to learn. You know, as we've had issues with BGP sessions and all that fun stuff, that's been pretty crazy. But seeing the growth of Omni, of course, has been awesome. And seeing our customers that have chosen, especially like we have an EV company that

we work with that does charging stations, right? And all of their charging stations are Tilos clusters. And they're all checked into Omni, right? So when they talk back to Omni over like LTE connections and parking decks and things like that. Are we talking hundreds? Are we talking thousands? Roughly how big? They've got 500 distinct clusters right now

and those are just like, you know, different edge sites basically. So again, that's just seeing some of the architectures that have come out of Omni growing the way it has has been really interesting for sure. Okay, okay. I think it's that flexibility of getting a Talos node anywhere

and then connecting it and connecting them all together, which is really, really interesting. You can look at them all together. How do you deal with unreliable network connections? Speaking of KubePrism, right. Okay, yeah, yeah. Okay, well, he keeps coming back. So let's talk about KubePrism.

Yeah, I mean, and that is the answer for some of it, right? Because, I mean, we have had, especially like some of these edge places that do have really crazy networking things going on you know a semi-truck parked in front of the in front of the parking station and so then it doesn't work right stuff like that so it is kind of interesting because I mean you know the way that Kubernetes is architected your workload should remain running if you do not have access to a control plane. Like, that should still work. Yeah. But what we found in practice is there's a lot of things around, yes, your pod may stay

running, but if you've got ingresses and Nginx ingress as your controller, essentially, right? Like, that's actively talking to the API server. And once that connection goes down, your ingresses go down. Yep. Yep. And so, basically, some of this we've solved with kubeprism, right? So the idea of kubeprism, and this was actually an idea that was built into kubespray back in the day because I maintained that before we started Talos.

But basically the way it works is there is a load balancer on every node in your cluster if you have kubPreset enabled, right? So that means that the kubelet talks to local host essentially on your workers, and then that load balancer is smart enough to use like Kubernetes itself and the discovery service and these things that we know about to say, okay, here are all my upstream control planes. This one seems to be down. This one takes longer than that one. This is the one we're going to go to for now, right? And it handles all that internally, right? It's just a really small load balancer. And so that means that, you know, your chances of just dying that way, right, are much smaller, right? So, you know, in the Omni sense, it was like, oh, you know, your workers

couldn't talk to the Omni API, and so they couldn't talk to the control plane. And so QPresum solves all of those problems, right? So it helps you kind of keep those things running in the way that you thought they were going to to start with, basically. So that's the story behind it. That's exactly how I discovered about QPRISM, which I didn't know it existed as a feature, when there was issues with my worker nodes connecting back to Omni, and services were going offline because they had an ingress, And I didn't realize about this thing.

And of course, there's documentation about it. There was a video out there, like why QPrism is important and how to use it and how to enable it. Andrei did a great job with that. So I really appreciated having that out there. So I've learned about it by needing it and say, hey, what's going on here? And then the solution was there,

but I had to upgrade and a couple of other things. So that came like a great time. That's why Q-prism, it came on my radar for a very important reason. And I knew I needed it then. Okay, are there other things that people should be aware of when they run Talos? Like little things like these that in practice,

like you realize actually the theory when you get tested, in some cases it falls apart, right? Like what you think would work will no longer work. Are there other examples like QPRISM that QPRISM solves that you have like the cluster API, sorry, ingresses and the nodes being able to connect to the control plane? Are there other such situations people should be aware of?

Yeah, I mean, some of it is still just, you have to know the way you're running and the way that you're going to operate this cluster, right? I mean, we still see issues. We saw an issue just a few weeks ago around etcd and connectivity across availability zones in a given AWS region with one of our clients, right? So there's certainly still like Talos is the culmination of like Andrew and I and some of the other folks on the team having run Kubernetes. In real life. In real life and doing it with KubeSpray and doing it in this like really painful way.

So a lot of what's built into Talos is fixing those things. But, you know, a little bit of the problem with Talos is that Talos is really easy, but unless you've run Kubernetes and had some of that pain, it's hard to see how it fixes those things. You know what I mean? Yeah. But, you know, there are a bunch of features around,

like, I mean, QPRISM, of course. The way that we do upgrades, especially of Kubernetes, like, we try to be really safe about that. We try not to knock you offline. We've got plans on making that even better and being able to be aware of things like there's a Rook-Seph cluster here, and we need to make sure that we're draining that properly and waiting on replication to happen and this and that. So we've got bigger plans on making that even better. But yeah, there's a lot of things built in like that that just kind of hopefully make the operation side easier.

I would say that just from a Kubernetes point of view, a lot of people don't seem to get the, oh, I gave that guy admin a kubeconfig and he's left the company. What do I do? That's a remarkably common question. And what do you do? Well, in that case, like what would someone do in that case? Well at that point it's a bit late. I see, okay.

Don't let them leave the company. Don't give them an admin kubemfig. I see, okay. But, you know, that's kind of the standard practice. You can put Dex and things in front of it, but Omni solves that. But just for a general non-Omni Kubernetes cluster, that's something you really have to think about. It's like you are effectively giving root access to your cluster to someone,

and because that's a certificate file, that isn't tied into your enterprise. So locking them out of your Microsoft Active Directory doesn't prevent them logging in and doing whatever they want. So Omni does completely address that. There are other ways to address that, but a lot of people just don't think about that. One thing which I would like to add to what Spencer said, going just a few minutes backwards, when you upgrade Kubernetes, the one thing which got me is to remember to apply the bootstrap manifests because maybe your queue proxy will go out of sync. That's like a gotcha. I was like,

why is my queue proxy on 127 when my cluster is on 128 like what's going on there and I think this was like especially like for for patch versions I'm not sure whether minor versions do the same thing but applying the bootstrap manifests is a gotcha that I mean once you know about it it's easy it's of course and once you understand how it works but it's something that you know you I found you need to get a little bit of experience with Talos to understand that part. But apart from that, I mean, I have to say production has been, apart from QPrism, we already talked about that, right? I've learned about, I need that and I know why. It's been like pretty reliable and upgrading things has been pretty reliable. And this is again, something that for the listeners, I chose it because I thought like

the technical solution is sound, the implementation looks good. There's like some very good choices being made, like some very good technological choices made inside of Talos. And are you telling me I can get Kubernetes just like that, just by booting a host? Is that simple? Like even just like DDing basically, like DD, lay down the partition, it has its own partition, boot it up, and guess what? It's like a Talos machine. That's all it takes.

And that was really simple. I thought, like, this is how it should have been from day one. Like, why don't more people know about this? So that was like my quest, right, since we last spoke, ShipIt episode 84, that I went on, I see, okay, so what can I do with this thing?

So I had one experimental cluster, single node, but that's okay. I've learned, like, what doesn't work in single node, but that's okay. I've learned like what doesn't work in single node scenarios. And by the way, scheduling workloads on the control plane is a little bit more involved based on when you configure it and how you configure it. But once you know what to do, again, it's very simple. And I think that the documentation has gotten a lot better around this as well. That's another thing which I want to shout out is the community. And I don't mean Sidero Labs, I mean the Talos community,

like users helping each other out and sharing their experiences. That has been very nice to see and experience as an end user myself. Is that by accident? Is the Talos community the way it is by accident? I'm sure it is, and it is a leading question. But we're lucky that it's the way that it is, I think, is the thing. I mean, we do have a really cool community and a lot of helpful people out there. And we keep, we've actually wound up hiring a bunch of the people that were, were community people early on. I mean, that's how we got Tim

and that's how we got Noel and some of those other guys around. But we've always like, I mean, Andrew and I talk about this all the time, especially as we grow. But it's like, you know, the thing that's always been important to me is that we never lose kind of the human element

of Sedaro Labs, right? And like like I want it to always be such that if folks need something from me they can Slack me directly right like they they know where to find me they know where to find Andrew they know how to get in front of us and ask questions like this because I think that breeds a much more collaborative community right and and even our professional services stuff is way more collaborative than just we're coming in and doing work and leave us alone, right? It's the collaborative aspect of that, the human aspect of that, I think is what leads to us having a great community. So that's always been, like I said, as we've grown and hired, it's always been an important factor in how we, you know, bring people on. So what is the number one favorite feature that your users are telling you about?

Something they cannot shut up about, like, wow, this is amazing, Intaloz. Is there such a thing? Well, I mentioned that a guy stopped me on the walk over here, and he was like, you guys just completely solved the provisioning problem. I Pixy boot, and I apply a a machine config and I have a cluster. It's like all these other systems where I have to install the operating system and install Rancher and then install Kubernetes and configure swap and turn it off and do all these other things. It's like, all that just goes away. That's a pretty common refrain, just how simple it makes the deployments.

But it kind of depends on the Kubernetes experience of the person. We have other people that are just like, I love the fact that it's the same declarative controller-based architecture on the operating system as in Kubernetes. It makes perfect sense. If you're not experienced in Kubernetes, that's like, I'm used to SSHing in and doing all congruent said, and none of that works. So what the hell do I do? So it's a bit of a learning curve. So one person's favorite feature is the other one's like i don't get this yet yeah but i think that's a keyword that's part of it being a little provocative on some of the choices we've made i guess i don't know what the right word is for that but like you know i mean shipping with no bastion no ssh of course is yeah

there's a ton of gray beard sysadmins that are like no way in hell would i use this right yeah do you know the first thing which i install on a telos cluster it'sard sysadmins that are like, no way in hell would I use this, right? Do you know the first thing which I install on a Talos cluster? It's my sysadmin. Demon's head. And it has like this, literally if you go to get help. And then you exact into it. Yeah, exactly.

I get SSH. I saw that problem. But obviously you have to have like kubectl access and that goes like through the Omni control plane API and that's great. But also have like a set of tools I need to have like kubectl access and that goes like through the omni um control plane api and that's great but also have like a set of tools i need to have right so for example i use uh iperf quite a bit iperf3 so how do you get that and i need like a bunch of networking tools like to understand what is happening in networking level and then before you know it well what about the volumes can understand what is happening on the host and then well how do i mount the host

you know namespace into so all those problems, I solved with my daemon set. That's why, like, the first thing, I don't have a neckbeard. However, the first thing, like, not a real one, is like, can I get my sysadmin please daemon set running on the Talos cluster? So that is a common thing, and I can relate to that.

We don't really recommend you do that, but yeah, that is common. Right, okay. It works for me, I'm happy. I'm not complaining. But no, I mean, it is interesting. Thank, okay. It works for me. I'm happy. I'm not complaining. But no, I mean, it is interesting. Yeah, it is interesting because, I mean,

yeah, sometimes the answer is what you want is not something that we have APIs for yet, right? And that's usually a kick in the pants that we need this API, but if it's not there, you can always schedule a privilege pod in the workload cluster

and exec in and do what you need to do, basically. There are always ways around it if you've got admin on that kube cluster. What about the number one most requested feature right now? I would say it's better local disk management. Yeah, that's very true. You can read this but i'll read it for you i have my notes managing local volumes that is my number one feature yeah for me it's something lvm based yeah i think lvm is great and it's a waste to not use it i never

really liked uh seph or any systems like that open esb open ebs anything it feels very heavyweight and we have lvm like what's wrong with lvm we can do snapshots we can do a bunch of things so on my talus notes csi lvm is one of the first things which i install and by the way i opened the pull request and got merged in that lvm driver and by this is not topo lvm i tried that one as well it was like too complicated. I can add it in the show notes. And I added talent support to the CSI LVM because that was the first thing which I needed. I have some

very fast disks, even NVMe disks in some case. Why can't I use them? That's what I would like to have, really. Number one feature for me as well. Okay, we're on the same page. You're not alone. Cool, great, amazing. That makes me feel

very good. What about number two? And how far apart are they? Is this like a distant number one, or are they fairly close together with something else that users want? I'm trying to gauge how important one is versus the other. Yeah, I don't know. I mean, I think the other stuff

that I can think that, I mean, I've already touched on a little bit, but we get a lot of questions about kind of what comes next, right? Like once you've got a cluster up and running, okay, now what, right? And so I do think, to me, at least maybe number two is something around, what does a day two stack look like? What does an operation stack look like for this? How does a new user come in and actually grok what's going on and how they deploy stuff in a way that makes sense. So I think that's probably, to me, maybe number two of the type of question we get. We also get a lot of CNI questions. I don't know. Does Cilium do this? Can I do Cilium here? Can I do Calico here?

You know, whatever. We get a lot of that stuff, too. How easy would that be, by the way, swapping out flannel, which is a default CNI for something like Cilium? Yeah, it's super easy. It's just part of the machine config. I mean, for folks that don't know about Talos, everything is driven by this one machine config that gets passed to each node. And so swapping out the CNI is as simple as saying, I want to use a custom one. Here's the URL for Calico's manifest YAML, right? Or something like that. It's really easy.

We have a lot of people that come in and say, I don't want to use flannel. And we chose flannel as a default just because it's super easy. If you don't need the network policies and the stuff like that, that, you know, Calico or Cilium is going to buy you. It just makes sense and it just works.

And we had a lot less headache with your total new user. But yeah, we made sure that, you know, because our clusters are all Calico-based. We run Calico for everything. Question for Steve. Is this somewhere in the docs? The Cilium part? The Cilium part in the docs?

Yes, it is. Okay, okay, okay. I need to look harder then. Search better, search better. That's actually a function, something that I think we do need to improve. Like our doc content is one thing, it's there and getting better,

but our doc search function only shows you five results and you can't kind of sort between them. We need to just like, someone needs to spend half a day and it's like, just improve our search. And we're using a search server, so it should be like two lines of code. It's this ongoing problem that Steve and I were talking about yesterday where

we have a lot of stuff that needs to be done but it's not high priority enough to get done you know what I mean with nine people that's why I asked number one and number two like how far apart are they usually there's a big fire I was trying to think about number twos and it's like most of the number twos which are you know like things

that the image factory is solving which is and it's like, most of the number twos, which are, you know, like things that the Image Factory is solving, which is like, it's cumbersome and if you have to write a system extension and how you deploy that, it's like, that's not a good user experience. How to enable GPUs wasn't a good user experience. With the Image Factory, that's like, oh, now it's like, I want this with this extension

and the USB drivers and the GPU support. Tell me more about that, I don't know about that. What is the Image Factory? We sent an email about it just like last week. Really? Okay. All right. Well, I'm not good at it.

It was straight to spam. It was KubeCon. No, no. KubeCon is like a bit crazy the week before and the week after, so I blame it on that. But yeah. So tell me more about that. I'll spend some time. Yeah.

So the idea of the Image Factory is... So the way that Talos works is everything kind of has to be there at install time right like you're not adding things to the system after the initial installation right so no loadable modules no and so all that stuff has to be there at install time or at upgrade time when when new things get laid down to this friend and and it has to be signed and it has to be signed yeah so what would those things be can you give us a couple of examples of what those things would be? Yeah, so like the systems extensions that, like we have an extensions repo that we kind of host some of the community submitted stuff, some stuff we've built ourselves. But it's things like GPU drivers is the big example that everybody, you basically say, okay, I install Talos, and then I say,

I do a same version upgrade, and also say, I want this extension as part of it. This is the way that works. And it just wasn't a very clean process, right? So we're building out this thing called the Image Factory, which basically allows you to, you can go, I think it's factory.talos.dev. I'll have to look. Yeah. We'll put a link in the show notes where people are sending to this. Yeah think it's factory.talos.dev. I'll have to look. Yeah.

We'll put a link in the show notes where people are sending to this. Yeah, that's perfect. Cool. But you can just go there and it's just a web UI that says, I want this Talos version and I want these extensions installed. And that will basically on the fly

generate an image for you. You would download it and you can specify that same installer image and that same image like manifest ID across all your nodes, whether they're bare metal nodes or cloud nodes or anything, right? Like it'll be the same image for all of this. And we kind of can generate that on the fly

and it enables people to create these, like we have some folks that have their own extensions for like internal, like we have somebody that runs FRR It enables people to create these. We have some folks that have their own extensions for internal. We have somebody that runs FRR as an extension for some BGP stuff. We have folks that are doing security agents as extensions. That has to stay internal. So having an easy way to build these images and not have to do the rigmarole of bring Talos up, patch it with the same upgraded version. Sign it all. Yeah and it just takes care of all that. So that's kind of

this new big feature and I don't think I'm doing it justice because I think it's gonna be really really cool. But it has to work its way into Omni first. We're not quite there yet. Like it's there but it's not worked into Omni. It's out in beta, it's great if you're just running Talos or probably Cedaro Metal. But the Omni integration is about to be released. It isn't there yet.

Okay. But it's gonna be really nice. I mean, you know, one of the things we're hoping to really enable is making like, I mean, we have a few folks doing it already, but making the experience of GPU enabledenabled bare-metal Kubernetes, making that super simple, which I think is going to be really nice. Because anybody who's installed the NVIDIA drivers knows it's not a fun time.

Oh, yeah. I had to do it recently. I know exactly what it takes. Oh, yeah. Oh, yeah. Okay. It's a whole new world. And the version has to match, and there's a bunch of things like that. Oh, yeah. Oh, yeah. Okay. It's a whole new world. And the version has to match. And there's like a bunch of things like that. Okay. Okay. But the other kind of beauty of the image factory is it's all declaratively driven too. So you have what we call a schematic file that describes.

You can do it this way or you can use the UI. But you can have a schematic file that defines exactly what you want in this image. And so that's also going to enable some of these, like one of the TalosCon talks from last year was this crazy talk. A guy from Docker forked Talos and got it running on some SBC that I can't remember. The Rexta Rock 5? Yeah, I can't even remember which one.

But it was like he had to build a custom kernel. He had to get all this stuff in the mix, right? And so the image factor is going to enable you to like plug that stuff in even too, like so okay i've got this custom kernel that talus would never support but i can you could plug it in basically and talus can run on top of it right so interesting um so it will enable some of these support of some of these things and at least a community level support for for those kind of things. We talked about the Eevee use case. I think that's a really cool one.

Do we have other cool use cases of Talos that you can talk about publicly? Yeah, I mean, some of the architectures that Omni has enabled specifically has been fun, right? I mean, I think that one has been really cool and having this like nodes and clusters everywhere. Yeah. We've also got an AI

robotics company that's doing like, at least as I understand it, there's a control plane and a factory and then all of the robots are Kubernetes workers essentially, which is really cool. You know, I mean, that's an interesting topology. What are the robots running? Like is it ARM? Is it x86? Like what is that? I think they're x86. I can't remember. But it's a pretty cool setup they've come up with. How big of a scale are we talking about?

These hundreds, thousands again? I don't think it's that big yet. From my understanding of them, it's like fulfillment robots. So picking stuff from a warehouse. So I don't think it's huge. Okay. Do you have any idea? I don't know.

Yet. Yet. It's growing. Okay. Exactly. Nice. No, but some of the other architectures have just been, you know, we've seen, like I said, a lot of Q-span enabling hybrid stuff, like control plane in the cloud, and then, you

know, running bare metal in the data center and doing it that way another client that i'm working with right now that we've been having a bunch of fun together they do like multiplayer game hosting for gaming companies and so the way that that that works for them basically is they have their control plane in amazon they have a bunch of bare metal that they bought but you, the burstiness of like a game release, you can't, you can't always anticipate how popular game really will be. And so then they've got additional bursting into Amazon if needed. Right. So it's like, a truly hybrid, yeah, cluster. And it's, it's been really cool because before CubeSpan came along,

I always was kind of like, I mean, I'm a cynic anyway, I think, but I was always kind of like, everybody talks a lot of a good game about hybrid. They want it, they want it, but nobody ever does it right. And I think that's still true with the exception of CubeSpan, honestly. But seeing people able to truly embrace like an actual hybrid cluster and not just, yeah, we have a bare metal cluster and an AWS cluster. No, these things are commingled, right?

And that part I think is powerful. Yeah, this gaming company I think is the best case study for kind of the expansion of KubeSpan. It's like, you know, yes, you've got a whole bunch of dedicated bare metal so that you have costs contained, but if you exceed that and you need to burst out the same cluster on demand into Amazon or Azure, then that just works. Okay.

Yeah, and I think proving that out, and I think one thing that I haven't seen a ton of our Talos users yet do that I'm excited about is seeing how fast Talos really is when it comes to an oh crap scenario like that where you're like, we need as many workers online as we can get right now. And how fast can they do in the cluster right that's something we haven't had a chance to really showcase yet and talus is going to be really good at it i mean well you know tell us it's fast there's nothing there's nothing on the there's nothing there so it's really quick to come online and check in okay are there any considerations for latency when it comes to where these nodes are placed? For example, if you have one in the US and you have another one in Europe, would that work? Could you cluster them with kubespin? Anything to be aware of when the latency exceeds

like 100 milliseconds, 200 milliseconds? It's definitely possible. I mean, kubespin makes all that really easy. But you are going to have to be careful in terms of where your workers are stationed if they're away from the control plane. If those workers require something from the control plane, data about ingresses, whatever, then yes, you're going to have a latency problem.

Whatever's running at this very remote spot needs to be pretty well by itself, I think. It's still going to be part of that problem. You can't solve that, I think it's still gonna be part of that problem. I mean, you know, you can't solve that I think right and obviously your control planes need to be co-located because it's it's ed does not like replication like Yeah, there's yeah, we had we had a pretty interesting Scenario with one of our clients around that like they were doing that CD and three different AZS in the same region and Amazon but the AZs, something was going on networking-wise there, and they were dropping packets all over the place. And etcd was even within the same AZs, and that's a supported, recommended architecture, right? And it still was like, yeah, it lost its

mind. So we had to... That was actually part of why Q Prism was built, I think, if I remember right. Okay. For those listeners that stuck with us all the way to the end, one key takeaway that you would like them to remember from our conversation? I think for me, I'd love to just hit on, again, just the community aspect and how awesome our community is. And so if folks are interested in Talos and Omni especially, come try it out. Give it a shot. There's free trials and all that fun stuff for Omni. And just collaborate with us, right? Like,

the more we can hear from people, the better of a job we can do. So, you know, join in. Yeah, I think it's similar vein. Talos is better than we let people know. We're not, you know, everyone's an engineer except me. I'm an ex-engineer and I'm the one responsible for marketing. So any fault in marketing is entirely mine, but we don't do a great job of pointing out all the features that, you know, Talos already does things like,

you know, tells you if the version of Kubernetes that you're upgrading to is going to break some of the API calls that you're in use. This is something we've been doing for, I don't know, a year or something or other. And it's something that I think Amazon is going to release to giant fanfare in about two months.

We've been doing that, and I don't think we mentioned it on our website. Nice. Okay. So we're worth a look. Better than you think. We're better than you think. All right. So that's, okay.

And is there something that people can do to help you with that or to help with that? Like what would you like to see happen there so you can be better at marketing? What would that look like? I mean, you know, the question you asked, what do people value about Talos? I would love people to tell us that. Like why are they enthusiastic about it? Why do they like it? Because that's, you know, putting it in the words of end users so we can convey that.

It's great. If people want to write blog articles, we'd love that. I mean, documentation PRs are always appreciated. I mean, we're a small company with limited resources. So if people help with one thing, that frees us up to work on marketing or messaging or all the other things. Got it. 10,000 other things, small things that need doing that don't rise to the level of today's priority that's a good one okay well on that we will end it here steve thank you very much for joining spencer it's been an absolute pleasure i look forward to next time thank you

so we're getting close to the end of the year is It's that season again. Yes, the season of State of the Law. Jared and I are going to come on the mics and talk about what we loved about this year in terms of what we shipped, in terms of topics we covered, and stuff like that. And we want to hear from you. So go to changelog.fm

slash S-O-T-L. S-O-T-L stands for State of the Log. So go there, drop a voice recording, and if we air it, we're going to ship you a free t-shirt. And of course, you get your voice on the podcast. And that's kind of cool. So of course, you heard on this podcast today, Jared and I met Gerhard for the first time.

I mean, we've known Gerhard since 2016. And what's interesting about that is how much trust, how close we've become with someone we've only met on the internet until recently. And I think that's just such a strange, a peculiar way for the world to operate. But it's quite normal. It didn't feel weird meeting him. And it was as if we just left off from the last time we talked. It was cool, but KubeCon was awesome. It was so big. The

expo hall was like a circus. It was just, you have to go and experience it. You really do. So of course, a big thanks once again to the team at KubeCon and Cloud Native Con for having us out at the conference. And also to our friends and our partners at Fastly, our friends at Fly, and our friends at TypeSense. And last but not least, Breakmaster Cylinder, the Beat Freakin' Residents. Those beats, just so good. So good. Okay, that's it. This show's done.

We'll see you tomorrow. Game on.