The Changelog: Software Development, Open Source - Kaizen! The best pipeline ever™ (Friends)

Episode Date: June 2, 2023

Gerhard is back! Today we continue our Kaizen tradition by getting together (for the 10th time) with one of our oldest friends to talk all about the continuous improvements we're making to Changelog's platform and podcasts.

We are going to shift in 3, 2, 1. Welcome back to Changelogin Friends, the weekly talk show dedicated to continuous improvement. Thanks to our partners for helping us bring you the best developer pods each and every week. Check them out at fastly.com and fly.io. Okay, let's talk. Well, we're here for ChangeLogin Friends, and we have one of our oldest friends with us here to kick off this new talk show. It's Gerhard Lazu. What's up, man? It's good to be back.

Everything's up. I was just telling Adam, everything that should be up is up. Nothing is down. Don't ask Gerhard that question. Nothing is down, okay? Oh, my gosh. It's all up from here. And, of course, Adam's with us as well what's up adam what's up

chasing the nines chasing the nines how many can we fit they get more expensive as you go don't they they do yes orders of magnitude each of them that's right what's our sla and slo's gear hard we're kaisening i don't know okay what do you want them to be we'll make them whatever you want them to be just i want to be just right not too much right not too little just right yeah i think for changelog.com it's 100 that's just right it's gonna be a billion dollars please because they didn't go down like changelog.com did not go down. That's true. And that's Fastly. Thank you. Yep, that's Fastly. Speaking of being an old friend,

how long have we known you, Gerhard? 2016? Yes. 15? Something, 16, I think. So 16 is when you started working on the code base. In fact, I've been creating this. I haven't got it finished yet

because it hasn't turned out very well, but I'm creating this visualization with this tool I found called Gorse. I don't know if that's how you say it, but it's G-O-U-R-C-E, like source. And it goes through your version control and the whole history, and it creates a visualization

of people working on the code. And I did one across the eight years of our code base, and it was like 45 minutes long. So it's too, like, who's going to watch 45 minutes? And I started futzing with it, trying to make it better. Anyways, long story short, I know exactly when you started contributing, because it was the summer of 2016.

I remember your little avatar coming in on this visualization and touching all the files. But I knew you before that, briefly at least, or I knew of you before that. I think you wrote a blog post for us about Ansible prior. I think we've told the story before on Kaizen's past. But what is that?

Four plus three, seven years? That's it. That's a long time. I think we started talking in 2015. I think it was December or something around that time. And then it took us a couple of months to figure out how and why. The why was important for me. And I'm glad that we got that right. Remember the 10 questions? I mean, I still have the email. We can dig it up. I think we've done this. I think we've done this before. We have. That's why I said we've told this story before on Ship It on a previous

Kaizen. So here we are now. We're on ChangeLog and Friends, but we're still Kaizen-ing. There was a big question on Ship It 90, where would we Kaizen next? And this is where we decided to put our Kaizen episodes. Of course, it's also on the Changelog. This is our new show. It's also our old show, but it's just a different flavor of the Changelog. We're happy to have you here, Gerhard.

We can maybe throw this on the Ship at feed for those folks if we want to, but we can talk about that perhaps offline. Maybe you'll be mad because it'll be like episode 91 and it'll ruin your flow that's okay i wanted fewer episodes so i think this is okay okay why not they're happening but just less frequently now well for those people who listen to the change log and we haven't kaisened on the change log and except for maybe a cross post years ago when we first started ship

it gerard why don't you give the conceit of what Kaizen is and then what we do on these episodes? So Kaizen stands for continuous change for the better. And it has a strong association with agile. And I know that somehow has fallen out of favor with industry. I'm not sure why exactly,

but I've seen that there's a lot of anti-agile feelings out there. Maybe people have been doing it wrong. I don't sure why exactly, but I've seen that there's a lot of anti-agile feelings out there. Maybe people have been doing it wrong. I know that's my joke. Okay. That's possible. You're holding it wrong. Yeah. And one of the principles in agile is to keep small improvements, keep iterating and keep continuously delivering those small improvements. So for us, what that meant was that on a cadence, for us it was every 10 ShipIt episodes. That was roughly every two and a half months. We would talk about the improvements that we're doing for ChangeLog.

So we don't have a release as such. There is no semver. We're continuously releasing. But what we do is we talk about all those improvements that we have shipped in that time frame, whatever it is, on ShipHit it was, and now on ChangeLog. So this idea of continuously improving something

and then pausing and taking stock of what we've improved, it's almost like a retro. For those, again, that have done Ag agile the right way or even the wrong way. You can do retros the wrong way. Anyways, it was one of my favorite meetings because it brought everyone together and we would get so much better for it. But this idea of continuously improving, if you just keep focusing on that and always be better today than you were yesterday that's all you have to care about it doesn't matter how big or how small it is they compound that's the beauty of it all those improvements compound over time so keep improving yeah one of my mottos so we're obviously fans of this process and we're putting our money where our mouth is so to speak because

we're not just talking about our improvements but but we're committing to Kaizen-ing every so often. Our new cadence we're going to try is going to be every other month here on Change. We were doing it once every 10 episodes, like Gerhard said, which was roughly two and a half months. Now we've been a bit on hiatus because we've lost our groove. We've got our groove back now. So it's been longer than a typical Kaizen time period. Doesn't necessarily mean I accomplished more

than I normally do. I think we did, actually. I think you did. Yeah, I think you did for sure. So let's start hitting through some of the stuff we've been working on in and around changelog.com. So for those first coming to Kaizen, changelog.com is our open source podcasting

platform. It's written in Elixir and Phoenix. It's deployed to Fly with Fastly in front of it, has a Postgres backend, and a pretty cool, I don't know, deployment infrastructure. What do you call it, a pipeline? I would say infrastructure. I think all of it is infrastructure. The way it's structured, the way we talk about it,

the way we capture it, even in documentation, the way it's documented, I think it's really cool. Yeah, and so we're doing this. We've been coding on this, like I said, since 2016 for many years, kind of in fits and starts. We go heavy sometimes, we go light, we continuously improve it and we also experiment. So one of the reasons we have this platform is so that we can try new services, try new techniques, hit ourselves on the thumb with a hammer and tell you all about it so you can avoid said hammer or give it a try if you like.

And so that's kind of one of the ideas behind this. It's not merely to make changelog.com a better website or make the distribution of our podcasts better, although that's definitely a huge part of it. It's also for learning, experimentation, and hacking, because we're hackers. And it's nice to have something to hack on together. I think that's very rare, right? Being able to do this in the open source, in the spirit that it was intended, to have the time, right? We are very busy during our day, and during our work week, and then the the weekend comes and there's like all sorts of pressure on your time being able just give yourself permission to try things just for the fun of it

it's so easy to just get down you know ruts or delivering off a backlog or whatever there's never time to try things out so this is again us giving ourselves permission to do fun things talk about them, but also keep improving the platform, the whole change of a platform in an open way. My favorite approach. Yeah, so many tools I come across, I don't have a good enough excuse to try them. Because at a certain point in your life, it's just opportunity costs left and right. And it's like, if I try this thing, I can't do that other thing on my list of things to

do. And so we almost need an excuse to tinker. And this has been, for me at least, in certain ways, one of my excuses where I can feel like I'm also pushing the ball forward while I do something versus merely futzing around with my git configure, my vimrc, right? Gosh, Jared. The old vimRC. Signed commits, finally, I see. I think since the last time we kized

and you have signed commits now, Jared. I did do something with that. We succeeded. I still see that there's, when we merge some stuff, there's still that PRs that come in. There's still like a, what do you call it? A DCO thing that's like failing because the sign-offs aren't correct.

And maybe it's because I rebased via the web interface. I don't know. I've had some problems where I'm like, why can't we just do it the old easy way? Why are we going to have all this security and stuff? But I'm, you know, you're dragging me along, kicking and screaming. All right, let's get into some of the major changes since our last time talking about this. The biggest one, it seems like, was the upgrade of Dagger and the switch from Q, C-U-E, the configuration language

for configuring our pipelines, to Go. And I think we had that last time, but now we're actually using it for more stuff. So obviously, you can tell by the way I'm talking about it that Gerhard should be talking about it, not me. So go ahead. So in the last Kaizen, let me go back to the beginning, I think it was November 2021. That's when the story started. This experiment started, it was a long, long one. And the idea was, why are we using YAML for all these pipelines? And at the time we were using CircleCI, I wanted to migrate to GitHub Actions.

We're just trading one YAML for another. So I came across this tool called Dagger, which was, the whole idea was like you write Q, you don't write YAML, and you can run your pipeline locally. And what that means is that you can run the same pipeline written in queue, whether it's locally on your laptop, whether it's in GitHub Action, CircleCI, it doesn't matter where

you run it, it will always run the same way. It has a container runtime, so all operations run in various containers, you have the caching, you have a lot of nice things. That was, again, November, was 0.1. We're very courageous, but it worked well. And it was a good improvement. We talked about it plenty. We wrote about it. Where do we write? It's all pull requests. If you go to our GitHub repo, there's the change of.com. Even the topics that we're discussing today, there's a discussion, 4.5.2, where you can go and see all the various topics. And this is the first one, migrate dagger 0.1, dagger Dagger Go SDK 0.5. And what that meant is that there was a big shift in Dagger from people that liked Q to people that wanted to do more.

They wanted to do Go. They wanted to do Python. Like, why should you write your pipeline in any one language? Why can't it be the language that you love? And for me, it was Go, right? We didn't have Elixir at the time. By the way, that's changing. We can talk about it later.

Oh, you're getting Elixir support? By the way, you came across Dagger, but then you also went and got a job at Dagger. So when you say we, people might be confused. When you say we, you're talking about you and your cohorts at Dagger, right? Yes, exactly. There's like we, these different things

depending on the context. But yes, I joined Dagger as well. I really liked it beyond just the tool. You can tell I'm really passionate about deployment, about pipelines, about CICD, all that space. That's the space where Dagger sits. And remember how we met, Jared? Deliver?

I do. E-Deliver. Deliver and E-Deliver. Exactly. E-Deliver was a fork of Deliver, the Bash framework Deliver? I do. E-deliver. Deliver and e-deliver. Exactly. E-deliver was a fork of deliver, the bash framework for deployment that I wrote. So there you go. 10 years later, boom, Dagger came along and the rest was history. So 0.5, which is actually 0.3, Dagger 0.3, it introduced these SDKs. You can write Go, your pipeline in Go, your pipeline in Python or Node.js. We pick Go

and we transition. We migrated from having our pipeline declared in Q to a programming language, which has, again, a lot of nice things, right? When you have a programming language, you have very nice templating, you have nice loops, you have functions or, you know, whatever you may have, right? Can't you loop and do other crazy stuff in YAML too? You can, but it gets very, very messy if you do that. Very messy, yeah. And to be honest, why not use your language as much as you can?

And again, we didn't have Elixir at the time. That's slowly changing. But I prefer to write my pipeline in Go. So the first thing which we did, we migrated 0.1. We were running it in the new one. So I think the last Kaizen when we talked about it, it was just a straight import, right? Like wrapping the previous version in the new version. A nice gradual improvement.

Now all of that has been rewritten in Go. We're using Mage locally to run the pipeline. So many things happened. Again, this was two and a half months ago. Now, I want to show you something really cool. I'm going to share my screen. I'm going to go up a bit. Do you see that? This is a pull request that has not been submitted yet. And this is Dagger Engine on Fly Apps version 2.

It's exactly what it says. We're experimenting with Fly Apps version 2, which is the latest implementation of apps in Fly. We're running Dagger on it. I'm connected here, experimental Dagger runner host, via WireGuard tunnel, and I'm running Dagger run, the CLI, and I'm wrapping Mage CI. If your mind is blown, that's okay.

I think you need to watch this video. So you're running Mage inside Dagger, inside Fly on their V2 platform or locally. I'm wrapping Mage in Dagger Run. Dagger Run is just a command that's a CLI that connects my local command to a remote engine. It gives me this very nice view, which is my pipeline. And it's showing my Dagger pipeline

that is running in this dagger engine on fly apps v2 that I'm connecting to via wireguard tunnel okay so what we can see here is we have three pipelines in one and this is something that starts becoming even crazier so we are building the runtime image we are building the production image which by the way makes use of the runtime image and down here we are also the runtime image. We are building the production image, which, by the way, makes use of the runtime image. And down here, we are also running tests, mixed test. But because nothing changed, everything is cached, it completes in seven seconds.

So let me go into application EX very quickly. And let me do a foo2, just a comment, foo2, okay? And I'm going to run the same pipeline again. So now what's going to happen it will detect the code change and now it has to resolve the dependencies compile the app run the tests right compile the assets and all this we have a very nice ui that shows us the different pipelines how they run and how they combine i'm really excited about this i don't know about you maybe you're still like trying to process what you're seeing.

Well, I'm still watching it stream by. And obviously our listener here is imagining this in their mind, in their mind's eye. But it does seem very nice. I like the fact that it's going to cache everything. So let's say I just update an image in my assets folder. I don't touch any Elixir code, and I deploy that out. And with this new code, it's going to run just the mix Phoenix digest command.

It's not going to run compile and stuff. Not currently. See, I went to the logical conclusion. That's the next improvement. That's the next improvement. You beat me to it. Okay. So I'm almost excited.

You'll have me excited later. Almost excited. Yes. Oh gosh. This is still good. Now, if you're going to run this from start, by the way, it should finish right now. It will finish in a minute and a half. Start to finish.

Start to finish in a minute and a half. With no cache. Well, you do have a cache. Okay. If you try to compile all dependencies from scratch locally, it will take you about six to seven minutes. Okay. To get the dependencies, compile the dependencies.

Remember, you have to do it for both test and prod. You have to digest the assets, build the image. There's like a lot of stuff happening behind the scenes. The next improvement would be to split the static assets from the dependencies from the actual application. So now all of a sudden you have three inputs. Right now we have just the application, and the application means the dependencies,

the static assets, and the application code. So it's all seen as one, which is why it would run everything if something changes in any of those files. One minute and 39 seconds, right? And just recompiled everything. And by the way, this would also rebuild the image if it needs to. Let's say we bumped Erlang, part of the same pipeline,

it would rebuild at runtime, it would publish to runtime, all of that would happen automatically. And that's why I don't have to worry. There's like a couple of good pull requests worth checking out. The first one is 454. If you go in our repo, you can see that whole migration, we removed the last make file, we introduced mesh. That's a fairly big one. There's a lot of refactoring there. There's another one, 464, where we are reading

the versions for all the different dependencies, the runtime dependencies, Erlang, Elixir, Node.js, from tool versions. And that's an ASDF configuration file. ASDF is, I think I need to refer to the page manage. So ASDF, manage multiple runtime versions with a single CLI tool. You could use brew to install Elixir, but then which version are you installing?

It doesn't do versioning very well right having multiple copies or multiple versions of a binary of a language runtime on your machine and being able to switch between them is not easy with homebrew i think it's maybe possible with homebrew but not easy or maybe not possible yeah i'm not sure which one but with asdf it's built specifically for this purpose. So this goes way back to the days of RBNV. And what was the big Ruby one? Because we had to do this in the Ruby world. RVM. RVM, Ruby Version Manager.

And then there was NVM, Node Version Manager. And then there's probably EVM, maybe not, Elixir Version Manager. And so each little ecosystem, each language had their own version manager. And then ASDF folks came along and said, hey, let's build one that can handle all these distinct things with one API, one CLI. That's a really nice tool. I mean, I've been using it for years and excited to see this getting further into what we're doing. So keep going. This also solves the one string to rule them all in terms of versioning too, right?

Like we had an issue where there was multiple versions of Elixir in the code base, and this is like one now, only one. Right, yes. If you're using it, so if you follow the updates of the contributing guide, you can see how to use ASDF to install all the things. Now we are versioning tool versions.

We capture everything, including Mage, for example. Now, you wouldn't want to use Mage other than if you want to run the pipeline locally. There's also PostgreSQL, right? So PostgreSQL is also versioned using this. If, for example, you're on Linux, there's like a couple of extra things that you need to do because maybe you don't have some system dependencies

on macOS, it just works. So we are capturing every single dependency in this file, every single dependency version, like down to the patch. So for example, with Brew, because you mentioned Brew, Jared, you're right. There is a certain flexibility in terms of which version you can run to the miner. Some have major.

So for example, PostgreSQL 14 or PostgreSQL 13, I think 13 is still available there. What you can't do, you can't say 14.1 or 14.8. And then there may be differences that you don't realize between your local development version that you're running. Everything looks fine. You push to production, guess what? Things start breaking subtly.

You don't know why. Well, we had one of those a long time ago where it was actually the Erlang patch version that was different. Not even the minor or major version, it was the patch which had this bug in the TLS library or something that didn't exist on my local but only existed in production. We ended up having to debug that sucker. Exactly. So ASTF to manage all our dependencies locally and you just need ASTF, right? That's it. i know there's like certain tools

which use nix but then i forget the name we mentioned it on changelog but if you have that then you have to use the nix package manager and you have to install that and i ran it for a while but to be honest you really want the nix os to get the best experience but then you're running mac you can get the nix package manager some Some things will be a bit weird, especially when you restart. At least that was my experience. So ASDF is a fairly lightweight for what it is and what it does. And we are reusing the tools versions file for our pipeline. We're reading this file in our pipeline, and that determines what version of Erlang we're using to compile in the image that we're using for tests, for example, and as well as building the final image, what version of PostgreSQL we are running in production,

and so on and so forth. Right. So is this, it's.tool-versions, it's a hidden file, tool-versions. Is this an ASDF creation? Yes. It is. Okay. And what is the syntax or what is the format of this? It looks like it's just like plain text. Do they have a spec? Like here's how it works or is it just simple enough they don't need it? It's very simple. It has like multiple lines.

On each line, you have the name of the dependency, space, the version of that dependency. So you have Erlang space 25.3.2. New line. Elixir space 1.14.4. And so on and so forth. Too simple to even write the spec down. Just look at it and you can see how it works. I know, right?

I do like simple tools. So as a Homebrew user and an ASDF user, I sometimes have to ask myself the question of like, what do I install this with? And my go-to logic is like, well, if I could ever imagine myself having to have two different versions on my machine at the same time,

for instance, I have this project over here requiring Ruby 2.3.7. And then this project requiring Ruby 3.2.3 and they're, you know, I don't want to deal with switching, then I go with ASDF. Because of that, I'm a homebrew install PostgreSQL guy and now we have it inside of ASDF.

And so trying to uninstall and install with ASDF, does that screw things up for me? It doesn't screw things for you, no. It will keep things as they are. You can, for example, it won't prevent you from running a PostgreSQL that you've installed via Homeroom. So there's like no such guard in place.

Because it's all containerized or at least isolated. If you're using ASDF, it doesn't use containers. When you run the pipeline locally, basically it will ignore whatever you have running locally. It will read the versions from this ASDF generated file, which is tools versions. And those are the versions that this ASDF generated file, which is tools versions. And those are the versions that it will use for the pipeline, both locally and remotely.

Now, if you, for example, want to switch to PostgreSQL, the same version that we're running in production, the same version that we're testing with in our CICD system, what you want is obviously to do the ASDF integration. Usually when you integrate ASDF with your shell, that will prepend the ASDF integration. Usually when you integrate ASDF with your shell, that will prepend the ASDF path, which means that the PostgreSQL from ASDF will have precedence over the PostgreSQL from Homebrew.

Gotcha. Same is true for Erlang, Elixir. So you can have Erlang installed via Homebrew, but as soon as you're in ASDF and you have it nicely integrated with your shell, then automatically anything that you have installed with ASDF, that's what you'll be using in that specific directory. Gotcha. So the pipeline, though, it does use containers, correct?

Yes. Everything, all the operations actually run in containers. Imagine all the commands that you have in Dockerfiles, right? You have like those one lines. Now imagine if you could write those one lines in code. In our case, it's actually Go. We capture the equivalent of those lines. They're operations, basically. We're capturing the equivalent of those operations in Go code.

They get submitted to the Dagger engine. They create a DAG view of everything it needs to run, and it can reconcile what ran, what still needs to run, what has changed, which parts of the cache need invalidating. It has volumes. Behind the scenes, really, it leverages BuildKit, which is at the core of Docker 2. So anything that you can do...

And by the way, the Docker file is just an interface to BuildKit. So I love this. I love having one place to specify versions and have all tooling and pipelining and production just do their thing. The one that always has scared me historically or been more complicated was Postgres because sometimes a Postgres upgrade requires a data migration.

How does this handle that circumstance? So currently the version that we have specified in tools versions upgrade requires a data migration. How does this handle that circumstance? So currently the version that we have specified in ToolsVersions is the one that we have deployed in Fly. So in Fly when you deploy Postgres you have you divide the Flyctl and it's the version that we had at the time of running this command. It is a platform, right? So you run, you have a CTL, and you also have a web interface. But in our case, we had FlyCTL, Postgres, deploy, create the cluster with the whole clustering set up and everything.

And at the time, we had 14.1 deployed. One of the things that are on my list to do is to deploy PostgreSQL again, like have another cluster using fly machines which is apps v2 and basically pick whatever the latest 14 version is. I think it was 14.8 or 14.9 when I checked, I'm not sure one or the other. So once we do that there will be like a data migration then we will capture this version in our development environment via SDF, which will automatically be picked by the pipeline. So that when we run any tests in our pipeline, in our GitHub Actions, GitHub Actions will obviously be connected to a Dagger engine, and the correct,

actually the same PostgreSQL version will be used there as well. So we have dev, test, and production, same version. But the production is what determines the PostgreSQL version. And that happens when we deploy the cluster to begin with. I follow. So if I want to upgrade Elixir, it's as easy as changing the version in the tool versions file. Yeah.

But if I want to upgrade Postgres, it's the other way around. PostgreSQL, yeah. Postgres is different because it's like a stateful service. You're right. You have data. There's like a bunch of things.

You can obviously change it in the SDF file, but that won't change what's deployed in production because you're right, there's a data migration part to that. Fair enough. It was not quite as cool as I thought it was, but it's still amazing, Gerhard. Well, everything except that, right?

Like the Yard, Node.js, right? I'm two for two on finding the one thing it doesn't do. You know where to look oh yeah i do oh i know what i like i know what i know usually what i want is the hard parts you know taken care of for me so still that's very cool so the reason why i asked if this tools versions thing was asdf or if it was broader because it seems like this little bit of our infrastructure at least could be generalizable enough to the point where maybe it's useful for people to say, here is an ASDF-based pipeline integration that you can use. Do you think that's the case, maybe?

That is possible, yes. That requires a bunch of things on the dagger side as well. So right now, obviously, you can get inspired by our pipeline. You can take it as is and change it and adapt it to your needs. It is a starting point. But really what you want is reusable pipeline fragments or components, right? So for example, if you had like Elixir tests,

like how would you run those? You would want a component that you can just consume. That doesn't exist in Dagger today, but it's somewhere there on the roadmap. Very cool. Well, that's progress and not perfection, but it's somewhere there on the roadmap very cool well that's progress and not not perfection but it's progress over perfection and besides we have to kaizen again soon so we'd have to have something else to strive for you can't just can't do it all now

you haven't asked me something important so i'm going to ask myself the question hey gerhard how long does our pipeline take to run good Good question. So it depends, right, if you have it cached. So even today, we are connecting from GitHub Actions to a Docker running on Fly. And again, if you look through our GitHub Actions workflow, you will see that. There is a Fly WireGuard tunnel setup, and from GitHub Actions, we connect to Fly.

That's where Docker runs. Internally, whenever you run an SDK, it automatically provisions whichever version of Dagger Engine it needs. So what that means is that whenever our GitHub action runs, if it's in our repo, we have everything cached. Even though we have things cached, we weren't parallelizing our pipelines. So we were basically running, build me the runtime image sequentially, then move on to building the test image, then run the tests, then build the production image, and so on and so forth. So the whole thing was like one long line. What we did part of 464, we parallelized these pipelines. So now they run all at the same time.

So the last pull request which emerged, there weren't any code changes. I just had to recompile everything, rebuild everything, just make sure everything is fine. It took I think about two minutes. There was just like a markdown file change. So what I'm curious is next time that you run it, Jared, I think it used to be six to seven minutes for the pipeline to run. I think it'll be around four minutes now, maybe even three minutes. The next thing is to switch, and this is the pull request which I haven't submitted yet, to switch to the fly apps v2 dagger engine that by the way we can stop from within GitHub Actions. So because these are very small, like they're firecracker based, you can start them in within 20 seconds. So you don't have to have this thing running all the time. You spin it up. The state is there. There's like a local NVMe volume. And

this is all basically managed by the fly platform. Super fast. You run your pipeline. The cache is there. The state is there. Whatever needs to change, changes. A few minutes later, again, in my tests, one minute and 35 seconds to deal with a code change recompile everything everything runs in parallel and the deployment part that's the one that basically just depends how long it takes for it to be deployed you can add another few minutes but within three to four minutes actually even like three minutes i think you can get the code change into production that's awesome and that's much faster than what you had to deal with, Jared, right? It was like eight, nine minutes. Yes. Yes. So much faster. Love it. So much so that I would forget it. And then sometimes the build process would fail or

something would happen strangely, mainly from my stuff, really. I don't know if it's like just a me thing or you thing, Jared, but like, I feel like. No, it happens for me too. Okay. Okay. I don't feel bad then. I'd forget about it. I'm like, okay, I have no idea how to restart that action I guess. I guess I can go back to GitHub Actions and say try again. But then there's another thing going on. So then Jared deploys and it succeeds and it takes mine with it. So that's fine. Yeah. Because we're committing to main. Very cool,

Gerhard. I love it. Chasing not the nines in that case, right? That's the anti-nines. Chasing the zeros, man. We're chasing the zeros. Chasing the zeros. How fast can we get this thing well i don't think we can get it down to zero but maybe we get zero in the minutes column you know you can dream it it's already there you know it will take some seconds yeah i think that will be that'll be very difficult because even if you run it locally right where everything is as

fast as it gets right you can have the fastest. If you're trying to compile some code, it will take some number of seconds. And it's not just that, right? You have to spin the process up. You have to reestablish connections. You have to do all those things, right? You have to check the health checks, make sure it's healthy.

So it will always take some number of minutes because you're bringing a new instance. And by the way, this is like running in production. So you have blue-green, you have like a lot of traffic, you're shifting traffic. So it will take some number of minutes to get the code change out in production, so you have blue-green, you have a lot of traffic, you're shifting traffic, so it will take some number of minutes to get the code change out in production. And that's only with one instance,

which brings us to the clustering part. To go to fly machines with our app, we'll definitely need to solve that problem. We will need to be able to cluster multiple instances of the changelog app. Without that, we may... I mean, if the host has a problem, right, with Nomad, with Apps v1, multiple instances of the changelog app. Without that, we may... I mean, if the host has a problem, right?

With Nomad, with Apps v1, the instance could be migrated. With Apps v2, the instance cannot be migrated because it's tied to a specific host. If that host becomes unavailable, there's nowhere to schedule the app because, again, that's how the platform was designed. So the idea is you need to have more than one instance.

So we need to soft clustering, Jared. It's time. So you're pushing this on me. I see what we're doing here. Yeah, this one's on me. By the way, as a follow-up to your last statement,

I just ran our test suite locally. It took 17 seconds to run. So I will accept a deploy of 20 to 25 seconds. No problem. Sure. Yes, I did not get around to this. Although to my credit, I stated that I probably will not get around to this during this Kaizen

period because most of my efforts have been in and around the migration of changelog news onto its own podcast and the meta feed, which is our three shows, which are all distinct shows. So they can have their own feeds, own subscriber base etc and then our the changelog show which is all three of those shows in one show and the one-time migration of stuff we had to reimplement how we do our newsletter and stuff and so that was like what i've been doing the last three months and it's pretty much finished now we do have a idea for how we can make changelog news's web pages better which i would love to do because it's quite the upgrade from what it looks like right now and it simplifies things as well so that's kind of like what i'm

thinking about doing before this but we have honeycomb tracing now from phoenix which we didn't have previously thanks to you gearhard so i'm now without excuse because I can monitor the speed changes as I make these caching changes. And I have a prototype from our friend Losh Vickman, who showed me a way of doing a clusterable caching solution, which doesn't completely rip out the guts of what we're currently doing, which was my previous plan so the skids are greased and the observability is observable by the way you guys have seen honeycomb has their new open ai integration in there i just saw it today no i didn't see this i haven't yeah man spill the beans this is my new favorite thing is like

never make me write a sequel query again. Never make me write a SQL query again. Never make me write a whatever honeycomb queries are again. Just let me explain in plain English what I want you to do and then you figure it out. They integrated that. It's still beta. It errored out 50% of the time I've

used it, and I've used it twice. So the first one errored, the second one worked. But you just kind of tell it what you want to see and it's going to have the OpenAI API come back with what looks like a pretty good query to get you started inside of Honeycomb. So kudos on them for rolling that out quickly and

I think this is just every tool. I mean, this was one of my complaints with Grafana was like Loki or whatever. Gerhard, the stuff that you wrote inside of there was like learning a new language. And I liked them because you could save them as dashboards and I could look at them, but I was like, I ain't never gonna

write one of these from scratch because I don't have time for that. But if I can go in and tell Grafana what I want, and it can put together the actual query, which GPTs are very good at. I mean, I haven't written a SQL query in months. I've edited some, right? You tell it what you want, it writes the query,

and you edit it to work correctly. And so anyways, it's cool that we're starting to see this stuff get integrated. So it's there inside of Honeycomb. It's limited, 25 a day or something. And it aired out a couple times. They did quote it as useful in terms of saying they don't know what useful is for you.

So they put the word useful in quotes. They hope it as useful in terms of saying they don't know what useful is for you. So they put the word useful in quotes. They hope it's useful. Right. Well, they said that Query Assistant will produce, in quotes, useful queries. Right. And they said it's a loose term because they're not sure. It's impossible for them to know up front what the perfect query is for you. So I guess airing out is one thing, but a bad query or something that's not useful.

Yeah, and that could have been a one-off. i literally have used it a couple times it aired one time no big deal i'm looking at this though it's pretty cool they you know they have a gif at the top and it's like users with the highest shopping cart totals results and slowest db queries results because that's what you you almost always know what you want to see, right? Or at least you have an idea of where you'd like to start. I need to know this, but it's hard to get from there to the query. It's not like hard, hard, but it's like speed bumps. Sometimes that pain too, as a user, will make you not even use the tool at all. Totally.

You know, that's what I love about the way that generative AI is, I wouldn't even say disrupting. I would just say like, man, fine tuning the useful experience of something because Honeycomb is so powerful, but only if you have certain keys to unlock doors. And this just lets you bypass the keys because they're just like, hey, go find me keys. It's kind of cool.

Sentry is doing something similar. I've been inside the Sentry dashboard, by the way. Those are both sponsors of ours. This is not sponsored conversation by any means. This just happens to be the tools that we're using. And Sentry has this thing where they're like, do you want us to tell you what this might be the problem? Obviously, that's not how it's worded inside their dashboard.

That's a terrible copy. Rewrite that. I know. I'm not giving it credit because I don't have the dashboard open. But it's like, click on this button, and we will try to determine not just what this error is but it's kind of like let me Google that for you but on steroids.

I just like the idea that all these developer tools are just getting these upgrades that make them more usable at clips that are really fast and super useful. Side combo. But I have what I need with Honeycomb. That was pull request 456 by the way if you want to go and check it out super useful. So anyways. Yeah, I concur. Side combo. But I have what I need with Honeycomb. Yeah.

That was pull request 456, by the way, if you want to go and check it out, how integrated you can see the code changes. It wasn't that many, but you can see how we added, how we were sending app traces to Honeycomb. I want to give a shout out to AJ Foster. Shout out.

And yeah, shout out to AJ Foster. He's AJ-foster on GitHub. He had a good blog post about how to do that integration, which, by the way, was inspired by Dave Lucia's blog post. And that's davidog187 on GitHub. So shout out to davidog187. Thank you, Dave. Between the two of you, that was super easy.

And I added links in that pull request, 456. You can check it out. Nice. So if anyone wants to do this integration again, it was super easy following these two blog posts. Super easy. Barely any convenience. Yep. Just took an hour. That was it. And most of it was

like figuring out where to get a credential, how to get it right, things like that. Right. Awesome. Thanks, guys. Just wiring. I guess I could have asked ChatGPT for that. That's cutoff date. It depends when they wrote it, you know? Yep.

Although the new version is getting browsing, which Bard has. So I've done a little bit of Bard versus ChatGPT. Just literally copy paste the same command. I don't know if you guys have tried Bard yet. Nope, still not yet. And Bard has access. I mean, Google has answered to a certain extent. I still think GPT-4 specifically is better

than BARD because BARD by default, A, no cost. So I am paying for the plus to get access to the faster features. 20 bucks a month on OpenAI. BARD is free to use. You have to be signed in, obviously. But it has default access to the internet.

So you can paste in a URL and say, summarize this for me, or whatever you want to say, and it will go curl that sucker and spit it into itself. So that's super useful for documentation purposes. And the new, I mean, OpenAI will be there briefly, like they have with browsing. But the same exact command, all coding stuff. I don't care as much about the other stuff,

but just the comparisons. And Bard has been 100% inaccurate on Elixir code. It's completely failed on every attempt. There's a lot of improvements that have to ship. There is. That's what I'm thinking. Yeah, Kaizen. So they need a Kaizen, that sucker.

But it's nice for other, I mean, it's cool to see the different responses. They both need to get better. But I just thought that was really funny because the best thing is that I don't have in RAM storage Elixir, certain functions. I use it enough and I know like, okay, the enum module has these functions,

but then like other ones are in list, other ones are in map. What's the function signature? And I'm so insecure about my knowledge that every time and chat GPT is done as well, that Bard was wrong. I 100% believed it. I'm like, oh, cool. I love this. And I went and tried it. I'm like, no, you can't do that. Oh my gosh. But I would love it if you could. So I think it's very optimistic. Yes.

Well, if you look in changelog discussion 452, which is again the ones for this episode. Yes. I asked chat GPT, if we deployed 120 times between March 8th and May 20th. Yes. How many deploys per day did we do? And it was accurate.

It could figure out how many days there are between March 8th and May 20th. It was so nice. The answer is 1.62. There's a screenshot. 1.62. That's how many deploys per day we did. Really? And most of those were Jira's changes. Yeah. Obviously not every commit results in a deploy because you batch them. So it's not quite that many, but one per day, I think we had one per day on average. Right. Well, there's definitely some heavy lifting there. So also, I am not a pull request person, as you can tell.

You keep referring to your PRs by number. You probably have little tattoos with PR numbers on them. Nope. This was the time that I, my first commit to Dagger PR number. No, just messing with you. But I'm just like a trunk developer. I'm just like right there. I don't, I just want it to go out.

Me too. I think this context is perfect. The change log, right? That's when we developed it from the beginning. Every commit to main, master, will go into production. That's it. Right. So yeah, that's why I just keep doing that.

I don't have the patience. The only reason why it's pull request is because of these Kaizens, right? Like what do we reference? Oh, you know that commit and there's that other commit and there's that third commit if you put them together that's how we did it it's a lot more difficult to talk about them right well that's why we're talking about your work more than mine because i don't remember any of the individual things that i did exactly and by the way jared did a lot more work than i did

just to make it clear okay i just have full requests to prove it. He has lots of commits. That's true. You got receipts. Fair enough. All your stuff is cooler too. The one thing with this Chad GPT question is you didn't reference the year. Does it matter? It shouldn't. With a year, I guess not like May 8th through, or sorry, March 8th through May 20th will always be the same no matter the year, right? As long as it's not a leap. Gerard knows leap years. That's right. I do, yes. He'll leap year baby. Yep, February 29th.

So as long as you don't have to worry about February 29th. Yeah, because March will always have 31 days. And I think May is also a 31-day month. And here's a question. Would it account for that if you went like January to May, ask for the day count? Would it say, well well it depends on the year because of leap years let's try it and see what happens all right follow back up on that yeah

all right in the meantime what else have we done during this time period what else at least is worth talking about well i would like to give a shout out to ken cost for w3c html validation fixes let's pull a request 462. Oh, yes. Thank you for contributing that. That was a good one. Ken would like to get more involved. He hopped into our channel and said,

hey, I'd like to hack on this. And I just honestly don't have a great answer to that. I'm just like, I would love help, but we don't know what we're doing. Pretty much. We're just making this stuff as we go along. Yeah. I mean, when I see a thing, I do a thing,

and I have visions in my head, but I don't write those visions down because a lot of those visions aren't ever going to be worked on. So it's tough having contributions. I love that he found a way of contributing, right? It was awesome. He just went through and fixed all of our HTML to be valid. He has an easy merge.

But I wish we had a better story around contributions. Something I want that he might be able to do is the ability to have Tailwind built into the app alongside our current SaaS pipeline so we can incrementally move away from old design to something that's Tailwind powered. That's a good one. And that requires a bit of infrastructure to do that, right, Jared?

I know you said you could do it. That one's on my hit list. So there's things that I'm just planning on doing that I'm not going to write up for someone else because then I'll have to tell them exactly what I'm thinking. You know, like it just becomes, now they like work for us or something. And I'm like, hmm. It'll be a task. You find your own job.

You make your own job around here if you want a job, okay? We're not making your job for you. That's the hard part, though. It is hard. Our repo is almost like open source, not so much anti-contribution, but we don't know how to tell you to contribute. Right. It's not quite Ben Johnson level where it's like my code, limited contribution.

Welcome, but in certain contexts. It's just we don't have a label out there for, you know, easy changes to make, for example. You know, and certain tasks just kind of like hanging out there. And we never have, and I don't know if we ever will. It's just a hard problem to solve without having a very clear and specified public roadmap of where we're headed with the website. But I mean, we make decisions all the time that of where we're headed with the website. But I mean, we make decisions all the time that change where we're headed. And they could be like 30 minutes before I code them up, you know?

There probably are things that we could have out there, but we just don't. So thank you, Ken, for finding a way. Even this show here. This show was named something else. No, not this show. Changed all news. It was named something else for a bit there. And up to like the legit the 11th hour hour was it renamed back to something that was more meaningful i

suppose to the brand yeah or more aligned at least and then all the you know you had done some schema some visualizations in obsidian with how the feeds would shake out and which one would be the primary feed like right it was until like the 11.5 hour of that change and then we were like okay cool because you and i both had because we don't always agree on things i don't know if everybody knows that but i don't always agree well we partially agree and we're just still sort of in that unclear state it's like well i kind of agree with that yeah we're just not confident yet yeah sometimes we just need like the other person to become confident before we will be you know i'm like well adam's

not sure i'm not sure but once he becomes more sure it's like this starting to feel good or i'll be 100 sure and have to convince him and if i can then he becomes sure and if i can't yeah i mean this is typical business stuff but it makes it hard and i definitely have empathy for people building in public you know and there are teams that have like sass products that are open either open source or open source ish sass products with public roadmaps where they're taking like their users input and stuff and trying to pick a direction and i think that that's just a very, very hard thing to do well. Tell me about it. I actually watched the Homer Simpson car clip the other day,

like the four-minute extract on YouTube of when they asked Homer Simpson to design the car. It's such a good little analogy for the software world. And the best part is at the very end, they go to present the car, you know, and it has all the features. I'm not sure if you guys have seen this. I know you know the meme. Yeah, it's ugly, but it's souped up. I mean, it's got tons of stuff. And they present it to this crowd, you know, because it's like a concept car. And the presenter goes to name how much it'll cost, you know, and the guy just whispers it into

his ear. He's like, it costs what? And it's like $80,000 or something and nobody wants it because they just, you know, just packed it to the brim. That's right. And so not only is it ugly and that nobody would ever buy, but it's also completely too expensive, which software, right? It is half the battle. Getting the price right, the right product, the right price, right availability, the right quality, all the rights. Gerhard, in that example you shared earlier, that visualization of all the dependencies and whatnot, the pipeline being built, you had Apps V2 mentioned. What's not in our list

to talk about, but I'm curious of since you were playing with Apps V2, was the machines versus Apps V2 consideration we talked about the last time we talked. Can you share anything on that front? Has there been experimentation, evaluation? That was the one that I was showing. So we have, we will ignore ChangeLog Social for now. There's like a whole ChangeLog Social side of our infrastructure that we can ignore. We have three things today important for ChangeLog that are running on Apps v2.

Apps v2 is the Fly. io platform that is based on nomad scheduling so there's a couple of limitations with that basically it's the scale that fly reached and that basically meant that various things weren't working quite as well there's like a whole thread on this course where kurt was very transparent about some of the issues they encountered with Apps v1 and what they're doing about that. So Apps v2 is a complete redesign of the scheduling. The way I understand it, it's like their own proprietary scheduling, which does not use Nomad, does not use Kubernetes, does not use any of the things that you may expect. And as a result, it's a lot more robust. It's built for their scale,

for what they need right now. And what it means in practice is for us, it's really, really fast to use and deploy things onto. So it means that you can get very fast VM-like containers. It's using Firecracker VM, again, my understanding behind the scenes. And they spin up very quickly, but they have all the security that you'd expect from a VM. So spin up time is seconds, really, even less depending on what you're doing. But obviously, by the time you do health checks and a bunch of other things, it can take, you know, in our case, some things, 20 seconds, 30 seconds to come up and be healthy and for us to be able to use them. These apps V2, they are pinned to specific machines, right? Hosts, like physical hosts in this case.

What that means for us is that we can't run just one. So right now we have a single instance of changelog running. It's been like that for quite some time. That's why the clustering is important. So we have more than one. But that's okay because we have a CDN in front. We do a bunch of things where even if the app goes down, we are still available, right? And Jared did a couple of improvements, which means like even the posts, for example, now they continue being available when you do certain requests against like news items and whatnot. So our availability really relies on Fastly. And what that means is

that it takes like once every five years sort of event to take us offline, which is again, we discussed, I forget which Kaizen it was, but we talked about that when half the internet was down. Like two or three, I bet. It happened once. Exactly. Only once, right? And that was like in seven, eight years. So I think it's okay. And even then it was fixed fairly quickly. It's kind of like the end of Cable Guy. Have you guys seen the end of Cable Guy where Jim Carrey falls onto the broadcast dish? Mm-hmm. No. And everyone's TVs just go to fuzz.

The moment of truth, Mr. Sweet. Samuel Jacobs, sweet. Has been found. And they suddenly realize there's a whole world outside. And they look out their window, they step outside, their feet touch the grass, and they're like, wow. That's what we did that day when Fastly went down. The whole internet was basically down

because Fastly powers so much of it that we just took a walk in the park. Exactly. Systems were affected that you didn't even know existed. And just texted Gerhard, just like, Gerhard, what are you doing? It's down. Sunbathing. That's what I remember.

That's right. You were sunbathing, weren't you? Exactly. It was like a day off. So it was good. Yeah. TMI. BBC was down and New York Times was down and Guardian was down.

Like, you know, changelog, it doesn't matter. It's okay. All the other big news companies are down. Anyways, the point was, coming back, that with Apps V2, if the host is unavailable, the app cannot be scheduled somewhere else. So we have to have more than one. We can't have just one.

With Nomad, with Apps V2, let's say the host was down, the same app could be scheduled elsewhere because like it was more of a, basically the apps could move around the platform, right? In this case, in AppsV2, they're pinned to physical hosts. So even though they're faster, they're better, they're, you know, like more self-contained,

like when one thing fails, it doesn't affect, there's like basically the, it's a lot more resilient. The platform as a whole is a lot more resilient. but that means that you have to design with this in mind right with blast radiuses and when one thing that goes down like how do you basically work with that with partial unavailability and it makes sense but it means we need clustering so moving something like for example dagger engine it's okay or, right? Because if that's down, that's okay. It can fall back to whatever is local.

And by the way, we do that in our pipeline. If PostgreSQL is down, well, you're already clustering that. You know, we have like, again, some sort of redundancy there. In our case, we had one primary and one replica. And by the way, I think we should look into that as well to see if we can get a proper managed PostgreSQL service. We keep talking about that. Maybe Crunchy, maybe Superbase, try a few and see what sticks. To continue, we can still run on fly for the PostgreSQL instance, even though I think they were mentioning at some point they will want to invest in that. So we'll

see where that goes. Things may have changed. changed again this was on their forum so we can move dagger engine docker we can move postgresql to apps v2 to machines but the app itself it's a bit more problematic not without the clustering part because of what they explained like the limitation with hosts so while it's a modern platform it's very performant again all the tests which i ran you get like very nice CPUs, the AMD Epics, they're yours, especially if you get the performance instances. You get local NVMe's, again, very

fast. But if the host becomes unavailable, it won't get moved. So we have to get the clustering to do Apps V2 properly. Exactly. Alright, Jared. The whole world is on my shoulders. The whole changelog world. Quantify that.

There's a lot of things too we just talked about. You mentioned news items. I know they're still in our infrastructure, Jared, but there's a lot with this change from weekly and news items and this news homepage to this world we're in now, which is sort of obsolete and primed to delete code. There's a lot of places that we could delete from the app code. We just haven't because, A, you want to make sure that you like this new home

that you're living in for a little while. Yes. And you're not going to be like, why did I throw that furniture away? I like it a lot better already. I got my feet up. I'm liking it too. I don't think we're going to go back, but I wasn't going to rush into deleting the code.

I do love to delete code, but my least favorite command is git revert. A lot of stuff that we built, even the whole impressions thing and all that, just it's obsolete. We're just not using it anymore. We're not doing news items like we used to.

In fact, my concept, which I guess before we started recording, we were talking about STI, single table inheritance. Gerhard and I were reminiscing on war stories. From the old Rails days, this was a feature, it's probably still in there, where you could do a classical object-oriented inheritance with a single database table

and use a type column to instantiate different classes. You could have a table called pets and classes called dog and cat, which both inherit from pet, but they get stored together in one table. And my concept when I built this site, you make certain decisions that are foundational to an application. And one of them, which served us very well for many years was everything's a news item and every news item points to something so it can be on our website it can be on somebody else's website it can be audio it can be video it can be text it can be a tweet it can be a whatever and we

decorate those differently based on information we have about the news item itself. Now, I didn't actually use single table inheritance to implement this. It's just similar conceptually. And that was great. And it served us very well for probably all the years until right now where we literally are abandoning the news item part of what we do. And we're only publishing blog posts and episodes, right? Audio episodes. And every other news item is just kind of like an appendage

that is still there in the data structures, but only represents things that could just be represented directly, you know, if I hadn't made that decision. So aren't comments also hang on the news items too? They are. Yeah, they're attached to news items. Yeah.

Because that was like the foundational, you know, atomic unit of content in the CMS. Like I said, it was good for us for many years, but as we simplify and change now, where we're just publishing, I write changelog news in Markdown, which I love, but changelog weekly was generated from a list of news items

that we published throughout the week. And that's a foundational change to the way that we do content. This one's simpler and I think more sustainable and hopefully produces better content over time. But all of our infrastructure is built for that other way. And so there's stuff that we can delete. There's also stuff that's just going to be there

because it's like ripping out your circulatory system. Well, when you look at, for example, changelog.com slash podcast, which is where this show is at right now, the list of things there are news items. Right. Everything we fetch is news items. Unless we change the design dramatically to rend this obsolete, this still is pertinent. This is still useful. Oh yeah, we're still using that for sure.

That's why I say we're not going to rip out certain things. But there's areas of the code base that are news item related that could definitely go away. Like in the admin for creating a new issue, which was once ChangeLog Weekly, that can go away completely. We don't use that anymore. Yeah, we're not turning back from that. And we attach the newsletter content.

So ChangeLog News goes out at the exact same time, both the episode and the email. And we attach the email content directly to the episode, not to a news item. And so, yeah, issues are gone. News ads can go away. There's a bunch of stuff around scheduling sponsorships. All that stuff was based on the old system. So, yeah, we could definitely do some spring cleaning, probably summer cleaning at this point. But then you're like, well, it's also not hurting anything. And they're sure waiting on me

for this clustering feature. So maybe I'll work on that instead. I love it, man. Digging the changes, digging the Kaizen, digging the commitment to incremental change. Like, you know, it reminds me of the book Atomic Habits. If you can make a 1%, a half a percent change today, and you do that for three or four days straight, well, what did you do? You made a almost 5% change. And you stress out about, I can't do all of it today. That's okay. Just do what you can today. Just enough to move the needle forward. Whatever it takes to progress, do that thing. We've been doing that for many many years now collectively we've embodied that with kaizen on ship it now we've brought it here to change like our friends

with our oldest friend gerhardt with a sheer commitment to chasing the nines and the zeros the best pipeline ever like tm that at the end right trademark that thing the best pipeline ever it is world class, honestly. Gosh. No one has a better one. Nobody. No one has a better pipeline. You hear that out there? Show us your pipeline. It's better than ours. If you check out the code to see like a pipeline as complicated as ours in

code and how it all combines with apps dependencies, with the versioning, with how the app gets compiled, we even have, and again, I haven't finished this, is there Erlang releases. Let's get that sucker done. And it makes their releases.

It's not there, again. And no one has thought of this, like, hey, Gerhard, can you please get to those Erlang releases? Or hey, Gerhard, can you please upgrade PostgreSQL? These things happen organically. Because you kind of get in the middle of it, and you kind of understand what the system needs. It's like a living, breathing thing.

And if you have half an hour, and talked about this like way back when what is the most important thing i can do right now that will make this thing slightly better an hour tomorrow whatever the case i have a weekend two hours okay so what can i do in those two hours and all those things add up and two months later it's amazing how much has changed and that's why we're trying to capture these changes in the kaizen because again if you look and all those things add up. And two months later, it's amazing how much has changed. And that's why we're trying to capture these changes in the Kaizen. Because again, if you look at it every day, it's not a lot there. And same thing with news.

I've seen like, there's like so many things that you had to do, Jared, like to get those news. It was way too much work. I don't know what we were thinking. It was worth it. We had great ambitions, you know? It's always good to have good ambitions,

but it's okay to admit your fail, right? You know, we just didn't quit soon enough, Jared. We you're going to do. And so yeah, stopping and starting and change making changes. These are hard decisions to make. And especially when they require heavy lifting, you know, and you're experimenting, we didn't know if people would like changelog news. So we started shipping it inside of our current infrastructure. You know, if we had done all that work, and people were like this show sucks why are you guys made please stop putting this into my changelog feed then it would have been a bunch of wasted effort so i feel like we went in the right order at least and we're here now there's a lot of things that we can improve and we are dedicated to consistency on kaizen every two months so we

won't have an episode number quite as sweet as every 10th like Gerhard had on Ship It, but we do want to do every two months. And we do have this discussions thread. So Gerhard, lay this out for folks. You can participate in our Kaizens. You can give us ideas of things you want us to try, things you want us to talk about on these episodes. You can obviously hop into the code and find your way where it makes sense

and how you can help out, take it as shout outs. We even have our most popular ChangeLog t-shirt is the Kaizen t-shirt. I wore it at Open Source Summit and got some compliments. So maybe you can get yourself a Kaizen shirt by getting involved. But each Kaizen episode has a discussion that we use on github.com slash the changelog slash changelog.com

and hop in there, tell us what we should do for Kaizen 11 we will be in there discussing what we're going to work on for Kaizen 11 and what else, anything else left to say here? I would like to give a shout out to Jason Bosco, he's the one that participated in Kaizen 10 with TypeSense. Thank you, Jason. Our partners, TypeSense, they are powering our search. And we had some questions around how it worked.

Sounds like it's working as advertised, at least at the basic integration. The key answer, direct matches will beat fuzzy matches every time, I guess. Makes sense. It's just unfortunate because LLMs versus LLM was the search term for using. And you tend to search for one, not the other, even though you tend to say the other one because there's many of them. So yeah, thanks, Jason, for hopping in there

and explaining that to us. It's a great example. And we can certainly delete the Algolia code. So if someone wants to help with that, it's right there. Well, I already deleted the account and everything. So we have no account there. So if the code's still there, then... I thought the Algolia code was all gone. What's left? Was it? I don't know. I mean,

that's what it says. April 9th. Oh yeah. I ripped out some Algolia stuff. Did you? Okay. So that's done. Cool. Yeah. See, no pull requests. I don't know. So it's a commit. 150. Good luck finding it. I'm a follower of that gardening principle, you know, like whatever area of the code base you're in, you do the same thing with upgrades and stuff. Like while I'm working on this, if I see something, I'm like, oh yeah, we can get rid of that. I just do it. Boy scout rule.

I don't have the PR, but I'm just like constantly improving any area of the code base that I'm touching. And once it became clear that we were done with Algolia and I was just futzing with the search stuff, I deleted it. The cool thing I want to plug real quick because Jason did so cool and Kaizen 10's mentioned there. If you're not familiar with TypeSense,

the cool thing about TypeSense really is it's super fast in-memory search. So if you can hold a terabyte of your data in memory, Jason can speak better than I can. I'm speaking out of turn really, but super fast and it's in memory, which is just the best. So you should check it out if you don't know about it, typesense.org. speaking on a turn really, but super fast and it's in memory, which is just the best.

So you should check it out if you don't know about it, typesense.org. Well, what's next? Where should we go from here? This is the first official changelog and friends, which is a brand new flavor of this show. And Jared, I'm excited about this because we've got lots of friends and we don't talk to them frequent enough. And every two months we're coming back here with you at least Gerhard so maybe sooner I know you have some home lab material coming up we want to talk about uh unify ubiquity networking maybe some

vlans I'm trying to talk Jared into vlans he's he's like I don't want to do that should I have vlans Gerhard uh you should especially for your kids you should have a kid's network yeah i found it the hard way that's exactly what adam's telling me i think that needs a drink that story yeah have a separate wi-fi for kids that's on the kids oh yeah vlan okay and iot yeah you want like an iot network for sure yeah okay sell me you want two networks yes always two two routers two networks two isps oh yeah you know how hard it was for me to get one isp out here in the in the boondocks getting two is not easy i'm sure it was yeah no starlink uh i could have starlink yeah i suppose but uh that's expensive right 500 bucks down and then $100 a month?

It's like $700-ish in just hardware alone. Then it's about $100 a month, I want to say, for decent speeds. As a backup, can I turn it off and turn it on? I don't think so. Doubt it. That's too much for a backup. My main line's costing $115.

That would be cool, actually, if they can have a plan that's just for backup service. Because, I mean, you already own the hardware. Right. How hard is it to just metered use, right? Failover. I don't know if your Dream Machine Pro supports it, but I don't think that has WAM failover. Does it have WAM failover?

Yeah, the Dream Machine Pro has it, yeah. It does have WAM failover. I know my USD Pro did. And I say did because I used to own that and I sold it like a fool. I want that USD Pro back so bad. It's so cool. The UDM Pro is pretty cool too. Unify has, is it Unify as the company or is that the device?

I always forget. Ubiquity is the company. Unify is the product level. Oh, Ubiquity is the company. Ubiquity is the company. They do live at UI.com though, which is like. Yeah, UI.com is a sweet domain, right? Phenomenal domain.

So they have a product that I really want. So, you know, I have a small acreage. I got eight and a half acres. Do you really want that still? Of course. You do, okay. Why wouldn't I? Wasn't sure if this was a long-term thing.

Oh, I still want it, so... What do you want? Wi-Fi Base Station XG. It's an outdoor basically dish that just broadcasts your wi-fi for hundreds of miles no uh 1500 plus 5000 square foot coverage so one of the things that i do around my land is that i walk and i listen to our shows for qa for clipping for stuff like that and sometimes i'm uploading stuff i'm basically working and walking, which is a spectacular aspect of what we do. I love that part of the job.

I get down to the orchard, and I don't have Wi-Fi. I'm on cellular. I want blanket coverage. I want Wi-Fi all over my land. And this sucker will get me there. The Wi-Fi base station. The problem with it, A, it's sold out. B, it's $1,500.

But it is powered over PoE. And I can run it up to my roof. I think you'd probably go with the AC mesh professional, though. That might do a similar thing. All right. Is that one cheaper? Check this out. Oh, gosh.

Check the wireless systems that Microtech has. Microtech? Garrett is sharing links with us. I have both. Of course you do. You have two? Of course, yeah. Ubiquiti and Microtech has. Microtech? Gerard is sharing links with us. I have both. Of course you do. You have two? Of course, yeah. Ubiquiti and Microtech.

So I have UDM Pro. I basically have like two networks, one inside the other and one alongside the other. So they are like interweaved. So I have like two 10G trunks, basically, 10 gigabit trunks. And I really like the Microtechs.

I think they are underappreciated. You need to be a bit of a hacker to get into them. But once you set it up, you have terminal access, you have nice auto-completion on the shell, you can run containers if you're crazy enough, but you can run containers natively on the router if you want to. They have container support. It's a proper operating system.

And if you use Ubiquiti, it's great. It's almost like a Mac. Microtech is almost like Linux. So that's the difference. Oh, okay. But however, some of their latest hardware is just so good, so, so good. The UI will not be as polished,

but do you need that? Or do you want the stability of, just think Linux for your network. You can do 25 gigabits, you can do SFP, you can do like all sorts of crazy things. If there's good docs and there's an LLM to power it, I would totally configure it like NetApp apply. I configure my static stuff on Linux every day. No GUIs here, you know, so cool with that. Their YouTube channel lately, it's just so good. If you look like the YouTube MicroTik channel, they have so much good content.

Check it out. Yeah, definitely have to check it out. That's a friend recommendation. Okay. Well, that'll just be a tease then. So what I was really trying to do is just tease what could be the future of this show. Like, you know, having friends like you

to come up and talk with us about things. And I'm a nerd and I'll nerd out with Jared and he's maybe slightly less of a nerd than I am on networking and IOT and home labby stuff but way more nerdier than I am in back-end coding and databases and stuff it feels nice to be in a place where I'm the least nerdy person you know it's usually I'm the most nerdy of all people in the room but not in this room I like this room not in this room. I like this room. Not in this room. Maybe we could talk about MicroTik and Unify and the Mac versus Linux of networking. That'd be kind of cool. Coming to a change

looking friends near you in the future. Okay, cool. We do have a little list here at the very end of this discussion, which says what else is coming up? Can you blaze through that in one minute Gerhard so we can close out this show and tease the next Kaizen so I really think we should improve our integration with one password and secrets in general right because right now it's copy pasta in our in our github actions and that's not good like now we have the code we can connect programmatically to wherever we store our secrets in a nice way and get them just

in time and keep them secret throughout right like no copy pasting none of that stuff um we switched off pingdom and now we have a honeycomb just honeycomb so we need to configure the sls just have a better understanding of what's happening there i have uptime kuma running locally and i think we should have an instance on fly io as well at least like a few you know just like deploy them see what's happening from the outside world there's a bunch of upgrades like postgreSQL that has to be upgraded apps v2 like roll everything that we can to apps v2 follow up with the fastly guys just like to share our VCL madness and our UI experience and just figure out what exactly we're doing wrong when it comes to Fastly. On my list, part of that

is to look into Cloudflare. Like what did Cloudflare add to logs and how easy it is to integrate with Honeycomb? Because if they have a good logging story maybe we should check them out R2 has no egress fees currently we're storing stuff on AWS S3 if we went to Cloudflare R2 if we did like a couple of things have like some amazing stuff like Cloudflare a bunch of things they are really cool company in terms of the technology which they develop and as you know I like running two of everything so why not two CDNs I know it's crazy but you know many things were crazy to begin with and they turned out to be great ideas I'm one step ahead of you on the R2 thing I have an open tab I have an account that I've created for us over there I begin testing the waters one thing that I immediately don't like about it is that it

doesn't support transmit like it's S3 compatible API, but not enough that you can use your S3 client Transmit, my S3 client Transmit. And that just hurts my heart. It's not going to be a blocker, but it hurts my heart. So it's going to be an R2 CLI though, right?

Where you could do the same thing that Transmit gives us with the CLI? Oh, there's definitely ways you can do it. You can use Cyberduck if you want to. Just like years and years of using Transmit. Technically, Transmit doesn't support it, is probably the way that they would say it. But they say they're S3 API compatible,

and Transmit is an S3 client, but it does not work. Their S3 does not work with R2, so it's not a one-for-one. It has something to do with streaming data versus chunked uploads, or I don't know. I didn't read the description. But it's like a known thing that Transmit is like, yeah, we might add support, but they haven't yet. So anyways, that just bugged me

because I just like to drag and drop stuff in there with the same SFTP client I've used forever. But what really needs to happen is can we simply change all the environment variables in our code and will the XAWS work seamlessly? Will all of our stuff work seamlessly with R2 to upload to R2? And if that works, then we're pretty much

good to go. I also need to go through and change a bunch of our hard-coded URLs in our text areas from the S3 URLs over to cdn.chainsaw.com, which we should have been using the whole time. I just didn't even think about that because then we could seamlessly switch the back end without changing anything.

So that's like a migration step that'll have to happen. But anyways, I'm down the road a little bit on that because our fees at AWS are going up and we would love to have zero egress fees. I was going to say, if we need some motivation, in March our bill was $50, $53.69 actually. And here in May 2nd, and June's coming here soon, May 2nd it's $132.32.

So we've almost tripled our bill in three months. Yeah, it's on track for like $450 this month. Is it really? I thought so, inside the Cost Explorer. $450 this month? You said May 2nd, it was $150 something. I have to check this out. Remember we did this last time when we set up shielding?

That was exactly like during a Kaizen. So I have to check this out to see what exactly is going on here. I used the cost explorer to try to figure it out and I just saw it was S3. But I didn't actually drill down on what changed and when. So if you want to do a little bit of research on that, you're hard to be good. But then I just started thinking, well now is the time, R2. Let's just eliminate this problem for us. So that's on my hit list also on my hit list obviously

is this clustering stuff with the caching stuff changelog news improvements will continue and then my big not for the next Kaizen but my very big desire this year this calendar year I'm not going to guarantee it but I'm going to shoot for it is I want to migrate off supercast

and I want to provide a first-party changelog++ offering that solves a few of the pain points for our++ members. Specifically, one feed to rule them all is just not good for people. They don't want one feed. They want their++ feed, but not the master feed, which it currently is. And so we can provide that if we're off Supercast. And so that's just a thing I'm putting out there

as a big item for me here soon. That would be good, because they want it. You've got to give them what they want. They do. It's like 50% of the people who sign up for Plus Plus say, hey, can I have just JS Party or just the changelog in JS Party? I don't want the other shows. And I always had to say, nope, you've got to have all of them.

And that just sucks to say because I don't understand it. Unfortunately, Supercast doesn't have that functionality inside there. We can only give them one feed. Whereas once we're on our own platform, obviously everybody gets their own feed and you can customize it and have multiples or whatever you want to do. So that'll be super nice. We've proved it enough, though,

that it makes sense to bring inside. But also it requires some of this caching stuff to be taken care of. It does. It's just there's so many little things with subscription services and Stripe integration and blah, blah, blah, blah, where you start to realize all of the emails

you're going to have to send. That is true, at least a few mean there's probably five or six different emails just to manage a subscription plus a dashboard where you can go and change things cancellations this person's card expired refunds refunds yeah stripe handles the refunds part but yeah you still got to trigger it though you got to get get that UI to do something or other. You got to just manage the fact that either you're doing it as an admin or they're doing it on,

they're requesting something. Then they're like, where's support? We don't have support. This is just. You're looking at him. Yeah, this is support right here. Yeah.

You're figuring out why our bill went from basically 30 bucks to 130 bucks. As soon as I get permissions to billing, I'll have a look into that. Back on you, Adam. Oh my gosh, okay. One more thing. This is important.

How can we run multiple instances of the change log with the same data without the second instance behaving as the live one, without sending any emails, without anything like that? Because when I'll be setting up the new 2023 infrastructure and that is right next on my list that starts with postgresql with the app with whatever we can migrate i do like a long blue green and that means getting another production and having like a nice easy way of basically migrating data migrating everything

so it's like very easy to switch back and forth if we need to. But always my concern has been when you have a second production, it shouldn't be sending any emails or anything like that. Right. I think that's a follow-up for us. Obviously, we don't have to solve it now, but that's something on my mind. We're getting long in the tooth. Let's call this a Kaizen.

Let's call this a ChangeLoginFriends. I'm very happy to have been here. Thank you very much for having me over. And I'm looking forward to the next one. Thank you very much for having me over, and I'm looking forward to the next one. Thank you. Thanks for hanging with us. Kaizen. Always.

Kaizen. Kaizen always. If you're digging changelog in friends so far, tell your friends. And if you have feedback for us, let us know in the comments. Link is in your show notes. Our intro episode sparked a great discussion.

Wow, so many shower listeners. I knew I was right, but I didn't know just how right I was. Thanks once again to our partners for supporting our work, Fastly.com, Fly.io, and Typesense.org. And to Breakmaster Cylinder for this epic new outro track. It's called Link-O- Linkarama and you'll only hear it right here on ChangeBlog. Monday is Apple's big WWDC keynote and they're expected

to unveil the Reality Pro and XROS. Watch along with us in the Apple Nerds channel of our community Slack and tune in next Friday. We'll be discussing all the interesting bits with homebrew lead maintainer Mike McQuaid. That's all for now but let's talk again real soon.