LINUX Unplugged - 493: Network Nirvana

Episode Date: January 16, 2023

Chris' sticky upgrade situation, and we chat with the developer behind an impressive mesh VPN with new tricks. Special Guest: Ryan Huber.

Brent, correct me if I'm wrong, but I don't think we got a single email about scale last week. Did you see anything about that? No. And I was expecting quite a, you know, at least one. But we didn't even get one, which leads me to question some things. But did we get any boosts? No boosts. No boosts about scale.

Did you see anything about scale? No. No. Now that you mention it. No. Hmm. Do we want to go to scale? I mean, we're going, so I declared last week on the app, but we're going.

But then, like, nothing. Radio silence. I mean, I think maybe a couple of things in Matrix, but otherwise, total radio silence. And I wonder, when do we go and when do we not go? Do we go just because it's a Linux event and so therefore we should go? I'm not sure what to do. Yeah, we probably need a draw, huh? I mean, speakers we want to see, community events happening, or perhaps a critical mass of audience members who want to hang out.

That could be one, too. Yeah, because I think, you know, now that we're independent, we can't just really go to anything. So it's like we really have to be careful about travel because that's very expensive cost. And the production while we're on the road is really expensive. And it, uh, well,

people have to plan around it, right? Like it's a, there's a cost to just organizing a whole thing in general. We want to make sure those count too. But then at the same time, it feels like, I don't know,

like how could we not go to something on the West Coast? So I'm really torn. And Brent's never been. It's true. It's embarrassingly true. I want to, I definitely, regardless of what we do, feel like we should do something here at the studio before scale. Because especially it's right around 500. That's just like, we got to do something.

But I guess, I don't know. It's something about the travel and everything around it. I don't want to do it unless about the travel and everything around it. I don't want to do it unless I get a strong signal that it's worth doing. And I don't want to have to like turn to a sponsor every single time we want to do something like this in order to just pay for it. I'd like to be able to pay for it ourselves or through like audience contribution. And I don't really feel like if there's no excitement around it, that's not going to happen. And maybe it's just easier just to stay home. Or maybe I'm getting old. Hello, friend, and welcome back to your weekly Linux talk show.

My name is Chris. My name is Wes. And my name is Brent. Hello, gentlemen. Well, coming up on the show today, we're going to take a deep dive into some really innovative open source networking tooling that you're going to love. And I will report back on my first major NixOS upgrade. Not one, but two machines had to be upgraded. And yes, NixOS does have releases.

And yes, you do have to take steps to upgrade. I'll tell you about where things went a little sideways. And then Wes and I came up with a really simple solution to do HTTPS for my next cloud, my jellyfins and all of that. So we'll tell you about that solution, which I feel like is my go-to solution for years. I'm really, really excited about what we got set up after last week's episode, so we'll tell you all about that. We have a great guest joining us, and then we'll round out the show with some great boosts,

some picks, and a lot more. So before I go any further, let's say good morning to our friends over at Tailscale, all of us together. Let's head over to Tailscale.com, a mesh VPN protected by WireGuard's noise protocol. Go say good morning and try it out for yourself up to 20 devices at Talescale.com. And of course, a big good morning to our virtual lug.

Hello, Mumble Room. Hello. Hello. Hello. Hey, Brian. Hello, Brian. Hello. Mumbling today, a strong and clear signal.

Absolutely. Hello, everybody in there. We also have a dozen just up in the quiet listening area of the Mumbled Room just getting the low latency Opus stream. What? Nice to have you there. So, yeah, this is the part of the show where I talk about us going to scale 20x and encourage you. I still encourage you to go March 9th through the 12th, even if we're not going to be there in the Pasadena Convention Center. I'm excited that scale is happening, regardless of if we end up making it.

I think that's why I wanted to go. Right. I just want to support the fact that it's happening. Keep happening. Let's get this energy back up. And great crew down in Pasadena. Really great crew down there. But love them.

Just saw them. I've seen some of the people in Pasadena more recently than I've seen some of my family members. I don't know if that says more about our audience or about your family, but I'm not going to get into that. So it's up in the air. I think we have to,

we have to kind of figure out if you want us to go, if you want to tell us that story, or if you just, maybe we shouldn't bother. I'm not, I guess I'm asking for feedback. This is like the most

unrefined kind of discussion point in the show in the last few years, but we'll go with it. Yeah. And please hit us up this is like the most unrefined kind of discussion point in the show in the last few years but we'll go with it yeah please hit us up at linuxunplugged.com contact for all of that excitement yeah let us know what you think i want to talk about something great that did happen before we get into the whole show today because we do have a lot to get into but um we've talked about nixos over the summer and i wanted to just do a few month follow-up now that it's been running in production for a while and i had to do my first nix os upgrade i was using i think it was 2105 and now nix 2211 has been released and nix is kind of a rolling os kind of because you can

actually mix and match unstable and stable on one system. But you can think of Nick's rolling as the unstable release, and Nick's stable as the official release. And so when they release a version, you do actually have to take some action to upgrade. Otherwise, you will just stay on the 2105 release. And a couple of my systems work. Right. You'd set them up that way.

Happily been using the packages ever since. Mmhmm jupiter tube not jupiter tube up on lit node uh is on the new release 2022.11 i went in to go after i'd figured all this out because at first i didn't figure it all out but when i after i'd figured it all out i was like well we're good and let's go apply this new skill set so i'm like all excited and i ssh into jter.tube and I go to do the upgrade and it's like five packages. I'm like, hmm, that doesn't seem right. So I go to look at the NixOS configuration. I'm like, oh, dang it, it's already up to date. Nothing to do.

So there is actually a few things you have to do. And it's pretty straightforward. But it's like all things with nixos i find the language around this confusing and it assumes a lot of uh well like you understand the nixos configuration language inside and out it assumes you have the context for where this should go in the configuration and all of those things but once you get through that for me it's an uphill battle but it's one i'm willing to do because it feels a lot like when I was first learning Linux, you know, when I didn't really understand what an RPM was or a dev was. Oh gosh, yeah, no kidding. Wait until you heard

about Dbus. I still don't really fully appreciate and understand Dbus, dude. I really don't. But it really was one of those like, kind of like, all right, I'm willing to do this. I feel like it's worth it. There's going to be a payoff at the end of this process. That's how I feel about Nix. And so I learned that I had to change my Nix channel and I had to point it at the new release. And then I learned I had to go on my configuration file and tell it to look at the new release. And then after I'd done that and updated everything, it was just a matter of telling it to rebuild and switch. It was actually really, really simple. So I went out and did it on a few other boxes and ultimately upgraded like three different systems to the new release. And all of them went really smooth, really smooth, including my Odroid.

That's nice. That's kind of a big test. I mean, it wasn't that long ago over the summer, I suppose, that you kind of radically swapped out all the all the pies, redid stuff on the Odroid. And after centralizing a bunch of stuff, I i mean it's kind of a big test for this new build out this was one of my first big upgrades since doing all that what would you i mean what i'm trying to get to wes is like i can tell i am still really ignorant to the way all of this works and just figuring out the language is still where i'm learning like what would you call that what what stage is that that's it's super like at the beginning. And that's why I had to lean on you

for the NGINX stuff later that we're going to talk about. Like, how would you describe that stage? Is it? Well, I mean, you kind of have to rebuild your understanding of how you put a system together, right? Like, you know how to do it if you were just going about and updating,

you know, the Ubuntu release or to the next Fedora. Oh, yeah. Yeah. And you have to, you know, the Ubuntu release or to the next Fedora. Oh, yeah. Yeah. And you have to, you know, find your new footing with all the different mechanisms, the way things are set up. And with Nix, because it's like starting from the build system on up, there's a lot of assumptions that just no longer hang true anymore.

Even though it's wrapped in a language, you know, some of the terms are the same or look the same at the surface level. So you kind of get like, oh, yeah, well, it's a package manager. You know, of the terms are the same or look the same at the surface level so you kind of get like oh yeah well it's a package manager you know i'm installing stuff and i'm doing upgrades but it's it's a functional system so uh it's really more like various pointers and hashes getting updated and reported yeah um and the end result though is that man man is it great to be able to just get everything totally current run on the latest kernel the latest packages and everything just keeps on ticking. And if there is a bug, if there is a problem, it catches it before it does the upgrade. And this happened on one of my systems.

I had Sparrow Wallet installed. It was depending on a version of OpenJDK that Nix is considered insecure, and it bugged out before it completed the upgrade. And for me, it was like, well, I can just remove that and install that later. So I just took that out of the config, reran the upgrade, and it went flawlessly. Yeah, and you've kind of got that double system, at least two, right? You can do the build and you can be pretty sure that at least most of the stuff, in terms of getting all your config stuff, getting the packages available, that's going to catch stuff there.

And then you can just switch right to it, or you can do a test and kind of see if you can start that new system without switching the main system to it and make sure all the services come up cleanly. I love that. That's like an insurance policy when you're working with Linux. And then of course, even if you do switch, I mean, you can just roll it back in the bootloader. It's wild. All right. So I made you use Nano last week for a little bit. For a little bit until I just made a little Nick shell to install the oven. Yeah, you just pull that right in. That is one of the nice things.

And I don't have to worry, right? I'm not like installing it permanently on your system. It's just there while I need it. Do you see any reason we can't share this config? No, no, no. I took out like the secret stuff. So we will link to the config that we came up with in the notes. But you and I were looking at ways to make SSL termination easy when you're running multiple services.

You know, like so many things want SSL. Like I'm running NextCloud, and I want to use a Node app, and I want to use a phone tracking app that tracks your location. All of them require that you have HTTPS to your NextCloud server. Right. The context here is you're doing everything over tail scale these days. You've got this mesh network set up. So you already have a layer of encryption. So for your personal needs, you don't necessarily need TLS involved in every single connection

because it's going over a wire card already. Yeah. But a lot of these third-party apps don't really realize that. And rightfully so, I think, at least as a default, in the interest of protecting the end user, want to have some encryption in place before they start syncing data. Yeah, they won't even accept an HTTP URL. They won't even accept it.

It's got to be HTTPS, which I get it. Yeah, it's fine. Then I also have Jellyfin on here and I want to be able to access that externally and that gets a lot harder if you don't have TLS. And I have a few other services on my box

and pretty soon you discover to do this, you need some kind of proxy or some kind of intermediary where the SSL will terminate. And then it will forward the request to your internal systems, your internal services. You and I looked at Caddy, which looked like it may be the simpler way to go. We, of course, seriously considered traffic. Yeah, another great option. And that's the thing, right? These days, there's so many different ways to, A, just get the certificates. In general, there's a ton of great, like, Let's Encrypt or, you know, Acme clients out there, and then a ton of the web servers that exist now, they either have modules for or

just baked right in the ability to get those certs and auto-configure them for you. Yeah. And so, like, you know, we were tempted to go with Caddy, to be honest with you. The simplicity appealed to us. We were tempted to go with Traffic, to be honest with you, because the integration and so many projects just kind of work assuming you're using Traffic appealed to us.

It's got real nice, obviously, it's got that Docker socket integration. There's a lot of things to like about Traffic, for sure. However, there is a beautiful simplicity to good old Nginx. Good old Nginx. And on NixOS, you can set it up so stupid simple that I don't really understand if you'd ever do anything. I don't know why you would. If you're just running your own little personal server, I don't know. I just can't fathom you'd want to do anything else.

It was so great. We'll put the configuration in the show notes, but you essentially tell NixOS that you want to set up one of these Acme certs and where it should stick that. You tell it what DNS provider

you're using. You give it the API or credentials necessary to access that DNS provider. You define a virtual host or two, like in my case, NextCloud. And you're done. And then NixOS will maintain this in perpetuity. Like if I were to wipe this machine away and reload with this config, it would set up all of my TLS, all of my HTTPS stuff all over again,

all of my proxy stuff, all of my forwarding, all of the tls termination right there all over again it's really really nice so wes tell me a little bit about why we went this way uh well so one for one factor because you're using tail scale here uh you're not really exposing any of this publicly so you didn't want to use the automatic default like uh you know a lot of these web servers they'll just handle the sort of oh i'll i'll manage since I'm already exposing port 80 and 443 for you. I'll automatically handle the little check, you know, and approve to Let's Encrypt or whoever's providing your certificates that you actually control and own that domain. Now, you can also use a DNS challenge here. And we looked at that and, you know, Caddy's got that, but it looks like it's in a sort of

third-party model module that you kind of have to load in, and that was slightly more complicated with Nix, and it was already a Nix system. Certainly, traffic can do that as well, but looking at the NixOS manual, they've got integrated with a client called Lego, which is just a little Go Acme client. This is already

baked in there, and it's really no more complicated than the regular setup, and so you can just specify right in your NixOS config here are the domains that i want please go get these certs whether they're you know wildcards or regular certs uh you can tell it where to go get all the credential files for logging into your tns provider of choice and cloudflare was baked right in so that was super simple and then it'll stash these all in like a regular place on your file system and then nixos also has mechanisms for referring to that so you can say like oh yeah yeah i already set that up for you

service that i'm using a nixos config go look in the usual place to go find the certs for this domain and the configuration is so simple and so straightforward that i can just link to it and share it with anybody and it to me what i love about it is I can read it as a simpleton and completely understand what it's doing. And I can reproduce it and I can even understand how to add additional services to that in the future. Yeah, I think the other part is it kind of abstracts things a little bit. Obviously, there's lots of tools to automate setting up Nginx and it's not that hard, but the Nginx config can be a little verbose these days compared to some of the competitors. There's a lot of options that you might not really care about, but it might be

in your interest to set. But wrapping it in the Nix language, A, it sort of summarizes things a lot. They've got recommended settings you can apply, like, oh, here's some optimizations. Here's settings you might want to use if you're doing a proxy. Here's some recommended TLS settings that'll make sure things are, you know, up to snuff security-wise. And then, it just lifts things up. So instead of having to worry about all the Nginx-specific syntax

and all these other things, you're just using the really comfortable Nixlang, which is quite handy. It's easy to work with. It's a little more, you know, has a little more affordances

than something like JSON. Easy to read. But it's really easy to read. And you just kind of set things out and like, here's the name of the thing. Here's where I want to send my proxy request to. And you're done.

It's like the easiest Nginx config I never wrote. Yeah, exactly. You and I actually did check the output of the Nginx config that it writes. And it does write a very concise, but probably 30 times more verbose config than what we ended up having to produce. And it just is so straightforward, Wes.

It's so reproducible. It's easy to back up that one file and have that entire configuration saved. And now I have a valid SSL cert and all of these apps that require that, even when I'm accessing a tail scale IP, are totally happy now. Yeah, it also made it pretty easy when we wanted to sort of redo the same thing and, you know, just say like, oh, yeah, but on another port using that same certificate, we are also hosting Jellyfin over here. So can you just reuse that, make it all work for us?

And you kind of have to like copy some stuff and be aware of like what duplication you need in the regular Nginx config, but that's handled by the abstraction present in the NixOS module. All right. So tell me about the proxy party. Yeah. So as I was looking for some links to throw in just, you know, various resources we pulled up, this was not what I found last week, but I found proxy party, which is an Nginx redirector and proxier for a group called the Hack Club. And they've got like all, you know all sorts of various old domains, new domains, or just like you've got.com,.eu, that sort of thing.

And so this is like one place to document and deploy all of their redirects. And I thought it was just a nice example of once you've sort of lifted the Nginx config into the Nixlang, well, you can just do normal programming language stuff. And they've written a module so that they just specify, like, here's the redirect I want. And then it goes and, like, converts that into the necessary Nginx config. Jeez, that's awesome.

And then deploys this whole thing. It looks like they're even using Linode, which was a nice bonus find. That's so powerful. Yeah, right? And then they've just got, like, a NixOS setup. It's got Nginx installed. It builds this config just based on this key value pair of, like, here's all the redirects and where we want to go. They've got some, you know,

a couple nice little options like maybe you want to strip all the path or things like that. Obviously, there's a ton of different settings and a ton of different tools that you could use to do all of this. It's just neat to see if you're already using NixOS, how far you can take that without having to go outside the ecosystem.

Not only that, but how easy it is to actually wrap your brain around once you set it up. Like it was very intimidating at the beginning. I thought this seems like to try to define this in Nix seems crazy. But after we got it done, I was like, this was simpler than actually trying to do it in Nginx. And on top of that, one of the number one conversations I see coming up in our Telegram group recently is, hey, guys, I have all these services. I think I need to get like an SSL cert.

Like, how do I do this? And so often the conversation is go install traffic, right? Go to like this really complex route. And what I found so refreshing about this is it was 15 lines of config and I'm done. And it is like you back that one file up and I can reproduce it over and over again. And Nginx is great for this kind of thing. Like there's no shame at all in the Nginx game. No, I mean, you know, it's performant,

you know, it works well. And then if you kind of don't have to write the config, even better. And, you know, there's the usual sort of escape hatches. So if you do need to just slap in a little bit extra config, there's an easy way to do that in Nix that just lets you sort of, you know, render raw dog some code right until like the end of the template.

Linode.com slash unplugged. That's where you go to get $100 in 60 day credit on a new account. And it's a great way to support the show while you're checking out fast, reliable cloud hosting. They've been around for nearly 19 years, and I've had friends that have used them for like a decade and recommended them to me. But almost about three years ago, I finally had the opportunity to try out Linode. I've been using them consistently. I mean, I went all in. I really loved the platform. I didn't get the hundred bucks either. I just signed up and tried it. But I think if you get the hundred bucks, when you go to Linode.com slash unplugged and you kick the tires, I think you'll become a convert. I suspect that's why Linode continues to advertise on this here podcast is

because they know that when they finally get to you and you do try it out, you go claim your a hundred bucks. You're going to love it. I mean, of course they have all of the table stakes. They have tons of distributions to choose from. They let you burn the system down to the metal and install your own OS. If you want, I've done that a couple of times. They have great documentation 24 7 support 365 like even on the holidays tier one like you you call in and they answer your questions

there's all these things that add up plus the features like s3 compatible object storage the backups a lot of that stuff you'd come to expect just a really really good solid implementation of all of it but just part of linode special sauce is how they bring it all together the ui the api all of it kind of comes together and makes it accessible to somebody who's been doing this for 20 years or somebody who wants to spin up their very first website blog portfolio whatever it might be and they have pricing to match to super fast systems and they got a dozen data centers a dozen more coming online this year. If companies were like mascots, these would be the flying eagles. These guys would be the flying eagles. Like they're just soaring to new heights. I don't know. I don't know. I mean, you could, maybe there's a better

animal out there. My point is great host, maybe not a great analogy, but absolutely great host, especially for us Linux users. And it's a great way to support your local podcast. Linode.com slash unplugged. And it's a great way to support your local podcast. Linode.com slash unplugged. Ryan Hubert joins us this week. He's the CEO of Define Networking. And we've actually had him on the show like forever ago because he was one of the original core developers of an open source mesh networking tool called Nebula, which is a totally self-hosted solution until just recently.

And we're going to get into that part of the conversation in just a moment. I first want to thank both Tailscale and Ryan because, you know, Tailscale also does a Mesh VPN. Yeah, of course. And Ryan is creating something over at Define Networking. And everybody was super respectful about that in this conversation and totally just recognizes like they solve different problems for different people. It's one of the things we're going to get into into in this conversation. But just something I wanted to take a moment and just say thank you to everybody involved, because that's a rare thing where everybody can kind of figure that stuff out. And I'm really glad it happened, because what Ryan's been working on over to find networking with Nebula is a tool I suspect a lot of you could get some value out of. Imagine building your own mesh network protected by the

noise protocol, completely hosted on your own, but scaling to potentially millions of nodes if that's what you want. And the last time we talked to Ryan, he was still working over at Slack, solving this problem for Slack. And now he's solving it for everybody. So Ryan Hubber, the CEO of Define Networking, is joining us on the show. And Ryan, last time you were here, I believe I introduced you as an employee at Slack. Yeah, that's right. And now the CEO of Define Networking.

And we've talked about Nebula on the show before, but can you just refresh the audience on what Defined Networking does and what Nebula is? Sure. So I'll start at Nebula. You know, back at Slack, we had numerous problems related to networking that we were looking to solve. One of which was we had an ever-expanding presence in the cloud, you know, in AWS and Google. And we wanted a way to connect everything that wasn't vendor-specific, and we needed it to be performant. But one of the main drivers was we wanted to encapsulate security groups in whatever we did. So we evaluated a lot of things and ended up realizing we were going to have to write something. That was the origin of Nebula. So we created it back in, I hope I get this right, 2017, I believe,

and quickly put it into production because it was actually needed to solve some problems with connectivity to remote regions. So I think within six months, it was actually passing traffic. remote regions. So I think within six months, it was actually passing traffic. And then, you know, by the time Nate and I, Nate's the primary co-author of Nebula, also co-founder of Defined, which you just mentioned. By the time we left, it was passing the majority of network traffic at Slack. And that's network traffic across regions, but also hosts sitting next to each other. So basically everything at Slack uses Nebula to communicate with everything else by and large.

That's impressive because you have to imagine that's a fair amount of traffic. And so that's Nebula, essentially the mesh VPN. And then Define Networking must have come along after this just started to gain so much traction. Yeah. So we wanted to open source Nebula all along. We got permission to open source it well before we did. And at some point, somebody put the idea in our heads that we should maybe start a company around this. And so, you know, we were actually pretty slow to do it. I mean, we had our first conversations about it a bit before it was open sourced. And we weren't sure we wanted to do that. But,

you know, over time, it became more and more compelling. And so we actually founded the company just after I spoke with you the first time. So I think we spoke November of 2019. And then the company was founded in January of 2020. And the purpose of the company is so Nebula, that the downloadable version that's on GitHub is everything you need to run Nebula is there, right? Like you can run your lighthouses. You can run a fully formed Nebula mesh. You could scale it up to any size you want. It's the exact same Nebula Slack is running. But the part that it doesn't include is the management layer.

And the reason it doesn't include the management layer is because we created core Nebula at Slack and then did everything through Slack's configuration management tools. And there just was no way to generalize that. So the goal with the company was to make Nebula more accessible for other entities who didn't maybe want to roll everything themselves. Maybe you don't have a sysadmin on hand who's handy with setting up configuration management and managing all the certs that you need for Nebula. So that's where something like an admin console comes in. Absolutely. And one of the other things is over the time we spent at Slack, you know, running Nebula, we gained a lot of insight into how to run Nebula and networks in general. And, you know, asking people to kind of reinvent that wheel when we know a lot about how this

operates in a production environment seems silly when we can, you know, we can give them a head start on that. So it's been a bit of time now. And on December 15th, you have a blog post that went live. And it to me, it almost felt like a reintroduction for defined networking to the public, because it seems like there's now a more consumer facing option using Nebula to create a network with some tooling provided by defined networking. Tell me about this next step. Yeah. So, you know, throughout the last couple of years, we've been building an enterprise focused product, but I regularly get questions from friends and, and, you know, old colleagues and, and they're asking, when can I use this? And I'm like, well, you know, it's more focused on enterprise. But the December 15th release date was actually purposeful.

We thought, what if we put it out there and then make this available for people who are going home for the holidays and then they can try it out, right? And it turned out to be actually successful. And it turned out to be actually successful. So it was fun to watch the graphs go up over the holidays as people were back home and, you know, installing it on different devices and trying it out. And here, I just thought you were crazy for releasing it before Christmas. But now that seems like a good insight. Yeah, we actually set that December 15th date well in advance. I think it was two or three months in advance.

And to be clear, like, that's not the market we're looking for. Right. So this is this is a nice side effect of how Nebula is built that we can kind of offer this this version of it to everyone for free. So you just touched on it there for a second. Who do you see as kind of the ideal user of Nebula? Because it seems like it's a I mean, if a company with the network size of Slack is using it, it seems like it's a, I mean, if a company with the network size of Slack is using it, it seems like it's maybe purpose built for very large customers. Yeah. So it's interesting. The history of Nebula is we created it for Slack, right? And so the reliability, the performance, the security, those were all top of mind. At the same time,

all of us that worked on it wanted to use it. So you had both sides of it, which is massive company that has these very important requirements about how it needs to operate. But also, I wanted to use it for my own network. And so it does scale up and down pretty seamlessly. One of the features we didn't have for a long time that I'm sure you noticed is now there is relays, right? And the reason we didn't have relays, that's when you can't hole punch to another host, right? The reason we didn't have that for a long time is Slack has absolutely no use case. And so, you know, their massive network, they control all of the network in AWS and Google and all those places. And so there's no need for them to really do what we're doing there. and Google and all those places. And so there's no need for them to really do what we're doing there. But enough users were asking for it that it became a top priority.

And interestingly, the thing that actually led to relays being implemented, one of the folks that works at Defined, Brad, he's in an office and he doesn't control the network there. And so he had connectivity issues. So necessity being the mother of invention, I think is the phrase. And so Brad actually went and specced it out and created it. And we had some pretty strict requirements

about how we wanted relays to work. Like we didn't want to run open relays. We wanted it to be authenticated, not encrypted. And so there's a lot to the relays that's actually pretty, pretty important to us. And to clarify the role of a, of a relay in this setup. So with Nebula, you have your mesh VPN, it's protected by, you know, the noise protocol. And then you may have, is, is the, is the job of a relay? Is it to allow devices in and out of that Nebula network that maybe don't have the Nebula client or aren't directly connected? Is that the job of a relay? No, sorry, I should back up.

So this is for the connectivity cases that hole punching can't get around, right? Oh, right. Okay, so crazy bad NAT or something like that. Yes. Crazy firewalls. The best example I have that I've dealt with a lot is AT&T's network uses carrier grade NAT. We actually also added IPv6 for the underlay. We call it the underlay. So if you had IPv6 enabled, it would work. But relays were the last bit we needed to deal with bad NAT. Yes. Ryan, I wonder if you could touch a little bit

on some of the intentions around designing what you are and are not managing in the new defined service and the admin panel. One thing I noticed right away was, you know, you're doing, you're taking some of the load off me, but I'm still hosting my own relays and lighthouses. Now, the process is really easy. You've got a little script to run, a tool to run that gets everything configured. But it seemed like that was a pretty intentional choice. tool to run that gets everything configured. But it seemed like that was a pretty intentional choice. It was Yeah, I'm glad you noticed that. So one of the reasons we we push for that, and we had a lot of internal discussions about this, right. And just for everyone's, for the sake of discussion here, lighthouses are the nodes that allow everything in a nebula mesh to find

everything else. So they just they're the only thing that needs a routable IP. And the reason we decided to have lighthouses as something that you run instead of us is, well, a few reasons. So first off, availability. The reason we never seriously considered any alternative at Slack was largely because we could not outsource our network availability to someone else. And the nice thing about having everyone host their own lighthouses is that allows them to own their availability. So you can host a lighthouse inside of each of your AWS regions. You can host them. Slack, by the way, as an example here, Slack has something like 100,000 hosts using eight lighthouses total that are globally distributed. So, it scales extremely well, but it's also about getting the answers quickly and positioning them. One of the nice

things about this model is imagine the case where you lose internet connectivity, even momentarily. Most of the time, that's going to mean you can't make new connections. But if you have a lighthouse that's internal to each network, they continue to work, right? They continue to work independently. And so a lot of this was standpoint, I stand up a lighthouse and then there must be an enrollment process and Define doesn't have any involvement in that. It's just all between my machines. So we do. So what we're handling is actually the CA side of things. And it gets a bit complicated here. And I mean, that's why we're managing it, right? But the thing to know here is there's a version of Nebula you install on your hosts that has our management layer baked into it. And what that's doing is

getting all of the roles, groups, keys, all of that, and populating them on each host. Now, if you go and look at the configuration directory on each host, you're going to see a Nebula config. In fact, if you stopped running, it's called DN client, and just started Nebula and pointed it at that config YAML, it would actually just work, right? It's that similar to open source Nebula, it's just a configuration layer. So with defined, what we do is manage the really the hard parts of what people are doing with Nebula. So, you know, one of the one of the things that we've seen is nobody wants to manage a CA, right? It's, it's difficult. It's, it's time consuming. You have to worry about where your key material is stored. And so what we've done is take our,

our experience with doing this at Slack and other places. And, you know, we've, we've stood up, we've, we stood up services that manage this for you and distribute the certificates to all of the hosts. And like when you go to add a lighthouse, you go into the admin console and provision, or you can use the API to provision it. And when you add it, then all of the other nodes are made aware of it. Like we handle sending out that configuration that gets populated on every host. But outside of configuration changes, those hosts and those lighthouses and those relays are acting

autonomously. So our service is there to populate the config and sort of at a high level coordinate. But again, speaking to the reliability aspect, if our service is down for a period of time, you can actually make new connections. Your configuration is frozen in time, but everything continues to operate, including new connections. And I think that's something unique we offer. Bitwarden.com slash Linux. Go get started with free open source password management when you go to Bitwarden.com slash Linux. For an individual or for a team, Bitwarden is the easiest way to store, share, and sync

your sensitive data. I migrated to Bitwarden a few years ago. Some of you might be considering migrating today based on current events. If you are, go check out bitwarden.com slash migrate. They make it exceptionally easy to come from password managers like LastPass. And of course, they do things better. And they're trusted by millions of individuals, teams, and organizations around the world. And their code's open for the public to see and has been audited. You know, Wes was a little ahead of me on this one. I have to

admit, Wes was ahead of me. He started using Bitwarden before I did. It took me a little bit longer, but I eventually switched over, and I love it. I'm an advocate in so many ways, and that's why I'm really glad they're a sponsor too, because I'd be recommending you use Bitwarden even if they weren't a sponsor. Don't tell them that though. Don't, don't tell them that. I think one of the things that gives us confidence to use Bitwarden is that community.

There's just a ton of support out there. I've had questions and I've been able to get it answered within seconds. The other thing I like as a Bitwarden user is I'm seeing new features that improve the experience that like focus in on the core experience of a like sensitive secret manager and make that better updates to the mobile applications now make it possible to have a unique secure username password and email address for every website app and service you use across mobile desktop whatever i've got bitward installed as a flat pack i've got bitward in my web browser

i've got bitward on my mobile devices i can unlock I've got Bitwarden in my web browser. I've got Bitwarden on my mobile devices. I can unlock my vault with my thumbprint and on iOS with my face. It's so nice. The reason why that's so nice is because you can be in the flow. You know, you're just trying to get to something. You just want to get in and get logged in. And to make it all integrated with the OS so that way it can use whatever authentication mechanism you prefer and then get you into your database, find that information and then provide it to the app or the website quickly. Makes it a very smooth experience. I don't want something that disrupts that and makes

it take way longer just to get to the thing I need. And Bitwarden has just been working at making that process slicker and slicker. And I really think in their October update, they just, they kind of just nailed it. It's just in such a sweet spot now. So go try out Bitwarden, support the show, and maybe recommend it to somebody who could be doing a little bit better. Because I think we all know, good password managers, like the low-hanging fruit of the security world, using a unique email, username, and password for every site, website, and service you use is probably the easiest and most straightforward thing you can do to improve your security online, in my opinion. And Bitwarden gives you the tooling to make that actually accessible for experts and for totally new users to this idea. So go recommend it to your work, your friends, your family.

Become an advocate like I am and send everybody to bitwarden.com slash Linux. That's bitwarden.com slash Linux. All right. So I have another kind of question that I was kind of when I was looking at this, how are you recommending people solve name resolution inside the Nebula network once they have something established? Yeah. So in open source Nebula, we've had a subdomain delegation for a long time, but in defined, we don't have that available yet. So it's, it's on a very short-term roadmap. I think it's actually landing this quarter, but it's not in there yet. So right now, the way most people are running it, including Slack, is they are using configuration management on a large network, and then they're actually

handling DNS themselves, right? So the magical side of DNS, we want to do that as well. Again, it's one of those things where Slack doesn't use the feature, and so we have to be motivated to do it ourselves. I suppose that makes sense. And you probably then need to understand what all the requirements of that feature are and how to make the right thing that's going to work for the most number of possible clients. Yeah. One of the things about our philosophy that I really want to make people aware of is I would actually call us feature averse. And so when you think about the deployments of Nebula, so there are actually some massive deployments, especially once we started the company, we started to get inbound interest from people that had deployed

it at scales we were pretty surprised by. And keeping all of that in mind, we are very, very, very feature averse, I would say. So unless there's a need for something, we are probably going to say no to adding it to core Nebula. And the reason for that is, again, surface area for security, reliability, and performance. So I'm going to knock on wood. You'll hear it in the background. But in five years of running at Slack, Nebula has not had an outage. Did either of you happen to read my blog post and see the footnote about the only Nebula outage? Oh, no, I don't think I got to the bottom. footnote about the only nebula outage? Oh, no, I don't think I got to the bottom. Well, spoiler, I caused it myself. Oh, no. So, it wasn't nebula's fault. But on January, I'll have to look it up, but give folks a link. There are news articles you can attribute

directly to me because I rolled a bad bill of Nebula. So that was about five, well, four years ago, five years ago, whatever it is. But actually, that was the last time Nebula caused a production issue. I see that now. What made me smile is that you publicly admitted to the original name being Cloudy Cloud, which I thought was pretty great. I loved that name. Yeah, that is pretty good. Because it's your own Cloudy Cloud. Yeah, it was fun. And I think Boaty McBoatface was kind of a popular thing at the time.

So yeah, just a natural progression, I guess. But yeah, one of the things that we really take pride in is Nebula itself. If you ever look at it running on a host, it's using very little memory. It uses a lot of zero copy and the performance is something that we focused on heavily. So, you know, there's, there's a lot to that. There's a lot to tweak. And one of the other things that we're doing at Defined is taking some of that knowledge of how we tweak Nebula to make it faster and, and really baking that into the configs that we put on, you know, on the hosts that we manage. That's great. I know you really do pay attention to the performance of it.

That's something I've definitely picked up. Also seems nice in that, you know, if you can reuse that config with the open source one, it's just sort of, you know, teaching users some of the options that are available too. Yeah, exactly. You know, one of the interesting things about Nebula is we actually default to AES for our encryption layer. So if you look at the noise protocol framework, they spec it for both ChaCha, Poly, I can't

remember, 1305. I can't remember the whole thing. But ChaCha is basically the same encryption that WireGuard uses, right? And is very popular. But noise actually specifies AES as well. And so, you know, WireGuard and Nebula have that common noise protocol underneath. The reason we actually used AES is pretty interesting. So at Slack, we created this for host-to-host communication. And on almost all modern CPUs, I think at least for the past decade, there are instructions called AES-NI.

I think that's native instruction, but I'm probably wrong, that actually accelerate AES performance. And so the reason we did both was we wanted to take advantage of that on hosts that had it. And so it's an interesting kind of sidebar, and I hope I don't go too in the weeds here. But the reason WireGuard went with ChaCha Poly is there are numerous reasons. Protocol agility has been a problem for ages in TLS. And so it's caused many of the TLS headaches over the years. One of the things about Nebula's AES and ChaCha implementation is you have to choose network-wide which one you're going to use up front. And so if you're using something like Nebula at Slack, everything is speaking to everything else using AES because all of those hosts have native AES instructions and it's going to be significantly faster than using ChaCha on

those hosts. The reason that WireGuard went with ChaCha is actually, you know, it's pretty smart. So back then they wanted mobile performance to be much better than it could be with AES and most mobile processors didn't implement AES instructions, you know, around the time WireGuard was kind of being created. Of course, right. And so nowadays, like you actually are seeing AES in almost every small CPU even, you know, starting to, you're seeing it in a lot of CPUs, notably not in the Raspberry Pi. So it's actually available on the die, but I don't think they pay the license fee for AES. Shucks. That is fascinating, though. I mean, I think that just shows the amount of effort that has gone into thinking about these things and that Nebula is ready to scale and meet some performance challenges when users have them. You know, one of the things that I'm focused on and

was focused on November and December is benchmarking. So we've always kind of done comparisons of Nebula to previous versions of Nebula and also against, you know, similar things in the space. I mean, all the way back to Tink and OpenVPN. And I've actually had this Ansible repo for going on four years now that tests Nebula. More recently, I've kind of decided it's time to open source this Ansible repo. And the reason for that is mostly because there are just no good benchmarks in this space. So most people, when they're comparing performance performance and Iperf3 is a tool that everyone's probably familiar with if they use Linux and networking, right? And it's the best and it's the worst because it's easy to get a top line performance number out of Iperf3,

but there's a lot of nuance there that gets missed. So what I'm hoping to do here is sort of give everyone a framework via this Ansible repo where they can recreate these results themselves. And one of the things that I'm actually doing with this repo to publish results is I'm running it on five identical Dell boxes that I've had for about three years. So they're the same Intel CPU, the same 10 gig NIC. And I'm doing all this performance testing between them to give people an idea of the nuanced version of performance, right? So just because something can hit near line rate doesn't mean it's not cooking your CPUs to do that, right? And so you have to associate a cost with how much work it's doing versus, you know,

the performance instead of just looking at the top line number. Yeah, more so than ever, really. Yeah. And for the types of companies that we're talking to, the scale of companies we're talking to, I mean, that really adds up when you think about the cost, you know, even the electricity to run a massive network like that. And so one of the other things that I've done with this, it's kind of fun. I have a TP link, six outlet power strip, where each outlet has wattmeter on it. And so when I run the benchmarks, I actually record how much power it's using to pass this amount of traffic.

And I think that's actually kind of an interesting metric. So I don't know if that'll be part of the published data. But you know, it's kind of fun to look at. I use a similar technique for when I'm benchmarking systems, you know, like laptops and desktops. Yeah. And I think like the maturity of some of that stuff in, you know, CPU benchmarks and GPU benchmarks has been around forever. But for some reason, network benchmarks are kind of, you know, just not really there. So kind of zooming out as we wrap it up and we talk about, you know, who Nebula is going to be probably appropriate for. Do you see this also useful for the individual case? We've talked a lot about how it's probably appropriate at the enterprise level, but what are your thoughts for the individual user? So, you know, I think if you're

willing to run something like a lighthouse, then yeah. So, Nebula itself is a relatively technical project, right? It takes some amount of work to stand up and it's got the pitfalls of certificate management and that's one of the things we take care of. I'd say that folks are welcome to use what we've created at Defined, the free tier that we've created, but just know that it comes with the responsibility of running your own lighthouses, right? And that is more than some folks want to do. But the trade-off there is less of a dependency on us to keep your network operating. I think that's a good balance. Ryan, I'm really impressed to see Nebula grow. I'm really appreciative that you've touched in, you know, just touched in over the years and

kind of let us know, Hey, there's something coming or, Hey, this has changed. Cause I, for Wes and I, it's a project we'd love to follow just personally. And so I'm glad, I'm glad we got an update on the show. I wish it could have been sooner, but let's try to do it again, but maybe do it before three years or so next time. Okay. That would be great. And I can't wait to, to not have benchmarks as the only thing on my mind so we can talk about other fun stuff, too. Sure. And I'm going to be really curious to see where it all goes. We, of course, of course, will have links in the show notes. Ryan, thanks for joining us.

Thank you. And now it is time for the boost. Chris, I know we certainly got a lot of boost this week, but did we get any baller boosts? Hey, Rich Lobster! Rotten Mood came in with 100,000 sats, and I'm going to do my best, guys. I'm not always great at this, but I'll give it a shot. And he just simply said, Angela!

I don't know what it's about. I think that must be in reference to our suggesting that we would have a meetup near the studio. And I think Ange was part of that. It got mentioned, perhaps. I think, or no, I think the other thing is perhaps the whole family members using Plex that I have to switch over to Jellyfin. Oh, yeah, right, right, right, right, right. How's that going? Oh, it's going good.

In fact, self-hosted the most recent episode, we did an update. And I'm now using the native Jellyfin client on my set-top box. That's Swifted? Yeah, you got it. Nice. And it's been interesting. That's exciting. Selfhosted.show.

I think we'll do our Jellyfin follow-up, too, towards the end of the month, too. So keep sending your Jellyfin feedback. We are collecting that right now. But I think, why don't we focus on the here and the present? And we got a great batch of feedback this week, starting with Brent. What do you think? I would say sept, which kind of sounds like seven in French if you're willing to go that far. Okay.

Sept boosted in with 18,000 sats. I've been thinking about switching to Jellyfin from Plex now that they have started renting movies. But I don't want to start over with my library of what I've checked and not, especially when I've run Plex for about four years. Can you relate to this, Chris? Because you've been running Plex for quite a while now. Speaking of self-hosted, we did link to some scripts that will sync your watch status from your Plex library over to your Jellyfin library. That sounds handy. Yeah.

I discovered that after I had made the migration. So like an animal, I just went through and marked. Oh, Chris. Can I just say, The Simpsons, knock it off. Too many seasons. Wait, like 33 seasons? That's a lot of like Mark watched. Can I just say, it's too much.

Dean L70 boosted in as well with 11,101 sats. Hey team, I'm looking at moving to an Arch distro. Oh, I think I've settled on Garuda, but I'm interested in your opinions. Keep up the great work. I know it's all appreciated. Hmm. If you were going to recommend a non-vanilla Arch distro, I assume that's the prerequisite here. Otherwise you would just, I was going to say, yeah, you know, distro i assume that's the prerequisite here otherwise you would just i was gonna say yeah you know shout out for vanilla arch well well i think it's worth having a discussion of why not vanilla arch okay the work i guess because we've talked to well we've talked about the benefits of of learning uh by doing an arch deployment before so it was tempting but i would love to know a

little bit more information but have either of you tried Garuda? Oh yeah. Oh yeah. Oh yeah. I have no issue. I mean, give Garuda a shot. There's, you may not like, you know, I think for some folks, the default theme is like a bit much. But there's obviously a lot of care and craft that's gone into the setup and optimizations and tweaks and, and options that you have with Garuda. So even if you end up maybe exploring other options, rolling your own later, might be nice to try Garuda on for a bit and see what you want to keep for yourself. Garuda is a lot of fun. It's kind of like if you were going to buy a used car and the person before you installed a turbo, put some pinstriping on it, put a spoiler on there, added some air intake. It replaced the sound system with like a real boom and sound system. That's Garuda Linux.

And if that sounds appealing to you, then I think Garuda Linux is the right distro for you. And I mean that seriously. If you like something that's a little more basic, like the stock car, that's like the basic version of the vehicle and you'll add the features you want over time, I might consider endeavor os endeavor os is pretty close to vanilla arch with just a few changes to make it a little bit easier i think also though you should consider at least giving arch from scratch a go in a vm just so you understand what these distributions are building on top of and maybe the tooling a little bit better. And I think you just need to make the decision

if you want the pre-souped up car or if you want the basic model car that you add the nice stereo, the nice tires, the lift system, the air intake, the turbo, the spoiler, then, you know, maybe you might consider in that scenario going with Endeavor OS. It is pretty powerful when you sort of like like oh i can not that you're i mean you're not compiling the software you're not making the

writing the software but just knowing and feeling that you can sort of like build up a very workable nice system sort of from scratch and understand at least at the high level the components that go into it it's it's something it is nice even if you just do it in a VM, it's worth it, I think. Yeah. I also wonder, you know, I guess we should mention Manjaro in here. Dan, who's part of the Manjaro ARM team, mentions in the chat room that Arch also added the Arch install tool to the live images. So Arch has never been easier to install. Great point. And, you know, Manjaro has a great community around it. Totally worth considering as well.

Oppie1984 boosted in with 2,000 sets. Plus one for the Linux and self-hosting community aspects of the network. I don't really say anything in Matrix, but I lurk and I learn. I have no one in my day-to-day

that is into these topics. Not even my GMRS and ham buddies. So, it's really nice to know I have no one in my day-to-day that is into these topics, not even my GMRS and ham buddies. So it's really nice to know I have a community of like-minded people out there, even if it's only virtual. Well, thanks, Abhi. We feel that too. Yeah, I totally agree. Our matrix community is such a solid group. It's so funny because I love our telegram group as well, and the IRC room even still persists. Anywhere where, you know, JB Folk congregate is a solid group. It's so funny because I love our Telegram group as well and the IRC room even still persists.

Anywhere where, you know, JB Folk Congregate is a blessed space. Yeah, it's special. It's a special colony space. But The Matrix has that, like, just extra edge to it that I really like. But also, I totally grok what you mean. Like, you kind of hope, like,

your ham radio folks would be more into free software and using Linux to do all this stuff than Windows. You kind of hope, though, because it would be more into free software and using linux to do all this stuff than windows you kind of hope though because it feels like in the venn diagram there's a lot of overlap there you kind of forget that or you know sometimes i run into people who are really into like some of the smart home like automations and iot things but they're they're not running home assistant on linux that's not like the part that they've explained you're just like oh wait you can't you can do this other ways? But why wouldn't you?

So true, Wes. This is so true. Wolfman 2G1 boosted in with 2,000 sets. I started out as a sysadmin, then became a network engineer, then became a Linux systems engineer, and now an engineering manager. Never have I ever written a Bash script. In fact, I barely know bash. Ansible and Python have been able to do everything I've ever needed to automate. All right. Now these never have I ever segments just got real interesting. I feel like we just got to a real confession here.

I also, Wolfman, took many, many, many years before I wrote my first bash script. It was simply because I needed to save myself from a greater woe that I finally tackled that problem. And I felt like a poser for years not writing my own bash script. I wouldn't even admit it on. I mean, this was years ago, but I wouldn't admit it on air. And even today, I don't want to tell you how long it was. I'm still very sensitive about it. So that is a great never have I ever.

Really appreciate that one, Wolfman. And yet somehow you're so fluent in PowerShell. Yeah, or you know what? AppleScript. Don't forget that AppleScript. Oh, yeah, well. Dark Matter PHP dev boosted in with 1,234 satoshis. Here are some stats for joy of hearing Wes read some terry bradgett a new regular feature perhaps that was choice oh i don't think you dark matter so the background to wes reading that quote

in last week's episode was wes sent that to me to make a point in a private chat and i saved it earlier in the week and put it in the doc to have Wes read it in the actual show. I was like, this is good. We got to put that in there. Uh, all right. This is a great name.

Legit Savage. Oh, legit Savage boosted with 10,000 sets. Uh, I use track.tv or track it.tv. That's T-R-A-K-T.TV to sync my watch status between Jellyfin, Plex and Cody. I had to build the library in each first though. So keep that in mind.

But this is a really good tip. It is a hosted service, but it's been around for years and it lets you just keep all of your different services in sync for what you've watched. Now, for somebody that doesn't need this, you're like, why would this be a thing? But then if you're trying to complete Enterprise, you totally understand why. So, thank you, Legit Savage. Yeah, and there's a lot of stuff where, like, you know, maybe you don't care that much about sharing what random sitcoms you've watched. Yeah, that's not why I would want to use TrackIt.

I just want to use it for myself because I'm lazy. Exactly. Anything to avoid doing what I just did last time with this damn jellyfin jellyfin challenge i'll tell you what i just had a remark it sounds like a kid task you know oh it was but it required them to be like nope seen it dad okay then i'd mark it watch and i'd play the next one okay have you seen this one oh my goodness nope there's a whole family affair no just bella but it still took forever. I just used one kid, but... It was a thing. All right, 47 boosted with 1,000 sats.

I just recently rebuilt our home server with an Odroid H3 with NixOS. That sounds familiar. Yeah. A datastore on ButterFS, soon to be a RAID, Jellyfin for media playback, SyncThink for backups, Tailscale for remote access for everything, and even NextCloud for experimenting with it. I'm pretty proud of it. And the themes of quality, ownership, longevity, and repairability are things that I just realized I was thinking about while setting it all up. Just got to get the offsite backups running.

Thanks for the show. Chris, did you boost into the show called Number 47? This sounds exactly like you. I know, right? 47 and I are in the same exact headspace right now. And I have to say, it really feels great. It feels like, you know, stuff is just fresh and new to learn, like just a whole new world of opportunity. However, one area I could legitimately use some help, and I feel like I've really just don't have a great answer is what are y'all using for offsite backup? Because I use duplicati and what I like about duplicati is it integrates with

cloud services really easily. It does local AES encryption and it gives you a web UI to back it all up. However, I don't think there is a backup tool that has more horror stories about being unable to restore your data than Duplicati. Why would you need to restore? That's not what you do with backups. You just back them up and never check them, Chris. Come on. If that's your goal, it sounds like Duplicati is aces.

But if you want to actually restore the data, it sounds like you need to be using something else entirely. In part because you've got to get Duplicati reinstalled, you've got to get the database restored, a bunch of metadata has to work. Like it just sounds like it's a big overhead. I've been thinking about RESTIC, AutoRESTIC and a couple of those tools. But I'd really like to hear the audience's feedback on what you're doing for offsite backup and something that could maybe be scalable.

Because for my photos, I experimented with storage or storage or whatever. But I'm just not comfortable recommending that to everyone just yet. It still feels too new, too new. Like I need a year or two of using it before I'm comfortable recommending that as a way you back up your stuff. So I'm curious what you're using today. So please boost in or send us an email and let us know, what are you using for offsite backup? Does it have a web UI? Cause that'd be a huge win for me. Probably doesn't because I've looked at all of them. Old hates the command line Chris over here.

No, you know, it's funny. No shame here. No shame. It's for like just restoring things and whatnot. I like having the ability to browse through the data set visually and select the directory I want to restore and just that directory because I never want to restore an entire system or all the directories. I always just want to restore a couple of directories.

I just want to click on them and hit restore and have them come out somewhere. And that's what I want. But I am willing to try something like Autorestic. I have it all installed right now. I just haven't set it up yet. So I'm thinking about that, but I thought I'll pause and I'll defer to the JB community and see what people recommend. Zack Attack boosts in with 30,000 cents. Whoa! I know, sneaky.

Keep the change, you filthy animal. Listening to you talk about Graphene OS finally made me decide to start looking at it more seriously. I got an old Pixel 3a running it right now and I'll be moving my primary phone to it soon. It'll be hard

giving up Android Auto, but I really like Magic Earth. Also, this is allowing me to move more into open source software on Android and find alternatives for applications that I probably never really needed to have on my phone to begin with. Zacatech, amazing to hear that.

I'm glad you're liking Magic Earth. I know it's a little bit of a transition, but I still think it displays stuff better than the proprietary apps in a lot of ways. It is hard to give up Android Auto, though. That has been rough. I do miss that. I'm planning to do a Graphene OS update soon, though. So do send in your feedback and your experiences if you've been using it for a little bit.

Because I'm kind of collecting all of that right now, so we can kind of do some meta coverage to see how it's been for other folks as well. Yeah, sounds like Blue Mojo also sent in 500 sats just to say thanks for talking about graphing. Thank you, Blue Mojo. It's actually a lot of fun. I mean, obviously it's fun just us both doing it,

but knowing that there's folks out in the audience too who are all trying the same thing. Yeah, it's been really nice to see. And to hear everybody's experience with the different hardware has also been fascinating because each one of us is trying it on a different device for the same thing. Yeah, it's been really nice to see. And to hear everybody's experience with the different hardware has also been fascinating because each one of us is trying it on a different device for the most part. Like we've all got our different backgrounds and stories. Yeah, different workflows, different needs.

We got 2,000 sats from Ice Cube just to say thanks for all the fish. And Funky B boosted in with 9,000 sats. Heyo! Hello, Chris and team. I'm a longtime JB listener from Trinidad and Tobago. Here is my first boost with Sats I've earned on Fountain. Keep up the great work and keep that content coming. Oh, thank you.

You know, I had somebody ask me recently, what's the difference between the Fountain Earned Sats and like the Brave Basic Attention Token, which I'm not a very, very big fan of. I thought that was a great question. So Fountain FM streams you sats as you listen from sponsorships. When people buy sponsorships in the Fountain app

to like feature their pod or their clip or whatever, they pay Fountain in sats and then Fountain streams those sats to listeners. So why is that different than the bat token, the basic attention token from the Brave browser? Well, that's a great question. So the difference is, is that the basic attention token is an artificially created token by a group of people that pre-mined that token and own that token and can buy and sell that token depending on market liquidity. And when things are up, they'll sell. And there's really not much you can do with that token depending on market liquidity. And when things are up, they'll sell.

And there's really not much you can do with that token, right? There's kind of only a few exchanges that even accept it. Doesn't really do anything outside that ecosystem. It's an ERC-20 token that lives on top of Ethereum. It's just not a very compelling token. Sats are a part of a Bitcoin. There's 100 million Satoshis for every Bitcoin. And every Bitcoin is unique.

It has a mathematically provable address. It is scarce. And it has been mined by a miner. There's no pre-mines. There's no group of people that created this coin and gave themselves a percentage of that coin before they made it publicly available. Even Satoshi, who's no longer around, didn't pre-mine Bitcoin. Satoshi had to mine every coin they owned. And that makes it fairly distributed and it makes it something that is unique in anything that's come after Bitcoin bitcoin there's somebody behind it and a team

of people that have created it and are managing that token and maybe there's something with what with what the brave browser is doing but i'm not really interested in a bat token you know i'm not really interested in the in like their cryptocurrency right i want a satoshi out of that that's something that is an open network fully distributed and available to anyone sort of like linux right it's kind of the difference of a proprietary piece of software that is created by a company and maybe they even maybe proprietary isn't fair maybe they even open source it right like that but they're the only people that use that and they're the only people that contribute to that open source project versus Linux, which is contributed to by thousands of people and companies.

Right. And it's used by thousands of individuals and millions of individuals for different things. And there's a difference there that I think matters a lot when you really get down to it. So I think that's a great question. And I'm really, really glad that Fountain has made it even easier to send in Boost now because now you can add you can top off your uh fountain wallet uh directly within the app and i don't recommend that you hold a bunch of bitcoin or satoshi's in your fountain wallet at all i'm recommending you buy a few bucks and you send them to the podcasters you like that's my advice uh spam proof at fisa.sat boosted in with a row of ducks.

Hey-oh! And said, I'm happy to provide a little bit of liquidity to the Jupyter 01 node on Lightning and Value for Value is the way forward for the ultimate open source self-sovereign right to repair issues on all the things. Totally agree. Thank you so much, Spamproof. I love it when people get it. Been doing this for a long time. I've seen a lot of things come and go, and I got a pretty good track record on this stuff. And I'm really bullish on keeping podcasting decentralized, not making any company like Shopify or Spotify or Stripe or PayPal or Apple or any of those companies,

kind of like a gatekeeper for any of this stuff. I feel like that'd be a bad direction. So I'm glad you get it. And then Linux Teamster came in with 5,000 sats. And this is a great one, guys. I mean, it starts with a great name and how could it not get better from here? I actually think this might be

one of my favorite boosts of the week. Not because of just this boost, but because Linux Teamster sent in boost to Coda Radio, Office Hours, Self Hosted, Bitcoin Dad Pod. Like they went on a boosting spree and they sent their very, very first boost into this show. And they write, I love this podcast. I especially love how willing you are to push the boundaries and take risks by trying new things. I've been loving the member feed this past year. I think it's great that you have an option other than Patreon and PayPal. And they also just sent a 5,000 sap boost and just say, Hey, this is my very first, very first boost,

by the way. And I think that's incredible because I got a note in matrix this week from, uh, B Bob, I call him, I'm so bad with the nicknames, but I call them B Bob and I'll put a link in the show notes. If you guys want to read it yourself and they write Chris lass, I'm not sure if this is the right place to tell you this, but, uh, I just wanted to say after checking out one of your recent office hours episodes this past summer, I have gone down the Bitcoin rabbit hole. To the point where I now have my own Umbral node, my own Lightning node, and I've opened up a channel to you. And I'm subscribed to the Bitcoin Dad podcast and have plans to pick up a hardware wallet. You mentioned that you were excited to be playing with all of this tech.

And I gotta say, I know how you feel. I just wanted to say, thank you. I'm also currently an SRE for the self-hosted show. And I'm trying to wrap my head around how to send in boosts and stream sets. My podcast player of choice is not part of the 2.0 spec just yet. Well,

good news. Podcast attic is getting in on the game here pretty soon, but you can always go, go, go grab Albie. And I got another note from an individual this week that said, Chris, I was super skeptical of the boost. I thought you were just shilling crypto scams on the show. But I listened to your Office Hours episode about we hate crypto too. And now I get it. And now I'm having more fun than I've ever had in technology

since I first discovered Linux. And Brent, I think you can echo that. Like when we discovered this self-hosted world of Bitcoin, Lightning Nodes and all of that, it was like being back in the late 90s for Linux. Do you remember that feeling? Yeah, that was last January. And I got to say, I came in pretty skeptical, like most people should, you know, but the more you and I dove into it, man, I remember just days going by you and I in your office just super excited. Like we had computers apart, like putting them together like it was, I don't know, a LAN party or something from the old days. And we had a lot of fun. And that fun continues, which is kind of amazing.

Yeah. And I really encourage people to play around with it. I'm not suggesting that you put tens of thousands of dollars, anything like that into it. You know, a couple of bucks, go have some fun, play with some open source technology, and support a local podcast that is supporting decentralized podcasting. And Wes, before we go any further, we got a live boost into the show, didn't we? Yeah, Todd rocks boat. Buy a fountain with 222 sets.

You are getting old, but that's okay. Skip scale, save your time, capital, and energy for something you're excited for. That's hard to hear. But that's the start. That's the start. I think Toad Rock's boat has kicked off the start of this feedback chain. So if you've got contrary opinions or you want to second that, let us know. It sounds like Toad is certainly rocking that boat. Yeah, for sure. No kidding. And our boat. So, yeah, if you've got contrary opinions or you want to second that, let us know.

It sounds like Toad is certainly rocking that boat. Yeah, for sure. No kidding. And our boat. Yeah, my boat feels a little rocked and I feel like there's some honesty and truth in there that's worth considering. You know, it's one of those things where I can tell you all of the reasons I shouldn't do something and I can quantify them and I can even put a dollar amount to them. And yet I could not quantify all of the great things that could happen and what they might be worth. For me, it's always about the meetups.

I know we always have like an end destination, but it's the meetups that are the most memorable part for me. So I feel like if we do the meetup trip and just do a bunch of meetups and don't go to scale, I'd be fine with that. I don't know what a meetup trip. We just did a meetup trip in California though. Yeah, I know. I don't know what it looks like. I don't know what a meetup trip. We just did a meetup trip in California though. Yeah, I know. I don't know what it looks like.

I don't know. Cause I do want to see people. Of course. Live show, live show. I, you know, and I think there's something, I feel stressed about LUP 500 around there too. LUP 500 is the weekend before scale. We did the math, we figured that out.

And I feel like if we're going to scale, my focus can't be on both of those things cause they're both huge and i and because scale requires travel and getting you guys a lot of yeah yeah right there's no way i just have to focus on that and then like we don't do anything for 500 and then i feel bad we just skip it 501 is the new 500 yeah 501 where we save it and we just really to mess with everyone and 500 comes out in like six months. Yeah. Yeah. Only the people in the know that have been listening really carefully will even have any idea what's going on. What could go wrong? That's never backfired on us.

If you want to send a boost into the show, the setup that I just love right now is get albie.com, go grab the Albie extension, throw a couple of sats in there, and then go to the podcastindex.org page for Linux Unplugged. And it's real easy to just boost from the website. You don't got to switch apps. If you do want to switch apps, fountain.fm just released a brand new version, blow away great. And then Podverse continues to refine and make the cross-platform GPL podcast 2.0 experience excellent. So those are my two top recommendations. And boys, I don't think we have any picks this week. So let's just wrap it up and tell people how they can get a hold of us.

Brent, where do they go? Well, I think it might be something like linuxunplugged.com slash contact if you want to get a hold of us there. Atta boy. And of course, you can always watch us live over at the Jupiter.tube on the Sundays at noon Pacific, 3 p.m. Eastern. See you next week. Same bad time, same bad station.

And, Wes, we've got to tell them about Linux Action News. Yeah, I mean, we didn't talk about any Linux news today, like at all. But there has been some. Yeah. There's always some. There's always stuff going on. And you can find out just what you need to know, LinuxActionNews.com. That's right.

There's more show over there. Things that change Linux and open source for this week at linuxactionnews.com. As for us, we love hearing from you. So that feedback, those boosts, those are all a big part of the show. And of course, links to what we talked about today at linuxunplugged.com slash 493. How about that, huh? Getting way damn too close to 500. Thanks so much for joining us on this week's episode of the Unplugged Program.

We'll see you right back here next Sunday. Yeah, so they've got their, like, this new little client, their defined networking client, DN client. But it's really using the open source Nebula, and it's basically just sort of like, they provide a service that manages the configs and handles the certs for you. And then it just syncs that all down and starts the Nebula client pointed against those things.

But you get, like, an API with that? And so they've implemented, like, this is just like a little Go client that actually calls their API to do it. And you can make the same API calls. So like if you want to set up something to automate adding hosts. So in other words, you could have a system set up that would deploy a host and then it would

run that and just join your Nebula network and be available as a node. Yeah. You make an A like you get like a rest API call, you get like an enrollment code and then you just like pass that as a command line argument to the DN client, Go binary. And they've got, you know, they've got them for ARM, they've got them for AMD 64. So it's pretty easy. That's pretty nice. So if you want to try it, and you want to see their managed product with the UI and all that stuff, you can go to Define Networking's website at define.net, and they do have a sign-up there. Yeah, totally free. Just join up. You get up to 100 devices, I believe it is. So, pretty nice. They also, I just happened to notice as I was poking

around the docs, which have been getting better. That's been one thing. That open-source Nebula. I mean, you could go look at the repo, and it was pretty clear, but not the best documentation, necessarily. Which is, you know, something that improves with over time for all projects. But they also have stats support

so if you're running something like graphbind or prometheus already or you've been interested in playing with that I mean who doesn't

want stats on their mesh network