LINUX Unplugged - 524: How Our Server Got It's Groove Back

Episode Date: August 21, 2023

Can we build an indestructible server that stands up to the test of giving out root login to the Internet?

I've been seeing more and more people ask for the best laptop to run NixOS. And I did a little looking around. There is a NixOS laptop device list. You can find one on the wiki. You can find them all over the place. You can find websites where people are talking about this sort of stuff. But there isn't like a definitive large reseller that just offers NixOS for a laptop, even though it's been around for like a decade. It's not really a very i guess

popular you know they're not getting requests for it or whatever it might be but i thought it was funny that before we got to like this is the common way you run nix on a laptop this is the go-to hardware before we got that we got nix os on mobile phones and of all the devices in the world one of the supported devices is Brent's phone. No way. What? Yes. Yeah, I thought I was missing out because all this Giraffine stuff on my lowly little

OnePlus 6 here. You're just going to leapfrog us. This is one of two officially supported devices. I would imagine probably because one of the developers just happens to have one, maybe. I don't know why they chose it, but I'm tempted. I don't know what the experience is like so far, but bring on the experimentation. Yeah. All right, Brent, you report back. Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris.

My name is Wes. And my name is Brent. Hello, gentlemen. Coming up on the show today, we're going to see if we can build the indestructible server combining the immutability of NixOS with an impermanence module. The theory goes, can you create a system that can be completely brought right back to its original state with a simple reboot, even with your files destroyed, your config files all messed up, your web server deleted, whatever. Can you just bring it right back? And to put it to the

test this week, we're going to give login credentials to a NixOS box that we've set up. We're going to let our live stream log in, break the system with a couple of rules, and then we'll see if we can restore it instantly here on the show. And then we'll round it out with some great boosts and picks and a lot more. So let's say good morning to our friends over at Tailscale. Tailscale is a mesh VPN that's protected by WireGuard. We love it. It'll change your networking game. It creates direct connections between all of your systems protected by WireGuard's noise protocol, and you can get it going in just minutes. Go say good morning and try it out for 100 devices

for free at tailscale.com slash linuxunplugged. That's where you got to go to support the show, tailscale.com slash linuxunplugged. And of course, time-appropriate greetings to our virtual lug. Hello, Mumble Room. Hello, Chris. Hello. Hello. Hello. Nice to have all of you there and also those up in quiet listening who don't chime in but are just enjoying that sweet Opus audio.

It's nice to have you here. And we have a little bit of show news right off the top. I'm finally making good on a promise that I made to our community two years ago. And it just didn't work out. We wanted to try to get over to Spokane washington for a meetup and i promised and i promised and i couldn't do it and we finally

have figured out a time so coming up saturday september 16th we're holding a meetup at iron goat brewing in spokane washington it's our default for now we will consider other venues we'd like you to be open to all ages, plenty of table space and room for us to have a group of people. We really like the low-key vibes. That's often why we do these at breweries. You know, they're okay with us just kind of showing up with a group of people and taking over a couple of tables and getting some drinks and food. There's no fee generally. A lot of places that are a little swankier, they want you to reserve and pay and that's just not really our vibe right now.

But if you perhaps are listening to this and you're an owner of an establishment in the Spokane area and you want to have some Linux nerds, hit me up. Chris at Jupyter Broadcasting dot com or boost in. We could always hold it at your facility. But right now, the default is Iron Goat Brewing, Spokane, Washington, September 16th, Saturday at 1 p.m meetup.com slash jupiter broadcasting i'm finally making good boys of course i'd love to have you there but i know i just dropped this on you like a bomb but it's i gotta do it we're gonna i think we're gonna take the rv we haven't made 100 decision there but i think we'll camp in the area oh fun we've got a little time right it's um it's the third week in september and that's about four weeks from now i think i think the weather should be pretty nice and mild at that point

school's back in session so maybe the area is a little less busy in some places and uh iron goat brewing looks like a pretty great place we had a few other good suggestions but again i want something so that way you know kids can come anybody who's interested in this stuff or wants to talk about linux or anything like that can join us so details at meetup.com slash jupiter broadcasting love to have you there and then not too far after that your fest is back buddy linux fest northwest october 20th through the 22nd yeah it's back and we have the official sponsor price sheet up on the LinuxFest website. We'll link to the PDF right here in the show notes too.

You know, it's a crappy, crappy, crappy advertising market right now, especially for the Linux and really high-end technical niche. It's devastated the sponsorship opportunities. And of course, LinuxFest being off for COVID and now coming back out of cadence, it's also tricky for sponsorships. And we had a discussion amongst the team and we decided the right approach is to really make the community sponsorship opportunities approachable. So if you're in this space and you want to have a nice presence, a good presence, get something seen at Linux Fest Northwest, they have

tiers for pretty much every price point. And I mean, I'm talking under a thousand bucks for some of these and you can get nice expo hall placement logo on the website t-shirts for your members you get identified as a silver sponsor for the 750 package they have gold tiers and title tiers and Friday party sponsors and you can sponsor the Saturday party which is a great deal and then of course the raffle donation too there's an opportunity there even if you don't want to contribute money but maybe you've got a cool little gadget maybe you've got some swag that you'd like to be involved in the raffle they have information about that as well really looking

forward to this and i think it's a great opportunity for the community to step up and make linux fest as good as we want it it's a real opportunity here to kind of create a new standard so links in the show notes at linuxunplugged.com slash 524. But let's get into our topic today. And to set the stage, we had to get NixOS up and running on Linode, which they do not offer a default image for. And I went shopping, and much like the laptop situation, I wouldn't say there's a standout VPS host that's offering NixOS right now. There's a few that I recognize on here. I've heard their names before. I don't know I'd want to run my business infrastructure on any of these though. And of course, we prefer to use Linode. Linode does have a guide for this,

which we'll have a link to in the show notes. But there's a strange gulf in the VPS VM world for supporting NixOS. And I don't know, Wes, you've really fallen in love with it. You're way down the rabbit hole now. What's your gut tell you? Why do I not have options on DO and Linode and all the other places to go spin up a quick NixOS VPS? I wonder if there's been sort of a simplification. Like you touched on, there's sort of this matrix you have to choose the right quadrant of, like the diversity of the, especially the Linux options, as maybe some of these providers have become more cloud-like

and focused on Kubernetes options or serverless or managed databases. I think maybe there's been more configuration space opened up there, and they've kind of simplified, still a robust set and lots of stuff, but maybe there's more interest in having CentOS Stream and Rocky and Alma and less these days on having Arch and NixOS. Now, NixOS and Arch are pretty different, especially in the server context,

but in terms of sort of niche non-standard use cases and the intersection with people that are able and willing to get it going themselves. that are able and willing to get it going themselves. True. Although, doesn't it seem like you could be almost a NixOS-first platform where the Nix stuff is an implementation detail, and the platform is doing all of the Nix orchestration, and as an end user, you're just getting the end result, essentially. Maybe it's a web server, maybe it's not.

Maybe it's something more headless serverless like to me it seems like yeah you could you could build all these toolings like all these vpss have different tooling for deploying a lot of them have their own scripting and all that kind of stuff but if you were to build something today you could actually build it around nix right and and the end user could be sort of unaware i mean yeah that's sort of a separate you know there's like the offering to the end user and then there of unaware. I mean, yeah, that's sort of a separate, you know, there's like the offering to the end user. And then there's like, how do you make it all work yourself? And NixOS is a implementation detail seems,

seems like it could be pretty powerful. But as you say, that might be what you would do today. I think you probably need the, there's also sort of the, you know, we've seen it as the number of people writing in, the number of folks talking about it, the general sort of interest in Nix seems to be up a lot in the last few years and even more just recently. So it might just be too that they're,

you know, we're not yet to the point where there's enough of a push on enough of these places that it matters. Plus, you know, Nix, the Nix community being what it is, which is just magical. There's a lot of options from the, you know, like the Linode guide has you boot up, you know, lets you load an ISO, a custom ISO that boots the Nix installer and just installs that way. But, uh, Nix has lots of options like the, the lustrate stuff to be able to take over a host or to use the same host to install, or there's NixOS anywhere, which we'll talk a little bit more about today. And I've talked about before. So I think it's, it's also well served by like, it's just, there's a lot of ways to get Nix running. is true it's sort of like the community has has built their own way but you touched on it there wes and bren i'm curious on your insights on this it feels like nix is sort

of victim of a typical cycle we see in the free software community and i think maybe in the tech community in general but it's sort of annoying that everything has to go through this cycle and i wonder if you agree. So a project, it builds and builds for years. Hardly anyone notices. This happened to Mastodon. Tiny, tiny numbers until Twitter screwed up. And then all of a sudden, people realized after a decade that Mastodon was a thing.

Right? So it goes through this period where it kind of builds more recognition amongst its hardcore users its diehards and then we seem to like cross into this new level where we just spend years debating if something is hype or not and if it has value and we compare and we spend a lot of this time comparing it to everything else and then after however long that period takes which seems to be unique to each individual cycle sometimes it takes a decade sometimes it takes a year. Then we kind of seem like maybe then the commercial companies and the hosting companies kind of start to jump on.

And then we just have to go through this process, Brent. And it seems like NixOS is maybe on the slower end of that cycle. I wonder if it's because it's such a paradigm shift from what everyone has been used to. I think there's part of it to that. The users need to be ready for it. And if we're not ready for it, or there aren't enough usability guides or YouTube tutorials or that kind of thing surrounding a project, then the adoption is slowed certainly by that. And if it's a community first distribution, then who's thinking about this stuff, right? The kind of periphery that makes a project more familiar, more common. That said,

once it gets going, like we've seen with Nex in the last couple of years, once it gets going, there's just relatively no stopping it, it seems, as people jump on and learn more about it and just keep going and going and going. So I will be curious to see, you know, even what the next two years brings for Nix and NixOS. It seems like some pretty bright stuff. Yeah, I think your insight there too is the user ready for it. That's one of the big determining factors of how long these cycles take is people have to come along to utility of it. And not everybody's convinced that an immutable or composable system or a reproducible system is even necessary for maybe 80 percent of the tasks they're doing. And we, you know, over time, as the project gets more capable, as the tooling gets more capable, the tooling gets more capable it fills more and more edge

cases it addresses more and more use cases and it starts to absorb it goes from like well it's only good for maybe 20 to 30 percent of your use cases to now it's like 80 percent of your use cases and we just that transition just takes time i uh someone clued me into a website that i think is interesting i think it's kind of a flawed metric, but it's just like a little fascinating to look at your favorite projects in this way. And it's a star dash history.com. It just looks like GitHub star history over time. Uh, that's like a flawed metrics to see how good a project is or how popular it is. But what is fascinating is to at the very least see like

the big skips and bumps in some of the projects that you're familiar with. And for us, Chris and Wes, who we kind of keep our finger on the pulse of how projects are doing and their popularity and where they are in their development cycles. It's interesting to see. Occasionally you'll see in a project that's just like massive spike and, you know, maybe they got to the hacker news front page, or maybe they came out with a new version that all of a sudden solves the problem for a lot more users, or, you know, they made it to some show, like maybe Linux unplugged. And it's an interesting way to look at a project. I think, you know, play around in there and check out some of your favorites and you can compare them to, which is kind of fun, but there's some insight there.

And I just wonder sometimes, you know, if you've got a favorite project, even as just a one user, what can you do to kind of spread the love? And I think there's certainly some stuff you can do, you know, tell friends and family, There's certainly some stuff you can do. Tell friends and family, tell co-workers especially, and get the news out for the stuff that is worth sharing to other folks. Linode.com slash unplugged. Head on over there to get $100 in 60-day credit.

That's on a new account, and it's a great way to support the show. And you can check out the exciting news. Linode is now part of Akamai. Yeah, the Akamai, but all the tools that we love, the stuff that we just used to build a new NixOS system on Linode, all of the cloud manager, the APIs, the command line client, that stuff that helps you build and deploy is there. But now it's backed by the power of Akamai and their global reach. And they're expanding their services to offer more cloud computing resources and tooling and more locations, giving you more reliable, affordable and scalable solutions for users, individuals, small businesses or even large enterprises. I mean, it's Akamai and they can really deliver. And as part of Akamai's global network of offerings, data centers are expanding worldwide. They're going to give you more access to's global network of offerings, data centers are expanding worldwide.

They're going to give you more access to more resources, including some micro data centers. They have all the plans on their website. It's so exciting. But the bottom line is you're going to be able to help grow and serve your customers or your project or your community. So why wait? Go see why we build everything on Linode. Go experience the power of Linode now Akamai.

Go to linode.com slash unplugged. That's where you get the $100 and learn more about how Linode Now Akamai can help scale your applications from the cloud all the way out to the edge. Yeah, way out there. I think like, you know, Brent is pretty much the edge. Canadian wilderness. Find out more and get that $100 while you support the show at linode.com slash unplugged. Now, as we sometimes do here live on the show, we've got us a Linode box that, Wes, you've configured so that folks can play with, but what are we doing today? And what the heck did you do to this box? Yeah, it all started out as an Ubuntu box that Chris had spun up and

said, yeah, here, Wes, here's your Nix host. Have at it. Yep. You know, I asked him first, do you even care what distro I put on this first? He's like, no, I don't. I don't even care what you start with. Yeah, and so if you follow the Linode guide,

which is a nice guide and has some stuff, you probably want to take some kernel command line options and stuff regardless of how you get Nix on there, and we'll have that linked. You know, it's kind of a manual process, which is totally fine, especially like they have an option where you can make a Linode image, you know, that's sort of like a pre-saved image that you can then deploy copies of to scale out or spin up new hosts, you know, as a base NixOS image that you then further customize. So that's where they're kind of thinking of it, like how do you solve this problem moving forward. But with NixOS Anywhere, you really only need, you know, a virtual machine that's running Linux that you have,

you know, root access to. And you probably want to set things up. So it's got, you know, like either you log in as root, and that's allowed, you got all the keys set up, or your user has like passwordless sudo, and you've got keys set up. But other than that, you write out your NixOS config, you write up your disk partitioning setup, and then you run the NixOS anywhere script, and you just point it at the, you know, whatever the user and SSH host name or IP address is, it's going to go in there, it's going to go figure out if you've got the NixOS installer binary script already present on the system. And if not, it's going to go in there. It's going to go figure out if you've got the NixOS installer binary script already present on the system. And if not, it's going to K exec into a little NixOS installer environment, but it's got a clever script. So it'll also go look at your

host system and like copy any SSH keys and stuff that you've set up on the host system into the new environment, which is a nice little touch. And then of course you can configure that. You can build a custom K exec environment if you need to. But what makes it so nice is it just feels like it's it's just one step of the script where it goes in there SSH is K execs, the script just waits for the host to come back online, the host just comes back online, opens SSH starts listening script SSH right back in and then gets to work. What's especially neat is it uses the disco tool. in and then gets to work. What's especially neat is it uses the Disco tool. And if you've installed NixOS by hand before, you'll kind of notice, right, you've got your configuration.nix, that's all declarative, you just kind of tell the system what it is what you want.

But you still have to do the disk partitioning stuff. That's all imperative. It's the old school, it's get out F disk or parted, you know, manually make your partitions. And it's a step that a human has to do where you have to carefully script it yourself. Disco is an attempt to fix that by bringing your disk configuration into your NixOS configuration. NixOS Anywhere hooks in and uses Disco via Flakes to then go once it's got your host up you're in the KXX environment so you know all you can unmount and format all your drives you're not relying on that for your OS anymore. Runs Disco to build out your configuration however you want it. Then Disco handles formatting, partitioning, and then mounting everything.

So it's all set up and mounted under the slash mount point. And then you just run the regular NixOS install, right? You're like, oh, here's the configuration I told you about. Please go put it on this mounted file system I've got set up for you and you're done and now your ubuntu box or whatever it was is now a nix os box and it even has your ssh key yeah yeah exactly it's all it's all configured and then like whatever configuration you applied is is there uh so that provides a really nice way to you know not have to fuss too much or even worry about the specifics i think that's's part of the power too, is you're, even though Linode provides a lot of great options and other cloud providers have similar sorts of setup, there's ever so slightly different.

And for the most part with NixOS anywhere, you don't have to care about that. If it's got SSH and it's running Linux, you're probably good to go. That's so great. And so, you know, an immutable OS is probably a pretty clever base for a server OS. You can specify exactly how you want the system built. Now with Disco, you can even specify via a config file how your disks are laid out, and it'll generate that system for you. And you can rebuild that system at any point. It'll generate the same exact system for you. So that's a pretty good starting spot for a server OS. And I've often wondered if maybe some of the more frequent issues where somebody gets in and they replace a file in Etsy or they add a user account,

would an immutable OS always solve those particular problems? And especially if you didn't rebuild, if you just rebooted. And that's where maybe something like an impermanence module for NixOS could come in that would essentially create a completely static system. And I think the way this works, Wes, but you're going to have to tell me if I'm wrong, is the impermanence system is essentially just wiping clean the system every reboot. Is that how it works? And so when it comes back, it's coming back from a predetermined state every single time. to work. So you get the sort of base immutability where you, you know, you build a whole system closure, you've built like a little hermetically sealed predefined system via Nix and the NixOS configuration. But there's still a bunch of stuff that exists outside of Nix on your box, right? Like, you can add users, maybe your users put a bunch of Docker compose files in their home

directory and started services running up that way, which is a popular way to do things. But that's all that's outside of Nix, which is nice. But it also means if that got deleted, if that hard drive fell off, you know, if you redeployed, that kind of stuff would all be gone. But it's easy to forget about, especially if you've got a box that's, you know, long running, maybe it's running some database stuff or an old legacy system and you're trying to keep it up to date, but you don't have everything fully documented. People have set things up. There's often all kinds of little like tiny things you do on the first time, like, oh, you got to get your certificate set up right.

Or you got to touch the special file or, oh, you know, like the automation got it almost there, but then we got to change this file permission and this one thing to like actually fully get it working. All that stuff just like kind of gets forgotten. And one approach to preventing that, you know, we've seen is just redeploying, right? Like you kind of just, you just erase things, you treat them like cattle, you shoot it, it's dead, new host spins up from your configuration and everything else is wiped out. configuration and everything else is wiped out. But there are still, you know, there are still long running hosts out there, especially for things like databases, things that need longer uptime, things that don't work in that cloud sort of auto scaling stateless service model. What about if you just wipe your system at reboot? Like you kind of take that, like I can wipe the system and rebuild it to a quicker feedback cycle level. And it's an interesting perspective shift to just think like all your logs,

any stuff that you've configured in Etsy password that isn't in your configuration. You know, like there's so many little things that just disappear when you do it this way. And it's fun to have to switch things. It's kind of like with a firewall where you switch on deny by default. Yes. It can be kind of painful, but that's the secure way to start. And then only allowing what you need. This kind of takes that approach to the state that you're storing on your system where things get wiped unless you explicitly set them up so they don't.

It seems like a really bulletproof way to just take a system that's been compromised and just go back in time and go back to a state before they had their files on there, before they had an account, before they had replaced a binary, and just reset or a configuration change. So what we thought we would do is put this to the test, and we want to release the login information, and we've given this user sudo permission for some reason, and they'll be able to log in. Now, we're asking at least a couple of caveats because there are some limitations. We're asking that you not delete the boot files

and the file system partition. The kind of attack scenario I'm thinking of too, that's just really just a fun experiment, is persistent threat, somebody that wants to use your box to mine some crap cryptocurrency, somebody who wants to take data off your system, somebody that wants to use it

for a command and control relay. You not really uh there to be detected they want to get in your box they want to persist and they want to be able to use its resources without you ever even noticing and ideally without your hosting provider ever noticing and they're not really there to be destructive they're not necessarily it's like the early dos and windows viruses were destructive but then later on they figured out there's more money to be made by keeping the system alive and hiding their presence. So, if you had a system that was impermanent like this, where every reboot, it's a clean slate, that probably

takes care of a lot of issues. Not everything, but a lot of issues. So let's test it. We're going to give our live audience root login, or pseudo login, with access to the box. We have got HedgeDoc running on this system. An absolutely critical web app that we use every single day for every With access to the box, we have got HedgeDoc running on this system, an absolutely critical web app that we use every single day for every show that gets produced, and for internal notes,

it is a real-time, collaborative, markdown document editor, the best of the best. And we have that running on this system, and our hope is that after our live stream totally trashes this place, without deleting the file systems, without deleting boot files, we can do anything else. We can recover this. Is there any other caveats you want to put on it, Wes, or does that cover it?

It's kind of made possible by the fact that Nix only really needs the boot files in place for the firmware and all the stuff before it to get the kernel booted, and then it needs access to the Nix store. So, don't delete the boot access to the Nix store. So don't delete the boot partition or the Nix partition.

But otherwise if you want to add new partitions or you know I think pretty much anything else can be fair game. Applications. Your own user account. Don't crash the box for other people too. But alright chat room. Let's go for it.

You're off. We've given them a login in but all right chat room let's go for it you're off we've given them a login in our live matrix chat room we'll check back in here in a minute and see how bad they've destroyed the box so wes what's the process of getting nix os to be not just immutable but to be impermanent is this just add a line to the config is there additional software you had to go did you have to jump into flake land how How hard was that? Well, I don't think, flakes definitely aren't strictly needed, but some of the tooling, like NixOS Anywhere is a pretty flake forward tool. And you can use the NixOS impermanent stuff as a flake. So flakes work fine, but the classic way works fine too. The biggest change, like, so if you start with a system where you're, you know, you've got it installed and set up on whatever provider, you know, in your virtual

machine environment, you probably have a formatted root partition. Maybe you've done more partition, but we'll just assume a simple setup with like a boot partition and a root partition. That's the rest of it. You need somewhere where your Nix store is. Cause that's where all the files of consequence are going to live. That's where the system closure lives. That's where everything really. But you also have, that's probably shared just on your root drive by default. So it's all kind of co-mixed with the rest of the stuff. So you need to put the Nix on its own partition.

It needs to somewhere that it can live that's separate from the rest of the files on root. Okay. And then you also probably need to separate boot out. I mean, those could be on the same partition or multiple, or you could do this with CFS or ButterFS subvolumes. Not shared with root. Yeah, but you don't want like slash var and Etsy and user and all the rest commingling with slash nix,

at least in terms of how it's actually stored on the underlying disk. All right, we're checking in and our web app is already nuked. That took them about 30 seconds. They brought down our web server over there. I wonder, Brent, can you still log into it? Can you jump in? Yeah, let me give it a try here. See if we can find out what's going on with this poor thing.

And then report back in the chat room. It looks like someone stopped the systemd service. That's an easy one. Oh, good. So I started that back up. Chaos. All right, systemd's back up one. Oh, good. So I started that back up. Chaos. All right. System D's back up. Nice try, Chaos Monkeys. They went right for system D. And with that, our web

application is now back up and running. Nice job, Wes. Didn't even have to flex the immutability muscles. Just boom. Restart that system D. You can do better chat room what do you think the feasibility would be of actually deploying something like a matrix server using impermanence yeah okay so there's kind of a couple stages to impermanence is you start with with um you know getting things separated and then you need something that actually wipes your root setup and there's a couple approaches one is you mount root as a tempfs, which is basically like a RAM file system. And that's a popular one. So it's just like from the start, when an early boot up root gets mounted, it's going to be empty because it's a brand new tempfs. But there's some other neat setups that can be pretty nice if you want to use something like butterfs or zfs,

is you can set up your root like subvolume, and then snapshot that in its empty state. And then so just like every time you boot, whatever was on there before you just roll back that partition or that sub volume to its previous snapshot, and then you're right back to your clean slate. Or you can even have something that manually went and like tried to RM everything in there every boot or before you shut down or whatever. So there's a variety of ways, but you basically end up with a system where as Nix, like as the Nix init stuff is all getting going, it's going to find an empty root partition. And what's so neat about that is,

if you look at a Nix setup, when you look at the command line, it's a little like the kernel command line, it's a little different. You don't see all the stuff that you normally kind of see, and you especially don't often see. You can still, but by default, it doesn't even have the UUID or the root equals flag to kind of tell it where to go find stuff. It just tells it like what in it it's, it's looking for. Cause you can bake yourself an in it RID that already knows for the system, uh, where all those things are, or it can like look by path. Uh, it's got various clever methods built in. Um, so then it can just like mount the root and then it knows exactly the in it it's going to go from inside the next door. And then the rest is all just like pre-scripted by the build system.

So that it just like, it knows which system D it's going to start from the Nix store. That knows which services they're going to start because they're all built and linked together. Wow. It's putting it all together that fast. Yeah. That's the nice part about the Nix stuff is once you've built the closure, it's all just ready to go basically. That gets you the root system. That gets you the system where every time you reboot, you don't have any state, but that, that can be a little rough if you do need state,

because you might still want some stuff. So Brent, you noticed one consequence of this when you were testing things out for us. So what I noticed was a bit of a different behavior, Wes, and I blamed you instantly, of course, was I had logged in via SSH and then you had rebooted the machine, you were setting things up. And when I tried to log in again, my system complained that some keys had changed on the other side and that I needed to kind of, you know, delete the known host from my list and kind of authenticate that part again, because it had appeared to be quite different. And while this seems like a brand new machine, and I guess in many ways it is. Yeah. So some of those files that you forget about and that are just going to disappear out from under you are things like, you know, your systemd machine ID or the SSH host keys that got generated for your box when it booted the last time. or the SSH host keys that got generated for your box when it booted the last time.

And so some things like that you can live with or work around. But if it really is going to be a longer running machine, you probably want to do something about that. And that's where the NixOS impermanence module comes in. Because confusingly enough, it's actually there to help you add a little permanence. Seems like a different name change would be nice. I just got booted from SSH here. Do we have? All right.

Do we have a problem? We're going to follow up on that, Wes, but we got to do a check in here because the web app's down again. D message is going off like crazy. And Brent just got booted out of SSH. So things are not looking good on the box. Yeah. I wonder, did it reboot?

Yeah. Let's find out. Maybe we need to reboot it. Yeah, it might be time. Let's see if it comes back after a reboot. So SSH went down. I wonder if the box itself went down, but this is the test. Well, I'm seeing in the messages here I think someone ran a NixOS rebuild switch. So I'm curious if there's a

Nix confident listener trying to pull the rug out from under us here. That's the route I would have gone. That is... Those are the powerful... Yeah. Well, all right.

Let's see if we can get it... Oh, Gamma says the box is actually still up. All right, we got to phone a friend and find out if we can get the system back on. What do you think, Wes? Can we get it back? We're about to find out. Let's give it a reboot and find out what we can get the system back on. What do you think, Wes? Can we get it back? We're about to find out. Let's give it a reboot and find out what happens.

Let's do it. If NixOS was involved, it's possible we'll have to boot to an older configuration to get back to where we were. But we're going to find out. And we're back. Nice.

How hard was it? What did we have to do? Well, thankfully, the system wasn't destroyed too much. Now, obviously, if you make too, you know, there's a lot of ways to really break this thing, but the changes that have been made by the audience weren't enough that we can just roll back to the Nixos generation before that and have everything back in action so i noticed that we had people do a nix rebuild we had folks knit uh totally destroy ssh and just completely reconfigure and change the keys uh we had people nuke systemd

on us and all of that was reversible yeah now the one thing if you do make a new nixos generation that becomes the primary that that will change things, right? That's because you're using the Nix tooling to do what you want. Right. And you're doing that with root bunch of services outside of that, if you're modifying things in place, if you're updating user details or, yeah, deleting SSH keys, that kind of thing, then, yeah, as long as the system can still reboot on the Linode side of things, it'll just come back. Well done, Wes Payne. I think that's a win.

I mean, we gave a whole chat room of people pseudo access to that box and they definitely messed it up system D was cute that was cute wrecking docker was cute and breaking SSH was cute that was all those are all good ideas guys all right yeah I think now should we turn it over and let them really destroy it

yeah what do you want to do well just let's uh let's have at it, gang. No limits? No limits now? Yeah. All right. All right. No limits as we round it out.

We'll give you a few minutes to totally break it. You should. I killed any scripts and stuff that were going, so there should be folks, and I made sure that the password for Tux is reset again. So you should all hopefully be able to get back in. Yeah. Good luck. All right.

We'll keep an eye on it, and we'll follow up here in a few minutes and see if they've completely destroyed the box. Collide.com slash unplugged. Go there to support the show and check out the demo. If you work in security or like me, I used to work in IT. If you work in IT and your company has Okta, you got to listen to this. You've noticed no doubt over the past few years the majority of breaches hacks whatever you want to call them they all have something in common right unfortunately it tends to be the employees and staff doesn't it you know sometimes the device gets hacked because of unpatched software sometimes an employee leaves sensitive data behind a thumb drive a laptop leaks credentials sometimes they get their account phished that stuff's pretty

unfortunate and it's not really their fault it's not really anything that you know they've done wrong intentionally it's just that the solutions are supposed to prevent these breaches don't really do the job but it doesn't have to be this way you see you can actually have your secure devices that only connect to your apps when they are checked out and approved. That's where Collide comes in. In this world where you're checking before they connect to your systems, phished credentials are useless to hackers because the system's getting validated. And you can manage every OS, even Linux, from a single dashboard. And you can get employees to fix their own security issues through some really clever systems and messaging without

having to create more work for IT. You don't have to imagine this. You don't have to pretend. This is a real thing that Collide does. It's a device trust solution for companies with Okta, and they ensure that if a device isn't trusted, it doesn't log into your cloud apps. It's really great, and they got a demo at collide.com slash unplugged. Go watch that and support the show.

It's collide.com, K-O-L-I-D-E dot com slash unplugged. Thank you to everyone who sent us in some feedback. Really, really appreciate it. Kind of collected a few here today. 412 Linux sent us a little note about Linux Unplugged 5.2.2. Says, I also wanted to give a shout out to NextDNS. This has been a great parenting tool. It doesn't matter where the device goes on the network.

It's always connected and my configuration always remains. That is nice. I think we had one cautionary tale about NextDNS and now we've had a couple of good positive. NextDNS made it on our list as an audience recommendation, so thank you, 412. That's good to know. I didn't really think about it, but the fact that it follows the device is really perfect. Linux Unplugged 522 was the privacy episode that we did recently, and we got a few pieces of mail for that one. Andreas came in with another privacy tip.

of mail for that one. Andreas came in with another privacy tip. Hey, you mentioned something about email privacy, which tickled my fancy in the latest Linux Unplugged regarding personal privacy. I want to share my practice since I think it might be useful to someone. I registered my own domain with domain privacy enabled, of course. On that domain, I have enabled wildcard support as well, so that whenever I register for a new service, I can just enter a custom email address that's unique to it. This is super useful for shipping, detecting data leaks, and personal data being sold, and blocking everything from one particular sender if needed. Everything is routed to a signup mailbox, so whenever I get an email from someone I don't really know, I can see that the email is there and which email they

sent it to. Probably not the most secure platform I pay for my email service from Fastmail since ProtonMail is just too darn expensive for custom domains and wildcards. Best regards from Andreas. That's a nice little workflow there. I like that he's kind of thought through that a little bit better than I have. I think that's an area that I now feel like a little, like I haven't really given that proper thought through. Appreciate that feedback, Andreas. Yeah, I started using, I guess, what my service provider calls plus addressing to do this very thing. And it confuses some local businesses that like, wait, how's the name of our store in your email address? They're like, wait, how's the name of our store in your email address?

That happened just the other day. But I appreciate this folder approach. I didn't quite, you know, I've kind of set that in pace, but didn't quite know how to manage it in a way that was going to be obvious. So thanks for the tip. I like, Sid's got a project on his hands. He writes, I'm getting my first home lab machine this Tuesday. Oh, how exciting. It's a Lenovovo think center i5 6400t i'm planning on running image plus pie hole and a few other things on it it has a one terabyte hard drive and a 200 which is i think spinning rust and a 256

gigabyte ssd for backups of my image data i was planning on creating an lvm setup and running daily backups by snapshotting the partition then having REST to encrypt and upload things to AWS S3 or Cloudflare R2 or storage or whatever. What do you guys think about this? Is this a good way to handle backups? Should I maybe consider Project Stratus instead of LVM? And by the way, I am writing in from India. I don't think there's a viable way to get boost here, but I'd love to know if anybody has some tips for that.

Love the shows. So two questions in there, really. Uh, he's got a question about backing up and where to go, I think in there. And then a question about, uh,

project Stratus versus LVM. I'm thinking our take on this show is LVM over project Stratus at this point, right? It's not, I mean, project Stratus just doesn't seem like it's been a huge hit. You know, I checked in with Alex about this one. He couldn't join us here today, but I thought, you know, I can get a sneaky little note in from him. And I sent him this piece of feedback and he said, Stratus, no, don't do that. Don't use Stratus.

So I think the self-hosted show's opinion is also lining up here yeah it i think lvm2 it's just so tried and true uh we have a link in here speaking of alex's perfect media server.com and he goes through a storage stack and he also goes through a backup stack the 321 backup methodology applied to different offsite backups definitely give that a read we'll put that in the show notes so you can get directly to the backup section i think the key part you're talking about here is kind of approach. If you think Stratus is the right thing that you need because it has the feature set that other solutions don't, or it meets your needs for other ways, then okay. I mean, check it out, look at the documentation, play with it, see if you feel comfortable using it.

But if you're just looking for sort of a rough level of the things it offers, yeah, it feels like it's probably the niche case, the least tested, the least well-known. And if you have to ask, probably better to use something that's a little more standard. I think I have a question here about methodology. It seems backing up the entire partition is the approach here. But considering it's image and pie hole and do you really need to go that far can you just do like a file based backup or a block based backup instead of the entire partition good point yeah i don't really know why you'd want to back up pie hole you could back up the config

you could you know i mean maybe there's something in pie hole you'd want but image i'm worried about he's got a one terabyte storage drive and a 256 gigabyte ssd and i'm just worried that image is going to eat that not right away but it's a silent eater right because every picture you're taking on a modern smartphone is huge and every video you're taking is like generally 4k video now and they're huge so uh you can very quickly just accidentally fill up your disk so that's something i think you're gonna want to keep an eye on and I think Brent, you've just tuned into something there too. Why are you backing up the whole dang partition? Maybe you got a reason,

but we would postulate to you that you just consider backing up the data. I will say the, the more fancy you get on the backup side in terms of like just picking and choosing, or, you know, then you do need to have an equivalent reverse step in your restore process. So if you only back up the data,

then you've got to reset up the stuff to get PyHole running and then load that data in. So I could see, if you're just starting out and you know that you've got this, it's not going to move servers or it's going to be on roughly the same hardware, at least if you do the whole thing, you know you've got everything inside it. You can go grab those files,

even if you don't restore that whole partition. So I think it's a nice, safe sort of like, you can start there, and then you can whittle things away, more optimize what'll play with it, or as long as you've got tested backups, and then you have some room to modify it. That's a good tip.

I want to make a little mention that Podverse has a bounty out there. We'll get into the boost, and then we'll follow up and see how destroyed our server is. So Podverse, one of our favorite podcasting 2.0 apps, it gpl it's cross-platform they've got a web version they've announced a bounty of twelve hundred dollars to anyone who can deliver them a finished version of android auto for podverse they really want to get across the line they got carplay nailed the android

auto is tricky and so they're willing to uh up a $1,200 bounty for that. I'll link to their GitHub, Podverse, P-O-D-V-E-R-S-E. They're also, that's the player that we use on the new community-built Jupyter Broadcasting website, too. We like it a lot because it's a web player that supports chapters. So we'll have information about that. And hopefully someone in the community can help Podverse get Android Auto going because I think that's kind of like the last major piece that people are really missing

in terms of overall app functionality. And it'd be great to get them across the line for that. Boost to Graham. We got some boosts this week. Tomato Deer came in with a big old baller boost for us. 333,333 sets. Hey, Rich Lobster! Coming in with

Breeze, actually, and he wanted to send us an email and put it on our radar, and Brent, you have broken it all down for us. It came in via email because it is quite an email. I will say, though, let me just make it a short

version. Let's see. Sometime in 2015, I discovered Docker, and my self-hosting journey really took off, as many folks. So in about, now that's 2023, I rebuilt my server, got a bunch of ZFS mirrors, and managing over 2,000 articles in Wallabag, and successfully migrating that instance across a few versions of Postgres in the process. Unfortunately, a recent upgrade that included Postgres 15 broke my Wallabag. Oh, shucks.

I've been using a custom-made, unmaintained Wallabag Docker image for years, and I have totally borked that setup from the beginning not really understanding the finer points of containers my suspicion is the database driver that wallabag employs cannot speak postgres 15 i've not upgraded and things are just going bad so i am boosting to post a bounty to the jb community 300,000 sats to get a NixOS config for Wallabag 2.3.8 behind Nginx and Postgres 14. And further bounty may follow, given how incompetent I am. That's incredible.

I also will, Jay from Philadelphia, I'll send you to the Knicks nerds chat room in our matrix too. There's a lot of helpful people there. I know somebody out there has this working though. So you may, you may, somebody may cash in on this bounty in self-hosted one Oh two. We talked really recently about wall of bag. And I heard from a bunch of folks that are self-hosting it after that episode.

So I bet you'll find somebody. And thank you for the baller boost. We really, really appreciate it. a bunch of folks that are self-hosting it after that episode. So I bet you'll find somebody. And thank you for the baller boost. We really, really appreciate it. And it's been a while since I've seen somebody coming in on Breeze, but that's pretty great. Cyber Gray boosts in with 46,912 sats. I hoard that which your kind covet. I don't know if the previous boost went through,

so I'm sending this one too. Thanks again. Cheers. Well, thank you, CyberGrey. I guess not. I don't see another one. So thanks for trying again, because it'd be a shame not to have heard from you. Yeah, we appreciate the support.

Active Shadow came in with 35,000 stats using Fountain. He said, I wanted to say hi and do my part to keep supporting this great show and Jupyter Broadcasting in general. Thanks for everything you all do for open source. Well you very much active shadow we really appreciate it complete noobs came in with 33 333 satoshis it simply says hey karma from complete noobs.com that's a great domain there's not very many good.coms left i want to know how long you've owned that complete noobs is that an early snipe or did you uh just recently score a dot com that's that awesome that's a rare thing these days todd from northern virginia comes in with a row of mcducks 22,222 sats from the podcast index is that looking up for all the duck my favorite part of linux unplugged is when chris pulls out his soap. That's worth a row of ducks right there.

Careful what you wish for. Yeah, be careful. I saw this boost come in earlier this week and it had me chuckling. So thank you for that. Eric Boosin with 22,222 cents. This old duck still got it. Huge shout out to Tailscale. I've been using Tailscale with Headscale for a while now, and it's been working great,

and I really love it. With NixOS, I configured Headscale in AdGuard with custom DNS, with a split DNS where the AdGuard DNS has the home internal IP address, and the Headscale DNS has the Tailscale IP, and it works great. Brilliant. That's really clever. Also, during the pandemic, we had to set up a VPN to secure access to our internal services. Well, the person who implemented it really liked OpenVPN, and this was pre-tailscale

pricing v2. Since he was implementing it, I gave up on using WireGuard-like solutions. Well, this past week, it stopped working and he's now gone. It wasn't certs or anything obvious, but it was a huge productivity loss. So we ripped it out and put Tailscale. It's now easier to understand, and I finally understand how to write a proper ACL with Tailscale. Looking forward to managing those in my personal deployment next. That is fantastic.

You know, we're really talking seriously about how are we going to integrate Tailscale with our backend So looking forward to managing those in my personal deployment next. That is fantastic. You know, we're really talking seriously about how are we going to integrate Tailscale with our backend. Just as we spin up a system, it just becomes part of the tailnet. And each one of us would have access to that tailnet in theory and then be able to get to any of those systems through that tailnet. And there's more and more ways now to just deploy a system. And it just basically has Tailscale ready to go out of the box. We're spoiled for Mesh VPN options. You also got stuff like Nebula and various others. Yeah, right? Like, if you have

any use for a Mesh network at this point, there's no reason not to have one, and it really does make things simpler. Good choices out there. The Leaky Canoe came in with 21,949 sets. Hey, got any tips for learning file and directory permissions in Linux? I just emptied the Albie wallet, so please excuse the odd numbers.

And cheers to a great show. You know, there used to be a great website tool for this. I wonder if I could find this or if anybody knows of it. But it was an interactive app to learn what the different permission settings did. And I don't know if I remember the name of it, but it was a really handy tool. Yeah, I know there's a few good ones. I found a decent one just here, quick, Chumad Calculator. You can kind of just like check the boxes and it'll show you the final octal form.

That could be a good way to learn. You might just want to play around in the shell too. You know, make yourself a temp directory where you can play around and then use Chumad with the non, you know, just start playing with it. You can use stat to figure out what the octal version is pretty easily and see what you can or can't access or what you can do to files. You also, you don't have to use octal numbers with Chumad if you don't want to. You could actually do like user is read right x group is read right and you can actually spell it out in natural language that'll all be in the man page and that also is another way to learn what those numbers mean is you can do it with like natural language and then over time you can

see okay i've set that do an ls what numbers did i get that's another way you could play without even going to a website it's not often we say this on the show, but with Chumad and Linux permissions, the man page is probably a really good place to start. It actually probably is. It's something that is simpler than it seems on the surface. When you come at it from Windows, NTFS has what's considered by us Linux users extended attributes with lots of different permissions and attributes you can do in ownership models. Standard Linux, like extended for permissions without extended attributes are really simple. It's user, group, and world. And that's really all you have to worry about. But you'll figure it out and follow up Leaky Canoe. I think that's a great thing to wrap your head around.

Yeah. You're on an exciting journey and there's lots of fun stuff to learn. BamHim 182 comes in with 21,226 sets from Podverse. I'm finally sending a zip code boost after giving up several times of getting a JB membership. Thanks for your courage, guys. I've fallen deep down the NixOS rabbit hole, and I absolutely love it. I've converted almost every box I have at this point, around 10, ranging from old Chromebooks to routers and my Threadripper NAS. My last bastion is my home assistant VM on Proxmox.

Oh, fun. Soon. All right, did you bring the map with you, Wes? I sure did. I don't go anywhere without it. Yeah, attaboy. All right, so let's look up this zip code and figure out where he's at. Which doesn't really make sense because, like, I need a West Coast map,

and it turns out here I needed a map of Baltimore, Maryland. Oh, hello, Baltimore. Thanks for boosting in. Thanks for boosting in, and thank you. I'm just loving it. I mean, NixOS adds to the top, but just these reports, hearing more about all the fun home and production setups, keep it coming, folks, because this has been great.

Nev boosts in with 9,001 sats. Say, whatever happened to that email server you guys talked about? Is it still rocking and working, or did you take it all down? I've been honestly considering hosting my own email for some DIY notifications. The number one rule of Linux Unplugged is you don't talk about the email server. You don't talk about the email server. Yeah, we did set one up, and it is actually still running. I don't think it's how we would set't talk about the email server yeah we did set one up and it is actually still running i don't think it's how we would set it up today though right wes it's

well it didn't use nix at all exactly no i uh we have an account or two on there too we probably should shut that thing down honestly we should probably just yeah you know it was it was worthwhile i think it's been you know we have like done a few updates here and there uh so it is all ticking along but we didn't really adopt it and maybe we should check back in and see see what it would be like what if we wanted to convert it to nix yeah how much work is it because i think it was not an insignificant amount of work but on the other hand if we weren't trying to use it as like a full replacement for stuff you know like so if nev's just trying to set something up that's going to be for diy notifications and doesn't need all the

guarantees of like i need my spouse's email to not bounce you know like that kind of stuff there's different problem domains there and especially with something like nix where you didn't have to worry so much about getting all the the state and the ordering right maybe it'd be reasonable it might be fun it might be a fun project, I've just been curious what happens if you leave it online, tend to it a little bit, you know, what happens to an email server like that in the modern era? And so far it's surviving. So, that's good. One thing that

hasn't survived, it seems, is our live box here. I just got booted and it seems like CraftNix has owned up to the problem in the chat room saying, I think I may have accidentally removed Tux from the wheel group. We'll find out. We'll follow up here in just a second.

Just a couple more boosts to round us out. SWAT, 2,317 sats. Wes, I'm okaying your pronunciation from the previous episode and wonders if we use OpenStreetMaps.org very often, you know, with its Wiki-like contributions. We all kind of use it with various different apps on our mobile devices. Oh, true, yeah. But not much on the web, on the desktop.

I occasionally use it on the desktop. My favorite way to get to it, because it's such a long URL, is to just use DuckD, duck goes like, um, quick bang shortcuts. Do you guys know about these? You go bang OSM and you just brings you straight to open street map. You can even do a search in there as well. It's one of my favorite,

it's like a launcher for the web. I really, really enjoy that. One of my complaints with open street map, at least on the web like that is often when they're showing you maps of a place, especially with provinces and stuff, I find their delineations between various counties and things like that really hard to distinguish.

So I find sometimes the, I don't know, map colors and contrasting sort of difficult sometimes. So if I'm looking for a broad overview of something, I can sometimes be a bit of a struggle. So that keeps me occasionally from going there to get that kind of stuff. All right, I'll round us out with a couple of more so we can get to that server update.

Root is good came in with 10,000 sats. Maybe a rolling distro is what is needed for the year of Linux on the desktop. It'd be pretty difficult for Windows or Apple to compete. And the golden dragon came in with a whole bunch of boosts, five boosts, 11,110 sats. And in here, he had a question.

He said, I'm looking into a hotel router for a trip. What I need to run an exit node at home. And how easy is that to achieve? He's talking about tail scale there. And I think that's really going to depend on if you have the resources you want to access, if that box is running tail scale itself,

or if it's on a system that isn't running the tail scale client. If you have a system you want to get if that box is running tailscale itself or if it's on a system that isn't running the tailscale client if you have a system you want to get access to remotely using tailscale that isn't running the software then you just one of the boxes on that same land turn that into an exit node but it'll depend on the application that uh you want to uh get access like jellyfin you could probably just run tailscale directly on the server we got a row of ducks from batvin123 coming with podverse he said i had i heard the golden dragon it suggested a 3d printed lup coin indeed i've got a uv resin based 3d printer and a standard plastic 3d printer i or someone else could design a coin that combines the two technologies very nice i i like that this is, keep this rolling, guys,

because this is something that I'm not getting going on my own, but I very much support. Do we need samples? Yeah. Of course, we've got to do quality inspection. Okay. Brent's going to have to do a little QA, you know. Faraday Fedora came in with several boosts, 6,000 sats in total.

Coming in hot with the boosts. And he relayed my question question on total like how high how hot did you get your home lab he said this weekend it was getting up to 34 c in my home office lab um yikes yeah you can use it to proof his bread though he had it even hotter in there in the 2021 heat dome but he turned off his stuff i'm still curious how hot people get their home labs and if they worry about it. Noodles1232 came in with 10,001 sats. He said, you guys called out for how hot you've run your gear here in Fresco, California. It regularly gets above 100 in the summer, so it daily can be 85, 90 in the house. Power costs so much more during the hot times of the day, so I can't really turn

my AC on much. Running low power hardware helps quite a bit since it kicks off less heat it all works great though i haven't really had any performance issues even when the ambient temps are in the 90s all right very good to know i'm still not like once it gets above 85 i'm still i'm still shutting it down i'm still turning stuff off the node i've left on so the node's been like a 90 degree and it still runs, but I've shut everything else off. Yeah, that makes sense. Paz boosted in 5,000 sats from Podverse.

Hey, JB crew, a longtime listener and first time booster here. JB really changed my life professionally and personally. More on that in a future code or radio boost. Keep up the good and inspiring work. Well, thank you, sir. I'm looking forward to that. I appreciate the support and you listening. Moonanite boosts in with 5,000 sats.

Re last episode's discussion about NFC payments after switching to Giraffine, I switched from iOS and the Apple Watch, where I had Authy 2FAFA, and NFC payments, credit cards, transit passes, all on the watch itself. Turns out the watch doesn't need an active internet connection or an iPhone after initial setup. I switched fully to Girafine OS a few months back, and still use NFC payments on my Apple Watch almost daily. Occasionally, I'll boot up the old iPhone to add new 2FA keys,

but that's all I need it for. That's kind of a fascinating little middle ground. I like it. I have been hanging onto the Apple Watch for iMessage. I never really thought about using it for payments. It just seems so silly, like holding my watch up to the payment terminal. But I also feel ridiculous when I do it with my phone. So one ridiculous for the other, that's probably a fair trade, I suppose. I am going to cut a little short so we can follow up on the server. But don't worry if we didn't get to your boost in the show. We have it in our show doc and we have read it as a team.

We really appreciate the support. We had 24 boosters. You know, I like radical transparency on this now. That in total was 30 boosts. Thank you. I mean, it's incredible that out of tens and tens and tens of thousands of listeners, 24 people stepped up and helped make the production

sustainable and possible for this episode. We brought in a total of 611, 870 sats, which is fantastic. Thank you, everyone. We'd really appreciate to sustain that support for the next episode. We always love getting your feedback, your notes, and your questions that way. If you want to boost in

and you don't want to switch podcast apps, go get Albie. Get Albie.com. You top that off with a cash app or they got actually a couple options directly in Albie now to top it off. You get the sats and go over to Podcast Index.

PodcastIndex.org. Find the unplugged program and you boost in over there. Or go get a new podcast app at PodcastApps.com. Podverse, Fountain, Castomatic, Breeze is listed over there.

You heard some of the names this week. Those are the ones our audiences are picking. And they're getting all kinds of great new experiences in there as well. Newpodcastapps.com or podcastapps.com. And of course, thank you to our members, unpluggedcore.com, for supporting the show directly and the Jupiter Party. Boost! Thank you everybody who boosts into the show directly, and the Jupiter Party. Boost! Thank you, everybody, who boosts into the show.

Now, why don't we check in on that server, Westpain, and see what's going on. It sounds like it's been trashed, because we've got reports of everything getting wrecked. I think something's got wrecked, but I just rebooted it, and I'm back in. Ha-ha! It looks like HDOX is running, too.

I think it lost, we might have lost what was in the persistence stuff as part of what people were doing. I'm not sure. But yeah, the server is online, at least partially. There's our web app. That's impressive. Wow. And we just let them go hog wild this time too. Oh, the logs I saw streamed by, they were scary. Yeah. A lot of error messages in there. Oh gosh. Many. A lot of error messages in there. It's been fun. Gamma's been hacking away here. And Gamma was trying to take a look at just ways to maintain

persistent access to the box and reports that the impermanent stuff did remove them adding keys to the root authorized keys file. So that's one thing. You can add that and it'll be there for as long as it's online, but then once you reboot and things go back to the way you've

configured the system, that drops out. It's pretty neat. That is a pretty hardcore test we just put that system through. I'll give a shout out to a package from Soltros who contacted me via Matrix. He's created the Nix package management script. I haven't tried this myself.

It's a Python script he designed to just kind of help assist with managing packages in Nix. He says it provides a set of functions that allow you to perform package-related tasks, installing, removing, searching for packages, listing installed packages, updating the Nix config, and rebuilding the system configuration. It's nixpkg.py. We'll have a link in the show notes. It's created by a fellow community member to help just manage your software on Nix. It's pretty nice. Yeah.

So after going through this experience, Wes, do you feel pretty solid about using Nix in production? What about in permanence? familiar with Nix, because you really do need Nix to make use of the system. And if you're not going to delete the boot stuff, like Nix, as CraftNix has been playing with, is the easiest way to actually break stuff, right? Because you can get Nix to write a new config that locks the old stuff out, or just never brings up networking, or boots in the least successful way possible. So there's lots of ways to work it via Nix. I think it depends on what you need, how much you have a problem with long, stateful servers, and do you write stuff? If you're already using Ansible

or tools like that, then not. If you're good at writing your documentation, maybe it's less successful. And is your team setting the box up with impermanence in mind, with data separation in mind, with putting NixStore

on its own partition, separating the boot stuff? Are you designing the system with impermanence in mind? That's probably gotta be a requirement. It is. I think it does seem like a really nice way, especially since it's not like,

you know, what the, what the impermanence module, you basically just tell it like, here's all the files I want you to make permanent. And then it uses like mounts and bind mounts and such to, to set that all up for you. So like it manages making it appear in the system as it should.

So it's a pretty low overhead, at least for things like relatively simple and static files to get that made permanent. So it might be a nice way to have a default because it just forces you to remember that you got to put that stuff into Nix if you want it to last. So even when you're trying to do something quick and dirty and just make it work, you can make it work for that time, but you know that it's going to disappear if you don't do it right. For it to stick around, you got to do it the right way, which isn't necessarily a bad thing

at all. Yeah. And, you know, the philosophy sounds great. What is it like in practice? So far, surprisingly workable. Yeah. I mean, we really put it in the fire. Thank you, everybody who showed up live and really banged on this box and made it

a lot of fun we appreciate you joining us if you'd like to show up live for a linux unplugged in the future we do it on sundays traditionally at noon pacific 3 p.m eastern we'll always have that in your local time at jupiterbroadcasting.com calendar if you go to jblive.tv we've got our self-hosted peer tube instance embedded and ready to go. See you next week. Same bad time, same bad station. And of course, we'll have links to all the stuff we talked about, documentation that goes into more detail on impermanence, the tricks and tools we've covered.

That'll all be at linuxunplugged.com slash 524. And there is a plethora, a whole network of shows over at jupiterbroadcast.com, like self-host. We mentioned that. And of course, Office Hours is undergoing a radical experimentation. We encourage you to check that out. And don't miss a single episode of Coder Radio. There's always some great takes in there.

All right, everyone. Thank you so much for hanging out with us for this episode. We really enjoyed it. We hope you did, too. And we hope to see you right back here for another Linux Unplugged next Sunday. Thank you. you