LINUX Unplugged - 494: Updating Our Fiddly Bits

Episode Date: January 23, 2023

Today we are finally taking on a project months in the making, and we're switching to an entirely new generation of Linux tech in the process.

I'm personally pretty pleased with the state of the Plasma desktop, and version 5.27 looks like it's going to be kind of a nice way to send off the 5 series. We'll talk a little bit more about it, but we just had the pleasure of deploying the most latest released version of Plasma in a new setup. And I think there were several times last night where Wes and I looked at each other and said,

Plasma's so handy! We definitely had that. I mean, also a few times where we thought, wait, is this broken? Oh, yeah. But it's both. And I don't, I was thinking about it driving home last night and like, could we have done all that we did in a day if we were trying to do that on Canome? Hmm. Hello, friends, and welcome back to a weekly Linux talk show.

My name is Chris. My name is Wes. And my name is Brent. Hello, gentlemen. Well, coming up on the show today, we're finally taking on a project that has been months in the making. And in the process, we're switching to an entirely new generation of Linux tech. We'll tell you about our wild weekend, what we have been up to.

And then we're going to blow your face off with some really great backup tools that people have sent into the show over the last week. And you're going to have no excuses. You're going to have no excuses.'re going to have you're going to have no no excuses you're going to need this is this is just you saying like after this episode you're going to back up you don't get to complain if you lose your backup no excuses yeah it's going to it's some good tools and then we'll round out the show with some boosts some picks and a lot more so let's say good morning to our friends over at tailscale you know tailscale is a mesh vpn that's protected by watercard and And it is awesome. Flat mesh network, put all your devices on there, you'll have your own private decentralized

internet. I love it. Go say good morning over at Tailscale.com and if you get a chance, tell them the Unplugged show. Are you still hub and spoking? I mean, come on. I hope not. Oof, oof, I used to be that way. Then I found Tailscale. But before we go any further, let's say time-appropriate greetings to our virtual lug.

Hello, Mumble Room. Hello. Hello. Aloha. Aloha. Thank you for joining us, everybody. Got some, wow, got 15 up there in the quiet listening. Extra hailed some today, too. Look at all them. Yeah.

Everybody's so fun when everybody just coordinates. It's nice hey so let's get into a couple of the great links that got sent into the show this week we had um a plethora of borg backup links that were sent in and i knew this was going to happen and i said it in the show i said don't bother sending in your dumb borg links i used to run borg backup i'm not interested in it kids you know i was that guy in last week's episode and then like they do they kept sending the links in we call you three of seven and then as they as they started coming in i started going huh oh maybe people are onto something here and then it started to change

my mind and they successfully the borgers have campaigned successfully and really got me considering Borg backup as my solution. And so I thought I'd tell you about the best of the best that I thought came in. Thank you to everybody who sent these in. A number one with a bullet that I thought solved a big problem with Borg backup is the backend component of Borg backup. You got to have something to have everything go back up to. And that's where Borg Warehouse comes in, a fast modern web UI for Borg backups, central repository server. The Borg backup software is just beloved by our community. They really seem to trust their data with it. And I think one of the trickier components is this repository that you have to set up and borg warehouse gives you a nice web

ui they say it's a fluid modern web ui that you can obviously self-host it's totally totally open source comes a little setup wizard makes it easy to get going and then you have something that you can send everything from a kind of straightforward command line backup client and send your data there to full-fledged graphical desktop applications like Vorta. V-O-R-T-A. And Vorta is one of the other highly sent in recommendations, which is a nice graphical front end to Borg. And it makes it possible on a desktop machine to go through and get your system backed up to your repository really easily, and it's up on Flathub.

That's hard to beat. Super easy to get going, Wes. And then last but not least, there was a couple of command line clients that got sent in, and I want to talk about those in a moment too because I think there's obvious advantages to using that as well.

The note I was trying to hit with my backups was I wanted something maybe one step less complicated than Borg. Because what kind of feels like the downside to using Borg backup is you do need kind of that server component. And you have the client component, server client setup. That's more stuff for me to manage and monitor. And if it goes down, my backups break. It does seem like you're in the camp. Like you're not like checking on these a lot. You're not like actively making new backups

and like adding to this. It's more of a, you want a system that's simple enough. You haven't touched it in six months. You're going to go like make sure everything's okay. And you remember how it's set up. It's easy to change. That's where like having a GUI probably comes in because like,

Oh, I don't remember the commands to do all this. Yeah. And I may only need it in anger, right? When I've lost something because generally the way I have my system structured, I'm not changing where the data stored very often.

It's it's there's, there's prescribed spots for data. There's prescribed spots for configuration. If I had a new application, it's config goes to this place. It's data goes to this place. Those locations are already being backed up. I don't go check the backups when I install it in application. They just should be part of that. And it's not the Borg wouldn't nail that. It's that if I went a year and I didn't check on the Borg server, maybe my backups are broken where I kind of feel like the nice thing

about a remote backup instance, maybe it's a service, is they're managing that piece for me. Even if it's just a bucket that I'm dumping files into, at least it's being managed and maybe I'm encrypting first. I don't know. Well, especially nice if you have, right, you want to have multiple locations for your backups and some of them are managed by you and some of them are managed by someone else who hopefully will mess things up at different times. Yeah. I wonder if anybody, maybe somebody knows of a Borg as a service. Like, you know, I have like rsync.net. I wonder if there's like a Borg.net where you can,

they manage that aspect for you. They list Borg on there, on rsync.net, by the way. And you might have to do some manual setup for it. That wouldn't necessarily be a bad compromise as long as I was comfortable with the encryption. That wouldn't necessarily be a bad compromise as long as I was comfortable with the encryption. That wouldn't necessarily be a bad route to go. Because it's that component I don't necessarily want to run. The nice aspect of it is...

You're not trying to set up a whole new backup server, that's what you're saying? Yeah. But the nice is, if I did want to, I could kind of run them at different locations. But, Tiny, you're familiar with a service named BorgBase? Yep. So it is an online service

where they set up a Borg repository for you and give you like the copy paste commands to work with. And they also have an Ansible role and they support the Vorta backup client if you need a GUI. All right. All right. They actually make Vorta.

Okay. I was wondering because I saw some reference to BorgBase. So I was, okay. So that would be a pretty nice solution as long as I'm comfortable uploading. Another nice aspect is if you become a member of BorgBase, you're supporting the development of Vorta too, probably sort of tangentially. Yep. And one other really good feature about BorgBase is they kind of have like a dead man feature where it will shoot you an email if it doesn't see new data in x number of days which has actually saved my butt once that's huge for me that's that's the kind of catch that's what

i need yeah all right i'm totally looking into borg base after the show yeah that seems like there's a lot of room for stuff like this you know it's like we value the ability to like have control of stuff or know that like right here like, like it's encrypted with your key and that kind of thing. But we do want to offload some of the like sysadmin stuff some of the time. But with a lot of the proprietary like traditional services, you just don't get to choose anywhere on that spectrum beyond like all of it or nothing. That's it, right? And I could also see a situation where maybe some backups like pictures go to Borg base, other types of backups, maybe I sync them between locations. Like Borg would give me the ability to have that kind of flexibility, which is appealing.

Because not everything needs to go to a cloud provider. But part of what I like about sending some of that off-site is it's kind of insurance against myself. Yeah, exactly. Who knows? I mess around with a lot of stuff so you never know gotta protect against that too um then there's a couple of command lane clients that got sent in that people really seem to love one was borgmatic it pitches itself as a simple configuration driven backup software for servers and workstations yeah this looks pretty nice

little little yaml file here to, you know, pick some source directories, you can set retention. And the nice thing about this one is I see, like, even the Vorta dev was saying that they use Borgmatic on their servers and they just have an Ansible role that deploys it on their servers.

Vorta on the desktop, Borgmatic on the server. That seems like a decent setup, too, because then it's the same back-end technology it's the same language everywhere but the tooling is sort of custom fit depending on the use case and then another one that's sort of along the same lines it's a front end and it's another simple command line tool that kind of makes it accessible yeah so maybe you don't need a gui but you don't know all the language you don't know all the commands that's where these comes in this one's called mborg e-m-b-o-r-g a front-end of borg backup they say it's just a simple command utility to orchestrate the backups lots of them

out there really but these are the ones that we got a lot borgmatic vorta and mborg and then it sounds like i have a little homework after the show. I need to go check out Borg base, which is really exciting. I have to admit, I was hoping to hear a little bit more from the RESTIC users out there because I know they're out there in the audience. I got one or two things about RESTIC and that's it. And it seemed to me from like what I know from a hundred foot view, it seemed to me like RESTIC might've been a little bit of a simpler setup for me.

And so much that it would just go to a bucket or something and I would just run it on my box. And that to me feels a little less likely to be breaky. But now that I know about Borg base, maybe I'm not as concerned, but I'd love to hear from the RESTIC users out there. And if they have any kind of front end tools, I know about like auto RESTIC, but I'd be curious to know about kind of front-end tools. I know about, like, AutoRESTIC.

Yeah. But I'd be curious to know about, like, web front-ends and stuff like that. I bet there's a bunch of stuff we haven't yet found out about, so please do write in. We definitely heard from the Borgs out there, though. The Borg Brigade. I suppose, really. I mean, if you wanted to be extra content-y about it,

you should set, like, two of them up. Hmm. You know, double backup for a month or something and then kind of guys got a time we got the time for that you need a studio intern i mean you're right though i would like to try so brian would say where's your second backup you're right or tertiary backup indeed yeah i mean if this is a starfleet system you know i wonder if i could that would be it might take a little while but it would be really nice to know and then just see which ones feel you know you've revisited in a band like which one do you like interacting with more?

Huh. Hmm. Yeah, because I'm just dumping more and more important stuff on my next cloud as long as I stay on DraftingOS. Pictures, and I'm putting my backups on there from my phone, and I got Dylan's laptop backing up there now, and it's just like, it's time to really

make sure I get that buttoned up. I'm sure the Odroid's reliable, but it's getting there, Wes. Linode.com slash unplugged. That's where you go to get that $100 for 60 days. And it's a great way to support the show while you're checking out fast, reliable cloud hosting built by Linux geeks for Linux geeks. And I mean that sincerely. It's where you really should try your next project. Maybe the next thing you want to learn. That's what I started with. I started just experimenting. And then from there, we've deployed it for production for just about everything that's

user listener, I guess you'd call it facing for the last few years, performance, the stability, the reliability, the support and the pricing. Those are like the key tenets of why I love Linode and why I am very confident in recommending it to you. And Linode has been turning it up past 11. They're rolling out a dozen new data centers this year to different regions of the world. It's awesome. And truth be told, with 11,

I already felt like they had something that was pretty much close to anybody we needed to get to, but not enough. They're going to keep building it out. And you know, they are their own ISP. So these connections, they're crazy fast. They got 40 gigabit connections coming into the hypervisors, MVME disks.

So you can actually take advantage of that speed and AMD EPYC processors if you want that. But all that aside, I really think you should go grab that $100 because everybody should try a NextCloud. I'm going to harp on this for a little while. Or if you haven't run NextCloud in the last couple of years now is the time to do it they also have a really easy to deploy mastodon instance so you can get an actual feel for what it's like to run a mastodon instance day to day without having to spend hours figuring out how to get

it all set up that's something nice because honestly i hear a lot of people setting up their own mastodon instances as their own home base with their own domain name so they kind of have like their own unique identity on the fediverse and with a hundred dollars you can probably get a rig that would last well i mean you could you could really play around with all kinds of things you could throw a really powerful rig at it if you want maybe mess around with photo prism have it analyze your photos why not throw jellyfin up there for friends and family all kinds of things you could do. Set up a VPN network.

It's just, I mean, seriously, a lot of fun. Some of the most fun to be had on these systems because they're fast, they're easy to set up. You can do it however you like, either like through the API or through Ansible, or you can just do a standard old click and deploy, you know, like we used to do it back in my day. There's no judgment over there.

Whatever works for you. Just go try it. Go learn something and support the show. Linode's been doing it for nearly 19 years and nobody does it better. And the pricing's great too. Linode.com slash unplugged. That gets you the hundred bucks,

lets you kick the tires for free, at least for 60 days. Try it. I think you're going to love it. Linode.com slash unplugged. Try it. I think you're going to love it. Linode.com slash unplugged. Well, Chris, on this week's last Office Hours,

Office Hours 21, Boiling the Frog, I think is properly named because I know our OBS machine was boiling at your feet, I think. You can hear the sounds of it in that episode. And it was, when we were doing it live, it was worrisome. And you both were supposed to find a solution to that, but I haven't heard anything. So can you update us? We were indeed up late last night here at the studio fixing that. You know, it worked out. We knew it was coming. And we literally ran that old studio OBS machine to the last moment. You know, I remember, I think it was like 2017, 2018,

Brown Bear and I were out here setting it up before LinuxFest. Actually, it was really Noah, Rikai, and me, but mostly Noah and Rikai setting it up. And we kind of just slapped it all together, and it has been a really solid. It's when we really went full 100% Linux in the studio. Right, yeah, which was a i mean a big transition yeah it was like 10 years in the making that's also about the time this guy

entered the picture yeah that's when we got the big mixer and um just sort of like it was like a big leveling up in our game it was a big leveling up in our technology and it was the one of the first times where we were taking advantage of things inside Linux that started to put it ahead of what the commercial operating systems could provide. And I feel like we've done that yet again. It's been a few years, and I feel like we've just done a whole other leveling up again. And we ran that OBS machine

literally down to the last possible second until we had to take it out of production. It was making some unhealthy noises. And, you know, when I banged on that power supply, like you do to take it out of production. It was making some unhealthy noises. And, you know, when I banged on that power supply, like you do to keep it running, I got a face full of ozone. And in my experience, usually when you get that much ozone in your face, it's in the power supply is in the process of a capacitor popping. And it doesn't generally last much longer than that.

And so I didn't even know if he'd get through office hours it's pretty funny you can hear it falling apart it was very hairy and the most embarrassing thing about it was i had a brand new system 76 thaleo sitting outside the studio in the box staring at you haunting you just hadn't gotten to it yet and so i placed the order for the thaleo on november 7th so that's and then you know it took a couple of weeks or whatever or a week to get here probably and so it's been here since well before thanksgiving no doubt about it and i was hoping to make it past the new years and we did and we made it just a little bit past the new years so i want to tell you about the box we got

so you can get a little idea of what we're working with the base price for the this is the the thaleo that's the smaller of all of them started at 587 us greenbacks for all these that was the base price i got it with the red martian strip on the front it really is pretty compact i mean compared to the other rigs underneath the oh yeah here like yeah they look huge and old it's funny the thalio is not only is it a really elegant, classy looking case, but you're right. It is so much more powerful. It's crazy. So this we got a 13th generation and I went with an i5 13,600 K, which boosts up to 5.1 gigahertz, has 14 cores, 20 threads. That was a little bit of an upgrade price.

So we paid a little bit for that. We got 32 gigabytes of dual channel DDR4 RAM. And we've got two disks in here. One is a 500 gigabyte PCIe MVME. That's the OS drive. And then the other is a two terabyte SSD, a fast SATA drive. And that's the recording drive for OBS,

the storage on our recordings. We got it shipped with a Radeon RX 6500. It just went for their cheapest Radeon because we're putting the Intel Arc GPU in this case, which we will get to. And it just comes with their 650-watt power supply, which isn't remarkable, but it's totally fine. It's really small, but it seems to do the job so as configured our thaleo total all-in price was one thousand nine hundred and seventy seven dollars now i have to say guys yes that's a lot of money nearly two grand at the same time we got a badass kick butt live, recording production machine that'll easily last us five years at least, doing that job, which is a high-end job, for under two grand.

Pretty happy with that. And it's buying it from a Linux-first company. Yeah. That's, you know, built here in Colorado. Employees people we like. I'm pretty happy with that. That's a pretty good win. I think the big reason was because I went cheap on the GPU and I went with an i5, but it's a 13th gen i5 with tons of power. So I'm

good. Yeah, we're streaming right now and the load average is 2.7. We're streaming and recording. And you got to remember, it's pulling in WebRTC streams for each one of our camera shots and then compositing them together. That's really impressive. And we are using 4.27 gigs of our 32 or 31.1 gigs of available room. So this should last for a while. this morning, we're going to use this system to the point of when we retire it, we will not even remember when we put it into production or how long ago it was. It'll be so foggy. That's how long these systems last for us. You should engrave it on the side of the case. That way we can remember. We should, right? Actually, Wes and I thought about putting a file on the desktop that says install date or something like that. Wes was here. You can find it if you look around. Something we had to consider, though, was the tech stack we were going to go with.

The previous system was reliable on Ubuntu 18.04, and it'd gone through a couple of updates that, you know, kind of brought up the latest edition, and it seemed to survive all of those. And just overall, it was a really reliable release of Ubuntu. No complaints, really. We did have a few issues, software-related over time, but never really because of Ubuntu. Held up quite well, yeah.

Yeah. And now you wanted to go with OpenSUSE. Yeah. That was on the list at one point about a year ago. This time, though, we decided let's do the right thing and put NixOS on there, which, I don't know know we thought was a little silly because we haven't really put nix anywhere that's in full production and definitely not as

much on like a like a desktop audio you know it's one thing multimedia host a server and run some docker containers it's great at that we know that jupyter tube is our nick server on linode running peer tube in a docker container it's in production but it's like this is different this is like if this machine breaks it disrupts recording it disrupts the live stream disrupts the member feeds it's it's a high high consequences machine and there's a lot of just fiddly bits about it yeah and a lot of little strange packages you need that i mean that's why we went to ubuntu 18.04 to begin with back in the days because some of that stuff was only packaged for Ubuntu. Yeah, or for like random PPAs that folks had made.

And the tech stack was X11. It was Pipewire and Jack and an old version of Plasma. And this time around, we wanted to do things a little differently so we wanted nix os because it's just a great mix of pretty modern packages kind of like you get with arch but with declarative configuration and where the rubber meets the road on that for us is the ability to try things like tools applications subsystems like audio subsystems, entire kernels, et cetera. And then we can just, with the change of a configuration file, rip it all out and replace it with an entirely different set of stack, an entirely different set of packages. And it was like it was never installed

on that system ever. Yeah. I mean, at one point we were building like a custom Electron app, or we were trying to compile a plugin for OBS. And, you know, as you do, you're like trying to hit your head against the wall, like, oh, nope, I need that to build dependency and that build dependency. And Nix, you can just, you know, pop into a Nix shell, add them. And then if it turns out that didn't work

or you don't like that plugin, none of it's around if you don't want it anymore. Yeah, you don't build up all these layers of, oh, that was a failed experiment. That was a failed experiment. Because a lot of times with these audio and video tools tools there's five or six of them that do the job and we have to figure out which one is in the right state which one's being actively maintained which one is compatible with the set of technologies we're using like wayland obs version

whatever it might be because of course some things change 28 yeah we're're on OBS 28.1.2, but there were things that changed from OBS 27 that dramatically impacted what we're doing. So that just dequalified a bunch of stuff and we didn't find it out until like the API wasn't working and we'd built the tools. And so Nix lets you just wipe the slate clean like it never happened. And then on top of that, you got rollbacks.

That is really compelling for a media production system now we went with the lts kernel and this this is where things went a little sideways this is where things begun to go a little sideways because we decided very much thanks to nev in our mumble room right now to try the arc gpu this is intel's gpu with an entire open source stack and long term i figure that's got to be the most stable linux setup possible intel cpu intel chipset intel video card all of it using open source software integrated into the kernel we know they have a long history of wanting to to work in that model and it's kind of just exciting to see a new like discrete graphics card, you know, vendor in the market. However, what I discovered is to really get everything working like OBS, there's updates on OBS's end. But you also need really Linux 6.1 and ideally Linux 6.2.

And to get all the other components working, we need the Linux LTS kernel. So we ended up being stuck in the spot of, do we go with the ArcGPU and then try to solve how to get OBS and FFmpeg and everything to work with it? Or do we go with the AMD GPU and have everything work today and we can use Wayland and we can use Pipewire and all that kind of stuff? Ultimately, we decided to put the AMD GPU back into the OBS machine and thought, you know what we should do instead? Because why not mess with Chris a little bit? Is we should put the Arc GPU in my system

right here in front of us. And that's an interesting idea because not only is this a critical system to the production, but it's one we can actually show on air. We can actually, yeah. So that is ultimately where the gpu is going to live but this and this version of neon already has linux 6.1 because i have that custom kernel tool

so i just i'm surprised i hadn't thought of it earlier but i do have to update the mesa stack first so i haven't done that yet so we'll get to there but um why don't we start with our wayland issues because this when we were trying to figure out going with the ArcGPU and we were trying to decide if we wanted to do X11 or Wayland, there was a lot to sort out. And there was a few moments where I think we legitimately thought maybe we should just give up and go to X. But we wanted this to be a future forward stack,

something that would last for years. And we really felt like maybe Wayland was going to be a necessary part of that. So Wes has been here for about an hour and 15 minutes so far. And I'd say we spent about 70% of that time messing around trying to get this thing to log into Wayland by default. So Wes, you figured out maybe there's a bug right now with SSDM or SDDM and Wayland auto login. Yeah, I think. I think it's one way or another. I'm pretty strongly convicted about it.

All right, so let's talk about why we wanted this. So one of the things we were trying to accomplish was auto-log it. Now, why do we want auto-log it? Chris hates logging in. No, it's because these systems get rebooted remotely when we're traveling, and we need them to come all the way up to the desktop, A, so we can get remote desktop sessions, and B, if we're in the middle of a stream or something, we need it to reboot and relaunch the applications and start streaming again. It's exactly the kind of setup where we don't have time to go turn our attention that way. It just needs to work. So why was this harder than we expected, Wes? And why couldn't we just use SDDM and just go to Wayland why do we have to switch to LightDM and debate GDM what went wrong yeah well I guess I showed up and you had it all kind of going you had a base desktop installed had SDDM going it was working but of course we were still running X11 and oh my gosh part of what immediately followed us recording

that clip was about a hundred different reboots and then checking, oh, is it Wayland yet? Is it Wayland yet? No, it seems like as the KDE environment and things like SDDM sort of progress on the Wayland path, not quite everything was working. So we wanted auto login and we needed it to choose the Plasma Wayland session. I guess there's an issue that that just doesn't work broken right now it seems it's not unique to nix os other systems are having the same people on arch people in the open source world so we weren't sure at first if this was a bug that we ran into on our own system or it turns out it was actually something going on with sddm and wayland so then it came to the question of, do we even bother with this auto-logging thing? Is it that important?

Because it did still work if you manually logged in. You could choose Wayland manually at the login screen, and then it would give you a Wayland session. That's what we were so confused, really, because it was like, well, clearly the Plasma Wayland session does work. Where's the missing piece here? Yeah. And ultimately, we decided maybe the route to go is just a different login manager and we wanted something that wasn't too resource intensive yeah i was like let's just do gdm see if that works and i'm like oh it's a lot that's a

lot just to log into a plasma desktop there's a lot of gnome shell in gdM. And so you were like, well, what about LightDM? Bing! Like a little light bulb went off and it worked out perfect for what we need. Yeah, and I mean, really, I said, what about LightDM? Because it was listed in the NixOS manual as another option that they have pretty good support for that, you know, automatically supported.

We could keep the auto-login setting and it would know how to adapt that for LightDM as the display manager. And so we just sort of commented out the line that it turned on stdm switched that to one that enabled light dm and it worked ultimately we discovered by going with waylon we were setting ourselves up for a whole other challenge that took the rest of the night to solve but we'll get to that we'll get to that we made we didn't know in at the time when we were getting wayland working that we were setting ourselves up for a job but one of the other things we were switching out so going from ubuntu to nix and going from x11

to wayland we also wanted to make the transition from pulse audio to Pipewire. This was our chance to put Pipewire deep into the heart of our production. And the way it works with Pipewire is you don't stop using Jack and Jack's tooling or Jack's APIs. You continue to use those as well. So anything that can talk Jack can now also just talk Pipewire. And OBS has support for Jack in there. And we really kind of, I think, probably had, I don't know, of, I think, probably had, I don't know, Wes, I mean, I guess I was expecting it to be this huge job of getting Pipewire to replicate the functionality we had.

And it really just almost did it all automatically. Yeah, you know, because we were previously on that 1804 Ubuntu base and we'd installed, you know, there's various sorts of like pro audio PPAs out there, things you can install that have pre-compiled a bunch of these plugins that have like later versions of Jack than you find in the normal repo. And so that was our previous experience was like, all right, I've got the stock system. I need to kind of go get special tools.

I need to go find weird, nontraditional sources to get them all running because this is kind of niche software that doesn't get as many updates and most users don't care about. And it all worked. But yeah, it was, you know, it wasn't just just a couple tick boxes to go use Pro Audio or anything. No, in fact, we had to do a little digging around. Like I said earlier, a lot of times you install five or six tools to figure out what works and then whittle it down. And so I step out to go hook up the old OBS machine to grab some files off of it. And I come back and Wes is like, hey, check this out.

Gosh, sure looks like Nix has a lot of audio stuff already packaged. And so far we're trying the pipe wire thing, right? So you're just trying all the different toys. Yeah, I installed a smattering of things. There's more tabs open here that we can play with, but here's the like minimum that seemed to be working. Well, I already know Reaper's installed. So we've got Reaper on here. I see Arduer's on here too. Are we going to be editing on the machine? No. But will we use these to route jack audio?

Potentially? Record live stream? Maybe. I think the one that's jumping out at me here is Helvium. I'm going to launch that right now. Because this is a pipe wire patch bay. And you can kind of picture like the analog version of taking wires and stringing them from one device to another. It sort of lets you do it in the software,

like a grid. So that's pretty cool. Let's try a QPW. It starts like something like that. QPW? Oh, no, it's another competing... Oh, we got another one on here. So we got, of course, in true Linux style we got four or five apps that do the same thing.

And we'll just have to figure out which one. QPW. Oh, I see. Yeah, here we go. So this is a QT version. I like this better, actually. We're doing the Plasma desktop. I like this better.

There's a lot. So I didn't successfully get the Jack files backed up. Everything else got backed up. But for whatever reasons, those didn't sync up to NextCloud. But I don't know if it really is all that important if we're going to do everything in Pipewire anyways. Let's find out. And we did. We did find out.

And that went pretty well. Then we got to the issue that we had created for ourselves by going with Wayland that we didn't realize. Although, honestly, if any of us had stopped for two seconds and thought about it, we would have known this was going to be a problem. Oh, for sure. Because Wayland is famous for improving the security model. It's just more secure. It's more robust. And that means you can't just send key presses to any darn window anytime you want.

And guess how most things like stream switching work. It's by sending a key press to the application. And that application takes the key press and you assign that to a to a scene and it changes the scene uh we have right here in front of us the elgato stream deck and we knew we couldn't use our classic for for you know i don't know 13 years as long as i've known i have been switching using uh a shuttle shuttle pro that kind of has like Star Trek, the original series style. Jelly bean buttons. Yeah.

Jelly bean buttons. You know, I love it though. And it's in a, it's in a grid layout that my brain could understand. And I knew where all of the shots were. You got good muscle memory going. Yeah. Quickly toggle between. Yeah.

I mean, you know, any tool you've been using for that long, you can just use it without looking. However, it's really old. And I think they built like their last Linux driver support for like Ubuntu 12.04 that we hacked together to get working on 18.04. Yeah. And it works by sending a key press, an F key. And so I knew we weren't going to be able to do that. So in preparation, we ordered a stream deck, but the stream deck also works out of the box by sending a key press. That just wasn't going to happen. So we had to come up with another way to send commands to a running application

that didn't involve keyboard input. I don't know if you know this, but I've been doing Coda Radio for like 450 episodes. That's true. That's impressive. I knew that. I was just waiting for you to catch up. That's really impressive. So sometimes you just got to build it yourself. And, you know, that's what we're doing here.

There's not a great solution. And so we're going to make a great solution. We looked at a Docker container that would act kind of like a proxy between the Steam, I'm sorry, the Stream Deck and OBS and it would kind of relay WebSocket commands into OBS and change scenes. And we wouldn't have to emulate a keyboard or any silly stuff like that anyways. However, it doesn't seem to be working and it offered a lot of stuff that we don't actually need. So, Wes, you're building us kind of like a lean-mean version instead, aren't you?

Yeah, we found a handy-dandy new Python library that's aimed just at the new WebSocket post-version 5.0 standard. Right. It has the capability in its library. It's like a little SDK. It makes the WebSocket connection. It's got a bunch of methods already defined, including one called setCurrentProgramScene. Yep.

It just needs to know what to do. So Wes is creating a little bit of Python to go with it. And look at that. You got it. So he has a Python script, and we just specify the scene that we want OBS to change to. You execute the Python script, and it does it. So far, so good. So, functionally now,

what we have on the Elgato stream deck, and you could use this for I don't know, macros on your desktop, like muting your audio. You could use this for creating directories. Anything a Python script could do. It doesn't have to be for controlling OBS. And so, these buttons that are on

this little stream deck, each one of them is now executing a Python script when I push them. It executes a Python script where we pass a variable to which scene we want it to change. And then it communicates that through this API to OBS and does the change, and it's instant. Yeah, it seems to work very well. But what I thought was really funny, Wes,

is while you were working on yours, you were like, were like oh that's it i'm getting myself one of these like because you don't have to even if you're not using it for obs they make that stream deck ui available for linux it's packaged even on nix yeah and i mean like we're obviously kind of late to this we've all been eyeing these for a long time and you know alex has had them and uh so i've thought about it but i just hadn't i hadn't seen how nice and good the Stream Deck UI software had gotten. It was pleasant to work with, easy to get things set up. And yeah, if it works reliably and can run a command, what more does a Linux script and type person need? And then when you found that Python OBS library that communicates in the new updated way that the newer versions of OBS use.

Yeah, that was a little bit tricky. I guess with version 28 of OBS, it came with an updated version of the OBS WebSocket support, which then has like an updated standard with slightly different commands, or at least a lot of existing libraries were like, whoa, that's version 5. I don't know how to talk to that. Because we'd seen like a fun little Go-based CLI

because we were looking for something like, is there already an OBS WebSocket CLI? Surely, right? And yes, there was. It just didn't work with this updated version. We found another library that looked neat, which was basically a WebSocket to regular HTTP proxy, so you could sort of just send curl commands.

We thought, oh, well, that could be a handy way to do it. Same problem. Yeah, this worked out, though. Nice and simple. Exactly. And then we realized, boy, this would be great on all of our workstations. Like I kind of want one upstairs now

and just have different Python scripts and bash scripts that it executes. The fact that it doesn't have to be a keyboard, that it can execute a command. And so anything you put at the other end of that command, you can assign to a button on a stream deck. That sells stream decks, my friend. That's what that does.

I mean, it's's great it really is super handy and uh we'll put a few resources in the show notes so you guys can see what we're talking about there was one major downgrade and i don't know what we're going to do about it and we really got to solve it yeah unresolved as yet and that is we can't use rust desk with this setup and in fact if you install rust desk on a wayland system and, in my opinion, a bit of a disqualifier, it'll come up with a dialog box that says, hey, friend. It's something to this effect. Hey, friend, you know, this app doesn't work with Wayland. Would you like us to run a fix to solve that problem?

And you say yes. And what it does is it goes and modifies your login manager to log you in using x instead of wayland and you no longer on our wayland system anymore that's what happens when you say yes fix it for me fix fix it yeah whoa and that's you know if you think about it a ginormous system level change fundamental change to one of the major components in your system and they're just going to go flip it and fix it for you real quick. Bit of a disqualifier. So until they get Wayland support, I think I'm out.

Unless somebody knows of a way to use RustDesk on Wayland, because I do love it. You know, Chris, I've done a ton of research around RustDesk and trying to run it on Wayland probably for the last year since we've been trying so many different distributions this last year. At least I have been. And there are long discussion streams about whether or not to do it and if there's development time to do it, et cetera, et cetera. And it seemed pretty

dire in that respect. They basically said, well, from last I checked, well, there's not much reason for us to put the work in because we don't have enough time and we need help and the help doesn't seem to come. So this issue will just kind of sit here until that changes, which is a bit depressing. Yeah. And if this was a GNOME desktop, I would just use the built-in RDP server that they've been working on. Yeah, right. We've played with that a little bit. But on the Plasma side, don't really have the options there.

But the tooling that Plasma gives us, like the KWIN rules and just some of the options, like when you right-click on a file and you can copy the URL right there in the browser. When we're digging around for stuff and then we find it and we need to go paste that into OBS, it's really nice. Yeah, that was one of those Plasma, you're great moments. Yeah. And it works. It served us so well on the Ubuntu version because we were running Plasma on that machine as well. And it just served us so well, it it felt like let's just keep that going but I would love to figure out the remote desktop on Wayland stuff and maybe figure out how to get that working in a way where we could run another other systems

that inevitably end up on Wayland 2 because I feel like that's kind of what we unlocked here was we answered the question is can NixOS be used in this kind of live stream production environment and I'm not trying to advocate go out there and drop your distro of choice and switch to nix i think what i'm trying to communicate in our coverage here is as a long timer who had kind of felt like you know things were sort of settled to come across something like this and have a completely upend my approach to deploying linux systems and to be so thrilled in the results 20 plus years into this journey. When does that ever happen? When do you ever get 20 plus years into a, you know, like 20 plus really like into a profession and you know it inside and out and then you come

along and it gets upended on you and it's a great experience. And that's what's so freaking awesome about Linux and free software is it's a, you know, we're always moving forward. We're always coming up with new ways to solve it. And when you get on board with that, it is an awesome ride. And for me, it has been incredible to witness a whole new approach to Linux and it solves fundamental problems that I have had with Linux forever. And it does it in a way that's really approachable to me. And there's other ways to do it.

I'm not trying to convince anybody to switch, but I just wanted to share what excites me so about it. I don't know about you, Wes. I know you've been around the block a few times. What is your takeaway? Not as a way to try to pitch people to use Nix, but why maybe I could see us putting it on the Reaper machine. And I'd like to know your thoughts.

Yeah, well, I suppose we've probably talked a lot about like the benefits. But when you're like weighing any of these tools, there's also costs. And one thing I was a little nervous about thoughts yeah well i suppose we've we've probably talked a lot about like the benefits um but when you're like weighing any of these tools there's also costs and one thing i was a little nervous about when we were making the switch was just yeah there was a lot of different software software we hadn't tried before and sometimes if you go a little outside the box on something like nixos you know you're not on the path trodden before you might have to go find some you know some forum posts some other folks,

or we've got a wonderful matrix room who has very helpful people. So you might ask in there to like cobble together, especially if you're not an expert yet on understanding the Nix language and how to write expressions and how that all ties together

to get what you need. Maybe you need to figure out how to add UDEV rules or create a new system D thing. Those are actually quite easy, but if you're new, it's a different way and you can't just copy and paste the solution that might work on another distribution yeah yeah you're not likely going to find your answer on stack exchange yeah exactly so you got to be prepared for that and i wasn't sure how that you know would this use case fall

into that value proposition very well um and there were times where we were kind of wondering like oh is this an xixOS problem? Or being a little fiddly with trying to get something to work. But I wouldn't want to trade that right now. Like, it's so nice. We just have that one configuration.nix. There's like a couple stuff, going to try to get more of the Python stuff

into the Nix side of things. So not everything is totally Nix, but so gosh darn close. And it's fun to iterate with. It gives you this nice, you know, like, it doesn't always 100% work, but if your Nix config sort of comes together, you at least know you're going to get the bits on your system. And sometimes with some of the software, that's half the battle.

It's really nice because when it passes the build, you know, okay, like that is half the battle right there. The build's passed. Now we just got to see if it does what it says it's going the battle. It's really nice because when it passes the build, you know, okay, like that is half the battle right there. The build's passed. Now we just got to see if it does what it says it's going to do. And that sets you up for like a relatively fast feedback cycle. And some things you may need to reboot, but a switch will often get you a lot of the way if it's just a systemd service or adding a new binary to your path or something like that. And so you can just really rapidly iterate. And before too long, you probably got it to work.

And then, you know, it to work and then you know it's going to keep working bren i think you'll appreciate this when we got done last night and i really i got to give so much credit to west because you know he's spending his saturday and sunday here at the studio working on this stuff and uh staying here late you know he's got doggos at home and stuff like that and it's a long drive and it's crappy weather, but man, did you nail it? I mean, like, you know, watching you crack out the Python stuff to do the camera switching, I couldn't have done that on my own. So super grateful for that. But I think the other thing you're going to appreciate Brent is it really felt in line with a lot of kind of the themes

we were thinking about this year. Like this is a sustainable system. That's going to last us a long time, the software and the hardware, like we're not going to need to reload. We're not going to have to nuke and pave. We're just going to be able to iterate on this. And the rollbacks that Nix gives us kind of gives us that parachute that if we wait a long time and we do a huge upgrade and something breaks and we got to do a show, I reboot, I choose the previous entry in Grub and everything's back up and running again. Yeah, we know you're not going to not do the pre-show updates. We've tried that part. We've tried fixing that side of it so you just gotta plan around the

other side of it but the other thing too brent it's like this really felt like the definition of an upgrade because we leveled up the entire tech stack and we improve the way we do things like the control mechanism with the stream deck versus the old controller and using the web socket commands instead of like a keyboard and switching all that. You know what I mean? It's like, it's not just, we got a faster system, but we got a better workflow too. That's going to last us longer. It sounds to me like, even though you still had to do a bunch of custom bits to get it all to work and maybe not all of it is solved like REST desk, it's still kind of from you

telling the story of the history of how that original system came to be, it still feels a little bit more solid because you had to do all those things on the old system and it lasted that long. So I'm going to be curious to see what happens, say, in five years when you get a new machine and you switch it out. And because in theory, you have a lot of this stuff declarative already. You know, you don't have to forget the backup files that you forgot or whatever. And I'm sure that configuration will also mature. I mean, you're on, what, day one right now? So not even 24 hours.

Yeah. So give it, you know, three, six, eight months. And it'll be interesting to see the history of that configuration just improve. Or maybe it won't change at all and you got it right the first time. I'm curious to see that. Oh, I imagine we'll iterate for sure. But your point that if it works, the idea that we could then pick, lift that config, drop it on the next machine and be that much further along. Or if you think like maybe our road machine that we take on road trips and stuff,

you have, we could just replicate the entire setup. Isn't that attractive? I'm real excited about being on Pipewire too. Like, I mean, we're still using a bunch of Jack tools and we're not doing anything, especially Pipewire specific at the moment, but a, you know, Jack's always been kind of its own weird little corner. Doesn't always see the most active development, you know, not weird in a bad way, just like normal users use Pulse and maybe you've never even heard of Jack. Whereas Pipewire, I mean, it's just got a lot of momentum. I don't see that stopping anytime soon.

It's got great compatibility with both Jack and Pulse now. So like you are seeing, we're not doing this yet, but if the OBS setup needed to grab audio from some random application on the desktop. Super easy now. We could do it before, but you know, you had to like kind of tie things together yourself. This, you just drag some lines in our patch bay and you're done. Yeah, I mean, you know, if you say you're maybe even just in a meeting and you want to pipe in audio from a video like VLC or from a music from a web browser, you could fire up, what was the graph manager

we ended up going with? It was- Oh yeah, it's QPipewireGraph, QPWGraph. Yeah, Q- I love the link in the notes. QPWGraph. You can pull it up and you can just drag a line from the output of Firefox or VLC into your browser.

And then the audio will just go from VLC into your browser. And then the people that are in your meeting or wherever you're doing could hear the video or whatever. It's so easy. It's so nice. And, you know, for a live stream, maybe we would want one day to pipe in Firefox to the stream for a moment.

And we just do, do, do. And we can just drag it OPS and we've got it. It's really cool technology. So I'm glad we were able to get it deployed. We've waited a little bit longer than I thought we were going to, but I think when we did decide to do it, it was ready to go. And it's the pipewire aspect is totally production ready for us. Bitwarden.com slash Linux. Go right now to get started with a free trial for a team

or an enterprise account, or, you know, as an individual, you can try it for free. And it's a great tool for that entire range. that's why i try to mention that right up front it is the easiest way for yourself or a business to store share and sync sensitive data like your passwords usernames login ids that kind of stuff tokens bitwarden is fully customizable and allows you to manage all of that and of course it's open source you know that's super important for us here on the show your vaults are end-to-end encrypted with zero knowledge encryption including the urls And one of the things that I think is absolutely great about Bitwarden is they take the cybersecurity week thing seriously. I didn't know about this because I think it's kind of new. First time I heard about it was last year.

And it's something that the federal government's putting on. And at first I was like, this seems kind of silly. But I actually think this is the ideal time to start the conversation. It's happening right now as we record and it runs through the 28th. So I believe that's Saturday as of the week that this comes out. And to me, it just kind of feels like cracking the door open a little bit and getting that conversation rolling about Bitwarden. Maybe at your place of work, maybe at the open source project that you participate in, maybe just with you and a significant other. I've been there. I've had to have that conversation too. Maybe this is your week. This is your opportunity. Send them to bitwarden.com slash Linux. Then you know they're going to the right place. Plus we're getting a little bit of credit here on the show and they're

getting Bitwarden, which I think is just the absolute best tool out there. Been using it for a few years myself. Wes convinced me to try it. He was right. I switched from LastPass years ago and you can too. Go to bitwarden.com slash migrate for that information. I think once you go through the migration process, you'll feel silly for not having done it sooner. Don't worry. I was there too. I just did it a couple of years ago, you know, like might be maybe three years now. So I'm just a little bit ahead of you. So I can tell you after a few years, I really like it. And I recommend it for you. I recommend it to my family, no matter what their tech skill is.

This is how we're doing it for them. It's a low hanging fruit that they can take care of that really improves their security online. Same for teams, same for projects, and even us individuals. Go try it out right now. Go get the confidence to use a unique username, password, and email address for every site, service, and app you use. Bitwarden.com slash Linux. Once again, to support the show, go to Bitwarden.com slash Linux. We saw several themes of feedback this week from you all. Thank you very much. Linuxunplugged.com slash contact

if you want to send us a note. One thing though, Chris, that you found, you got a special message in. Do you want to take this one? Oh, right. So LinkedIn wrote in, or it was actually in the matrix and they say, I have a media server that is written in Yeah. And the idea is it's very heavy plug-in based and modular. And he says the tagline is you shouldn't have to adapt to the media server. The media server should adapt to you. It's not entirely usable, but it is up on GitHub. But I'm beginning to have less and less free time with school, and I could really use some help to bring it back to life.

So I think it's Carino, K-R-I-I-N-O. Carino. And we will have a link in the show notes. If you want to try it out, maybe you've been waiting to get your hands on Rust. Seems like kind of a neat idea. Kind of like rethinking the media server. Something that's maybe not as complicated that just helps you get to your stuff real quick. And, you know, written in Rust, kind of a neat idea.

Yeah, I'd try it. On the same theme of like sharing listener successes, we got some notes in from Scale. Sam wrote in saying, speaking of Scale, I'm going to be a speaker at Scale for the very first time. I want to thank you guys for the knowledge you've provided me over the years. I've been listening to you guys for years, literally. Just in case you were interested, the topic is about showing people how to own their own data. One of our themes this year. So if you guys are going to scale, hope to see you there.

And thank you for everything. That's wonderful. That is nice to hear. Good luck in your talk. I'm sure it'll be great soon. You know, we also got a note in from a friend of the show, Carl George, saying the exact, almost the exact same email. We'll be speaking and we'd love to see you there. Hi, Carl. Oh, Carl's going to be there. Oh, man. Now I really want to go. Okay. Good to know. That's good. Thank you for letting us know. It helps do a very hard calculation. We also got an email in

about Backup PC, a high performance, quote, enterprise grade backup for server to server disk. Holy crap. I can't believe it's still going. When was the last release? July 20th, 2020. Honestly, the age of this project, that's pretty recent because I've been using this probably since the mid 2020s. So I've been using this for a very, very, very long time.

And my most successful data recovery story of all time is thanks to Backup PC. Had one of those situations where an administrative user in the HR department who had too much file access, but politically they'd managed to engineer their access to the file system of the share. And they deleted years worth of hr records and backup pc had it all backed up and i i my breakthrough backup pc was i realized ages ago that hard drives were getting bigger and cheaper than the tapes were and so if i could leverage a backup system where i just rotated hard drives in a raid this is also one of the reasons i began using riserFS, then I could come up with a solution that might be larger and cheaper than tapes.

And I used backup PC for it. And one of the, so what backup PC does, Brian, is it uses your favorite rsync over Samba or SSH or whatever. And it does hard links to avoid duplicating files. And it checks them, hashes them. And if it sees any duplicates, it just symlinks them, right? And the beautiful thing about this is in a bank-type institution,

there are so many people that have the same templates and whatnot in their directories that you actually get a significant savings. So you could actually back up many times more, in theory, than if you looked at it and you did the math and you said gosh you know my users back then my users have two terabytes of data oh man it's so that's so much i don't i can't afford they'll never afford they'll never buy me a two terabyte backup server that's never going to happen well in in practice the backup might have been 600 gigs because it

removed all of the duplicates really really love backup pc it might still be a little bit in that category of a bit more to manage than i want to today it does give you a web ui to manage all of it i almost would love to just try it out out of nostalgia purposes but it is so it is one of your softpot i know but i love it so much and i could see setting up like a little raspberry pi with us with a a USB disk hanging off of it. That's just running on the network, backing up the Odroid. But then I'd still like to get that out of the rig because I'm in a... All of this isn't something that goes down the road.

Like I got to get it off site. You need some off sites. Yeah, for sure. Guys, you are not making this any easier at all. Replicate to Brent's house. It's hard to get to. It'll be really safe. Replicate to Brent's house.

It's hard to get to. It'll be really safe. And now it is time for the boost. NorCal Geek comes in, I believe, as our baller booster this week with a row of McDucks. So thank you, NorCal Geek. Quacka quacka, it's a treasure. Yippee! They write, dear JB community, in all upper caps, come to scale to scale y'all need to come i want to see your faces we're back in pasadena that's true it's gonna be fun so

seriously come okay you got a boost about scale chris that's true gene bean also sent in a boost sent in a row of ducks saying we should go to scale and then bh32 sent in a thousand sat saying skip scale prep for linux fest northwest and then toad rocks the boat sent in some baby ducks to say uh skip scale and save your time capital and energy for something you're excited about it's okay that you're getting old so we're all over the board we're all over the board internally we're all over the board too and i've been fantasizing you know again about a lean mean just jump in the car do a cannonball run down kind of a thing and uh part of the thing one of the things that's tricky in here

is we're trying to figure out how much we want to invest in what conferences and events because i think this is a year to kind of be a little more careful and be a little little more conservative with what we spend our money on there's a possibility down the road that there's a situation that could come up that I think everybody would agree would be worth investing in, but I won't know until I already need to have plans in place for scale. And that's a tricky, tricky meatball right there. So I got to think about that. So I'll have more, but right now don't plan on us being there.

But if there's a way like we can do a cannonball rundown or something like that, maybe we do. What happens is we start doing the math on the Airbnb and the hotel room and the gas and it gets real expensive. And so then my options are, do I turn for a sponsorship? But then like there's a deliverable if I do that and that puts up a whole other layer on top of a trip and it kind of splits our focus. I can, you know, sometimes it makes sense depending on the scale of the trip. And kind of splits our focus i can you know sometimes it makes sense depending on the scale of the trip and i think it's worth going to a sponsor but

other times i'm not so sure and because we're looking at maybe funding it internally it's something we have to be just really super careful about but we'll do the math on that team seven tux or elite tux i suppose came in with a row of sitting ducks not so long time listener second time booster i decided to send some sats i have earned from fountain i decided to try a micro text editor because you guys talked about it in the tuxes honorable mentions and i'm loving it that's great uh greetings from the land of linus torvalds try that pronunciation two worlds two worlds i don't know if you can see that one wes but we got another boost into the show this week i'm going to just skip ahead for a moment on this one and we'll get back into it that also also said, you know what I love?

This was files copying, which is great to hear from you again, files copying. I've missed you. 10,000 sats. Thanks for the great show. The tuxes are the cheat code for software recommendations. Oh, that's fun. It is.

I actually think it's our cheat code for doing a best of roundup in a way that's just a little more fun than just sitting down and doing a roundup. It lets you guys kind of tell us what you think the best stuff is. And as we always see, you have excellent taste, Eron. Yeah, it's a good signal.

It's a good signal. We got a boost in this week from friend of the show, Wes Payne. 5,000 sats. Hey, that's you, Wes. That's me. B-O-O-S-T. First boost from the Pixel 7 via Podverse via Albi.

Nice, Good setup. Yeah. You know, I have that one set up too, but I've been using Podverse as a regular player, but I'd yet to actually get it all connected for value. That's nice because then you can also just boost from the web browser if you want. And there's other podcast apps that are coming online that are going to use Albi. If Antenapod does finish their integration with Boost, Antenapod's also going to go the Albi route.

So then you could use one wallet across all those different scenarios. Yeah, that sounds nice. It is. It's quite nice. Sam H boosted in with 10,000 sats. First-time booster, mid-time listener, about three years. Thanks for the great shows. They have helped keep up my enthusiasm for Linux and open-source software. I've enjoyed trying NixOS along with you and even move my desktop over after my Fedora, originally a 33 install, failed to boot recently. I really like the NixOS snippet

in Linux Unplugged 493. DNS challenge always seem complicated, but this makes it look easy. Too bad I just finished moving my server from Ubuntu to Fedora and fixing up all my Ansible playbooks. Hey, a Fedora Ansible server sounds great too. BN boosted in with 1,212 sets. New listener, still new to Linux, learning how to set up a persistent live boot USB key. Thanks for the show, Billy. Oh, that brings me back. That's a fun.

You know, I played with doing that years ago, and I think I got it to work, and then it was like, okay, well, now what do I do with this? And I never went back to it. Have either of you played with that before? Yeah, I mean, it was kind of one of those first things that blew my mind about Linux, right? Like, one, you could have a USB boot at all. Okay. Yeah, you know, I can like get a full distro going just from this like installer USB. It's not some janky Windows PE environment,

right? But then you can work with all my different hardware, right? It just just works. But then that you could actually abuse and or use that as a basically full system with a little bit of persistence. And, you know, there's been like different methods of how the distributions kind of hang that together over the years. But it's kind of fascinating. I wonder if anybody in the Mumble room has a current way they're doing persistent USB at Minimac. Do you have any thoughts? Yeah. In fact, it's easy. In fact, you have that installation ISO files

that you take on one USB stick and then you take the second USB stick and use that as a target device and then you start installing. It's as simple as that. The one thing that you can optimize in the fact is that you can, for example, your browser, cache that this is in fact

handled by ROM and not by the USB stick to avoid writing cycles. But installation process is easy and with a few optimizations you can use them for years. I have one that I use in my kitchen for the computer

for two years at least. No problem. Even with suspend one that i use in my kitchen for the computer for two years and at least no problem no problem even with uh suspend and and wake up no problem if it is usb3 i see no problem using that ah that's a good one yeah make sure it's usb3 it's a good tip i love that idea just boot from one and install to the other why not it seems so simple now that you mentioned it. Yeah, yeah. Neil, it sounds like you're working on something for Fedora. Yeah, so I don't know if you took a peek at the Fedora Linux 38 change set, but me and now former colleague of mine developed some technology at Datto around improving automatic persistence on USB when there's detected to have enough space for it.

And we developed it and contributed upstream to Drakeit. And as a consequence of that, we are both integrating it into Fedora Linux 38 as part of the next release. It's part of the modernized live media change that is listed in there for 38. So that's still going through, and we're going to try to make that happen. But as part of that, GA38 media, once you boot it up, it detects that there's enough free space for it, automatically creates you a persistent partition, mount the overlays, configure it, and you're just done.

Automatic persistent USB. That's going to be so nice. Right. Like a lot of people don't realize that almost every major Linux distribution actually does support persistence. It's just not configured because such a pain in the butt to set up, or it has to be done manually, or yada, yada, yada. So this feature was so important for us for a particular thing that we developed for our customers at Datto and our partners, that we had to do the work to improve the experience in Drket and we contributed it upstream and now uh we're making it available for all fedora linux users that's great because you know i mean last time i did an install i think my usb thumb drive was like 128 gigs yeah that's not an issue these days yeah it's not at all it's actually really hard to buy a small thumb yeah well that's gonna be and it'll

just work one day that's gonna be fantastic be fantastic. My favorite kind of feature. Purple dog boosts in with 3000 sats. Ah, giving us some feedback about our Jellyfin January challenge. Oh, great. For remote access to my Jellyfin, I'm running traffic on a Linode to reverse proxy over tail scale to Jellyfin in my LAN. Yes. DNS in my house resolves to the local IP,

but outside the LAN, it resolves to the LAN host. This? No open ports on my firewall. Okay, this is so slick. And you and I were talking about maybe setting this up as a tech demo a couple of days ago for our coverage on SSH, because this is one of the big differences

between Plex and Jellyfin. Plex makes remote viewing and sharing a lot more straightforward. Jellyfin, you can do it, but you kind of got to roll the solution yourself. Yeah. And, you know, to be clear, this isn't going to turn Jellyfin into Plex or anything, all kinds of other things you're going to have to solve for. But if you don't want to modify any of your firewall at home and you're already running something up on a VPS somewhere. Or maybe you just don't want to expose your home IP. Sure. You know, you're comfortable exposing a Linode IP maybe you just don't want to expose your home IP. Sure. You know,

you're comfortable exposing a Linode IP, but you don't want people to know your home IP. Absolutely. Yeah. This will essentially just forward the Jellyfin traffic from the Linode over Tailscale back to your actual Jellyfin server and then back to them. And so they think they're talking to a Linode and the Linodes on the back end talking to your Jellyfin on your LAN. One of those things that a mesh network just makes work so nicely. Yeah, yeah. Also great to hear from you again, Purple Dog. Forward humor boosts in with 20,480 cents. I hoard that which your kind covet.

That's a great username. Digging the segment on Nebula and defined networking, I'm interested in ACLs to control what my various family members' self-hosted setups can access on each of our LANs. Jellyfin, yes. Photo backup, yes. File server, no. That kind of thing.

Is this part of the free tier? Thanks for introducing me to this tool. Yeah, I believe so. You've got access to roles. You can sort of define firewall rules and then, you know, say like this role here is only allowed to do X or access these sorts of ports or from this, you know, from this area. It didn't appear, at least in my limited plane so far, that you really had any restrictions there from what you could do in a Nebula config. Yes, but I don't know about the defined hosted part.

I know you had a chance to look at that, but especially when it just comes to Nebula itself, that's all open source. So you could definitely do it at that level. I don't know about the defined part. I imagine that's also just part of it. Yeah, I think so. Okay. Yeah, that's a really nice feature, those ACLs.

I use a similar thing with Tailscale, and that's very, very handy. But I'll do some double checking on that after the show. Okay. Also, you know what? He passes along forward humor, passes along a hot tip. Cash app is now 100% compatible with the lightning network.

This is what's so amazing because like strike and cash app and these podcasting 2.0 apps, they don't have to work together. They're just supporting an open standard. And now the cash app is 100% lightning compatible. And forward humor says

it takes just a matter of seconds to grab your sats with the cash app and then send them over to like the fountain fm or podverse albie wallet instantly like literally instantly he says it works even faster than the now integrated moon pay top off in albie and fountain where you can top off directly says it took a lot longer doing it that way but using cash cash app. Oh, that's so cool. Great to see the Cache app jumping on the Lightning network like that. Yeah, just nice to have easy, simple options.

And I mean, it makes so much sense. We all have these various payment apps. Why not let them talk to each other with Lightning? Well, and it's a big app, dude. It is. That's millions of users that just became Lightning clients overnight. That's huge. Yeah, okay, just following up here.

It does look like you can define roles with the managed Nebula, and in the firewall rules and stuff, you can say, like, allow access to this port from this role, that kind of thing. So I think that should work. You overachiever, Wes. Look at you already checking it out. Okay, well, let's round this sucker out.

Active Shadow came in with 5,000 sats. He said, I'd never heard of rsync.net as an off-site backup solution mentioned on the show. I'm kind of surprised. I use rsync on my daily. A lot of us do. I self-host GitLab, Mattermost, and Vault Word and Data, and I want to back it all up, including documents, config files, and my daily driver work laptop. Well, looks like you ended the math. Did you grab that, Wes?

Mm-hmm. Okay, tell us a little. So it looks like it's based on, what, storage or transfer? What are these numbers? Yeah, well, this is just, I quickly took a look at the pricing page because, you know, I'd heard good things about rsync.net. I've been thinking about, you know, where should I put more of my long-term data? It's always interesting to take a look at various prices as they change. And since ActiveShadow boosted in, I thought, why not? Take a look?

Yeah. And so it's based on different tiers here. So if you have like between zero and 99 terabytes, then that's sort of the starter tier. It's 1.5 cents per gig per month with a 680 gig minimum order. So that sort of works out to a little over $10 minimum spend a month. For 99 terabytes up to? Yeah.

Well, that's if you get the minimum order of 680, and then it scales above as that. Ah, I see. It's 1.5 cents per gigabyte. Yeah. And then it gets cheaper if you have more than 99 terabytes total. Different tier. But what some folks might like,

they've got this little blurb on their website. We give you an empty Unix file system to access with any SSH tool. Built on ZFS for data security and fault tolerance. So then they list some tools like rsync, sftp, scp, borg, rclone, restic, getannex. So it seems like a nice, and just in that Unix way, if you don't mind, maybe, you know, they handle the storage of things. I think this tier gives you seven daily snapshots as well, because it's got ZFS on the back end.

And if you can handle using SSH to send it data, it seems like a pretty flexible solution. Yeah, it's a shame that they're not. I mean, they're well known in our audience, I think, overall, but not like I don't think they're as well known as they should be for such a cool service that you could just build your own tooling around. That's really awesome. Spamproof came in with a row of ducks. It said, I have iDrive for off-site backup. Yeah, it costs a little scratch, but for my Linux, Windows, and iPhone, everything in my home life, really, it solved so many headaches. So iDrive is a recommendation.

I don't think it's an Apple product, actually. Gene Bean boosts in with a row of ducks. I'm using RESTIC right now and storing it in Backblaze. All right, there's one for Restic. Now, I don't know, maybe there's better solutions, but that to me feels like a long-term solution that I could count on. And I mean, as folks know, Backblaze often seems to have cheaper prices. Hydragerum boosts in with $4, 4930 cents a note on restic it supports mounting

your backups directly on your local machine from the remote which allows you to browse and selectively restore files using your file manager choice which is maybe something up your alley chris have you kind of just like oh how do go in? You don't want a complicated command line to go fetch this one remote file that you wanted to restore. Yeah, I almost always just want like one file. Right. That's a great tip.

That's a great tip. Thank you. Cos Peeland boosts in with $3,690. I'm doing on and off-site backups all on ZFS using SnapZend. SnapZend. SnapZend. It looks kind of legit.

They say it's a high-performance open-source ZFS backup with mBuffer and SSH support. Hey, you know, hard to not like tooling built around ZFS. Yeah. If you're using it, it's a great solution. I mean, you just send your snapshots up there. It looks like it's got a UI around it. The whole thing looks pretty nice. That could be cool for a lot of people. It looks like it's got a UI running the whole thing. Looks pretty nice.

That could be cool for a lot of people. It seems like you need a lot of bandwidth, but I don't know. I guess if they're actually, if they're just changes, maybe not. Maybe not at all. Well, okay, I'll round us out for the very last ones. Vectron came in and mentioned that I've been talking about tiling window managers and whatnot.

As you do. They say, I've landed a new job recently, and I'm now using Manjaro with Material Shell Extension for Genome, which implements a tiling desktop workflow. It's got its quirks. Its behavior, though, is very predictable, and it combines the best of a full desktop environment with window tiling. You might want to check it out.

Cheers. I'm watching KWIN. I think in 5.27 there's some tiling stuff coming in. And oh-ho-ho-ho coming in and i can feel my prediction coming true now i agree the moment it's out this neon box boom boom i'm putting it on there i mean the moment mr noodle writes in with 5000 sats i run the genome version of jeruda and uh it's garuda let's not be weird let's not it incorrectly. I found it a bit more approachable than the funked out KDE version. Another quite niche Archbase distro you guys should check out is CachyOS.

C-H-Y-O-S. And a plus one for more Sir Terry Pratchett on the show. People liked that, Wes. Heard loud and clear there, Mr. Noodle. You snuck that one in. You snuck that one in. Pooh Bear sent us 18,820 sats. Not bad at all.

Thank you, sir. Almost the baller booster this week. They say, happy Winnie the Pooh Day. People say nothing is impossible, but I do nothing every day. Inspiring. It is. It is. Yeah, that is.

We also had 16,000 sats from Magnolia Mayhem. But I read their series of boosts in the pre-show. So Magnolia, make sure you check that out if you're a member or you can catch a one-off version on Jupiter 2. If you'd like to send a boost into the show, we love it. It's a great way to support independent media on a decentralized network that isn't owned by any company. And to get a message into the show, it's a great way to share a little value you can boost with albie and podverse or you can go get a new podcast app at newpodcastapps.com and we appreciate everyone who stream stats as well or maybe you boost in and there's not a message those are all very much appreciated and they're all read almost generally in real time

with you you know occasions like when wes and i were working in the studio late into the night last night we we had the boost dashboard up. Sure did. And we had the speaker on. So every now and then, pew, a little boost would come in and we'd read a message as we were working on stuff. I mean, talk about keeping you motivated while you're going. No kidding. Thank you, everybody.

All right. We got some great picks this week in the Jellyfin lane. HDL6C in our Matrix chat is putting together a flat pack for Jellyfin server. Oh, cool. You don't normally see server software flat packed up, but this has been really successful for Plex and Snap. And there is an MB flat pack, but there's no Jellyfin flat pack. At least not yet.

And so this makes it approachable to a whole other category of Linux user that just wants to grab a Flatpak and go. Now, it's new, and they're asking for people to test it. We'll put a link in the show notes. It's about a 130-megabyte Flatpak. But, and I quote, they have really, really jazzed up FFmpeg. I can't remember exactly all the details, but I remember it being like a heck of an FFM peg set up to really make it do all the stuff you might need to do in Jellyfin. All the extra stuff. So that's really great to see.

The Jellyfin challenge is going on for a little bit longer. And Jellyfin just put out a new bug fix release today as we record version 10.8.9 is going on. And, of course, we'll have further coverage in this show and in the self-hosted podcast we'll see how all that goes we'll follow up on that so you still have an opportunity to chime in and uh send us a note let it know let us know how it's going for you it's never too late to switch to jellyfin proud of you too you guys have been you guys have been testing you know what i can tell i can tell you've been you've been doing the jellyfin hours doing the hard work we have some more to do i think we got to put in a little more time, complete our research. Fair enough.

Fair enough. But you know what this means? This means the new rig, the new setup just got us through an entire episode. We've officially done it. It's awesome. The stream's still up. Everything's going. If you're hearing this, it all worked.

If you'd like to watch it live, like this is one of those weeks and office hours, you could have watched the whole thing literally catch on fire. Jupiter.tube. We do this here show on a Sunday at noon Pacific, 3 p.m. Eastern. And of course, if you want more show, go check out Linux Action News. Wes and I did our homework this week, and we do a breakdown of a net filter flaw that can be attacked with some VLAN shenanigans. Turns out it's affecting multiple Linux kernels that are shipping right now. The details are in linuxactionnews.com.

There's a lot more going on in the world of Linux and open source. We talk about it over there every single week. Lots of links this week. Everything we talked about, that's linked over at linuxunplugged.com slash 494. Of course, you'll find our contact page, our RSS feeds, and all of that over there. But with that, we'll wrap it up and just say thanks so much for joining us on this week's episode of the Unplugged program

and we'll see you right back here next Sunday Thank you.