LINUX Unplugged - 539: Rollback Required

Episode Date: December 4, 2023

This week, our embarrassment is your entertainment. Then, we check the age and health of all our disks with one app.

I'm definitely looking forward to hearing your stories this week, boys. I know it's going to be a little embarrassing, so I appreciate you sharing your pain with us. Perhaps more than a little. I had two stories I wanted to pick because they're starting to fade. And one I've told you guys privately, but I never shared it on air because I didn't want to get any legal trouble. That company is no longer around, so I think it'd be safe to share. But I'm going to save that one for another time. And I'm going to share another story this week that really was the quintessential experience

of dealing with Windows servers that finally got this company to make a migration to Linux. So there will be a happy story, but there was a painful journey getting there. Well, hello, friends, and welcome back to your weekly Linux talk show. My name is Chris. My name is Lawrence. And my name is Brent.

Hello, gentlemen. Well, stay a while and listen, because it is our worst mistakes and bad situations that we've gotten ourselves into. This week, our embarrassment will be your entertainment. And then, our new favorite way to check the age and health of all of the discs in a system at once with a beautiful GUI. And then we'll round out the show with some great boosts and picks and a lot more. So let's say good morning to our friends at Tailscale. Tailscale is a mesh VPN protected by WireGuard. That's right.

It'll change your networking game. I've got no inbound ports anymore. Everything's on a mesh network protected by WireGuard. Go check it out and try it for free on 100 devices. Go to tailscale.com slash Linux Unplugged. 100 devices and three accounts too. I mean, that is a lot. I'm on the free plan, and I use the snot out of it.

Tailscale.com slash Linux Unplugged. Yeah, I kind of want to see that, you know, what listeners out there got such a home lab that the free plan doesn't work. Yeah, let us know if you're out there. What are you doing? I mean, I totally understand the enterprise. Sure, yeah, of course.

And a big-time appropriate greetings to our virtual lug. Hello, Mumble Room. Hello, Chris. Hey, Wes.time appropriate greetings to our virtual lug. Hello, Mumble Room. Hello, Chris. Hey, Wes. And welcome back to Europe, Brent. Yes. Well, thank you. Hello, everybody.

And hello up there to the quiet. Listen, yeah, Brentley, you're back in Berlin. I am. It feels like home. I have many of those scattered all over the world, but, you know, I'm adding it to the list. I mean, you definitely must kind of know your way around now pretty well. You know how to get to the places, all that kind of thing. So it's getting to that level, I would imagine.

Yeah, I got from the airport to my hotel without needing maps or searching or having to ask kind strangers how I get to places. Or actually, our mumble room has helped me in the past quite a lot. That's a nice feeling. Yeah, and I have all the right apps. Not the official apps. Those aren't that great. You need, like, the community apps that are doing things better and faster.

So it's, yeah. It feels like, oh yeah, I'm back again. Back into the groove and everything's going well. Well, we'll have more on that in a little bit in the show, but I want to take a moment and let everyone know big news this week. LinuxFest Northwest

2024 announcement. Just on the heels of the mini-fest that went fantastic. LinuxFest Northwest, the full event, has been announced for April 26th through the 28th, 2024. Oh boy. It's here. It's here, and the call for sessions

slash speakers is open. They're looking for like 30-minute lectures or 90-minute hands-on labs or even full-day workshops or mini events. Oh, fun. Nice range. They'll have a link to the call for speakers in the notes. And then also a call for sponsors. Linux Fest is free for everyone to attend.

It's run by volunteers, and sponsors are kind of how we pay all the bills to make it. And you know what? There's things like, of course, you have facility costs, but then there's like little events and stuff like that. So there's a lot of opportunities for sponsors to kind of come in and make a nice local impact. We'll have links to all that in the notes. It's a weird feeling I'm having right now, which is I remember just giving a talk like it feels like yesterday and now that they're already open.

Maybe on behalf of the listeners, maybe I can ask the hard questions. Will it be at the usual event? What's going on? Is there a new location? Yeah. So we do have a confirmation that we'll get access to some of the venue. We don't have 100% confirmation of all of our regular spaces. So we're kind of still keeping things open and up in the air.

But we do have the green light for the Belly Knob Technical College. Well, that sounds wonderful. It'll be good to be back. Yeah. I mean, things are kind of still up in the air in terms of, like, do we want to have other locations as well for certain things, or do we want to have mini events going on other places? So we're still kind of sorting that out as a team.

But the actual fest is back, and it's going to be busy for us. We're going to be hitting scale and then LinuxFest and then Texas LinuxFest. It's going to be a busy Q1, Q2 for us. I hope you guys are up for it. And very excited if you didn't get to make it to the mini-fest. Well, the full fest is back, and I think it's probably going to be quite a hoot. I hope we can see you there. We'll have, of course,

meetups and all that getting together as it gets closer, and we'll have details on that. So this week, we were inspired to share some of our more embarrassing stories, because an individual that worked on one of the Mars rover programs back in 2003 at JPL, which we're familiar with, shared their story that nearly, well, potentially could have led

to the destruction of the rover. Say it ain't so. Derailing the launch and probably costing, you know, NASA and JPL over $500 million. And it all happened during a routine test. Testing the motor, they have this complicated system where they have a test motor and they wire it all up. And then they put the actual production motor in there and see if the terrain they just put it through damaged the motor at all. They don't really take the motor apart because it's very intricate. But they can check the voltages on the motor and see if it's properly ramping up and ramping down as they would expect and if it doesn't if there's some blips in there that kind of indicates that maybe there's a problem they need to look into and so as they were hooking all this up they're

following all their procedures they got a checklist everything's looking right they flip the switch and a ginormous amount of electricity accidentally, in theory, gets dumped into the Mars rover instead of into the test engine just because something got kind of mixed up in the wiring. And at the moment that happens, they lose all test telemetry from the rover. Of course. And the dude that does it, his name is Chris. He's like, OK, well, I knew that bad news only gets worse with time. So I informed the team right away and they're like, yeah, we can verify. We lost telemetry around

the same time everybody stop what you're doing let's have a sit down let's just replay all this and document it and they're panicked because they're on a super tight deadline 500 million piece of dollar piece of equipment this guy you know this is his first job he's hoping this is going to become his career he's just out of school he wants to have a career at j hoping this is going to become his career. He's just out of school. He wants to have a career at JPL. This is his first position. He's working in that clean room that we got to see from up above. So they're in there. And he's just supposed to be validating the motor, and he thinks he just destroyed the rover. So they go through in this post-mortem type meeting trying to figure out what happened.

And they work through everything. They're like, okay, all right, well, let's boot it back up. Maybe it just caused the system to lock up, and'll clear itself and everything will boot fine because if if you're going to dump this much electricity into the rover this is like the one spot that's got a circuit breaker that could probably handle it so we might be okay so they wait till the next morning they boot the rover up no telemetry no data nothing happens and happens. And they're like, crap. And everybody's heart just sinks. They get that horrible feeling in the pit of their stomach like we are so screwed.

So they're like, all right, another meeting. Let's go through every single step. What happened? What did we do? Where did this thing break? And they're going through the beginning of the process before they connected the motor and everything like that. And the individual that was responsible for this accident notes you know i i was actually i was told to to go grab a power meter and i just disconnected that power meter and it had a positive and a

negative and it was connected and it was just metering something on the side that wasn't that important and i grabbed that and i was using it for my test and one of the tech puts his hands up and he says, Oh, actually that was completing the circuit to the telemetry computer. And so they put the, they put the little meter back in, you know,

they connect the little two sides that connected back up. Sure enough, telemetry starts coming back. It was just totally, even though, I mean, it was, it was almost disaster,

but it was a completely different issue. You know, they thought maybe they'd burn something out, but it turns out it was just that that got disconnected and that circuit breaker did actually catch it. And it got me thinking of situations that we've probably been in because we've all worked in the field over the years in different tech companies and different production aspects. And so there's always stories. And I know our Mumble Room will have some, and I know our audience will have some that we'd love to hear. So, Brentley, I wanted to start with you. Do you have any kind of disaster stories that maybe were partially because of your actions? Partially.

If not fully. I think, unfortunately, this one's a fully. I also feel like we all have multiple stories to pull from. Oh, yeah. Oh, sure. So I tried to include one that i think we could all relate to the most and which is on my mind recently uh and has been very frustrating it's on my mind because in one of the latest self-hosted 110 episode 110 alex and i kind of dug into some

upgrades that i'm doing to a local NAS box that I'm building. So I bought some hard drives recently and threw those in, and that box is working great. But I didn't think about this topic for like four years, and that's because this next event scarred me for a time, and I had to just let it sit for a while before I thought about it again. Yeah, I've been there. So back in 2019, I purchased, I did a very similar thing than I did this October.

So 2019 October, I also bought a new hard drive, an eight terabyte MyBook, Western Digital MyBook that I shucked. And this was going to be like, okay, this is a big upgrade. I'm out of hard drive space. This is like the new frontier, right? I'm going to use this hard drive for a while, but I got to put this somewhere. And so I shucked this thing. I have to slide it into some computer I could keep on. So the situation's almost identical. I still have the same need four years later. And so I got inspired and I thought, okay, I have an old computer case and I've got an old motherboard I haven't used in a long time. Maybe I could just get those up and running. But at about the same time, my brother gave me one of his old computers

and had quite a newer motherboard. And I thought, okay, the case that my brother gave me with the motherboard and everything in it that's more modern is just one of those tiny like mini ATX cases. There's no room for extra hard drives and like a hard drive bank. If you've heard the self-hosted, I've got like, what is it? Eight drives or something, Chris? Like more than that, probably. Yeah. A man's got needs. I want to use that old case that I had that was a tower. So I could fit tons of hard drives in that thing, but I just wanted to do a motherboard swap with this newer one that my brother gave me. It seems reasonable, right? How hard could it be? you'll see so i decided to swap everything over so you know pull out the screwdriver and start pulling the motherboards

out of each each um case and doing you know i took my time to i put music on and i thought i've got all evening there's no rush that's the way to do it and i took my time and knew you know i'm probably gonna only do this once so i I'm going to route all the cables and stuff in a really nice way. And, you know, I had to find extra screws because the motherboards had different mounting locations and I had to change all the pins and all that stuff. So it was going really well and I felt inspired and things were going great. And so I plug a bunch of cables in, you know, but I don't know, I guess I'm not a hardware guy a hardware guy but when you're doing this when you're building a computer you two have done this a ton there's so many cables inside that case like it's like usb port hub cables for the front of the

computer you've got power cables of course there's like four different style of power cables for whatever reason like i don't know who designed these things but anyways i plugged all the things in to the right places and i thought all right this is gonna go great this is pretty easy i feel really good about where all these cables are going and like i'm actually finally getting this project going so i hit the power button and the drives spin up but only for a moment and then everything kind of just goes dark, except for this little red blinky light that is on the motherboard. Uh-oh. And I thought immediately, that's probably not a good light. Yeah.

And it's not really booting in the ways that I'm expecting. Did you get like a post beep at all or anything? Anything on the screen? There were four beeps, if I remember correctly, and nothing on the screen. Okay. So you do have a screen connected and nothing? Nothing. Oh, boy.

So I thought, oh. Yeah. Well, maybe I just plugged something in wrong or something. So I unplugged all the drives immediately. Well, okay, I turned the power off first and unplugged all the drives immediately because I thought if there's something wrong with this motherboard, I certainly don't want it to affect the drives that I have information on that I care about. So I took a long time, probably like an hour of like double checking all the cables and making sure. Yeah, yeah.

And I was smart enough to take photos of each motherboard before I took them apart. So I double check. Yeah, yeah. All the cables are in the right spot. And so then I'm like getting desperate and I thought I gotta pull the manual out and read about this new motherboard like maybe I'm doing something wrong that I'm not catching so I read and read and read and in fine print I do find this little statement that maybe is helpful and it says something like Something like, do not plug a USB hub into the IEEE 1394 port. I thought, I don't even know what that port is.

So I had to look it up. And sure enough, you're probably giggling, Chris. The IEEE 1394 port is a FireWire. FireWire. FireWire. Long live FireWire, I say. And sure enough, I had done that so i just unplugged it and i thought okay well i'll just unplug it and reboot it right so i hit reboot

nope get that blinky light with the four beeps again and then so there's more fine print and it says oh yeah if you do this it will just fry the motherboard and you're just sunk. Oh, no. It's the exact same connector for the USB port versus this FireWire port. Oh, man. And I knew at that moment that it was just, you know, this is the end, like the doors say. Project stops. Hard stop on the project right there. I gave it a day. And the next day, I started taking the motherboard out,

swapping it back with the old one. So I did swap it back to that older motherboard that I had, and everything seemed fine, except I went to plug the drives in, which is eight drives. You plug the first one in, plug the next one in, and I get to the eight-terabyte drive that I had just purchased. It was the last one because, plug the next one in, and I get to the eight terabyte drive that I had just purchased. It was the last one because, I don't know, that's just how it worked. And in the process, I had accidentally nudged the data cable,

the serial ATA cable, and I broke it. Oh. So on this brand new drive that I installed into this machine, I had broke the serial ATA cable off of it by, I don't know, accidentally hitting it. I don't even know when it happened, but it was a dark day. That is a double whammy too. That is my story. Wow. All right. You just had to build back.

Build back better. This time you know better. So this time do it right. Right. This time. I mean, the drives are at least running at this very moment doing sort of a stress test. So I've gotten that far. I've gotten further than the last iteration. So I think we're worth celebrating. So you'll get home and those things should be well broken in. Toasty. I want to hear your story, Wes. You got a kind of embarrassing story for us? Yeah, I got two. Maybe a quick first one and then sort of the real one. And I guess really it starts early. Apparently I just have a penchant for changing stuff that you really don't need to change.

I recall as a kid, you know, we had a Windows 98 maybe became ME. It was definitely ME at some point. Strong memories there. And I knew just enough that I thought it'd be fun to play with changing the startup image. Sure. You know, and that you'd like patch some boot files. It was like a bitmap image you could like overwrite in there. Literally a dot BMP. Yeah. So that was my first experience pretty young of, you know, just breaking the family computer and then having to spend all day repairing it when my mom was trying to get things done. I had a, there used to be like a Norton Commander UI for DOS. Oh, yeah.

That was kind of pre-Windows. And somehow I put it in Split View. And it was on my dad's computer. And he could not figure out how to get it out of Split View. And he hated it. And he gave me so much crap for breaking the computer. And it stayed that way until they got Windows. That was, yeah.

It also reminds me that for a while I had this Dell laptop. And it was pretty decent for the time, you know, well-specced, kind of, you know, consumer-y grade. But other than that, a good machine. Had one of those hybrid graphics cards that, you know, never worked great under Linux. But you could kind of make work. It was also in the time right as, like, the start of the EFI transition was kind of happening. And while this shipped with like a traditional BIOS, there were some models you could get. There was like an alternative firmware.

And then enterprising folks on the internet had figured out how to patch that in a way that was compatible with the EFI support that Apple was kind of like one of the first to really roll out. Oh, I know where you're going with this now. Yeah. And so I was like, okay, well, I'd really like, you know, my girlfriend at the time was doing a bunch of After Effects and other things, and I was thinking, it'd be interesting if we could turn this into like a render farm for her

or otherwise have another machine that she could use. So I was walking down that Hackintosh path that you know so well. And I thought it'd be a good idea to just take this firmware, I downloaded it off a forum on the internet, and flash it on my main laptop at the time and and had that go it actually worked great oh yeah same firmware that's on there damn it i don't use it it's one of those old laptops but uh thankfully nothing went wrong but it just makes me think what were you thinking did you even have a rollback plan you know what one of the things i did it's even a little bit crazier i think is because i spent money is there for a hot minute there was this company that was producing firmware images for certain machines that you could buy on a USB thumbstick from them and plug it into your computer.

It would reflash your system to look like a Mac, to Mac OS. And it worked. And I think you even had to have like a USB thing always plugged in or something for it to check. And Apple cracked down on them so fast. But I got my hands on it. It's like 50 bucks or something. And it worked. And it was really awesome.

And so you could have Windows, Linux, and Mac OS on one x86 machine. Wow. But you're right. Like I didn't even think about back then the security aspect of some hokey, you know, fly-by-night company building some hacked together firmware. Also, Chris, you literally got a USB drive in the mail and just immediately plugged it into your main rig. Heck yeah. Heck yeah. Well, I mean, there was nothing on it, right?

Because it was a built Hackintosh. But yeah, it just didn't think about it as much back then. And today, replacing your firmware would be a ludicrous idea. Except people still do it. Yeah, I mean, sure. Oh, yeah. Oh, I'll do it still. You know, you just maybe think about it a little more first.

I know, though, Wes, you have one more for us, don't you? I got two more. Oh, two more, Wes? One's quick, though. Yeah, okay. Just in the theme of me, you know, updating or playing around with firmware things I shouldn't. Yet he continues. Yes.

I know. Yeah, it's a pattern. I think I've stopped. I think I've stopped. I think I've stopped. This was after college, and I hadn't yet really got like a tech job, but my girlfriend at the time had got her sort of like professional job. So she moved for that, and I was going to move in with her for the first time. That's exciting.

Yeah, it was very exciting. Big step. Mm-hmm. And her parents were nice enough as a sort of like moving present to get her a new flat screen TV. Oh, sweet. And I'd seen that it was new enough that it supported like the Miracast stuff or at least like a proto version of that. And, you know, being the at the time unemployed nerd I was, I was looking for weird random hobby projects. I was like, oh, well, it'd be cool to be able to stream from my Linux machine.

Yeah. I was like, oh, well, it'd be cool to be able to stream from my Linux machine. So I found out how to get into the, like, you know, I think the admin firmware, sort of like what the TV tech would use to, like, go get in and make adjustments. Because that's how you had to go, like, enable this. Do you remember how you found that? I mean, got to be more reading sketchy forums on the internet. You know, like, oh, here's this model. And, like, here's the, like, the right commands you can press at the right time when it's booting. So I get in there and, you know I enable stuff, and I'm playing around.

And I accidentally click on a button that enters into the state where you can load in new firmware from a USB drive. Right on. But I didn't have any firmware to load. I didn't have a USB plugged in. But I didn't know how the interface worked, and so what I thought was going to be the back button was actually the button to trigger loading,

and so it just bricked itself right then and there. Because it just tries to load like an empty image. That's terrible. And, you know, I turned it off and on. I tried to press whatever I could. Nothing was working. This is a gift from her parents, from your girlfriend's parents. Did you own up to it?

Because, like, did you go like, I don't have to admit to this. We could just claim technical failure. Yeah, I did. I did. But I didn't own up to the tech that we called. And it was still under like the one year warranty. I don't know if that would have covered what I was doing,

but I just played dumb. And I was like, you know, it was working. We went to bed after watching a show. I tried to use it the next day, and it just wouldn't turn on. Because you could see on the board the power light after the converter and stuff was all working. It's just like no output. The rest of it was never going to work. So thankfully, the tech just replaced it, and it was okay.

Well, I mean, technically, the firmware went bad. That's a component. You know, it's funny because you kind of would hope today when they maybe because consumers are a little more savvy today, they would try to do some sort of sanity check and maybe see if there's a file there. See if you got a drive mounted. See if it's a valid file before you start writing, you know, those kinds of things. That would be nice. But it's extra blow, Chris. You just don't need it.

Linode.com slash unplugged. Head on over there to get $100 in 60-day credit. And that's just a nice way to support the show while you're checking out the good news. And that is Linode's now part of Akamai. Yeah, that Akamai. That Akamai. But they're keeping all the tooling that we love, like their beautiful cloud manager, their well-documented API, and their command line client that's like a Swiss army knife that is so useful when you're just even locally on your box or if you're on one of the servers.

All that stuff we love about Linode, it's still there. But now it's combined with Akamai's power and global reach. I mean, man, Akamai, the Akamai. And they're expanding their services to offer more cloud computing resources. And as part of Akamai's global network of offerings, they're also expanding data centers worldwide. We recently just took advantage of a new data center in Switzerland. It means you get more resources to help you grow your business. Or if you're an open source project, you get closer to your users. Just gives you more optionality, even just for like offsite replication. You know, it's just all these areas that are just so positive about it. And

the pricing has remained fantastic. You still get that hundred dollar 60-day credit so why wait go experience the power of linode see why we've been using them for years and how now with akamai they're going to help scale your applications beyond what they could have ever done before because this is akamai we're talking about go check it out see why we love them support the show and get 100 in 60-day credit it's linode.com slash unplugged that's linode.com slash unplugged. That's linode.com slash unplugged. All right, well, a few years later, I find myself helping to manage

the Chef infrastructure of a company. And this was a lot of fun learning stuff. I have no particular love for Chef, but it was a system that worked at the time and was doing a lot of configuration management, and they'd rolled it out to pretty much most of their fleet. I came on board as this, you know, they were kind of switching over from their legacy in-house system

and adapting stuff to Chef. It wasn't a huge fleet, but it was a couple hundred KVM servers running a few thousand VMs. We tested on those. Those were working. We were trying to be pretty diligent. We had various QA environments and this was running there first and all the dev teams were involved and they had to kind of like

if not right, at least approve of the Chef recipes that were going to be running the servers that was running their software. But of course, not everything was in configuration management. Of course. And one of the things I was working on was a recipe that was going to help

updating our host file. So, you know, I would go make sure, write the host file, make sure it matched, make sure, like, it had the right you know, the right name for that machine it was also searching the right domains, everything that needed to be happening. Sure, okay. Unfortunately, some of the machines that had not yet made the complete migration

were still running Chef. They just weren't using it for everything. They were using it to install some of the base that had not yet made the complete migration were still running Chef. They just weren't using it for everything. They were using it to install some of the base stuff, you know, the common libraries that all the hosts needed to run the machines. But they weren't using it for the hosts file. And I didn't know that. So I go and it's the data rolled out. We'd sent all the notices. We told people we were doing it.

I hit that button. Machines start updating. Looks like it's all going well. I log send all the notices. We told people we were doing it. I hit that button. Machines start updating. Looks like it's all going well. I log off for the night. You go home? Yeah. Oh. Oh boy, so you've left. Because it looks like the deploy's gone well. We've checked things. But you don't know. It's broken and you've left.

Yeah. And then I get paged at three in the morning because the chef server's not working. So one of the machines and this was probably wasn't even supposed to be running that recipe that you know that had a problem with it was the chef server itself so okay i go figure that out i roll those things back i think you know i look around it doesn't look like anything else is really affected that at least that i have purview into get the chef server up and running think it's like five at this point i'm like okay

i can go back to bed. And then it becomes clear that, well, I guess a little context, this company was in the, related to the telecom space. They were kind of doing analytics on top of phone calls. And that meant that the phone calls actually had to flow through our servers to let people communicate,

to have a working call connection. Oh, man. And yeah, some of the machines that were still using custom host files were the sip systems and that meant i broke calls oh jeez now thankfully once the people who actually you know those systems i wasn't an expert in i didn't really even touch or at least normally didn't think I was touching, but of course this rolled out to like the entire fleet. Those people alerted me to what's going on. I was able to roll back that host file change for everything. And then they had to go manually

restore. Wow. You know, my story kind of starts with an unattended update that I wasn't aware of that causes chaos in the background that I have to kind of like put together. that I wasn't aware of that causes chaos in the background that I have to kind of like put together. Oh, that's what's happened? Yeah. So it starts like any other day. There were some days where I would drive down to Seattle and I would see a couple of Seattle clients. And it was December-ish, right about this time of year.

So it was kind of cold, but it wasn't freezing, but it was very dark and rainy. And I had to get to my first client in downtown Seattle up in one of the buildings at 7.30 a.m. It was a gynecologist office and I had to get there before they opened because I had to get things fixed for them before it started business day. And, you know, they specialized in cancer treatment too. So like the type of work they did was really kind of time sensitive and really kind of doesn't really allow for downtime because they never really take a break. And so I was like, okay, I have this hour from like 730 to 830 a.m. And then I will be able to test and have everything ready by 9 a.m. So I get down there and it's almost 8 a.m. And my BlackBerry, because it was like, I don't know, 2003, my BlackBerry, whatever it was, starts buzzing like crazy, starts going off.

And I'm getting tickets and I'm also getting phone calls and I'm getting emails and it's all coming in at once. And I'm like, it's like 7 a.m. You need to calm down, right? I'm like, what is going on here? This is horrible. Well, it's a new client that I'd been assigned a week or two and at most I'd gone in there and I'd kind of done a little like walkthrough of their network. I kind of documented what servers they

had, what applications they were using. You hadn't even gotten to the stage of implementing any new changes yet. Hadn't even logged into a server yet. I was, in fact, I don't even know, I don't even think that morning, I don't even think I had the admin credentials for their Windows domain server. I'm like, okay, well, tell me what's going on. And so what they report, and I

remember it so clear because I always hated these user reports. The internet is down. I can't get on Internet Explorer. The internet is down. And so because the internet is down, we can't schedule. Well, this client was Kenmore Air. They're still around and they're an airline, a local airline.

And they have a lot of scheduling that they try to keep track of because they also are like a subcontractor for Alaska Airlines. So they do some local flights for Alaska Airlines. They have like certain commitments. And if they don't meet those commitments, it costs them a ton of money. So everybody's in a panic. All right. So I wrap up with the client and I spend like what feels like the longest 20 minutes navigating five miles in downtown Seattle because I had to get like on the other side uphill, of course. And compounding matters worse is that it's raining and it's dark and people are trying to get to work and I'm trying to get to this office.

And it was not fun. But when I get to Kenmore Air, which I can just – I still can picture walking through the front door and the gal there is laying it on thick like, oh, we're so grateful you're here. You've got to fix this for us. The boss is really freaking out. OK, well, tell me what's happening. Well, we don't really know. It's just internet isn't working for the pilots. They can't check their schedules.

Like, OK, why? We don't know. So I get on the phone with one of the pilots because they're already calling in. Because each one of them is like discovering this as they get to work that morning. Of course. And so they tell me, yeah, I can't open up Internet Explorer. Like, okay. Because they got like some crappy ActiveX-based booking system. I mean this is old school stuff.

Yeah, it was the time. And he tells me it's urgent. They've had to ground all flights. So you got to fix this for us. You got to fix this. Time is money. We got contracts They've had to ground all flights. So you've got to fix this for us. You've got to fix this. Time is money. We've got contracts. You've got to get this fixed.

Like the pilot is really worked up, right? And I'm starting to kind of get a picture. This is a bad situation, but I have no idea at this point why their crappy Inter Explorer won't launch, right? Like I don't know. I don't know why you won't launch. So I have the pilot kind of walk me through step by step, boot the computer up and walk me through it step by step. Oh, you're doing this remote too, right? Yeah.

You don't even have it in front of you. No, because I'm at their headquarters. Right. That's tricky. Yeah, the pilots are all over the place, right? Some of them are up in Alaska and yeah. And so what becomes clear as he's going through the process that what's actually happening is his Citrix Metaframe connection for Windows Terminal Services isn't connecting. Okay.

So he can't get to his desktop where Internet Explorer is. So therefore Internet Explorer is not working. So we're working it back. Look, I'm supposed to Internet Explorer. I can't Internet Explorer. Yeah. So the Internet is broken. And that's why I always love those reports.

So, I mean, and, okay, so I always love those reports. So I mean – and OK. So I'm figuring out he can't get into his remote desktop. So that way they can't book a flight. So why would the Windows Terminal services be down? Do you have any guesses? Like I mean off the top of your head, like with the information I've provided at this point. I'm going to say update gone wrong. Yeah.

Yeah. You could be close. You're in the zone. It took me about five seconds to actually reproduce the problem. I also, even from the local office, could not log in to the server. So there was definitely something wrong. And there were message boards and stuff, sort of, online. And I was certified in Citrix Metaframe at the time.

So there was this form I was a member of and all of this. And I go in there and I probably literally find hundreds of various reasons why a Windows terminal server might not be accepting connections. It's a lot. It's a lot. It could be anything. But this was the beauty. And this was, you know, again, this is early aughts. This was like, holy crap, the internet's going to change my career because i found a form thread that was fresh that day and it clicked it was sort of the same kind of overall experiences and they also were a mcafee antivirus user oh and mcafee antivirus this is built before intel owned them and all that stuff they had they had sent out a new set of updates, like an engine update and a new pattern update.

And then at some point in the night, after McAfee had updated itself, it did a system scan. And it scanned all the Windows servers because they had McAfee on everything. Because you got it, right? Because it's the diligent thing to do. Of course. Security. And McAfee helpfully detected some critical DLLs that Citrix Metaframe used as malware.

Oh, no. And it quarantines them. It quarantines the file, which causes like the login coordinator to crash and all of this, right? And this was an interesting situation because you could see McAfee realize they had a problem. And I think they were a West Coast company at the time. So they're kind of like on the same schedule I was. And you can see them like like their support line's jammed. And that's when I knew I was on to something

because other Citrix users were having the same problem the same morning I was. I mean, it's a dark place, but that's a good feeling. Right. A little light in the dark. Yeah, it's like, okay, they're working on this. So I'm telling the pilots, this doesn't make them much happier though, right?

So they rush out a patch. And I think there was like a specific program we could run on the server where we'd like download the latest stuff and install it. And I run that McAfee program that does that. And then I can recover those DLLs and reboot everything and the terminal services came back online. But when we were doing the postmortem and I was explaining what happened, it really opened up that doorway to have the conversation around using Linux. And ultimately what I did for this client is I took that one application that was a super old jank ActiveX application that required IE,

and I virtualized that server in place and moved it onto a Linux host and moved all of their other back-end stuff onto Linux. And it worked. That was one of the few times I'd really taken Microsoft, I think, made the tool to convert an existing Windows box into like a VHD file. And then I could just drop that onto a Linux box and just run it. And so they came in the next day and they still logged into their terminal server, but it was a VM. That's so smooth.

It was like it opened up, they cracked that door and they're like, this is ridiculous. Right. But you've been telling them there's problems, but finally you're like, no, they're, I mean, you can't deny it. These systems aren't working for you. And I'm like, let me tell you the good word about Clam Antivirus. I did actually end up putting Clam AV on their file server because they still wanted antivirus protection. And Clam AV never deleted DLLs, causing the system to crash. How did I know that the moment you mentioned McAfee that this was going to be one of those kinds of stories? The bad old days, man. It was the bad old days. It was the way things were.

Jeff, I know that you have a story that involves more like hard, hands-on tech that could have gone pretty bad. Oh, not could have. Did go pretty bad. Uh-oh, what happened so my first two weeks as a commercial solar technician which was also my first two weeks living in the pacific northwest or rather you know northern california we were tasked to uh re-terminal re-terminate a whole bunch of inverters on a large site again this is commercial there's something something around 50 inverters on this site and we'll re-terminate a whole bunch of inverters on a large site. Again, this is commercial. There's something around 50 inverters on this site, and we're doing somewhere around a dozen a day. Each inverter has six independent strings,

and we have to re-terminate all those. Take them all out, strip them down, put them back in, tighten everything up. So on a lovely day, nice and overcast, luckily, I was down to my last inverter. And these inverters are sitting pretty low on a metal roof. And I just repositioned myself as my screwdriver tended to push forward a little bit and arc out the strings. Uh-oh. AC to DC.

Sorry. Positive to negative DC arc. You know, all the ac was off but dc is always on these things positives negative dc arc this thing went and it went hard typically i mean i've arced out stuff inside inverters you know residential stuff and you might get a little bit and it'll just go out real quick now this thing wouldn't go out it just kept going and going and going and luckily everything's metal the whole roof's metal everything there's nothing to catch fire but uh you know it was a nice flash and it just kept going we we did get it to go out with a fire extinguisher and um about 20 minutes

later a cloud broke and it kicked back up that's when we traced down the the pipes going into the inverter, traced them all down, unplugged all the home runs, which we should have done in the first place. But yeah, that was fun. And I'll share some pictures and mumble. Yeah. Okay. Wow. I can imagine one day getting myself into a similar situation as I keep trying more and more stupid things. Okay. We would love to hear your stories. If you're listening out there, please boost them in. Remember, we are trying to raise funds to get to scale. It could be a great way to support the show and tell your horror stories out there. I know some of you listening have got them.

Share your pain with us. You're amongst friends. Now, Brent, being in Berlin, we got a meetup coming up soon. We do. I would like to say, hey, this is low expectations, low effort meetup. Last time, well, the last several times I was here, we organized some more sort of formal meetups. I want this to be an informal meetup. I mean, I haven't even really told the venue or anything. So, but at Seabase here in Berlin, there's a regular Tuesday NixOS meetup that happens.

But you don't have to know anything about NixOS. It's really just an excuse to all hang out in the same space. If you've never been to CBase before, it's an amazing experience. If you have been to CBase before, well, you just want to go back. So not this coming Tuesday. So not the Tuesday you're hearing this if you're listening to this as it first comes out. But Tuesday, December the 12th, I'm going to be at that NixOS meetup. So it happens at C-base, and it usually starts at around 7 p.m., but you're very much allowed to be late to this particular meetup.

I always am. And it goes on until quite a bit later. So if you want to join, if you want to join us, the crew, please do. There's a Matrix room that we usually link to as well, the Berlin Buds Matrix room. We kind of organize in there. If you've never been to Seabase before, you will get lost. So you might need a friend to help you find the door. But if you're interested in hanging out, please, please come.

I'm bringing stickers and they have a whole barrel filled with stickers there, too. It is your opportunity to attend one of the renowned Berlin and Brent meetups. You don't have to use NixOS or like NixOS. You just have to tolerate people talking about NixOS. And if you've got this far into the show, you've already managed to do that. Now, the Tuxes approach. Very, very close. It is your last moments to go vote. Tuxes approach. Very, very close. It is your last moments to go vote. Tuxes.party.

They are open. Of course, we also have a form linked in there if we miss something. You can put it on our radar. Nominate your favorite projects. We do use that term loosely for our 2023 awards. What is our number, Wes Payne? 2,060. We got to our 2,000 goal.

If you want to just put it over a little gravy on top, you still have a small window to get your votes in. And of course, we'll have a link to those. You voted. Tuxy stickers by our golden dragon. Two dollars for the digital version. We'll have a link to that in the show notes. It's all coming up. We'll have links in the show notes so you can get to that and get your vote in there because we want your opinion represented in the 2020 Tuxes. Again, it's tuxes.party. Collide.com slash unplugged. Hey,

if you're in IT or security, you got to listen to this. The problems have only gotten worse with bring your own device and now work from home. I mean, there's a lot of nice trends about it. I enjoy choosing my own device. I enjoy working from home, but there are security challenges. There are technical challenges that come with that. And often it's employees that don't mean to cause any harm. Some old software on their machines at a date, they get a vulnerability. Their credentials get phished. Maybe they don't have the right software on their machine. They're connecting remotely to be compliant, but nothing prevents them from actually connecting. So you're technically out of compliance these are just like you know the top four or five issues that come up and i could think of like a bunch more but the true problem

isn't really the intent of the users they don't mean to do harm it's just kind of modern technology especially especially in the work from home and bring your own device world, which I don't think that's going away. I think Collide's solution here is pretty slick. For those of you that are dealing with this, if you work with Okta, Collide ensures that only secure devices can access your cloud apps. So you don't have to deal with phish credentials or software that's out of date,

and it gives you a single pane of glass, one dashboard to manage your Windows, Linux, and Mac boxes. That also means that you can run reports to make sure everything is in compliance if that's necessary to generate those kinds of reports. But also one of the really slick things that Collide does, and this is the part that I think would have really extended my life in IT, is it empowers employees to fix their own problem directly with just really intelligent, simple messaging. Taking advantage of your messaging platform, giving them the message to how to fix this.

Hey, you know, you need to have XYZ antivirus installed before you can connect. Go here to get that. And you can do it in a way to make sure that everything looks clear by your company's policies and communications and procedures, and they can solve the problem on their own without having to contact IT. So it's just this really comprehensive solution so go to collide.com slash unplug to see a demo because it really it's like wow right how can it do all this well go see k-o-l-i-d-e.com slash unplug get a demo and get some insights into how seamless all this is and it's a great way to support the show, too. So it's collide.com slash unplugged.

Well, we've arrived to the feedback section, but Chris, I think you have some exciting news in this regard. Can you share? Yes, the 32-bit laptops that were sent in by Spazzy C have arrived, Wes. Do you want to do the honors of the unboxing? Oh! I've pre-opened it a little bit here. So I cut the tape. There should be two Dell 32-bit machines in there, which... This is a heavy box.

Yeah, it is a no joke. You know what, Wes? They built them big back then. Oh, yeah, there you go. Get in there. It reminds me of the map sounds. The map sounds very similar to this. Yeah, that's, well, you know, whenever Wes gets ASMR-y.

Oh! I see gold! Or silver, I should say. Nicely packed in there with some bubble wrap. Oh, yeah, look at that silver and white beast. Oh, hand it over here, Wes. Hand it over here. There's two of them in there. Oh, God, it's heavy! It is really...

Oh, here's a power brick. Oh, yeah. This is back when the laptops were definitely bigger than the power bricks. Dell Inspiron 6400 with media direct keys and, you know, hard drive LEDs. It's a big boy, Brent.

It's a really big boy. Oh, yeah. It's got VGA on the back. It's making me a little jealous because the little 32-bit laptop I have is like a tiny little netbook. So now I feel inadequate. Wes pulled out a second one. It's got a DVD drive, so does this one. It's got an

Ubuntu Mate sticker on it. Yeah, these are nice looking rigs. These are going to do just fine. It has a Mate sticker. That's so perfect. Thank you, SpazzyC. He sent these in so we could do this. That's a huge value-for's so perfect. Thank you, Spazzy C. He sent these in so we could do this. That's a huge value for value contribution. And I was looking at my notes because I keep notes. I think Spazzy has been listening since like 2014. Yeah, how about that? So that's really exciting. We're going to be digging into this. We will have details on how

the 32-bit challenge is going to work next week so you can participate as you listen along at home. I don't know exactly because the holidays are coming up, but we are roughly targeting, this is a soft target, but we are roughly targeting perhaps our first episode of 2024 for the 32-bit challenge. We don't know for sure yet because we're still working everything out. I guess that means we've got work to do. We also got a little piece of mail from Soltros

giving us an update on a project. A little bit of an update on my NixOS config builder tool. I'm still ironing out some of the bugs, but if everything works right, anyone should be able to take a plain old NixOS ISO, boot it up on a Steam Deck, run my config builder tool, select steamdecksupport.nix in my tool, my config builder tool select steam deck support. Dot Nix in my tool, and it should build valves game scope, including the steam deck firmware and set up gnome shell,

which is a more touch friendly desktop. It loads up through Jovian Nix OS that project. This is a really neat project. It's getting some traction in our matrix room too. We'll link to his config builder in the show notes as well. So yeah, getting a little Nixification on the Steam Deck. All right.

Well done, Siltros. I like it. I like that a lot. I'm going to be checking it out after the show. I don't know if I'm going to nixify my deck because one of the things I really like about my deck is it just works. You haven't tampered with it.

However, if we were to have like a spare deck or something like that, yeah, it's definitely a contender. And now it is time for the boost. Yes, it is. And Devator, sounds right, is coming in with our baller boost this week. 450,000 stats. Sending in from Podverse, which by the way, I think I was looking at her stats. Podverse isn't the number one app used for boosting, but it is the highest boosted value.

So the Podverse users loving the GPL app out there. And Dev writes, first time booster, party member, and I've been listening since the pandemic started when I found self-hosted. Thank you. Thank you. Thank you for all the ways you support and for listening. JB Shows are the only podcast I've ever listened to more than one episode despite trying several others. In honor of Chris's one-year anniversary with Graphene OS, here's a boost to reset all the silly naming conventions back to best guesses. I love it. Yes.

What a game changer. All right. We're going to have to think about this. This is a big moment. No more giraffes. All right. Yeah. Yes, this whole boost is just for the hard or softness of letters. Oh, and I guess scale, too.

That is so great. Thank you, DevTor. Really, really appreciate that boost. Appreciate that support. Thank you for listening, too. And I've been loving my graphing oh it feels so weird to say it os huh now somebody just needs to get us to stop saying genome that would be great the do to buy boosts in with 340000 sass.

Boosted in from the podcast index. First time boosting. Here's a little something for your trip to NixConf. If you divide by 10, you'll find my current zip code. And a tip, it's in the south of Europe. Well, Wes did bring his worldwide map this week. Thank you, The Dude Abides. See if we can find you.

Appreciate that generous boost. And thank you for taking the time to set up boosting. I know the real hurdle is just getting it set up. You've done that part. Now, can you find him, Wes? Yeah, looks like it's a postal code in Montpelier, France. Well, hello, France. Thank you for boosting in, TheDudeAbides.

Appreciate that. And he says, P.S. More to come. Well, I'm looking forward to that. I look forward to hearing from you. VT52 boosted in with 150,000 sats from Podverse. I'm not sure about your fuel efficiency, but I figure this should at least get you to about Portland on your scale trip. Best of luck. And I'm looking forward to excessive amounts of NixOS coverage.

Wink, wink. Who, us? Never. VT, thank you. You know, I was just doing the math again, too. And when we did the math for driving down there in car or RV, we did it at $6 a gallon because California is extremely expensive. But by March, it's possible gas prices will be down because they're coming down here right now.

It's like regular. The lowest I found right now is like $3.39, something like that. That starts changing the math a little bit. So we may get in a nice spot where we get a good amount of support. We get gas prices come down a little bit because we're either driving cars or driving RVs one way or another. Fingers crossed.

Yeah. Thank you very much for that boost, VT, and for that support. Bam Ham 182 comes in with 100,000 sets. Hey, rich lobster! Now, this is a pet peeve. It's not really a hill I'm going to die on, if those semantics matter, but dockers, they're not a thing. You're just managing containers with Docker. Here are some stats towards the scale,

because Nix, and on that topic, an awesome feature snuck into SteamOS 3.5, and that is that slash Nix is now persistent, which means you can manage your Steam Deck with Nix. It's everywhere, and I love it. Oh, that's great. Okay, that's great to hear. Bamham coming in with the handy SteamOS 3.5 info.

And I saw you nodding along, Wes. Do you agree with this pet peeve that people should stop calling them dockers? Yeah, I don't know that I have to hear it too much, so thankfully I'm spared. But, yeah, I like that take. Hey, gentlemen, I have been keeping track of the sats we've read so far, and we just crossed the million sats mark. Oh! One million space bucks.

A million! That's... Wow, thank you everybody. We're on the way to scale already, I can feel it. And the boosts keep coming. Droopy Draco sent in 100,000 sats. Hey, rich lobster! Ah, from Castomatic. Hey guys, first time boosting in? Certainly not the last. Oh, great. Hey, Rich Lobster! to the void, about to take the plunge. So thanks for spurring this on. I have little doubt that some Nyx coverage from this event will be the final little push I need.

That's great. Thank you very much for boosting in there, Droopy Draco. I like that username, too. And I appreciate you taking the time to do that first-time boosting. I know that is the uphill battle there. Now, the only challenge, the only thing I'm worried about a little bit is if the Nyx experience goes bad, is Droopy gonna

blame us? That's worth the risk. Okay. All right. Okay. I think you're probably right. Mick Zed P boosted in with 100,000 sats, also from Castomatic. Wow.

Hey, Rich Lobster. Simply saying, let's get the crew to scale. Heck yeah. Yeah, we're getting excited about it. Thank you. This is awesome. Before we got connected on the live stream today, Brent and I were talking about, yeah, it's been too long since we've gotten together and done an event. Like, it's

just been, you know. Since we flipped an Airbnb upside down. Yeah, right. We did Linux Fest, which helped, but it was just a small amount of little taste. Very much looking forward to getting down there and covering that. Rotted Mood comes in with our last boost of 100,000 sats. Hey, rich lifestyle!

Using Castomatic as well, and Rotted Mood just says, Scale boost! B-O-O-S-T! Thank you, Rotted Mood. Appreciate that. Woodcarver boosts in with 81,474 sets. All right. To scale and beyond. Thank you, Woodcarver.

I like that. I wonder if 81,474 means something. I feel like 474 is very intentional. Did Wes look that one up? Sneaky. I think Woodcarver will have to let us know if I'm on to something there. Well, Moonernite boosted in with 66,666 Satoshis. Ooh.

That's a lot of asses. Get yourselves to NixCon. I bet these sats will be worth more fiat by March as well. Could be. As we approach the habit. That would be a really nice little development. You never know, of course. That's something we have to go into with an open mind.

It could go both directions but chances are we could have a situation where the these go to work for us as the having approaches and which would be april so you never know we'll see we'll see kiosera comes in with 22,666 sats using podverse they write, when it comes to paywall bypass, we were talking about ladder last episode, my view for certain content like news is they need an escape hatch. It'll give me like a single article. Maybe I could send them some sats or fiat fund coupons and just buy a single article for a reasonable price.

If I find myself purchasing enough articles to make it worth it, well, then I would just buy a subscription at that point. And here's some sats to get your arses to Mars. I mean scale. Well, thanks. Yeah, thank you. Torped boosts in with $22,222. Things are looking up for old McDuck.

Recently added storage to my system with two different types of disks, NVMe SSD and regular SSD. I got to see what it's like to add storage to a Linux system, and it was glorious. I added the SSD as a physical disk, added it to my existing LVM-LG, and was able to expand my root, temp, and var partitions across the disks. Then I added the NVMe as a new VG and LG, used DD to move my home to the NVMe, and updated the mount. I did all of this hot and live. Yeah.

Only had to reboot once I moved the home partition. I know how this feels because it was like, what, a few months ago, six months ago, I had that big storage victory on my workstation where I redid all the storage hot, you know, did it live. It's so awesome when you get it all done and everything's the same. It looks the same, but yet your storage situation is much improved. Feeling you on that one, Torb. Thank you for the boost.

Well, deleted boosted in with 12,345 sats from Fountain. Hey, since we needed a second Spaceballs boost. Is this our second one? Oh, this must be from last week. Yeah, it was. Yeah, I think it was a live boost that maybe we didn't get filtered into the show. Well, he gets it. 1, 2, 3, 4, 5.

That's the stupidest combination I ever heard in my life. Thank you, sir. Thank you, sir. Sir Alex Gates, the podcasting 2-to-0 consultant comes in with 10,240 sats using the index. Well, I officially tried NixOS. It required me to compile a custom version of the latest kernel

to release a custom device tree and a custom bootloader in order to run the Orange Pi 0.3. The documentation assumed too much inherent Nix knowledge, and half of it used flakes. Needless to say, Fedora came to the rescue, and I'm following back to Ansible for automatic configuration. The majority of the deployment configuration will be for Kubernetes anyways. All the Nix talk always reminds me of Haskell programming. It's ironic that I found Rust easier to learn and more practical. Serving down

a solid criticism there. I will also agree. Sometimes the configuration sort of is written in a way that assumes domain knowledge and they're like yeah just do it with doing the thing and you're like but i don't know how to do the thing i don't even know what the thing is i think that's sort of the point of like expert level things like nix are are kind of geared around like these platforms are all built around this idea that you rtfm and you build up a wellspring of internal knowledge before you do anything meaningful with it. And completely agreed.

And so it's like, it's a way of life. It's a way of life. Yeah, and that's a tough first sort of Nix project. You know, maybe trying like NixOS in a VM and getting the lay of the land will be a little bit easier. But if your environment means you need, you know, good Orange Pie 0.3 support,

then Fedora and Ansible sounds like a pretty nice setup. The Golden Dragon boosts in with 12,345 sets. Baseballs boost. So the combination is 1, 2, 3, 4, 5. That's the stupidest combination I've ever heard in my life! Scale boost. Won't be able to make it myself, but here's something to help you guys get out there. Let me know if you need your sticker game to be

on print. Oh, we probably will. I think that's the same. We should talk more about that, GoldenDragon. Thank you for the boost and yeah, we should, yeah, we gotta stickers. And I'm gonna bring down some LinuxFest Northwest stickers to spread the good word about LinuxFest so yes, good thinking.

We'll have to circle back as they say. Southern Fried Sassafrasises boosted in with 12,345 sats. And you know what that is, Chris? One, two, three, four, five. Yes. That's amazing. I've got the same combination on my luggage.

Keep up the good work, gents, from the neighborhood mog. Aw, thank you for the boost. Using Podverse there, too. Like to see that. Takiro comes in with one, two, three, four, five sets. We're going to have to go right to ludicrous speed. About the 32-bit challenge. I hope one of you plays to the strengths of NixOS,

considering that you could apply the same configuration that you have right now, automatically utilizing the packages that are also supported by 32-bit x86. The same is very true also for utilizing Nix-Darwin as well as NixOS. Same configuration worked nearly 100% when applying on my MacBook. Something to consider since you could likely get started with a nearly identical environment easily. Kind of sounds like one of us could rescue the other. One of us should definitely try this. I feel like Debian's on that list.

I think Tumbleweed, right? Well, maybe Brank can handle that. Yeah, yeah. No, I've done it already. He's our dedicated expert. You're our Seuss correspondent. Yeah, but have you done it with 32-bits? Right.

It might go better. Is that what you're saying? That's half. Oh, definitely. Yeah, using Seuss on 32-bit hardware that's super old is going to go way better than it is on your brand new dev one, I suspect. Oh, my gosh. The facetiousness is so strong here.

You know, I think we should go the totally opposite way. We should try some crazy things we haven't even tried before. Like, I don't know, some more obscure distros, perhaps. Okay. All right. We're going to have to have a little meeting. I think so. Davril and Boosin with 3,000 cents from Castomatic. The best way to not have to deal with paywalls is the cookie auto-delete enabled with Firefox. As soon as I close the tab, the cookies are deleted, and they have no idea that I'm a returning user.

And then we get a little link here to Firefox add-on cookie auto-delete. Might not be a bad idea in general. Could mess some stuff up. Could also be good for privacy. Yeah, for some people who don't want to install an add-on for this, using private mode, you know, if you can think of it while you're doing it, that can sometimes work. Not always, but sometimes.

Faraday Fedora boosted in with 2,222 sets. Hey, sorry I haven't boosted in a while. I'm trying to stack some sats for Christmas. Oh, what does that look like? I want to know more there. I sure hope you guys can make it to Knicks Fest. Here are some ducks to help you get there. Just need 3,599

people to boost the exact same. Well, you're starting it off fair day and we sure appreciate it. Yes, we do. Yes, we do. Shajara comes in with 5,000 sats using Fountain. First time boosting. Try it again. You don't think I got that one right, do you? I don't. I mean,

I kind of like just seeing what comes out. Shajara comes in with 5,000 sats using Fountain. You like that one? They say, first time boosting and here's the sats I've earned using Fountain. Wow! 5,000 sats earned using Fountain. Well done! Hope this can be used towards the scale trip. I've always loved the content and figured I'd help out.

Thank you. Thank you. And it's another first-time booster in this batch. So great to see. Thank you, everybody. E-Rock boosts in with 3,456 sats. Duplicati user here. I run nightly backups from my on-raid server to OneDrive

through my personal Microsoft 365 subscription. Duplicati has been mostly reliable, except for one filter, non-critical data, thankfully, that has had to be recreated a few times. I don't verify as much as I should, but the files I've had to restore in the time I've been running it have been restored without issue.

Okay. I do appreciate that boots on the ground report, Iraq, as I am, as I said before, a duplicata user. We're using it right now to back up my configs, my app data, my pictures. So I'm pretty, pretty tied in. So if anybody's had any horror stories, I'd like to know. Drag it boosted in with a row of ducks. Hey, I'm just on unplugged 5.3.2, so a couple episodes back. Though I will say I'm an avid Gentoo user for 10 plus years,

and have it on a couple dozen devices over time, including a 200 MHz Pentium One and a PS3. Ooh, but there's some expectation management. It is not easy. It is not fast. But damn, do you get a sexy system in the end. I'm actually right now playing with Nix since I am a little behind on the declarative idea. But why not all the way, like CoreOS? I have actual reasons why Arch, apt-based distros, and others are for me, and I still have to make up my mind yet about Nix. Fair enough.

That gets me thinking, thinking actually quite a bit man i can't imagine running gen 2 on a pentium 1 200 megahertz i might have done it back in the day maybe but i don't think it was pentium 1 generation it must have been pentium 2 3 generation must have been i did it on a pentium 3 yeah way back in the day because painful but all night builds well it was all i had right like i didn't i didn't have anything better for a long time how long would that take to build like do you have any well i remember kde took three days i do remember that i also was doing this on dial-up chris worth it yeah oh yeah three days to download three days to build no no

no no no no longer than that to download. I think for me, getting the base system downloaded, plus just X386, because we're way back in the day now. I think it was like three weeks to download all of it. You know how we care now about how our systems, like if they're using more than 10 watts, we really get bent out of shape? Well, you spent three weeks with an ancient thing just running like crazy just to download some stuff. We've come a long way. Yeah, that's true. That's true.

Dark Matter PHP Dev comes in with 3,900 SATs using fountains. It's great to hear the discussion on managing dot files. I'm just starting to explore this for the first time myself. I've been looking at Sto with Git, but I'm early enough into my journey not to be committed to a particular solution at this point. I'd be eager to hear solutions to keep up the awesome work. Now, this is a big topic. We are also discussing this in our pre-show about dot file management for KDE. This is an area where I could do better in general. So I would also be interested in hearing people's suggestions for what's

worked for them. I know this is a rabbit hole. I think you had a suggestion, Chris of a cheese. Moi, what did you call it? Was it cheese? Moi?

Yeah. Yeah. Cheese. Moi is one we discussed. Yeah. Yep. Cheese. Moi,

of course, which is the proper pronunciation as everybody knows. That's one point against Stovas. You can't mess it up. Oh, I could. But I'm curious. Maybe we can get something from fellow boosters. Help Dark Matter PHP dev out.

Help us out. Be interesting to see the wide array of different solutions folks have to this problem. Soltros boosts in with 2,349 sats. I know you guys use the fish still. So I was wondering if you've ever written any fish functions in your configs. I have one for rsync backups, one for mounting said backups over sshfs,

another for backing up deconf settings, and the works. Oh, that's a fun question. You know, I'm sure I've written a couple over the years. I think I tend to just end up putting scripts in a folder on my path and running those, but fish is great. I'm taking notes right now, though, Soltros. That's a great idea. I think I tend to just end up putting scripts in a folder on my path and running those, but Phish is great. I'm taking notes right now, though, Soltros. That's a great

idea. I like that. I love Phish. Control-R is so much better on Phish, too. It is really so fantastic. Well, at user came in with 6,500 sats, saying, hey, just clearing out the fountain wallet, and another 6,000 sats with no message

provided. Well, thank you for thinking of us. Appreciate that. Tyler? Tywer? TYR comes in. I know I always have a hard time with that one, but appreciate the boost anyways. 3,000 sats using podfers. Debian is good and all, but the NixOS talk has finally pushed me to try it out. NixOS is now my primary

OS and my daily driver. I'll be keeping it for the foreseeable future. And the Nix nerds community in Matrix is a lifesaver. Debian, though, is a good base for Nix. Oh, man. So fun. So fun. Thank you, everybody who boosts in.

We had a really incredible week. We are trying to raise 8 million sats to get to scale. We had 28 boosters, 32 boosts in total. We didn't get to all of them on air, but we raised 1.6 million sats to get to scale. We had 28 boosters, 32 boosts in total. We didn't get to all of them on air, but we raised 1.6 million sats. Thank you, everybody. You have brought us to the first stage in our milestone to 8 million sats. I am amazed at that initial result. That is very great. We'd love to keep that rolling. If you'd like to support us, you can get a new podcast app at podcastapps.com, and then you get a button right there in the app to boost. You could also get Albi, and then you can just use your web browser, getalbi.com. And I also discovered recently, I was playing around with the Bitcoin dad pod.

You can use just Strike or Cash app directly if you use Fountains. If you go Google search Fountain FM, Linux Unplugged, there'll be a boost button right there on their website that gives you a QR code. You could just scan that with Strike directly. I tested this. I didn't even have a SAT balance on Strike. I just have it connected to my account, my checking account. And I scan the QR code with Strike. It instantly grabs the SATs, sends them over, does the whole transaction.

You don't even have to have an intermediary app. You can keep your podcast app. You can just go use Strike or Cash app directly from the Fountain FM website. It's really neat. And now Strike's available in 36 countries. I'm hearing from a lot of listeners that are using it to get the Sats to boost in, and it's working really good. But there's a lot of options. Podcastapps.com will have links in the show notes.

We really appreciate these boosts. We are hoping they'll go up a little bit. We'll see because that could help us get across that milestone as well. But we're also trying to prepare for the fact that the price could always go down as well. So your support can help us do that. Also, we had a lot of you sign up to become members. Had a good group. We could use another good group this week. If people want to become a member, unpluggedcore.com.

You get an ad-free version of the show. And you can also pick the bootleg version of the show, which at this point is clocking in at two hours and 42 minutes. There's a lot of content in there that we make just for our members, including news stories like we talked about RHEL phasing out X11, the new release of SimpleX, Brent's travels this week, as well as our NixOS upgrade experiences are all in the member show this week. And that's another way we can just say thank you for supporting us. We really do appreciate it.

It is a value for value podcast. So if you got a little value out of it and you want to send us some value back, the boost or membership are a great way to do that. The app picks this week could have been the entire show. It's one of these weeks where it really could have been the whole show. I want to talk about scrutiny. It is a hard drive smart monitoring system that is so simple to get up and going. You can run it directly on your box or you can get a Docker Compose going

or something like that and have it up and going in just mere minutes. It'll use the smart CTL tools and stuff like that to scan your disk and then give you a web UI with all of your hard drives listed, with how they're doing, their age, if they've had any issues, their average temperature. There's actually a surprising amount of data inside there that these disks have that you just don't really have a UI for. And when I ran it, I discovered that some of the disks

have been running, not just in age, but have been online for 10 years in our Arch server. The youngest is four years old. The average age is like seven. There's some nine-year-old and ten-year-old disks. We all bought them used. They've been running for... That's online.

Online continuously for ten years. That's when I realized it's time to start budgeting some new hard drives. I think so. Yeah, scrutiny's great. You know, Chris, this whole episode was about things we did that we regret, and I feel like maybe this is starting to suggest something.

Yeah, foreshadow a little bit. Now, we did note that if you have MVME disks... Yeah, they call it out in the readme as well. There's an additional, at least if you're running Docker, it's really easy to get started. They've got a command. You probably will want to put it in a compose file or something else, but if you just want to try it out, there's a one-liner Docker command you can run. But if you've got NVMe. You probably will want to put it in a compose file or something else, but you just want to try it out.

There's a one-liner Docker command you can run. But if you've got NVMe disks like I do, you've got to add an additional capability to the container so it can properly see those. Good to know. But I was impressed. I mean, super smart looking. Looks like it's got webhooks too.

So if you want kind of getting alerts out of this thing, that's neat. Yes, there is some alerting mechanisms and all of that as well. And the dev makes some great stuff. Analog J is behind this, but they also make a library called Lexicon that Let's Encrypt and Certbot uses. We talked about it back in LUP 307. It can be also super useful if you want to roll your own, like, dynamic DNS support. You know, get your fresh IP and then have Lexicon handle talking to your DNS provider. Thank you.

That guy's an all-star. Something maybe we should check out, which I haven't, working on a project called Fasten Health, which is an open-source, self-hosted personal family electronic medical record aggregator designed to integrate with 10,000 different insurance hospitals, clinics, etc.

This is so needed. Wow, that's impressive. Yeah. That's exactly the kind of thing that should be self-hosted. Indeed. I remember, go vote in the tuxes, tuxes.party. Just a little bit longer, that window is about to close. And then we are fishing for some boosts. Have you had a bad moment?

Did you RM-RF something you shouldn't have? Did a backup fail just at the wrong moment? Share your pain with us. Share your pain with us. Also, we'd like you to start sending in your 2024 predictions. That way we can steal a few of them and get inspired as well. You could do that by going to linuxunplugged.com slash contact or, of course, please consider boosting in.

I'd love to start getting some prediction boosts, start getting that kind of just percolating in our minds. Who knows what could happen in 2024? It's coming up. It's coming up quick. See you next week. Same bad time, same bad station. Yeah, that's right. We're wrapping up. Why not make it a Linux Tuesday

on a Sunday by joining us Sunday at noon Pacific, 3 p.m. Eastern Live. Got that mumble room. We got that chat room. We just love that live energy too. Gives it that special flavor that requires you as an ingredient. Details and times at jblive.tv and jupiterbroadcasting.com slash calendar. That mumble room is open.

That's at jupiterbroadcasting.com slash mumble. Get in there and get a low latency Opus stream. Links to what we talked about today. Well, that's linuxunplugged.com slash 539. Bunch of great shows over at jupiterbroadcasting.com as well. Thanks so much for joining us on this week's episode. We'll see you right back here next Tuesday, as in Sunday. Thank you. you