LINUX Unplugged - 355: Chris' Data Crisis

Episode Date: May 27, 2020

Chris' tale of woe after a recent data loss, and Wes' adventure after he finds a rogue device on his network. Special Guest: Drew DeVore.

The internet cannot figure out how to quit Vim. How do you get out of Vim? Well, friends, how about a simple tool? Wes Payne brings us the Vim Killer. I mean, I was thinking of you here, Chris, because as you know, I'd like to alias nano to Vim on really any system that you're going to come across. So why not install a hardware kill switch?

You're familiar with the mute pedal you've got down below you for taking a cough. But what about when you run into trouble with your least favorite editor? I can never get out of it. It's just too darn hard. If only there was an easier way. Well, Vim Killer is a button on top of an Arduino wrapped in a 3D printed enclosure. It's expertly crafted, and with one button click, you can finally quit Vim. What could be easier? I just 3D print a case, build a tiny computer, install software,

develop that software, connect it to my host system and set up software on that side, and then when I'm all done, one button press to end Vim. It's easy. And yet, it's still easier than learning Vim. It's easy. And yet it's still easier than learning Vim. Hello, friends. Welcome to Linux Unplugged episode 355. My name is Chris. My name is Wes. There's no kill button for this show, Wes. This is just too much show. I'm a wounded soldier. I'll tell you my tale of woe and how I managed to completely bork up my home server last night. And Wes discovered a rogue device on his network.

We'll tell you how he managed to track that down. We have a bunch of community news and some feedback. But before we get into that, we have some pleasantries. I'd like to say hello to Mr. Cheese Bacon and Drew. Hello, gentlemen. Thanks for joining me. Hello, Internet. Hello. Hello, guys. Hello. You know, this is nice. You, me and Wes here could just have a little picnic while we do this. We should do that sometime, a podcast picnic. We'll all order the same meal, have it right here snacking, but don't get it on mic.

Yeah. Although we'd have to figure out a way to include that virtual lug. Time-appropriate greetings, Bumble Room. Hello. Did a bunch of you sneak in during the intro? I feel like that's a lot of people that weren't in there before, but it's good to see all of you. Welcome into the show. I apologize. I'm a little sad today because of a data loss situation that happened last night. And of course, I spent the morning refusing to accept what happened, trying to recover the file system. Data loss? You said data loss? Yeah.

I am so sorry. Before we get there, this actually is really cool. We were joking around in the pre-show. I mean, it's silly, but this Vimk killer button on top of an Arduino wrapped in a 3D printed enclosure. I kind of like this idea for just a general close the application function. If you could expand it beyond Vim so it supported any application and you just hit that red button. And we'll put a link in the show notes so you guys can see this. You just hit that red button and it just closes whatever app you're using right there. Maybe it just kills it even.

How great would that be? and it just closes whatever app you're using right there. Maybe it just kills it even. How great would that be? Kill it. What if I told you that you could easily do that with a few buttons and like an Arduino Pro Micro? What is it, the Elgato that all the streamers use these days? Yeah, right. You can do the same thing with just a little Arduino Pro Micro, which can act as a HID device and boom. That'd actually be kind of a fun segment. You know what else would be fun is to make a homemade OBS streamer rig switcher kind of thing that is way cheaper than that Elgato because that Elgato OLED or whatever it is panel is like $300. And I don't even think it works with Linux.

I might be wrong. I've never tried it because it's so expensive. That's why you got to go with cheese brand products. That's right. I got you, fam. Cheese printed and approved. Yeah, I think the Elgato works with Linux under OBS, but I think to program the buttons, you need Mac or Windows.

I've never really tried it. I'd be curious to know if anybody has. This is the streamer's life. So anyways, let's talk about a little community news. We really just have one story we wanted to dig into, although there are many. Maybe next week we'll cover what happened with the Gnome Foundation and their patent suit, but I'm waiting to get a little more details on that. I'm hoping to get more information. So let's talk about this Raspberry Pi firmware update. I got a lot of people telling me about this one. People knew I was looking out for this.

This is an update that will be official eventually. Right now it's in beta that enables the Raspberry Pi 4 to boot from a USB SSD. No SD card required at all, which we were just recently talking about, Wes. Yeah, that's maybe where it's a little bit confusing because, I mean, you and I have already been talking about using a, you know, USB mass storage device for your root file system, but you've still got to have the bootloader and all the requisite files sitting on that SD. But really, since launch, Raspberry Pi 4, I mean, there's been promised functionality around booting directly, you know, right off with no SD card required, right from USB. Finally, as of, I think, May 15th, there's a firmware out that lets you do

just that. Well, that is, if you can get that firmware installed. So in theory, a very simple process. You edit a Raspberry Pi EEPROM update file, and there's just one line in there about firmware release status, and you change it from critical to beta. And then you do an apt-get update and there's a package that gets updated and they're called rpi-eprom. And that package will pull down the latest firmware and put it on the file system. And then there's a pretty easy rpi-eprom update command where you just point it at that bin file that's now sitting on your file system

and it writes it to the Raspberry Pi's firmware. You reboot, and then in theory, from that point forward, it will boot from a USB device. For us, on my test Pi, because I've got one just on the bench here at the studio that I use for testing these kinds of things now. This is how you know that you're using these Pis in production, is you've got a whole staging environment set up for your pie. You're dang right, Wes.

I'm taking this serious here. I'm not going to deploy beta firmware on my production devices. That's what operating systems are for. I admit, I did put Ubuntu 20.04 on my pies before it was released, but that was different. Several weeks before. Okay, yeah.

But I'm not doing this firmware, but I will put it on this test device. And this is kind of a nice candidate. I got it two, three weeks ago, and I have a fresh Raspbian SD card on there, and I got a fresh SSD on there. So it's like the perfect candidate to try this firmware. And no matter what we did, it would write the new firmware in theory, it would seem to, and then it would say, okay, reboot for it to take effect, and you would reboot or power off, power cycle, and it would still be the old firmware. Yeah, I'm not quite sure what's going on. I mean, I think to the best of our knowledge, we followed both the official advice and there's a handy article that sort of summarizes things,

skips some of the intermediate steps over at Tom's Hardware. It sort of summarizes things, skips some of the intermediate steps over at Tom's Hardware. And in theory, the new firmware includes a bunch of defaults that sort of do exactly what you want. They try the SD card and then, well, if there's no SD card, boot from USB. So all you have to do is install the new firmware and let it use its defaults to overwrite any sort of bootloader customizations you've got installed.

That's one command. But after rebooting, we got nothing. Nada. Yeah. And I'm not really urgently trying to fix this because for me, having just slash boot on the SD card and then the rest of the file system on a USB SSD is perfectly functional. You know, that's one of the things that's using Linux,

like that's a behavior that's entirely supported in all of the models we have. So like just having this separate little boot drive is really not a big deal. It's just sort of, they mentioned that this support would exist and, you know, therefore we want it. Now, I have a tale of woe about my Raspberry Pi home server last night. If you want the full, like, multi-Pi setup, all of that, it's well-documented in the self-hosted podcast, so I'll just make that disclaimer. So some of this is a shortened version. Oh, God, I'm so sad about this.

It's actually still hard to talk about. So some context is soon I'm heading out in Lady Choops, my RV, to go to Austin. It'll be about a nine-day drive down there because we'll be stopping a bit along the way. And then I'll be there for about a week and then I'm driving back. And so that's a long time and I'm bringing my three kids with me. You've got a full load. Yeah, seriously. And including production gear, wife and dog, and that's food and clothes for all of us. I mean, it really is a full load. And it's a lot of time on the road.

And I wanted to sort of re-engineer my home storage setup so that way I could get it ready for this. And during the LUP plug, Neil convinced me I should try ButterFS. I was going to use MergerFS for my cousin, and I thought if it worked out well for him, I'd use merger FS at home to accomplish this. Okay, but there's a way to do it with butter FS, and it seemed like a superior route to go, and it probably is, and I'm going to still try to do it. I come into the situation kind of in a rush because I'm trying to get everything ready to go and get more storage copied over and trying to just get it done really fast. And so I ended up doing it late at night when I probably shouldn't have because I had that thought cross my mind. Like, I'm not feeling my sharpest right

now. Been a long day, been on, you know, some Zoom meetings during the workday. And this was probably about 930 at night. So that's late for me. I mean, that's the only time you have, right? Where you're just like, I'm sitting here, I'm thinking about the problem. I should just fix it right now. I have to drop the RV off for some work before we leave. So I only have it for a couple of more days before we're gone. So I really only have like last night to do it. Although it's going to be tonight too. So I'm doing it later than I should. But my idea was, is I need to shut down a couple of my Raspberry Pis. They're producing too much heat and I want to shut down two of them and consolidate the functionality onto one. I'll shut down one of them.

I removed the hard drive, and on this hard drive contained all of my media for home, like movies and television and all of that, and my Docker containers and, you know, all of their data. All my other, like, my Compose stuff, and really there's nothing else installed on the host OS. That I have separately and of course i have backups going for all of this i'll make that disclaimer up here so i disconnect the drive hook it up to the new raspberry pi that i'm going to be consolidating to

and i'm doing all of this through an ssh session on my laptop and it's about a 45 to 50 minute conversion process when you run the butterFS convert from Extended 4. So I sit down to run this because I know it's going to be a bit... Oh, you're converting. Hmm. I mean, that's a feature that is advertised, right? Yeah, and it, in theory, is a fairly straightforward process, unless you manage to screw it up in the dumbest way possible, like I did. Interestingly enough, no matter what I did,

the new USB device I hooked up would show up as dev SDA and my root now became dev SDB. But because I'm using labels for all my mounts and for my bootloader, everything was fine. I tried changing different USB ports and I moved the new disk to a hub and the main root disk to the still plugged into the back of the Raspberry Pi. So it should be like directly connected to the Pi and it still shows up as dev SDB. It's fine. So I had to verify I was working with the right disk. And I was. I verified it was dev SDA.

So I set off to do the Butterfest conversion. Step one is I did a full check disk, you know, FSCK, whatever. Made sure that the Extended-4 system was in good shape. Verified everything was right and made sure everything was unmounted. This sounds so responsible. I kick off the ButterFS conversion. And, you know, it's going, it's going. It's on a Raspberry Pi over USB, so it's going to take quite a while.

So I decide I should probably start working on, you know, it's going to be tomorrow morning. I'm going to need to have a show ready to go. I should probably work on it. I start opening up the dock and I start working on it and things are just, they're going real slow, Wes. And this is where impatience and time crunch and sleepiness came together. And I made a horrible mistake. Oh, no. And then that was compounded by another failure. So first stupid thing I did is I was trying to load some webpages to do some show research,

and just the internet was so throttled up where I'm at at this campsite that I just couldn't even get a webpage to load. Literally could not even load webpages. So I decided, screw this. I'm going to tether from my phone. Yeah, you've got this LTE plan you pay for. Yeah. And the Wi-Fi is running on my phone all the time. So I just jump Wi-Fi networks. Boom, pages start loading. I immediately start filling out information in the doc.

About a minute or two into it, I don't know, I lost track of time. All of a sudden, I just stop. I have this realization, oh no, I've just disconnected from my SSH session in which I was running a file system conversion. What have I just done? The heaviness of that hits me. No screen or TMUX session in play. Nope. I have, and this is something I'm going to work on now,

I have developed a habit of always using, in the past it was screen, and now it's been TMUX or even TM8 sometimes. I think TM8's pretty great too for what I'm trying to share. Whenever I'm in a remote session, I at least have MOSH and then I'm almost always using screen or TMUX. Now, that's just a habit I've built up anytime I'm remote because I'm on this MiFi so often that connections are unreliable. But for whatever reason, when I'm on the LAN, and it doesn't matter if it's a box here at the studio or if it's at Lady Joop's, when I'm on the LAN, I just don't think to use those tools because I'm on the LAN.

It's not going anywhere. It's steady. It's stable. Right. You're not leaving, so your session won't break. Right. I realize the mistakes that are stacking at the moment so i switch back over to the local land bring the terminal up then i just sit there does it say anything does

anything happen do i get dropped back to my local box so i hit enter and it returns to the command prompt with no error but i'm still logged in like it resumed the session i'm not using mosh it just resumed the session i don't know like if it timed out or what because it might be i don't know it's all happened really fast in this point because i was beginning to freak out you know calmly freak out but i was really starting to worry and so i immediately thought well i believe the way the conversion works is it really doesn't do the final stuff until the end. And if I interrupted it mid-conversion, it's building that tree, but it hasn't switched over to the ButterFS file system yet. And so perhaps the extended file system is still fine. So I run another check. I run an

extended for file system check. It comes back clean. Oh, did you see any processes still running from your previous command? Wes, I was in a fugue state. I was in a fugue state, Wes. I was like, oh no, what's happening? Right. And plus it's like 10 o'clock at night now and I'm exhausted because it's been a long day. And I, it was Memorial Day. And so I was, I did a bunch of chores, which I felt really awesome about. Reorganized a bunch of our storage to get it road ready. Felt awesome. Had like a two-hour power hour meeting. Felt drained as hell from that. Like just, you know, one of those big days. And then I'm working on this after a big, big meal, which was delicious. But also draining. This is where I should have stopped because the extended four check came back clean.

But I thought to myself, I got to get this done. I'll run ButterFS convert again. And I'm sure if something's screwed up, the tooling will give me some kind of air. This doesn't sound good. You can tell I was exhausted. So I set it up to go. I set, you know, caffeine, the little gnome extension caffeine. I clicked that. So that way my laptop stays awake all night long. You got it running this time in Tmux, ready to go. Ethernet into the cabinet. Like I am dialed in, man. I am not letting anything happen again, right? I hit enter and it's going and I'm sitting there. I'm getting tired. I'm getting tired. And it's just sitting there clock, just no feedback because it's just creating image or whatever it says. And I decide, all right, I'm going to bed. And I set my alarm for 545 a.m. because I figure I'll get up, try to finish the work, and then run into the studio for the show.

Because, you know, it's quite a bit of a long drive. So I got to, like, get up early. So I set the alarm clock. I wake up before the alarm goes off, a half hour before my alarm goes off. I'm up, and I'm working on this thing. I'm sitting there chugging a Red Bull while I'm, like like one hand on the keyboard, the other hand slamming a Red Bull. I'm down to clown, like getting this thing going. Look at the output of the ButterFS convert completed successfully. It's got all the stuff on there that all the guides say it should have. Looks good to go.

I'm thinking at this point, oh my God, it worked. It didn't do it until the very end, which I interrupted. So that way it just started over from scratch and it took hours. So I assumed it must have. It's got to be good. So I think, okay, now it comes down to this. Mount file system. I get an error that it can't be mounted. I check the logs. A bunch of errors are in there, but one of the ones that I've pulled out now after running ButterFS repair and check is could not find extended items for route 256. That just keeps coming up every time I do a check. So I brought it into the studio. I ran like all the rescue things against it. I ran all the different stuff. I cannot get this file system back up. It's there. You can see the

size of it. You can see how much is used, but you can't get to the data. It's infuriating, You can see the size of it. You can see how much is used, but you can't get to the data. It's infuriating and it starts to dawn on me. I'm going to need to recover. All right, let's start that process. I've got backups. What a mental shift though, right?

You're like, okay, well, this system is lost. What do I have in backups? And maybe, you know, I still, I have all my Docker compose files. I've got the configuration and backup and the media files I have other copies of. These are not my only copies of them. So I can recover. So I go to log into Google Drive, download my last Duplicati backup, which happens every night at 3 a.m. So I'm just going to go get the previous night's backup. And I see a red bar at the bottom of my Google Drive where it says I am using 886 gigabytes of 15 gigabytes allocated. What? Yeah.

You see, back in March, my debit card expired. And the way Google works is you can go add your new form of payment. Maybe you have a YouTube Red or a YouTube television or Google Play subscription. You can go add it back to one of those services, but it will not add the payment information to all of the services. So my Google Photos, for example,

like that stuff in there, which I still have those old photos on there. I still want them stored on there. That's why that number is so large. My YouTube Premium, my YouTube TV, each one of them I had to reactivate individually. I guess I just didn't think that I would need to go manually check on Drive. I thought I would get some sort of email saying, hey, man, your Google Drive storage subscription is up and you're using 850 gigabytes or 870 almost gigabytes of space,

and you're only getting 15 right now. Can we do something about this? None of that. I didn't get none of that. I had no indication that Duplicati wasn't saving to drive. Lesson learned here, never using Google Drive again. I was simply using it because I'd already paid for it ages ago, so why pay for something else, I thought.

So my backups are basically beginning of March for all of my configuration, my Plex history, my Smokeping history configuration. All of it is basically March because nothing was generating errors, neither Duplicati nor Google Drive. So I don't have anything current current so i'm gonna have to re-stand up all the applications and recopy over all the media files while i'm trying to get out of town and it stinks because it's months and months and months and months of tweaking and perfecting. And also kind of a blow to this is all of my documentation was on that system. Oh, no. Yeah. And I could restore like the March backups, which I will. But anything you've done, you know, all the helpful hints you've added.

Ironically, I needed that documentation this morning and I had written it locally on my desktop, so I just had it. But it was like, here I was this morning. I already needed that documentation because I was installing software on Dylan's laptop remotely. And I, you know, needed my credentials and all that. And it's just such a shame because it was such a dumb mistake. Ultimately, my mistake was not using something like screen and then switching networks like that. And then I think you could back it up one level and saying, I even had that moment of, I'm not really at my best right now. I can tell I'm a little tired, a little punchy, but I had thought this through for days and I had it all written out, all the steps that I was going to take. And I had it all just sort of by

the numbers at this point that I thought, I mean, all I have to do is just follow my list of procedures. Like I'd already kind of gone through and looked at all the commands I was going to run and how I was going to structure it on this new machine and kind of just made notes for myself to follow. So I thought, good to go. But what happened was, is I was that impatience and being under too much of a time constraint and the slowness of the internet where I was at, because I'm on a MiFi and it was way throttled. I thought, you know, I'll just switch over. I'll do show prep while this conversion goes for an hour or so. You were trying to optimize. Yeah. And I think the lesson learned there, too, is when you're doing something at the file system level, as excruciating as it might be, just,

I got other systems I could have used. I could have used other computers. I just, that's a unitasker at that point. I think it's hard because we've done so much to move things off that sort of unitasker model. And most of the operations you do, I mean, especially if you have backups, you know, they're not unreplayable. You can go back in time or you've got the backups for it, but a giant media filled file system is one of those, it's too expensive, you know, so you have fewer copies of it. Yeah, I think I'm still going to go this route that I was set it out. I'm still going to go with ButterFS on this disk because, as Neil pointed out when I was comparing it to possibly using MergerFS, is it's pretty straightforward to take a new disk, add it to

the system, and then just add that storage to an existing mount point. And you can do that with RAID or without. And that to me is a very compelling way to just grow and grow a media storage pool by doing this ButterFS volume add or disk add, whatever it is. When you look at what is feasible on an ARM device like a Raspberry Pi or a Pine64, the overhead of a file system like ZFS doesn't seem very reasonable. And Extended For and XFS are absolutely, totally great and usable. But maybe you want something a little more, like maybe you want snapshots, or maybe you want this ability to just add additional disk storage and you want to integrate it at the file system level. Then ButterFS becomes a pretty compelling option on something like the raspberry pi and so i think even with this experience i think i would i'm still going to move forward if there

was one thing that i wish butterfs could have done for me because this was really all 100 my fault but if there's one thing is maybe that file system conversion tool could have had some checking because the file system was never usable. Like when it said all done, everything good to go, it didn't leave me with a usable file system. And maybe there could have been some kind of error checking or butterFS repair scan that happens in there or something to give the user feedback saying, this isn't a functional file system I just gave you. It wouldn't have changed anything, but it seemed like an opportunity for the tooling to be slightly better in that regard. Right. You were left thinking, oh, yeah, I have a usable file system that I can just go

work with now, instead of having some sort of universal way to query, like, is there still a conversion in progress? What's happening with my file system? And should I touch it yet? Yeah. It's just, oh, Wes. I'm so sorry. But that's enough file system talk. Let's do a little housekeeping. Remember, we're live every Tuesday. Although I'll be traveling, like I mentioned earlier, to Austin,

so there will be some time in there where we are not live, but we'll still have a show for you. In fact, there's a website you can use to find out more. That's right. Jupiterbroadcasting.com slash calendar, where they'll be robots to convert it in your local time. How about that? Robots, Wes. And we didn't make them do it.

We asked, and they said they'd be happy to because they are far superior to our limited human minds. Yeah. They've got plenty of time, so they figured, why not? What else are we going to do? And a little pay it forward going on over at a cloud guru. They're giving away 1000 subscriptions. How about that link to that in the show notes? Go check that out. You can find out more information. And I believe it's running until June. So for the end of June, anyways, details are over there. They're awarding an annual subscription to those applying, they'll receive a free ACG subscription.

That's pretty awesome. What a great opportunity to skill up and link in the show notes to pay it forward at linuxunplugged.com slash 355. Also, a mention for the LUP plug happens every Sunday. Same bad time at noon Pacific. Just a different bad day. There's a chat room, too, on GeekShed at irc.geekshed.net pound lup lug. And you can jump in there.

We do it every Sunday. And that's when I decided well, I was going to convert my cousin's systems to Butterfest and I'd start with my own before I did his. Now I'm not so sure he'll be down for it. But stay tuned in the lup lug for more. And that, my friends, is the housekeeping.

So Wes, we had a discussion recently about different network troubleshooting techniques that we had been both recently employing. You've actually had opportunity to go through and fix up the home network. And we thought, well, what a great opportunity to share our recent experiences here on the show. And this is one of those times where we get to go with Wes into his home on a weekend. I imagine it's a rainy evening in Seattle. Am I right? Of course it is. Yeah, it's because it's just been raining. You've all seen Frasier.

It's pretty much true. And it has been a rainy weekend. And Wes discovers something strange. I think we've all been there. There's something strange on my network. An unknown device. You know, Wes, we all have these projects that seem to crop up when you're trying to do something else. You're going about your day, and then you notice something's not quite right. I didn't ask for this. Really, this all started over the weekend when I noticed I was having some internet problems. I was trying to download a new Steam game to, well, let off some Steam with some friends, and I was pretty embarrassed when I found I was only getting about two megs down. Now, sadly, I'm not on fiber at my current place, but I do have a pretty decent

Comcast business connection that reliably performs well above that. Now, I should be the first to admit that my network, well, it's not in the most pristine state it's ever been. I move more than I'd like, and unfortunately, that means things sometimes get messy. I'm well overdue for some firmware updates and some password rotations. Since I was already over there giving the modem a reboot, I thought, well, I won't be playing this game anytime soon. I think it's time for a home network checkup. It's always a little embarrassing when you get all set with friends and then technical issues.

Yeah, right? Like, come on, I'm supposed to be a computer person and I can't make this work? Yep. And tip one in Wes's quest is check your logs. You may be surprised by what you find. As I sat there doing the usual updates, I decided, well, let's take a look at the log files. That's when I started noticing something wasn't right. There, right in the DHCP leases, was a device I didn't recognize. Now, as I said, my network is messy and there's plenty of random IoT gadgets,

wireless devices, and just general computers plugged in. But this? Well, this was really unknown. And of course, there's only one solution. It's time to break out the networking tools, perhaps the king of the networking tools. To get a better sense of things, and really because at this point I was just a little nervous, I thought I'd fire up Nmap and take a look at what responded on my network. And yep, there it was, the same anomalous MAC address. Nmap didn't know what it was.

And plugging that MAC address in online, no results there either. 10.1.10.212. What are you? And where? Your mind must have been racing at this point. Yeah, I mean, come on, this is my home network. I've got VPN set up to several secure, or supposedly secure, locations. I've got VPN set up to several secure or supposedly secure locations.

I can't have that. Did you go through, like, various ideas on what are my options on how to track this thing down now? Yeah, you know, I was kind of just thinking, like, well, what can I find out about it? What do I have that I can sort of eliminate from the list? But I was rolling around my house looking at devices thinking, like, okay, is there anything else I can unplug to try to discover if that's it? But no. If you're in a situation now where maybe you're working from home, I've noticed this with a lot of the meetings that I've been in people that are working from home. All of a sudden, they're a lot more reliant on that home network than they were before. And so you also wonder about the security of it. So these kinds of things really kind of can kind of nag at you. So I like the Westpain drastic action approach.

NMAP revealed the port 9080 was open, but unfortunately it didn't respond to HTTP requests or anything over TCP. At this point, I was getting desperate. I'd have to do something dramatic. It was time to change my Wi-Fi SSID. No going back. Having to manually set up all my connected devices again. It's the real scorched earth approach there, Wes. Yeah, right. Burn it all down. We'll start fresh and only the things I want will be added. That's definitely a way to just guarantee that it's a clean setup.

Did you ever get a sense of what device it was after you did this? Did you go around and find that something wasn't working? Oh my gosh, yes. Okay, so this is also pretty funny. Here I was frantically searching my house, you know, for IoT devices, rogue Google devices plugged in, which, let me just say, so many ports open. I'm a little disturbed. But I never thought to check the thermostat sitting behind the fridge. I mean, it's not plugged in. I didn't

plug it in. It's just a little wall ward. But of course, it has Wi-Fi. So useful, obviously, too. You know, I actually am glad that it supports Wi-Fi, and there's a home assistant module for it. So honestly, that is something I will be playing with because, well, I'm just curious, even if I didn't need it over the winter. Yeah. And now you can put it on the Wi-Fi when you're ready. That is the one nice thing about changing the SSID. I've played around with like trying to do MAC address and matching it to a vendor. But I agree with Past Wes from The Weeknd. This is his bit of advice. Don't let this happen to you. So there's a couple of tools and whatnot that I think could help in a situation like that.

A tried and true one that I know our friends at the Ubuntu podcast also love is NetDiscover, which is a really handy tool to take a look around your LAN and see what's going on. Yeah, that makes it so easy. And it's a handy sort of, you know, just little NCurses interface. So it interactively keeps pinging out to your network and then updating with whatever it finds. I like that. So that's a really good one to get a lay of the land. Also, Nmap, you heard Wes use that tool in there. You could read a book on Nmap because there are several and it is totally worth it. But a really simple command is sudo nmap tack, lowercase s, uppercase S, and then your IP space, slash 24, or whatever your subnet is,

and just let it go to town and discover all the devices and talk to them. And a lot of times it can tell you the operating system and whatnot. So that can help you get your arms around what's going on. I want to mention if you have Android, you owe it to yourself to get Wi-Fi Analyzer. It's such a great tool.

And you can look to see if there's maybe a channel that isn't quite as busy in your neck of the woods and move your Wi-Fi network over to that if it's not doing that automatically. And that may improve your signal. So that's something to consider. There's just maybe basics like that to help improve just Wi-Fi with something like Wi-Fi Analyzer, which is on F-Droid. to help improve just Wi-Fi with something like Wi-Fi Analyzer, which is on F-Droid. Then on your Linux laptop, take a look at WaveMon, an N-curses-based monitoring applications for wireless devices in your network, which is really useful.

And again, another beautiful N-curses interface. That's right. We love that N-curses up in here. I use WLAN Scanner, and that one also gives a nice overview of your 2.4 and 5 gigahertz on your Android. I'll also give a recommendation to a tool that Wes and I love. Wes may be a little more than I. NetData, which is useful for a lot of reasons, but one of the neat things that NetData will do is it will generate errors and alerts if there's something funky going on packet-wise on your network, I actually had this happen maybe a year or so ago when we first started playing with NetData. It was really early into it

because I didn't even really understand what it was telling me at first. It's just spewing errors and information about how broken your setup is. Yeah, and then I realized, oh, it's telling me that there's a whole crap ton of discarded UDP packets on my network that is way beyond the normal in a one-minute period. And that began a journey of troubleshooting that eventually improved the Wi-Fi network. Thanks so much, Vizio. So tools like that, even that aren't necessarily for network troubleshooting,

can help surface interesting information. Boy, that's a story for another time. I think what we're trying to advocate, though, if we just could extract away from the individual tools, trying to advocate being aware of what is on your network and how much resources it's consuming and what that might be from a security posture standpoint. And I think what we're kind of suggesting you might consider doing is do what Wes did and just say, hey, let's do a network checkup

and let's go through and check the firmware versions on the Wi-Fi or on the router or on this IoT device. And maybe it means you've got to go recover a password so that way you can log into the WeThings portal or whatever the hell they need you to do so that way you can manage your light switch. But do it. Get those updates done. Or maybe take a weekend and see if you can throw a few of those things in Home Assistant instead. Or maybe take a weekend and see if you can throw a few of those things in Home Assistant instead. And maybe reduce some of the internet traffic and improve your Wi-Fi that way, too. Now, Wes, you dug out a few tips from Jim Salter over on Ars Technica on some simple ideas for reliable Wi-Fi.

Yeah, I mean, at the end of the day, you've got to remember that Wi-Fi is a shared spectrum. So your performance is often the same or limited by the worst performing device on your network. So if you've got one AP sitting somewhere and you've got someone trying to stream YouTube across the house, that's going to be a lot of retries. And those are competing for your latency sensitive packets in your Zoom call. So you've got to start thinking about things like

where are your APs placed and when are you using it the most and who else is on your network? in your Zoom call. So you've got to start thinking about things like where are your APs placed and when are you using it the most and who else is on your network? And what are they doing, right? Also, things like the layout of your house, no more than two rooms and two walls is a pretty good rule of thumb. Also, don't necessarily get too hung up on transmit power. Too much transmit power can be a bug, as Jim says it. And make sure you're really taking placement into advisement as much as possible. He suggests above your head height is actually the most preferable location, which I got to admit, most of my APs are down near like foot and leg level,

maybe knee level at best for a couple of them. Knee level? What devices do you even have down there? Extenders that are plugged into outlets. Oh, you. Although I will say, you know, Mesh, I think, gets maybe a bad name in our circles here on Linux Unplugged, but it can legitimately be useful, especially in an environment where you don't want to run a bunch of Ethernet. Although I will say, if you can have a cabled connection, well, that's just going to help everything. Okay, now let's talk about the router.

This is one where I think all of us have a lot of thoughts. And Wes and I are constantly debating appliance versus hardware device versus build-it-yourself Linux box? Yeah, well, it's a tough question because different people have different needs. How much time do you want to spend babysitting it or having to migrate from backups like you've just experienced, Chris? But at the same time, do you want to be familiar with how your network works? And do you want to know the ins and outs?

And thankfully, there's a whole bunch of solutions that range from something like OpenSense or PFSense all the way down to just running IP tables commands yourself. Although, I'd probably recommend Firehull over doing that. Hmm. Okay, well, let's take inventory here in a moment, see what everybody's using. I had this thought when all of this went down. If I had some sort of QoS or something that was throttling all of the other devices, and maybe I had priority, like my IP or whatever, maybe I wouldn't have had this issue to begin with. Maybe I wouldn't have needed to switch Wi-Fi networks, and I wouldn't have disconnected to that terminal session and lost the data. Yeah, you know, that's a good question, and a little bit of quality of service can go a long way.

Obviously, it can't repair problems that you're going to have upstream, right? I mean, if you're just hitting the limit with what you can send or receive from your provider, well, you're kind of stuck. But for optimizing what goes in and out of your home network, yeah, those tools can be really handy, especially if you're in an environment where, all right, well, the kids are there streaming a movie.

It's okay if they buffer a little bit, but I'd really not like to lose my connection to this conference call. That's pretty important. Yeah, exactly. All right. So just around the horn, I've recently talked about mine, so I don't need to go into it too much, but I'm using that GE Slate mobile router still. Love it. Think it's a great little device. Probably will replace it someday, but I haven't decided what yet. What about you, Wes? Are you replacing your router? Do you have a new one? What's that area of your network like right now? You know, I was just thinking those might make a great little sort of just standalone guest or alternative network Wi-Fi where you're not too concerned about performance, but you just want

to connect it up with your, you know, whatever you've got upstream and then share it out, especially if maybe whatever you're using for Wi-Fi doesn't support multiple networks. If it does, you know, you've probably got ways to define more than one Wi-Fi network or set up a guest network, and those can be useful too. For me, well, I'm just still using Firehole because it's a great balance of configurability and programmability, but not having to dive into all the complicated mess of IP tables and the intricate commands to really do a good job of being secure,

a deny-first policy, and making sure that you optimize things for performance. What would make that even better, though, is if it was on a low-power, low-heap, kind of quiet or even totally silent device. Yeah, that's true. I am using a not super powerful but but not arm level Celeron chip at the moment, which meets my performance needs, but does have a big old x86 style case. Well, let's ask Drew and Cheezy, see what they're using. Cheezy, do you have any particular router or firewall that you've put in or you got the ISP box? I have a Netgear Nighthawk router that's been flashed with DDWRT. You know,

when talking about auditing your own network, one thing that a lot of consumer routers do by default is they turn on WPS by default, which is something you should definitely disable because you could possibly have your network breached using WPS. And if you're still using WEP, there's something wrong with you. All right. Drew, what about you? What are you rocking for your home router? Well, for my router, I am running an Edge Router Lite, the ER Lite 3. It's a few years old now at this point.

Actually, I think like five or six. But it's still rocking great, still getting support and firmware updates. And it's surprisingly versatile for essentially what is a Debian-based router. And then I've got the UniFi access points in a couple of different spots in my house. I am very impressed that you're sitting around five plus years now with this device. I mean, I think that's the stamp of success right there. I wonder how Bruce feels about this, because Bruce, you've been rolling a traditional Linux firewall since 1995. Yeah, that was the year I graduated high school, and one of my friends had gotten me into Linux, and I set up a FreeBSD run-of-the-mill bare OS firewall just to do masquerading way back when, and I've evolved since then, and right now I'm running PFSense. Okay, so what did you like about running a full Linux box as a firewall, and why did you switch to PFSense?

So two questions there, really. Well, originally, I just didn't like any of the existing firewall things that you would get from an ISP provider or something like that. I wanted something I have control of, all that kind of stuff. So the normal kind of ideas from our kind of target audience, I guess. And switching to PFSense, it was more just generally not having to do IP tables and all that kind of stuff. I have a point-to-point connection for VPN for work, having all the graphing tools and stuff that it has. Although I'm seeing in the chat that OpenSense sounds pretty good, so I might be checking that out. Yes, the call of OpenSense. I've heard that song as well. Well, that's interesting. And I think I kind of align with your thinking. We'll have some links

in the show notes to Firehole. Of course, other things that you're familiar with, you can probably send us in a few ideas as well, because there's probably more that we're missing at linuxunplugged.com slash contact. I'll do something soon. I'll do something. I think for me, key things I want are reliability, wire guard at the edge. The only place I'd like a GUI to really is just for setting up simple DNS entries and maybe static DHCP mappings. That's what I really like about the Pihole UI right now, is it gives me that kind of stuff now. You know, Pihole does a pretty good job and it's, you know, it's built on top of their own fork of DNS masks. So if you're already familiar with using that, which is super handy, especially for doing like split horizon DNS setups or just any sort of DNS customization, well, you can do all that same

stuff on Pihole, either through the GUI or, if you want, on the command line. There are a couple other things before we get off Wi-Fi. Consider at least two or three Wi-Fi networks. Wes, you kind of touched on this, but maybe consider a kid's network, an IoT network, and a guest network. And then, of course, the main production network where your your devices and maybe your spouse or whatever, your significant other's devices, whoever might be family. I have not done the separate kids network, but after this experience recently, I'm considering it for the QoS aspect of it, but I have done the guest and IoT network. And I like that a lot. So for me, I put all those people on two, four. And it's a separate SSID with a separate password. It's a separate IP space.

The IoT network can't talk to the guest network. The guest network can't see any of my automation devices or any of my IoT devices. They just get a route out to the internet, and that's it. And then I have a production SSID that I connect to that is the LAN that has all the devices that can talk to each other. And I like that a lot. I think it's a little additional complexity up front, but if you just do it in a really clear, simple way, it's pretty easy to maintain and easy to explain. All right, so Wes Payne's hot VPN tips. Hot VPN tips. You know, one thing I like doing for myself is if you have access to it at the router, you can set some of those things up so that if you've got multiple devices you need to use on

the VPN, well, you can set it up that way, right? Just make sure that you only allow access to the VPN connection, which you've established on your router from those particular devices. And you don't have to fiddle with clients on each of those things, configuring all of that. Now, this could be a security risk. So make sure you've got your firewall configured correctly. But it can make things pretty nice and simplify a config where you might want to do something like a split horizon DNS setup where instead of forwarding all your VPN traffic over to work and having everything resolve out that way, well, just send the things that you need over that network. And if you use something like DNS Mask or many other solutions, you can have it so that DNS queries that are meant to be resolved on the VPN network go that way,

and everything else just uses your standard resolver. That makes for a very slick setup where you don't have to worry that everything you do is going through the works network, or just, you know, getting that delay. Everything you do is going through the works network or just, you know, getting that delay. Some VPNs also, you know, you can specifically say just this IP block and only traffic to this very particular IP block goes over the VPN. Not true for all of them, though. I think the first couple of times I ever really had a work VPN, it was one of those where there's a client piece they'd have to install or one of these web things that have to go through like an ActiveX connection. It was the worst.

And it would just, by default, do all the traffic. It was just the worst. It was always for doctor's offices, too. If you are running a VPN locally, just watch out for IPv6. Sometimes you'll get an IPv6 address, which is just totally fine on its own, but you'll have an IPv4-only tunnel configuration, so you'll think all your traffic is going over the tunnel. But in fact, IPv6 is still going through your local gateway. Now, a big plug again for the self-hosted podcast. This is table stakes for that show,

amongst many other things, including running a bunch of services on your LAN. And I talk a little bit about more of my server setup in there. You mean just how many pies you have? Yes. And I think you'd probably really enjoy it if any of this appeals to you. So check that out at selfhosted.show. But let's do a little feedback. You ready, Wes?

Lay it on me. Michael writes in. He says, I wanted to thank you guys. I've been primarily listening to other Linux broadcasts and in their Telegram groups, but I switched to your Telegram group in the last two days. Controversy? What could it be? And this is why we're reading this, because since our last episode,

there has been quite a bit of feedback around our discussion regarding Microsoft. And Michael writes here, it seems that the closer we get to WSL2 being released and the more software Microsoft open sources, the more hatred against Microsoft I saw in other Linux groups. Ironically, it made me feel alienated and pushed me towards trying out Windows 10. Oh, no. You know, I can kind of understand that when like when group think starts to set in. I sometimes have like this rebellious nerve that just gets pushed on and then I go, I rebel against it. But this is something that I said would happen in the post show last week and definitely bared out. Like I got a lot of people who are not happy about the way we talk about Microsoft.

There has even been clickbait blog articles that have been written attacking JB and me for talking about it. I mean, it's pretty intense. And you'll hear so much rhetoric about how this is the worst thing Microsoft has ever done. This is embrace, extend, and extinguish, take into the next level.

And it seems to be yet another divide that is developing in our community. And that makes me very sad, because I think every situation with every company should be taken on its own individual merits. And any commercial company, from Red Hat to Microsoft, that is participating in this space has to be monitored with a certain level of suspicion, although maybe that's a little extreme, you know, but you have to be aware and paying attention.

A recognition that they've got their own motives and their own drives, and that's okay. That's part of the design of the whole open source ecosystem is that we all have those, but we can agree that some things are better done shared. And this truth is applicable to any company whose goal is to earn revenue. We kind of individually apply it in the Linux community. There's some groups and companies that get a blank check and really nothing is questioned. And then there's some who, probably because of past actions, there's just a relentless suspicion. Now, that of course, past actions, there's just a relentless suspicion. Now, that, of course, eventually had to result in,

unfortunately, division. I'm trying to consider how we can accurately and level-headedly cover this stuff without exaggerating that situation and causing more division. And that's something I've been chewing on that I don't have a great answer for. Michael continues, he says, I've been using Linux since 1997, but when a community becomes all about hatred of something rather than the love of Linux, then you push people away, even your own community. I'm so glad that you guys have managed to create a healthy community that does not fall into that trap. I mean, I think we're struggling with it too. I wouldn't give us too much credit. We're struggling with it. Yeah, absolutely. And I don't know quite what the right answer is here.

Some of it feels like reasonable suspicion, and some of it feels silly. The one that feels sort of silly to me is the embrace and extend and extinguish discussion that has cropped up around Maui and around DirectX being available on the Windows subsystem for Linux. I think the premise is flawed. There's no embrace and extending of something that's never been a success. The Linux desktop is not a success that threatens Microsoft. In fact, it's entirely possible that the quote-unquote vast success of the Linux desktop is barely even represented on any Microsoft market share chart. Now, the server, sure.

Mobile, absolutely. IoT, yeah. But desktop? I think people need a bit of a reality check. Microsoft is not embracing and extending the Linux desktop with adding DirectX APIs to WSL2 only on Windows. The entire premise is silly, but yet that doesn't prevent YouTube videos and blogs and people going on Twitter, going on and on about how this is obviously Microsoft trying to extinguish Linux.

It's a silly premise to begin with. There is no threat from the Linux desktop. I'm sorry, I love it too. But it is probably not even statistically measurable by Microsoft on the Windows market share bottom line. Let's be honest. The Mac gets a few percent. Chromebooks, yeah, that sucks. Not Linux desktop. We're too busy fighting amongst each other.

We're the last thing they're worried about. We're too busy fighting amongst each other. We're the last thing they're worried about. And WSL is absolutely more about taking on the Mac as a development platform than it is about Linux desktop. But people are so trapped in this bubble, they can't see it. They don't see the bigger picture. It's not some grand conspiracy to keep the Linux desktop down. It was never up.

I'm sorry. I love it too. But it never got up. And it's going to be fine. This will just result in more Linux users. And it's not going to stop the Plasma or Gnome Shell developers. Certainly not. And I mean, if anything, it's aimed at the success of Linux on the server, right? It's a recognition that people are using Linux, not on the desktop, but to deploy their production workloads. And that's the world Microsoft wants to cater to. Yeah, and this isn't a defense of Microsoft.

I think this is a defense of rational thinking and taking each individual situation on its own merits and evaluating it. That's all I'm advocating for. And I just, I plea for this not to become a divisive issue in the community. It would just, there's so many of them. It would just be so nice to have something that doesn't develop into this. We've got enough. And a good time to practice acknowledging that, I mean, Microsoft is built up of a bunch of people and they've all got their own opinions about open

source. And there is a company line, but I think we've seen how much that has changed even with their president admitting as much. So it's a flexible universe and they are a company, but they're made up of people who have different opinions about open source than maybe once they did. What are you? Yeah, we don't know though, right? It's hard to evaluate how a company change and over time. Yeah, that is part of it. Here's my pitch. Screw Microsoft. Who cares what Microsoft thinks or feels? Like, I don't think that even has to be part of the consideration for the Linux community. It's probably good to do that, but roll with us for a second. Forget Microsoft. Forget what they're trying. Forget if they're open or forget any of it. What the strategy is, forget what they're trying, forget if they're open, or forget any of it. What the strategy is, forget all of it. Here's the truth. There are going to be more people using Linux coming into our community, looking to connect with other individuals, either for help, for

inspiration, for guidance, for mentorship. And we should really try not to blow this. We shouldn't let our prejudice and our feelings towards Microsoft, whatever they may be, we shouldn't take that out on these new users. We need to be welcoming to new people when they come into this community and show them that it's a collaborative space. The true, true open source nature is here. Like you found something that is compelling, that maybe draws you in, that makes you want to go even further. And we're just so close to blowing it. I think you've seen that with Canonical's support for Ubuntu on WSL, right?

I mean, they recognize it as a supported platform for their operating system to run on. So we should do the same. If people are coming in with, you know, questions about things, we can't just disqualify them. Yes, there might be some caveats, but we shouldn't just outright disqualify them

just because they happen to be running Linux on Windows. All right, well, I never meant to talk about this this long. Michael just got us talking. That's one of the great things about feedback. So my final thoughts on this are, and I apologize this turned into such a soapbox, I would ask you to see if this all comes from a place of insecurity about Linux's future on the desktop, about the desktop in general, about what it has still yet to get to, and if maybe that isn't influencing your reaction. Wrapped up in moral superiority or technical advocacy,

but the reality, the nut of it, might be insecurity. I know that's a harsh thing to say and a harsh thing to consider about yourself, but I mean, people come after me. They write articles about me about this stuff. And I think it has to come from someplace deep that is afraid of where this is going. And I think that insecurity, while understandable, is misplaced. Windows is still Windows, and you can try to run it. If you've been a longtime Linux or Mac user, you're still not going to find Windows satisfying. It's not going to destroy the Linux desktop. And I would encourage us to maybe consider what's driving this defensiveness and try not to think about it from what's best for Microsoft or best for Linux or what Canonical's

doing and try to think about it from here come the users, how can we make them successful? How can we welcome them to the community? Yeah, really our goal is to convince them that Linux is a useful technology they should be excited about learning. Yeah, really our goal is to convince them that Linux is a useful technology they should be excited about learning. Yeah, and there's a community behind it that makes it kind of fun and maybe something that the Windows and commercial platforms don't really have. In fact, I'd say they're kind of envious of. Come on over. The water's fine.

Most of the time. In the right areas. Maybe don't use ButterFS. It's too soon, Wes. It's too soon. All right, well, that is it for this week's episode of the Unplugged program. We'd love to have you live next week because probably the week after that,

I'm going to be on the road. So if you want to make it to a live show, it's going to be one of your last chances, Tuesday at noon Pacific, jupiterbroadcasting.com slash calendar to get that converted in your time. Don't forget about the LUP plug on Sunday coming up.

We'd love to have you there. And links to everything we talked about at linuxunplugged.com slash 355. The show is at linuxunplugged on Twitter, at Jupiter Signal for the network. We'll see you back here next Tuesday! Thank you. No, Computer Kid, just no live shows while I'm driving to and from Texas. No. We're going to miss it twice.

You said we would never miss anymore. Oh, don't worry. It'll be like we were never gone for the download audience. I'm not the download audience. Well, you will be.