LINUX Unplugged - 425: Sad Server Stories

Episode Date: September 29, 2021

Sometimes things go wrong; this week, we admit we've got a problem. Plus new details about the Steam Deck everyone has missed, and an old friend stops by the show with an update. Special Guest: Danielle Foré.

A robot. Yes. What are we going to do with a robot? Well, Astro, follow me. Yeah, who doesn't want an Amazon dog following you around your house for just over a thousand dollars? Their new Alexa on Wheels has been announced today. It probably runs Linux, but I don't know if it even matters. I don't care at this point

if it runs Linux or not. I don't want a robot that follows me in my home. I have too many dogs as it is. You do have a lot of dogs. That's a good point. I would like them to just actually have a program that connects us to dogs. How about that? Let's just skip the whole robot thing. Save everybody money. Yeah, quick short-term, you know, prime delivery dogs. Yeah. Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris. My name is Wes. Hello, Wes.

You're coming in looking great today. I love that all-purple outfit. This episode is brought to you by A Cloud Guru, the leader in learning for the Linux, cloud and modern tech skills. Hundreds of courses, thousands of hands on labs. Go get certified. Go get hired. Get learning at a cloud guru dot com. Well, coming up on the show today, it is time for our server crash postmortem. It struck at the worst time possible and then it crashed again. It's been quite the journey. We have talked about our server on the show a lot,

and this week the chapter is coming to an end. I'm a bit sad but also excited about where we're going. We have a major update for you this week. But we will start out by telling you why our Arch server crashed recently and then why it crashed again even more recently that we haven't told you about. And then what we're going to do to get things right, to do the right thing here and get going and what we had to do just for now. We're going to tell you all of it. It's a full post-mortem today on the show. And then we have at the end a really kind of a fantastic surprise twist.

We'll tell you about some really big future plans. And on top of all that, in case the server stuff isn't quite your bag, we've got some nice thought-provoking community news, some old friends stopping by the show, and a lot more. So there is, as they say, a big show today. So before we go any further, let's say time appropriate greetings to our virtual lug. Hello, Mumble Room. Hello. Hello. Hello, Wes. Hello. Hello. That is a good looking Mumble Room. We got 18 of you in there. Nice to see you. Mostly everybody in the on-air room

rocking out with us today. So I want to start by asking the mumble room how they feel about a news story that I came across just before the show started. Pharonix has noted that the first sign of Linux support for what Intel is calling, quote, software-defined silicon are now landing for Linux. Michael writes, there's been talk of Intel moving to offer more license-able slash opt-in features for hardware capabilities found within a given processor. And so we are now seeing the Linux signs of support coming with a driver for the Intel software to find silicon to allow for secure activation of such features baked into a processor's silicon, but only available as an upcharge option. Yeah, Michael Larable did some digging on this

and kind of makes it clear the basic idea is that without having to purchase a new processor, Intel software-defined silicon will allow for activating additional hardware capabilities just by purchasing a license or upgrade. Unfortunately, right now, the exact details don't appear to be public for what these upgrades might actually have in them, but Intel is currently preparing the Linux kernel driver

support. Known as SDSI, this driver exposes a per-socket interface so that Intel's user space application can provision an authentication key certificate, which is then written to internal NV RAM. But it also allows provisioning what they call a capability activation payload. And it also handles the boring stuff of just reading out what configuration the CPU state is right now. Oh, OK. So what we have here is this software defined CPU that allows these hooks for authentication of various add-on packages,

initially targeting the Xeon line of Intel CPUs, which I think you could probably understand why. I'm curious how people in the mumble room though feel about software defined CPU, essentially a piece of hardware that ships, you buy it, and there is features and capabilities in that chip that you just don't get access to until you pay. This is not a new concept. We have seen this before, but this seems like a whole new level. And you could almost kind of see this taking off in a way.

You know, maybe an LTE modem that has extra 5G capabilities if you license those capabilities as an end user. You know, an in-app purchase, just $29.99 a month to activate these in-CPU or in-chip features. I mean, I think we've already seen this to some extent with some of those ARM boards, right?

Where you got to get a license to do some of the video encoding or decoding. Minimac, we're probably seeing this because Intel is going to target businesses initially. Definitely. I see this as a business feature,

so probably it's just a must that that thing has Linux support. I don't think this is really for private users. But didn't they try something like this with Itanium? Yeah, so HP had a system where they want to give the CPU

a sort of offload of the processes. So moving those kind of functions to the software, the software defining what instructions had to be executed, and then the software could just run through all the instruction needed. And so thereby creating a more efficient processor. But yeah, you know what happened with the Itanium, right? I mean, this whole thing, I'm sort of sitting here going, is this what Intel is spending

their time on right now? This is what they're worried about is licensing additional features instead of just making the absolutely most kick-ass processors possible and shipping them as fast as possible. But the reality is, I think, there's like a cold business calculation here because making these CPUs, they're producing these things in these massive quantities, and it is cheaper and easier to produce if they're all exactly the same. And then they just turn things on and off via software, which is not much further than we are today, where they're all manufacturing, like say the CPU, they're manufacturing tens of thousands of them. And a few of them, a core doesn't light up properly. Or maybe a couple things just don't pass a specific set of tests.

And so they will re-bin that as a lower brand CPU. Oh, this is the four core version instead of the eight core, because maybe one core doesn't test out successfully. So they'll disable them, several of them, and ship it as a four core instead of an eight core processor when it's essentially the same silicon. I mean, that is something that already happens today. This feels like an extension of that, but there is something I think initially when I read these kinds of stories as a long time free software user, I have this really like screw licensing, you know, like licensing and get the hell out of here. I am so sick and tired of

software licensing. I would like just no licenses and everybody just essentially operates as if the GPL were just cultural practice, but that's never going to happen. But anyways, that's at the core of me. I just don't like licensing and I don't like it in software and I really don't like it in hardware. Well, it's a reminder that at the root of these computers that we try to fill with free and open source software is this black box that we, you know, we don't really have full control over. And it just feels like a little bit kind of, you know, we've done this huge shift to services and this isn't quite the same, but it feels like it's kind of in that direction of

changing the relationship of, I bought this thing, it's static and I know what I've got. And that's how I calculated the cost. Whereas now that's a little more complicated. Yeah, that's totally it. That's totally it. We put all this energy and time and thought into the software stack that we run and try to get it as free as possible. And then we have dozens of black boxes baked into our motherboard. That's absolutely part of it. Oh, there was an interesting Steam Deck update this last week. We talked a little bit about this on Linux Action News. You probably heard that Epic and Battle.i have updated their anti-cheat software to support games under Proton on Linux actually functioning.

Now, there's been so many top-tier games that Linux users could theoretically technically play because Proton was capable of making that game run, but it would fail to run because of anti-cheat software. And as we noted on Linux Action News this week, that has been addressed by both BattleEye and Epic. However, game developers still need to patch their games. But no matter how you look at this, this is huge news for Linux gaming in this last week. This is ginormous news, and it immediately means that some of the most popular games on Steam are now theoretically within reach of Linux users. Yeah, personally, I'm hoping Dead by Daylight season updates soon. I suppose maybe now is time

to reach out and voice our Linux user opinions about games we like and would like to see these updates, although hopefully we do it politely. Yeah, I'd love to see Destiny 2 get support. That's what I would love to see. And I think that's bad a lie. But all that aside for the moment, we'll just have to wait and see. There was one small update from Valve that seemed to have gotten missed in all the anti-cheat coverage, but it's one that we're kind of excited about.

In the recently updated Steam Deck FAQ, there are two new questions and answers that really did stand out. The first asks if the Steam Deck will support booting from the SD card, which Valve answers really quite simply, yes, Steam Deck supports boot from micro SD. That's great. You're going to be able to boot different distros off of the SD card. And there was a second part of the FAQ that backs us up. The specific question was, this is a quote from the FAQ, which we have linked. What can you tell us about the BIOS and does it allow dual boot?

And Valve answered, multi-boot is supported. You can have multiple os's installed and choose which one you want to boot into users will have access to the bios menu what do i even say to this this is suspiciously open i mean wow huge this means we're going to be able to see community takes on that that steam deck experience that will have essentially zero barrier to entry. I'm very excited about this. I can't overstate it because it's going to allow a lot of people to participate in creating

a software experience for this hardware. What if it's too easy, though? It's not like hacking Linux onto a Wii or something. They're almost begging us to do it. Yeah. So, I mean, really, what we have to see here happen is a couple of things. We have to see game developers update their game to support these new anti-cheat updates. It looks like it's crazy easy. Epic seems to indicate it's very simple.

They say, quote, starting with the latest SDK release, developers can activate anti-cheat support for Linux via Wine or or proton with just a few clicks in their epic online services developer portal that's from epic battle eye says well we already had linux support but now we've specifically done an update to support proton that's number one is game developers are going to have to update the game to support this anti-cheat stuff number two and i'm not 100 but this looks pretty solid from what we can tell, you're going to need Linux 5.11. So you're going to need a Linux kernel or you're going to be able to upgrade your kernel

to 5.11 or newer because you need a kernel that supports syscall user dispatch, which is some work that was done to make these anti-cheat and other things that are running in wine communicate better on Linux. And if you don't have that support,

it looks like some of this anti-cheat stuff might not work. I haven't been able to test that, obviously, so I can't tell you for sure. But from our digging around, that looks to be the case. And from what you can tell, that seems like that's probably likely, right, Neil? With Linux 5.11, Collabora, on contract with Valve, made some changes that were upstream that called syscall user dispatch, which provides a way for user space to register hooks to the Linux kernel for syscall interfaces and APIs and things like that. This is particularly necessary for games because a lot of anti-cheat

software work by directly calling Windows NT syscalls instead of using like the standard C library and going through user space the normal way. And they would fail in Wine because Linux doesn't know what that syscall is and they would just break. So what this does is it allows Wine to register a handler in Linux for syscalls, just like it does for binary formats. So like when you load in an executive, a Windows EXE file, Linux knows that this actually needs to be run by the Wine bin loader rather than the glibc bin loader. And so that same process then allows you to run applications

that would normally talk to Windows NT directly to then successfully be redirected back to Wine for Wine server to handle and the Wine fake NT kernel to give it what it expects so that the applications will run properly. That's what's so impressive about all of this, is the long-term work where we got changes actually upstreamed in the kernel to better enable playing proprietary games on Linux and that Valve at the same time negotiated behind the scenes with these anti-cheat vendors to actually get that working. Yeah. And I think it's just nice to remember that Calabra is behind the scenes in some of this, doing some of the work, making some of this possible at some of the lower levels.

And you've had Valve clearly pulling the strings now for about a couple of years, laying the groundwork, this particular groundwork for the deck. And you couldn't look at it necessarily from the outside and see that's where it's going. But now with hindsight, it's obvious they were building towards a device now. It's just obvious. It just stands out. So that's probably what you need to know is you're probably going to end up needing a kernel 5.11 or newer. And that's, you know, by the time the deck shipping and all that, that's probably not going to be too hard to achieve.

So we'll have links to information about that in the show notes at linuxunplugged.com slash 425. But we need to talk about a little community news. In fact, something coming up very soon. Daniel Foray joins us from the Elementary OS Project. Welcome back. Hey, how's it going? It's going really good. I'm curious. You know, I want to talk about the hackathon that's coming up because it's just in a couple of days. But before we get to that, tell us how version 6

is going because I think it's been about a month since elementary OS 6 came out. There's been an update. How are things shaking out? Yeah, it's been about a month and a half now since release. So we do updates every single month. So at the end of last month, we did a smaller updates post, and we're packaging up all our updates to go out now with tons of new features for online accounts, tasks, mail, app center, like everything's getting lots of goodies. And I being dense, because I'm sure it was probably it was probably communicated pretty clearly, but I completely missed that there is like a monthly update cadence thing now. Yeah, actually, we started that last cycle. last cycle. And so we're kind of continuing that over. But we wanted to make sure that we're not just pushing out bug fixes and feature updates like every couple of years, because that's not

a super great experience. We want to push them out monthly because then we can get that feedback loop really tight where, you know, people are reporting an issue or they're requesting a feature and we want to get that out to them as soon as possible so they feel more involved. Okay, well, let me ask you this. Is there anything that stands out in your mind today that is kind of different or changed since elementary OS 6 version 0.0 shipped? Has there been anything behind the scenes that has changed in the last 30 days? Anything that stands out? I mean, I think some of the biggest things are that we're making a lot of progress with

App Center and with Flatpak. We've been working with the wider community on some of the free desktop stuff, like with the dark style support across all the apps. And so we're going to be working more on portals and that kind of stuff. We kind of redid our CI and CD system recently. So we're now building our flat packs for ARM. So we're going to have App Center apps building for ARM and they're all flat packs now. So we're like, we're going to have this really great collection of apps that you can go run on any distro. Now, if you go to appcenter.elementary.io, you can go download those flat packs for whatever distro. And we're just

going to keep growing that and make it super awesome. Wow. That sounds rad. Yeah, very impressive. That's some serious work you're putting in. That's really cool. Okay. So let's talk about this Hacktoberfest live stream that's coming up. Isn't it this Thursday? Yeah. So what's going on with that is that it's a really great time in the cycle right now because, you know, we just released. So it's not really like the end of the cycle. It's like the beginning of our development cycle. And so we have a great opportunity to get people involved and try to talk about what projects we want to pursue during this cycle. And we have Hacktoberfest coming up, which is the event put on by DigitalOcean and GitHub. So we're going to do a stream on Thursday, and we're going to talk about how to get involved with hacking on elementary

OS and working on new features and cool stuff like that and we're going to try to tie that into Hacktoberfest so if you've never heard of that and you want to try to get involved you can come find out how you can get some free swag for contributing to open source projects on GitHub Right on, boy you know it's fall when

Hacktoberfest rolls around it's like a milestone of the year so what is an easy place for people to go to get involved if they're interested and they have some time on Thursday? If you want to go to our Get Involved page, actually, elementary.io forward slash get dash involved. We have links there for all kinds of stuff. But if you want to watch out for our YouTube page, we have a link to our YouTube page on our website. That's where we do our live streams on the last Thursday of every month. So you're saying if I'm already subscribed to the elementary YouTube channel, I'm good to go. Because, of course, why wouldn't I be subscribed? Yeah, exactly. And then you can come hang out and do some little Q&A with us.

We've got the whole live chat, live stream thing going on. Great. Well, Danny, thank you for coming on the show. It is always great to catch up with you. And it's great to hear things are moving right along with elementary OS six. And it sounds like the Hacktoberfest thing is going to be a lot of fun. Thanks for joining us. Thank you. And I'll see all of you guys on Thursday. Linode.com slash unplugged. Go there to get $100 in 60 day credit on your new account. And you go there to support the show. Linode keeps getting better. I have been a customer for about two and a half years,

and I can absolutely testify to this. I'll admit, though, I was watching them for a while. You know, when these new, like, VC supercharged fly-by-night VPS providers came along and were offering all these different things, I sat back and wanted to see what Linode would do because, you know, they're kind of like the old timer in the Linux community. They've been around for 18 years. So I was like, are they going to fumble this? Are they going to get passed up by the new kids or are they going to stay competitive? And sure enough, they really, really have. In fact, now they're lapping those kids. Linode just upgraded their block storage with super fast MVME PCIe storage, and it screams. You can get systems with AMD EPYC processors that benchmark faster than any other major cloud provider and pair it with MVME block storage. Okay. It's amazing. And then they have 11 data centers you

can choose from, and they have become their own ISP. So they have secured super fast links between their data centers, 40 gigabit connections coming into the hypervisors, and then they wrap it all up with a beautiful, easy to use dashboard that lets you get under the hood when you need. And I mean, I actually have a Linode that I completely replaced the image that they ship with it. Like I wiped the box and I put a custom Linux distro from a router vendor to create an LTE mesh VPN endpoint. And they actually Linode actually let me like wipe it off and set up my own custom box there.

And now the crazy cats are launching bare metal servers. If you really need like the ultimate performance. And it doesn't matter if you've never like just the ultimate performance and it doesn't matter if you've never set up a box before because they've got easy one-click deployments so say maybe you're ready to switch to next cloud they got a one-click deployment for that or maybe something just like a little more like in between like are you done using zoom and you'd like to use something you can trust and open source they've got a one-click deployment for jitsi and there's really no excuse and they're 30 to 50 percent cheaper than the big duopoly hyperscalers. You can get like a $5 a month box that

runs Jitsi. That's a pretty justifiable business expense in my opinion. But I really encourage you to go play. Turn it up to 11 because you can get $100. And when you look at their pricing, that's incredible. That really really is great so go try it out i can tell you all about it all day long but you need to go see it for yourself and you can support the show it's an opportunity to go play learn and support the show you can call it like the trifecta of awesome so just head over to linode.com slash unplugged get a hundred dollars for your new account and try all this stuff out build something learn something test something there are a lot of ways you can host.

I admit it. There's a lot of them out there and there is some big, big hyperscalers that they'd love you to just put everything you own into their systems, but nobody does it like Linode. That's why they've been around 18 years. And that's why I completely recommend them. Go see why we choose Linode every single time by going to linode.com slash unplugged. There's a few things I'd like to mention in the housekeeping section. First of all, thank you to our members over at unpluggedcore.com.

The live stream feed has just been delivering. Mr. Roach joined us in the mumble room and told us about his time interning at Pixar and about the Linux boxes there. That was awesome. We had so many great conversations. That's available to our members in the totally uncut bootleg feed. We also create a limited ad version. It's the same type production, just a little bit shorter. And you support the show at unpluggedcore.com. I also wanted to let you know that our buddy Joe has a new show, a new member of the Late Night

Linux family. It's called Linux After Dark. Go check it out. It's in your favorite podcast catcher already. Episode one is already out. I'd also like to encourage you to join our Luplug. It happens every single Sunday at noon Pacific in our Mumble room right there in the lobby. You can figure it out at your time zone at jupiterbroadcasting.com slash calendar. And all of our mumble info is at linuxunplugged.com slash mumble. And the nice thing is you get it set up. If you're ever around on a Tuesday, you can come join us and hang out in the show and get your word in on your weekly Linux talk show. That's linuxunplugged.com slash mumble for that. Okay, Wes, I know we've talked a lot about the server recently, but this is kind of,

you know, it's what's happening right now, and there's some big stuff in the works. You think it's okay if we just maybe one more time touch on this subject? Well, we have to, right? It's clearly becoming an issue, and the audience needs a resolution. Yeah, we need a resolution, too. Well, yeah, that, too. So just in case you haven't caught previous coverage, just super briefly,

our goal with this Arch server that we have here in our studio has been a bit of a long-term experiment. And I think, you know, too often you'll see online reviewers who spend 30 or 40 minutes with something, if that, and then they go make their video

or their blog or whatever. Maybe it's just a forum post, somebody, you know, talking smart in a forum, giving you their opinion. People speak with authority after they try something for a little bit. They kind of, because they speak with authority to it,

it kind of gets accepted as maybe expert opinion. And then before you know it, it's all over the entire internet as fact. And so we love pushing back against that kind of stuff. And we also know that how something holds up long-term is a totally different story than how something is during your first impressions, right? I've been like, I am already preparing myself

for our Fedora 35 review because I've been running Fedora 35 for the last two weeks. So that, by the time it ships in a month, I'm going to have a pretty rounded opinion of that distribution. And some of it will be, you know, it's during, it ships in a month, I'm going to have a pretty rounded opinion of that distribution. And some of it will be, you know, it's during it's a beta cycle. But some of it's like I'm forming an opinion now about GNOME 41. And I truly believe whenever we can, we try to really experience something as long as possible.

And when we can't, we come right off and we'll tell you, hey, man, this is a first. This is a first opinion, you know, first impressions kind of thing. With this server, we wanted to try actually running Arch in production for an extended period of time and see if we could make a few tweaks to make it keep running, like, you know, using the LTS kernel, file system snapshots, utilizing ButterFS on the root file system, and then using ZFS on the large external data pool. Not that we necessarily recommend you do any of this, but we wanted to test it to see how it would hold up. We wanted to find out so you didn't have to run a rolling Arch server in production. Well, we both had such good experience with Arch, you know, on the workstation,

on various machines in our life. And we both knew that it had all the software and kind of got out of our way in terms of managing. It's like a pleasant environment that we were both familiar with to admin. So it was tempting to just see like, yeah, can we make it work here too? And I should have looked it up before the show, but I'm pretty sure we've crossed the year mark running this thing in production. In fact, we might even be rounding the second year, possibly.

It's all out there. We've tried to put all of it out there as much as publicly possible. So if this thing blows up on our face, you know. And, you know, because that's our singular motivation is really just to prove if it's possible, and you know, cause that's our, our singular motivation is really just to prove if it's possible. And if it's not, we're going to find out for you. And the reality is this is a production server. We really do use it. So when it breaks, it breaks in front of everyone and it hurts. And you know, it's out in a garage. It's not

particularly well thermally protected. In fact, it's not at all. And there has even been some power outages recently. And that's what's led us to some of the more recent issues. Initially, we had some lockups that we traced back to NetData, but we weren't sure. Right before the road trip, about a week before the road trip,

the server started locking up. The worst timing possible. Yeah. We got it up and running again, suspecting it was NetData. We turned off NetData and the system remained running for a month, maybe. Then about three weeks into my road trip, the server went down.

I checked in. I couldn't wire guard. I couldn't talk to the server at all. But I was able to use Rust Desk to get to a couple of the studio desktops. And using that approach, I determined, oh, power went out for a while. And the funky way we have this thing wired up is one power supplies into the wall and one power supplies into UPS for stupid technical debt reasons. And I think when that happened, the server got into a funky state

and it eventually lost power. Then when it rebooted, it came up in a bad way and never finished booting. And I couldn't tell why, because I was gone. I was on a road trip and the old server admin in me was dying. I had to know why my server was down. Plus I wanted my server back up. Right. I mean, here was this situation that we hadn't actually encountered, right? I mean, we've seen it power cycled. We've come back into the studio after the power went out and it's always been fun. I mean, it takes forever to do so, but but it gets there. Yeah. And of course it happens while Wes is camping. So I'm I'm gone. Wes is camping and then moving. The whole thing

was just like the worst time possible. So we didn't know why it was back up. And so like the last day of the road trip, I legitimately just made a beeline for the studio. I was just, I got Lady Joop's to her spot. I got Lady Joop's secured and parked. And then I jumped in the car and I ran down to the studio to figure out what was wrong. And I mean, I literally, I maybe was back in town from the road trip for a grand total of an hour before I was in the car and headed directly to the studio. Uh-oh. That's not a good sound.

Well, I've been in town for about an hour, and I've already made it to the studio to try to figure out what's going on with the server. Getting a beep. That seems bad. Gonna shut her down. Seems that the console's locked right now. Take it down, take it down. We did have a power outage here. Seems like I didn't expect it to be on.

I thought it was going to be off. I'm going to pull the power completely. It's got a dual power supply. Go hook them back up. Both power supply fans are working. Immediately getting the beep. Hold on. The beep just stopped.

The red light on the front of the chassis just cleared. Still waiting for console. It hasn't posted yet. There we go. We've got post. Yes! Okay, keep booting, you bastard. Keep booting. And it did. And I was sitting there thinking, man, I expected some sort of catastrophic hardware issue. And the entire time I kept thinking, how could I have done this differently?

Let's do a ping check. Yes, I am getting ping. All right, let's check if containers are up. have done this differently. Yeah, they're definitely up. There's a lot going on. Look at all that. Okay, great. Okay, I'm going to go applications plunking, see if everything's working. And it was. I mean, for the next 24 hours or so, the server ran great. We were able to connect in over WireGuard.

It was chewing away on a backlog of work, just like it was supposed to. It started syncing out results to LadyJupes. It was glorious. Everything, I was just like, God, I appreciate this. I don't know why it crashed, but I appreciate it. So that was, you know, Thursday or so, Friday, things seemed fine. Saturday, I kind of, I got wrapped up in stuff we were doing with the kids and I didn't really keep an eye on things. But Wes and I arrived at the studio on Sunday to record Linux Action News, and just as we sat down to get started, we noticed something wasn't quite right,

and it didn't take us long to figure out the server had crashed again. Let's go check on it. Server's down. We're going to go check on it. We're recording LAN, but we've got to stop. I got fish sticks air frying. We've got to go check on it. We're recording LAN, but we've got to stop. I got fish sticks air frying. We've got to see what's wrong with the server. Oh, it's dark out here too.

None of the disks are flashing. It's not beeping. But this is like the state it was in previously before my road trip where it would just kind of lock up. And then I could tell immediately because the fans are low and none of the hard drives are flashing. It's not working at all. And when you've got that many hard drives,

there's always at least one flashing. Let's turn the console on here. What will we see? I bet you it's just totally locked. Just locked. It's hard. It's just hard locked. I wonder if NetData started back up.

Maybe it did. And what is it probing? So we got to power it off. And we'll start it back up. And make sure NetData isn't running. Pillow over the face. Yeah, there's nothing that feels better than physically powering off your server over and over again. That's not going to lead to problems.

I recommend you do it all the time. I think it's off. I think that's it. This is as low as the fans ever go. Yeah. Here we go. Also, I like how all of them light up red. That's great.

Really reassuring. Don't freak me out at all. All right, well, I think we'll just watch this boot up. It's going to boot up fine, I would imagine. And we'll make sure NetData isn't auto-starting. Maybe I only stopped it and didn't disable it. Sounds like you. I can't believe it was freaking NetData again, Wes.

So I had not disabled it. I had only stopped it previously before the road trip. I mean, you say NetData. I surmise that, you know, anything poking those might do the same thing. Really, we have to figure out, do we want to keep using net data as a monitoring agent? Do we want to use something else to scrape the metrics? Do we want to just try to disable some of that, poking the hardware monitor functionality for the time being?

I don't know. We have bigger fish to fry first, but we'll have to solve that down the road. Yeah. I think you and I both kind of suspect there's some sort of hardware issue and NetData is poking whatever that issue is and causing the lockup. Because when you get hard locks like this, it almost always ends up being hardware. And I remember now, I hadn't disabled it because my intention was after the road trip to come back and fire NetData back up and see if it locked up again and so accidentally we ended up testing that theory and it did seem to hold so as long as

we leave net data disabled the system continues to run and assuming the power doesn't go out and i don't really know why it was in an air state other than i think one power supply is plugged into the wall one's plugged into the ups they lost power different times and it just sort of went into this failed hardware state and had that beep you could hear. But you know what? We have got some major stuff in the works. Very excited about this.

It's clear we need to replace the server. We knew that before this latest round. We bought it used. It has served its term. It started life as a free NAS box, and then I think it lived life as a fedora box for a while. It has seen some stuff. Yeah. And we have figured out that we really need like a three phase approach to replacing this thing. So phase one is going to be replacing

the server. It's well, actually, I'm going to flip this, Wes. Phase one is going to be replacing the server. Well, actually, I'm going to flip this, Wes. Phase one is going to be breaking out WireGuard onto its own dedicated box. So that way in the future, if our server crashes, we can still get access to the network. Now, we solve that with Nebula and RustDesk, but we'd like to have the VPN capabilities removed from the server itself and on a different machine. The debate there is, and I'm curious where you fall down, Wes, we could do it on a Raspberry Pi, like we could have a WireGuard Pi, or we could replace that Comcast router with like an OpenSense box and do OpenSense and WireGuard on one box. What do you think?

Yeah, there are a lot of options. I mean, I think we have to decide how many different things we want to administer and how much. I am kind of curious to see how well a Pi would perform, and that does have some nice options in terms of, you know, you have a whole bunch of Pis laying around. I do too, so it'd be easy to drop in and test,

and we could do that real quick. Do you think a Pi would be fast enough to be the firewall? Like, I'm picturing a Linux box with some basic firewall and routing. We might just have to try, right? I don't, I don't know. I mean... Maybe put PyHole in a container

for the DNS and DHCP. And then, you know, if it wasn't, we could always migrate it to another rig later. Neil says the Pi won't be fast enough. For performant network stuff, like almost always, you need hardware acceleration

for packet handling. But we're just on a Comcast connection. It's like 300 megabits. It's not just that. It's about how fast you can push it, not how wide you can go. So, yes, sure. Like you can blow out the bandwidth of it any day of the week. And that'll also be a problem.

But like the problem you're going to have more often is that you need it to be low latency and you need those packets to go out as fast as they can, as close to line rate as possible. Especially when you're live. Yeah, so Raspberry Pi would not work. This is just why I kind of want to test it, though. Yeah, I wonder. And the other thing I was thinking, and maybe people could write in at linuxunplugged.com slash contact, but like a really low power x86 box that has IPMI. It's like a micro ATX, itx kind of like tiny box there are a few mini pcs

that do actually have a bmc built in that you can use that might be something to look at as well but so i think phase one is going to be figuring out our vpn stuff and you can see there's some some stuff we got to sort out then phase two is replacing the server itself and the biggest challenge here has become the disk. We need about 40 terabyte of new disk to come in. That's clearly going to be the largest cost. Then the second challenge with a new server is what distro should we use? Obviously, we've been using Arch for a while.

It's been great. And there's certainly a part of me that would like to just continue to use Arch to see, like, how far can we take this thing before it breaks? You know, we already have some more updates on the server I just checked. Oh man. But also I recognize this is an opportunity for us to try something else as well. So in the show notes, we'll have a link to a straw poll and I would like the audience to vote on which server distro we go with next. There are only a few options because we limited it to genuinely rolling distributions. Rawhide's on the list right now, but Neil brings up a good point that we could have significant issues with ZFS if we went Rawhide, but

not necessarily impossible to solve for. So we have Arch, Nix, Tumbleweed, and Rawhide, So we have Arch, Nix, Tumbleweed, and Rawhide, all as an option for you to vote on right now. The live audience has gotten 42 votes in so far. These often will indicate the trend. And right now, Tumbleweed is in the lead with 45.24% of the votes. Rawhide's in second place, Nix is in third, and Arch is in fourth. If you have an opinion about this, we're going to leave the poll going for the next week so go to the show notes we'll have a straw poll in there it's a crazy url so i just tell you what it is and go vote and uh we're

very likely i mean like 99 likely unless some weird reason comes up gonna go with the distro that you guys choose rawhide gives me the most pause so not saying we're, going to go with the distro that you guys choose. Rawhide gives me the most pause, so I'm not saying we're not going to pick it, but it's going to require Wes and I having a strong discussion if it does, if it wins. I like that this is the option you're the least sure of. What a funny world. But you're right. I mean, there will be some complications. We might have to learn a fair

bit about the internals of Rawhide. Or Tumbleweed, if the trendline continues, my friend, which... That's a whole new world. What have we done? Snapper. So I guess like in that sense, you know,

maybe it makes sense, but true, true. There's also the hardware component of the server. And I cannot believe this, but the UPS tracking numbers don't lie. A listener has reached out a mr real zombie geek is sending us some new server gear new to us he's actually decommissioning it but he has in route

right now should be delivered tomorrow at the studio a dell power edge r820 with and and I just can't even believe this, four Intel E5-4657 CPUs, each with 12 cores. Four of them with 12 cores. And 384 gigabytes of ECC RAM. Hey-o. Right? I mean, even if, like, maybe it's only two Intel, I can't imagine it's four, maybe it's only two Xeons.

That's going to be an amazing, amazing amount of compute for us. He's shipping it with no caddies or drives. So we're going to have to figure out what kind of caddies that R820 takes. And I'll go on eBay probably and buy like a whole batch of them. So we've got some prep work to do here. Yeah. And then JB's got to buy some disks, right? But I mean, he's saving us a huge cost.

And I was considering going with another pre-owned server anyways. And he's had this in production. It's been a solid box. He's just us a huge cost. And I was considering going with another pre-owned server anyways. And he's had this in production. It's been a solid box. He's just replacing it. And he offered it to us. And I just, I can't believe it. That's wonderful. Best audience in the universe.

There's just no doubt about it. So that'll be the hardware. Whatever distro you all vote on will be what we put on there. And then there'll be the third phase. The third phase is going to be to set up some kind of remote console. We were initially trending towards PyKVM, and I think we still may try that because I ordered a little PyKVM kit. Oh, look at you. But this PowerEdge has a built-in iDRAC enterprise, which may be just fine, Wes. We'll see. Yeah, it'd be interesting to compare and contrast. So with iDRAC, there might be even some fun automation potential that you could work with. Because I know that in RHEL and CentOS, and I think also

in Fedora, and it would be in Tumbleweed as well, there's a lot of tools around integrating with various BMCs, ILO from HPE, generic IPMI, Redfish extended IPMI, and iDRAC as well from Dell. So there's all kinds of fun opportunities there if you have like built-in management interfaces like that. Hmm, I love that. Yeah, I can't believe it. I mean, what a gift to us right now because we just caught back from that road trip.

So expenses are tight right now. And so that's a great thing to have come along and should be great to play with. And I don't know what I'll, I may put the old server on a test bench and see if I can't narrow down what the issues are,

or it may just sit there for a bit. We shall see. But either way, that's the plan. We have a three-phase plan to get this server going. And Wes and I are already beginning to play around

with VPN options. There's some really fun stuff out there. We often talk about the desktop stuff because it's fun and we love the desktop, but there's so much awesome stuff going on at these layers too. So it'll be fun to deep dive into that. And I'd love to get your opinion

on which distro we should go with next. So go find the straw poll link at linuxunplugged.com slash 425. Oh, it's time for feedback. And hey, maybe if you've got some ideas on what we should do in any of those phases, well, yeah, right into the show. But in the meantime, Aaron's already done that with a self-hosted location tracker he really likes. Now, of course, we had that set up at colony tracker dot live, a different piece of software that we've been

testing out for the road trip. But Aaron has been using a Nextcloud app called PhoneTrack. This is an app for Nextcloud. I think I've seen this before. Yeah, I think we just need to install it on the Jupyter Nextcloud instance that we've already got running, as Aaron suggests, although I haven't actually tried it yet. I like this as an overall approach to like taking something that I think a lot of us think is, is taboo, like user tracking and cameras, right? Having cameras in your home. Ooh, I like taking that and putting it on its damn nose and saying, well, what if I do the tracking and I control the data? Well, what if I run cameras, but they only work on my land and they go into something like Shinobi and

I record the cameras and can technology that we often consider invasive be used by me under my control? And that aspect of phone track really appeals here because it's part of just hosting something yourself. And if you've already gone to the trouble of setting up NextCloud, you install this app, and I think it has an Android companion app too. Yeah, you know, Aaron hits on that, that he'd been on a journey to de-Google-ify, and that location tracking was one piece of that. And I think it kind of makes sense, in some sense, to have it integrated with NextCloud, because, yeah, NextCloud can be a big part in getting off a lot of those popular Google services. And for his part, he does notice that the phone app has worked pretty well, and that he hasn't seen a hit to the battery life.

Hmm. Yeah, you can have it set so it sort of gets the background location update from the Android operating system. We opted not to go with this just because we didn't really want to put our Nextcloud instance public and own tracks was sort of perfect for our use case. But Bytebin, if I recall, you gave this a go for a bit, right? Yeah, so I use it to track my phone. And for when I lose it, then I can find it again. And it works really well.

It also says when it's collecting its GPS points, how many it's already sent, how many there are in the session. And it's also a really good platform. For example, if you've got drivers or delivery guys driving around, you can see it really easily on a map and a nice overview. And you see they have put some thought in multiple use cases. But what I also noticed is you really need to get a good use of these sessions because I have, for example, one session of my phone tracking and once i pull that up my living area is just one big red dot about yeah you've been there there are all the lines

and i don't know if it's my amount of data or my setup but the filtering isn't that really great for me at the moment but i assume they will be fixing that in later versions. But yeah, you can select specific sessions you want to record to, and you can put devices in there, but you have to look at if you need to select the session from your device itself. So that is for me at least a question mark if you can put devices in there from the server side. Okay, that's good to know for people that might be interested in using it.

And we'll have a link to that in the show notes if you want to try it. Actually, no. You know what? No. You just go in your Nextcloud, you look in the apps, and you just install it right there. No link for you. No link. No link for you. I will put a link to this, though, because I am on the search for... I'd love something that's a server-side web app

that I hit up on my LAN, and I paste a YouTube URL in, and it downloads in the background and puts it somewhere on my file system, and then I open up my media player of choice, and it's connected to my NAS, and bam, there's the YouTube video. And I've been looking for a web app to do this for a while, and I don't know if I found the perfect

one, but I think I found a real contender. It's called Tube Archivist. It is a self-hosted YouTube media server. You give it the URLs and then it will archive them and then present you a front end to review all of your downloads, kind of give you like your own YouTube subscription feed, but on your own box where it's pulled down the videos locally. Oh, this looks nice. Yeah, man. And it keeps track of like which ones you viewed and which ones you haven't viewed.

And it's using that new YouTube DLP that we've talked about, YTDLP instead of YouTube DL. Oh, we made that a pick a couple of weeks back, right? Yep. Yep. So this is kind of like taking and putting it all together. And then I like this because I thought I've got a couple of YouTubers I've followed that have like put up videos and then pulled them. But I also have a ton of videos I watch on YouTube for like RV maintenance stuff. And sometimes I'm in an area where I want to do some work and I'm offline.

This is what I was hoping for is something I could throw on my, my box and just throw these links in there and then build a little offline YouTube collection. The only spot where I kind of got hung up and why I haven't pursued it and Wes, you're going to laugh at me, but you know, you got to set up like elastic search and do some of that stuff. And I just, I didn't want to go that far, but I wanted to give it a link. So you're saying you need someone to write you a already composed Docker compose file. So you're saying you need someone to write you a already composed Docker compose file? Absolutely, Wes. That's exactly what I'm saying. And also, if anybody has another tool to accomplish this same thing, do let me know about it. However, you want to tell me because I'm always looking

for a way to do this. Yeah, you know, I want something like this, but just if it also had some like Chromecast support on the flip side, you know, it could kind of be the one-stop shop for all the videos. That would be nice. Yeah. Yeah. And for me personally, I don't really need a front end that tracks the watch status because I just would like them on the file system and then I'll throw Kodi or Jellyfin or Plex or whatever at that directory and manage it with my existing media application. So you just want the ingest side.

Ingest and maybe a little bit of sorting, you know, maybe a folder per YouTube channel. And so, you know, if you download from this YouTube or you put it in their folder automatically, I don't know. I guess I'm asking for the moon here, but it seems like there must be something out there because, I mean, hell, there's so many. Back in the day when we all watched our television via cable, there was like a dozen different PVR, DVR solutions, and there was Myth TV, and we just need something for the YouTube age. So if you know of one, let me know.

And if you'd like to find our friends, A Cloud Guru, on social media, they're just slash A Cloud Guru at pretty much any social media website, the YouTubes, Instagram, slash a cloud guru. Go find them over there and see what they're up to. If you do the Twitter thing, you can also follow the show at Linux Unplugged. Wes, you're probably over there. I'm at Wes Payne. Remarkable. Remarkable.

The show's over there. The network's over there, too. Yep, that's true. We have a show and a Twitter for both of them and myself. It's a lot of Twitter accounts, really. Who would have known I'd be managing so many different logins in 2021? And then don't even get me started on the show mailboxes. Holy smokes. Just bypass all of that and come join us live. We do this show every Tuesday at noon Pacific, 3 p.m. Eastern. See you next week. Same bad time, same bad station.

And if you work in the tech industry or you consider yourself a Linux fan, don't miss Linux Action News every Monday morning, really Sunday nights. Wes and I break down everything that matters in the world of Linux and try to get you the information in a concise, accurate manner. That's at linuxactionnews.com. Keep Linux rolling after that by joining us on Tuesday or go subscribe at linuxunplugged.com slash subscribe. We absolutely love them subscribers

and our members at unpluggedcore.com. Thanks so much for joining us on this week's episode of the Unplugged program. It's not a Monday. It's not a Wednesday. No, no, my friends, we will see you right back here next Tuesday. so along with their robot amazon has has opened an invite-only program for their previously announced home surveillance drone,

the always-home cam that flies around and surveys your home for you. Oh, man, Wes. Oh, I think I'm about ready to go move out into the woods. It's really just such a shame because it's like, in some sense, if I felt like I was actually in control of it, yeah, maybe I would want that, you know, if it was open, I knew where the data was going. But of course, that's not what we get. That's very much like the philosophy I was just trying to articulate in the show. It's like, if it's cameras that I have complete control over, and I can replace the firmware, and it's something I can check and monitor, then yeah, I'm okay with

it. But if it's something that I have no control over that could be easily perverted, I find that gets dystopian really quick. And it feels like there's a lot more of the latter these days. Well, with the dog, you know, I was just imagining here you are, you needed to sit and then suddenly the connectivity goes out at just the worst time.