LINUX Unplugged - 577: Summer Kernel Corn Roast

Episode Date: September 1, 2024

Sixty vulnerabilities and exposures disclosed in one week sounds like a lot. We'll explain why it's just business as usual.Sponsored By:Core Contributor Membership: Take $1 a month of your membership for a lifetime!Tailscale: Tailscale is a programmable networking software that is private and secure by default - get it free on up to 100 devices! 1Password Extended Access Management: 1Password Extended Access Management is a device trust solution for companies with Okta, and they ensure that if a device isn't trusted and secure, it can't log into your cloud apps. Support LINUX UnpluggedLinks:💥 Gets Sats Quick and Easy with Strike📻 LINUX Unplugged on Fountain.FMToronto Meetup — Thursday, Aug 29, 2024Berlin with Brent — September Meetup @ Nextcloud Conference, Saturday, Sep 14, 2024Check out Alex’s “Building a Colo Server” videoMicrosoft’s latest security update has ruined dual-boot Windows and Linux PCs — The cause: an update Microsoft issued as part of its monthly patch release. It was intended to close a 2-year-old vulnerability in GRUB, an open source boot loader used to start up many Linux devices.What the f*** is an SBAT and why does everyone suddenly care — This update was not supposed to apply to dual-boot systems, but did anyway.SBAT Revocations: Boot Process - Ubuntu Community Hub“Something has gone seriously wrong,” dual-boot systems warn after Microsoft updateUbuntu Will Be Skipping Non-Critical Linux Kernel Updates For September - PhoronixSRU Mailing List AnnoucementCanonical Moves To Shipping Very Latest Upstream Kernel Code For Ubuntu ReleasesKernel Version Selection for Ubuntu Releases - Kernel - Ubuntu Community HubLinus Torvalds Begins Expressing Regrets Merging Bcachefs — The bcachefs patches have become these kinds of "lots of development during the release cycles rather than before it", to the point where I'm starting to regret merging bcachefs.Re: [GIT PULL] bcachefs fixes for 6.11-rc5 - Linus Torvalds — No one is being jerks here, Linus and I are just sitting in different places with different perspectives. He has a resonsibility as someone managing a huge project to enforce rules as he sees best, while I have a responsibility to support users with working code, and to do that to the best of my abilities.LINUX Unplugged 545: 3,062 Days Later — Kent Overstreet, the creator of bcachefs, helps us understand where his new filesystem fits, what it's like to upstream a new filesystem, and how they've solved the RAID write hole.Linux is a CNA — As was recently announced, the Linux kernel project has been accepted as a CNA as a CVE Numbering Authority (CNA) for vulnerabilities found in Linux.The Linux security team issues 60 CVEs a week, but don't stress. Do this insteadWhat is a "good" Linux Kernel bug?Keynote: Linux Kernel Security Demystified - Greg Kroah-Hartman - YouTubeMembership Summer Discount — Take $1 a month of your membership for a lifetime!added pihole nix module by Tdback · Pull Request #3 · JupiterBroadcasting/nixconfigs — Recently, I wanted to start 'nixifying' some of my docker-compose setup. I've created a simple module for spinning up a podman container running pihole as a systemd service, so that way I can just stick it on any NixOS machine and easily make it my DNS server.NetworkManager cli (nmci) wrapper to easily create a new network connectionDistrohopper WheelNo idea where to distrohop next? Let the ultimate distrohopper decide for you!Proxmox Virtual Environment - NixOS WikiPick: SaunaFS is a distributed file system — A robust distributed POSIX file system meticulously designed to revolutionize your storage solutions by offering unmatched efficiency, security, and redundancy. At its core, SaunaFS is a distributed file system primarily written in C++, inspired by the pioneering concepts introduced by Google File System.Google File System - Wikipediasaunafs/INSTALL.md

This week really, really made it feel like summer is coming to an end. Not necessarily because the kids are about to go back to school, not necessarily because it rained all week here in the Pacific Northwest in August, but because the news just really came rolling in. Yeah, we're not supposed to get this kind of stuff in the summer. No, so it's kind of great. So I think it's high time we do a little end of summer news roundup because there's a lot going on. Even though technically summer's not over, it seems like everybody's back to work. Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris. My name is Wes.

And my name is Brent. Welcome back to your weekly Linux talk show. My name is Chris. My name is Wes. And my name is Brent. Hey, gentlemen. Well, coming up in the show today, we are doing an end-of-summer news roundup. Some big things have slipped in there.

Plus, we have some thoughts on distributed LAN file systems. And then we're going to round out the show with some great boosts, crazy great pick, and more. There's a lot in there. So before we go any further, before we get started, let's officially say time-appropriate greetings to our virtual lug. Hello, Mumble Room. Hello, guys. Hello, Brent. Hello. Hello to the on-air and hello, everybody up there in quiet listening.

It's nice to have you on the show today. We'll probably have a lot we can get into later. Also, a big good morning to our friends over at Tailscale. Tailscale.com slash unplugged. It's the easiest way to connect devices and services directly to each other, wherever they are. Even if you've got, like, that horrible double carrier grade NAT, let me tell you, it still works. It busts right through there.

And you can get it for free for up to 100 devices and three users. Not a limited time thing, just 100 devices ready for you to use at Tailscale.com slash unplugged. Replace that legacy VPN infrastructure with something powered by Wigout. Tailscale.com slash unplugged. Well, after the show, I'm jumping on an airplane and I'm heading over to Toronto to rack a server with my buddy Alex. And we're going to have a little meetup while we're there. Thursday, August 29th at I think it's 6 p.m. local time.

I don't know because the meetup page is in Pacific time and it even confuses me, guys. 6 p.m., 6 p.m. Thank you. Brent's going to be there too. That's true. Wes will be back here like the operator, you know, looking at all the screens, watching all the metrics,

and then when we have a problem, Wes, I need you to do something for me. I need a switchboard or something. Yeah, you do. Especially the big clunk of the cable. Of course, yeah. And then, of course, you just SSH it in. Have you seen the number of attendees for a little Toronto meetup? Oh, no.

Should I go take a little update keys on it? Oh, it's fancy. Now, is the attendees count in local time or Canadian? Yeah, I think it's metric. Metric, yeah, yeah. Yeah. 41. Wow. yeah, yeah. Yeah. 41. Wow.

Oh, man. 37 members are attending for the first time at the KB event. That is really something. Pent-up demand. So what's kind of scary about the number 40

is it means it could be 65 or it means it could be 30. I made a reservation for 25, so I think I might have to call them back. Oh thank you for doing that yeah and parking with that many people might be an issue so we might want to ask them about that too if you wouldn't mind. Because we could always update the meetup

with another venue if we had to. It could be a problem. You get into like block block parties? Since it's on the Toronto waterfront on Lake Ontario I think and they're right next to Harborfront Canoe I think you can also take your canoe there if you can't find parking. There you go. And then later on, you could roll your way over to Berlin with Brent in September on the 14th.

It's true. Doing a little meetup at the next Cloud Conference. I know, I think a couple of mumblers are attending as well. Are there a couple of you there? Yes, a couple of you, Dave. So I'll be meeting for the first time which is really exciting and uh i think same same deal we've got like 30 or so folks showing up something like that so it'll be a it'll be a party i wanted to mention too we are

going to be racking and stacking servers i should be back before the next episode so maybe i'll have some tales to tell of how all of that went but in the meantime you can check out alex's building a colo server video that we'll have linked in the show notes. And he does a video walkthrough from power supply to disk of the new JB server and kind of what our thinking is and how we're trying to build in dual remote redundancy and things like that. So we'll have a link to his YouTube channel where you can watch that video and figure out how it's going to work. where you can watch that video. You're going to figure out how it's going to work.

Okay, so let's get into some news that's impacting Linux users. And this one was caused by Microsoft. You may have heard, I think it's just kind of getting the rounds this last couple of days, a Windows update has wrecked dual boot for a lot of Linux users. It seems to be that Microsoft issued an update that, I don't know, is doing something to the secure boot layers. And in doing so, they break Grub for systems of a certain age and a certain Grub setup.

Yeah, we can get into more of the deets in a bit. Yeah. And this happened Tuesday as we record. So one Tuesday ago, as we record, they did their patch Tuesday and started rolling out to everybody. It's for CVE 2022-2601, which is a bug. Yep, as the CV indicates, was discovered in 2022, but Microsoft patched it last Tuesday.

And when Linux users try to boot after they've had this patch installed via Windows Update, they get an error saying, Verifying shim SBAT data failed, security policy violation. Something has gone seriously wrong. SBAT selfAT data failed, security policy violation, something has gone seriously wrong. SBAT self-check failed, security policy violation. And we have, in our own community, we've seen reports from Linux Mint users and Ubuntu users. But, you know, essentially... Does that just tell you who's still dual booting? Yeah, maybe.

A lot of users are impacted by this. 17 hours after the post went live, Microsoft kind of recognized the issue. Yeah, maybe. A lot of users are impacted by this. 17 hours after the post went live, Microsoft kind of recognized the issue, didn't really own it. And so far, it really seems to be it's up to the user to fix it. Most people are just suggesting fix it with like a Fedora boot environment or something like that. There is a command that we can link to in the show notes. It's like a one command via Fedora boot environment to fix it if you have this problem. Thank you, Fedora boot environment.

So what the heck, Wes? How did a Microsoft update break Grub? Are they patching? In fact, if you read some of the stories, some of the explanations are, well, Microsoft was patching Grub and it broke Grub for Linux users. Microsoft is not sending out Grub updates via Windows updates. So what really happened here? Yeah. So SBAT is Secure Boot Advanced Targeting. And I guess it was actually developed collaboratively

between the Linux community and Microsoft. And, you know, there's more details, but kind of the example is there's a lot of boot components that need to be signed and trusted in a secure boot chain. Yeah, sure. And then in particular on a lot of Linux systems, right? So like Microsoft is usually the ones with the default enrolled Secure Boot keys.

You can enroll your own if you want or other manufacturers can have them or whoever else. But Microsoft has made agreements where when you go buy your new Lenovo and it comes with the Microsoft key and it can boot into Windows. And you'll recall classically, Microsoft has really been at the center of Secure Boot.

Initially, they were one of the few people that could actually even sign anything. And that was kind of controversial about Secure Boot initially. They were one of the few people that could actually even sign anything. And that was kind of controversial about Secure Boot initially. And over the years now, the Linux community has developed Shim, which sits sort of between things and handles Secure Boot and keeps making things work. Plus there's Grub in the mix. And all these components got to be signed and kind of work together. And there's a sort of social contract in the mix, too, because Microsoft is, in a way, helping us out by being willing to sign things like Ubuntu and Fedora so that those distros will just boot without having to add their own key or have a complicated sort of setup process.

Like I said, controversial a little bit. Indeed. controversial a little bit. Indeed. But so to have the good experience, we kind of don't want to break that trust because Microsoft signing us is saying, I'm trusting you because if there is then a flaw or a bug or vulnerability in the signed component, you can use that to attack Windows. Right. So they're vouching for us. Yeah, exactly. And so because there's so many components, it can be kind of a pain to like, you just don't have enough storage for all the hashes that you want to have. Other things have to kind of delegate trust and have this chain. And you have a problem around like handling revocation, right?

Because you've got to like, when you do have bugs, like Grub is an old project that was not designed when boot security was like a big thing. So even with best of intentions, we continue seeing issues. And then you've got to update things. And then you've got to roll, you know, you got to update the distros and you got to roll out of these updates and it just became mechanically, it became a lot to manage. So SBAC comes in as a way to have something called a security generation. And then instead of having to go blacklist individual components, you can say, I want a security generation of a minimum number. And then anything that's too old in the security generation will no longer boot. So this feature was a way to sort of make this a little, improve things in the ecosystem,

collaboratively between Linux and the Microsoft community. The problem here is that Microsoft rolled it out, and some Linux distributions were shipping a version of Grub with a known flaw. Yeah, with a known flaw that should not have been trusted. And then also therefore had a security generation below what Microsoft Update was telling the system to check. And so shim is the thing giving you that error message because it sees that SBAT was enabled. And now it's doing its thing, trying to be a good citizen, saying like, oh, well, I got to follow this policy. I'm not going to go boot into this grub

because the security generation is too old. So then you get the problem of these are old vulnerable grubs that you probably don't want to boot in a secure environment and those haven't been updated for some distributions. And then the Microsoft side, they clearly didn't do a good enough job testing this because it actually was only supposed to be enabled on single boot systems.

So they're going to detect dual boot systems. You know, that's always been a problem of theirs. Yeah. It was clearly not working. The plan, I think, I guess, was that they would roll this out for Windows sides and then sort of wait for those distributions to make the updates. And then the Linux side could turn on SBAT at the firmware level. Yeah. OK.

Yeah. So that is definitely a part of the story here is that their dual boot detection didn't work properly. It impacted more modern distributions than they said it would in the scenario where it does detect it as well. And Secure Boot, you know, has been a pain in the butt. I'm curious to know where the audience

kind of lands on Secure Boot. Is it something you live with? Has it caused you problems? Let us know because I'm trying to take a poll on where the Linux community is on this right now. I have, you know, in modern times, I've managed to wreck a dual boot setup kind of unintentionally.

And, you know, they're a pain in the butt to fix. You know, it takes time and you never want to do it. And it's like you just get like a pit in your stomach when you realize you just busted the dual boot. Yeah, I wonder like, I don't mind using it on my systems if I've enrolled my own key and I know what I'm doing. Yeah.

I feel like I don't know if I was setting up a computer for someone else. Would I even want it on? Well, sometimes you don't have a choice. Like that's what I ran into. Right. It's like, it's not my computer. The vendor's got it locked down.

And obviously there's a lot of benefits. If you do have data you want to protect and are taking your laptop around. But I don't know, for a home PC or something, is the complexity worth it? I don't know. I want to be the person that says yes, but then I look at it in actual implementation and I want to say no. It's complexity is there. I have concerns about vendor lock-in. Maybe it depends on the machine too. Like, do you want it for your Bitcoin node? Okay, sure. Do you need it for the machine you use just to browse the internet while you watch TV? See, that's where I fall down. I feel like for a server, yes, it kind of makes sense. You want

to make sure you've got a trustable OS all the way down the chain as far as possible. You know, that server's responsible for who knows what. Like, that's absolutely where it makes sense. Ironically, the places where it gets implemented are devices where it seems to matter the least to me. Actually, I would flip it all the way on its head, right? Like, so if you think about it, the system that is the most controlled and most physically secured is probably the one that needs Secure Boot the least. Yeah. And so that would be servers although there are other reasons to have secure boot on servers you know having lots

of data that people would want to steal and all this other stuff and avoiding the the whole boot time malware thing which is a real thing now i don't disagree and i think that's why you know people have had a pretty tolerant approach to secureure Boot, but then we get years into this and we're still... I mean, look, there's folks that just got their dual boot setup completely nuked. But if you remember, in the pre-Secure Boot world, the pre-UEFI world, it's always been a problem.

It's always been a problem. Even in the NBR days. Yeah, Windows likes nuking other operating systems. That's normal. But I think that it's important to understand that the whole Secure Boot design is bad. All right. I think that's enough of our energy on Secure Boot, but I think Neil makes a good point. And I also co-sign that there's some situations like mobile devices where you absolutely want

this. But Brent, I wanted to shift gears to talk a little bit about Ubuntu. You and I have been watching a seemingly large ship attempt to change direction a little bit. And a couple of them that I think are more intriguing is some of the changes that seem to be happening around the kernel, including a story that we saw this week that kind of seems to me to indicate some sort of infrastructure changes are needed at Canonical. Did you see this? I did see this with a mix of excitement and also cautious curiosity. Yeah. It seems like Ubuntu will be skipping non-critical Linux kernel updates for September,

which is kind of an interesting change here. With the exception of critical security issues, they'll be skipping shipping stable release updates for the Linux kernel in Ubuntu until about early October, they say. The decision is due to a critical infrastructure change, in quotes, that after September 2nd, things will basically be on hold until that infrastructure is in place, predicting about the beginning of October.

That's kind of a big deal, a month of not working on patches? I guess it's about a half from some of the deets. They say there's a security cycle from the start of September to the 16th. And then from the 16th to the beginning of October is when they're doing this break. Now, Chris, you've been around for a little while here. Has Canonical ever done something like this? Not to my recollection. I find this to be interesting that they're talking about this. Canonical's Roxana Nikosola said that, quote, please be informed that we'll be skipping the SRU cycle. Our next SUR cycle will start when the infrastructure

is back online. You kind of wonder what that changes. It's good to see, though. You know these things get built up over time. They get technical debt. We heard that some of these infrastructure issues were what kind of led to some of the changes around the RHEL source RPMs. They finished by saying, we are committed to delivering fixes only for critical security issues.

Okay. This comes, I think, though, in kind of a bigger context around Ubuntu, because Canonical also recently announced that they are moving to shipping the very latest upstream kernel with Ubuntu releases. So in the past, they would freeze, and it might mean that you get like one version behind or so, sometimes worse, when the new Ubuntu release comes out. when the new Ubuntu release comes out. And it's just kind of, I think this was inevitable, right? Because Ubuntu follows that hard set, with very rare exceptions, they are always released on a schedule.

And the kernel doesn't. It follows kind of like a looser time-based release process with major kernel releases happening every two or three months in there and kind of depends on when Linus decides if it needs an extra RC or something like that. And so what they're committing to is if they hit the release date for Ubuntu and the kernel's still on like RC5 or something, they're going to ship it. They're going to ship an RC. So that way they ship the absolutely latest kernel. And I would presume they'll update it later. So the way they are taking it, in their words,

they're going from a conservative wait-and-see approach to this new, we want to have the absolute latest support for our users approach. And they say really it's like hardware requirements and whatnot and users expect to install the latest Ubuntu and have their latest hardware to work. And they've been burned a couple of times. Yeah, we've seen them get caught the last few releases, but missing that window just

by a week or two and being stuck with something that's a little aged. Yeah. I mean, we've seen tech reviews go out, you know, for like the new AMD. How's the new AMD Radeon card work on Linux? And the reviewer tries the current version of Ubuntu, but it doesn't have any of the support in the kernel or Mesa. And that is part of this, too. That means you also have to be shipping new Mesa.

Like, it's not just the kernel you have to ship. You have to ship a wider range of stuff as well. But, Wes, doesn't this kind of seem to suggest a pretty significant strategy shift when you take they're shipping the latest kernel, they're obviously retooling some internal infrastructure. Is this the kind of stuff that you see as signal that, yeah, there really is a focus on making the desktop more performant, making the desktop more competitive? They've certainly been talking about it, right? I mean, also with a focus on making gaming

sort of a first-class citizen on desktop Ubuntu. You got to have good hardware support for that. Yeah, I think there are definitely steps. I do wonder, I think some of the coverage of this kernel change noted that, you know, this specifically wasn't them committing to shipping upstream kernel versions more often as part of their release cycle. So it's not maybe all of the investment that some have hoped for from their side, but it does seem like very active changes made with desktop use cases in mind. I mean, this benefits other use cases too, but I think that's the kind of mentality that

folks for a while had wondered if maybe wasn't a priority. So seeing these changes, seeing investments on the underlying infrastructure, I don't know if it gets me where I hope for or where it would change my perspective, but it's building. Well, here's what I would hope for. What if this meant that, I mean, we'll see, but wouldn't it be great if it meant that the canonical team was kind of writing more with the kernel team in finding and discovering bugs that are relevant right now and then maybe even engaging in the process to submit

those bugs back to the kernel team? Like there could be a situation here where now canonical is much more or could be more involved with the process if they wanted to engage. We'll see. I overall take this to be a pretty good sign. I think it means that 2410 should ship with Colonel 611, which will be nice to see. And my question, and boost in and tell me what you think about this, because I'd love to have a conversation in the next episode if we can. What would make you switch back to Ubuntu? Assuming you were once an Ubuntu desktop user, what would bring you back?

What kind of changes do they still need to make to make you consider coming back? Or if you're thinking of leaving, what do they need to change to keep you from, you know, bouncing to another distro? Or have you come back? Maybe there's been a reason you've come back. I'd like to get into this before the next Ubuntu release. So boost in and tell us your thoughts on that. And we'll pick it up maybe in the next episode.

So what could Canonical Tweak to bring you back? Or are you thinking of leaving? So what do they need to change? Or have they already brought you back? I'm going to have to ponder this one. 1password.com slash unplug. The number one, password.com slash unplug. Go there to support the show and take a moment to picture your company's security. Imagine it like a college quad. Stick with me here. You've got those nice paved paths. They look great. That's like your

company devices. Those are your IT approved apps. And then there's the shortcuts, the worn through the grass by the students late for class kind of shortcuts, unmanaged devices, shadow IT, contractor devices. That was my world when I was in IT and it was very frustrating. Most security tools only work on the paved paths as if, as if we're all going to stay on the paved path all the time. Let's be honest. All of the real action happens on those shortcuts. That's where one password extended access management comes in. It's like, if you will the real action happens on those shortcuts. That's where 1Password Extended Access Management comes in. It's like, if you will, putting security cameras on those shortcuts.

It ensures strong passwords are used, healthy devices are only allowed to connect to your network following your policies, keeps an eye on all those applications that you require. It solves the problems that traditional identity management and MDMs just don't handle. It's security for the modern workplace that actually works. And it's available right now for Okta and Microsoft Entra, and it's in beta for Google Workplace. It makes the whole thing a lot smoother for your end users and for IT. Go check it out and support the show. They've got a demo over there, too.

It's 1password.com slash unplugged. And no, that's not one of those shortcuts. It's the real deal. 1password.com slash unplugged. And no, that's not one of those shortcuts. It's the real deal. 1password.com slash unplugged. Regular listeners will know we're pretty excited about the up-and-coming BcacheFS file system. Normally on the show, we kind of highlight development updates. So far, it's been good things with a few bugs here and there. development updates. So far, it's been good things with a few bugs here and there. And now, well,

now, unfortunately, it's time for a little bit of BcashFS drama. Yeah. So on Friday, a set of fixes was submitted by Kent Overstreet to Linus for the current 6.11 cycle I just mentioned. And there was two pretty big things in there that I think caught Linus's attention. And he, you know, sometimes gets a little spicy towards the end of a release cycle, especially when you're making big changes that don't necessarily fix bugs. Do you know what it was that Kent was trying to get in there at the last moment? Oh, well, I think it was a lot of stuff. I don't have a specific list. We could probably go look at the patch set.

No, it's okay. I know that some of it was bug fixing, but some of it was like other changes that were sort of featured. Yeah, the objection from Linus's side for a lot of it is like we're in the stabilization process in the RC phase. Bug fixes, yes. Small fixes, yes. But new development? No. So here's what he wrote when Kent sent the poll request in. Linus's response, quote, yeah, no, enough is enough. The last poll was already big. This is too big. It touches on non-BCashFS stuff, and it's not even remotely some kind of regression.

At some point, quote, fix something just turns into development. And this is that point. Nobody sane uses BCashFS and expects it to be stable. So every single user is an experimental site. I'll pause there for a second. I read through this thread, and I feel like maybe Kent took a little exception to that statement. Because it seemed like he felt a need to defend it.

Yeah, and I think that's unfortunate because it kind of... Maybe derailed isn't the right thing, but it's not really the core. I mean, it's related. It's an important component to this is how you view the file system. What is the support model and what is expected and all of that. But I don't know if it was core to the, what was otherwise being discussed. You know, I think Kent finds himself consistently in advocacy mode just simply because it is hard to get people to care about a new file system. And he does have users that have terabytes and terabytes and terabytes of data stored on vCacheFS.

Yeah, it is a good point, right? I mean, he spent years and years building this himself out of tree first, you know. I know there's developing a bit more of a community, but pretty much a solo dev type model. A file system is something that right after 10 years, people are like, okay, yeah, I guess I'll use it. So how do you get enough? How do you onboard

that? How do you bootstrap that whole thing? You can understand why he's excited and wants to press forward to get to a stabilization stage. Linus continues, the Bcash FS patches have become these kind of lots of development during the release cycles rather than before it, end quote,

to the point where I'm starting to regret merging BcashFS. Ooh. Ouch. Yeah, that's the headline phrase right there. If BcashFS can't work sanely with the normal upstream kernel release schedule, maybe it shouldn't be in the normal upstream kernel. This is getting beyond ridiculous.

That really hurts because we've talked to Kent. We're very excited about Bcash FS episode 545 if you're curious and it hurts to hear Linus frustrated with it you know and Kent's reply is you know he's

an advocate of his baby he writes Bcash FS is definitely more trustworthy than Butter FS comes in swinging. He says, I'm working to make it more secure and robust and reliable than XFS and Extended 4. Yes, it will be. And he wanted to make the point that he has a bunch of users on it. He's trying to make it something really solid.

He wants a good reputation, so he doesn't want users out there using it with problems. You can see both their points. And I know from talking to Kent, I think Kent understands Linus's position. He's not angry about it. He understands Linus has a job to do, and Kent views it as he has. I'm projecting here a bit, but I would characterize Kent as believing he has a job to do. He's got users out there. I've got a quote from you, actually. In response to someone else kind of jumping into these threads on the mailing list, not to Linus, he said, no one is being jerks here. Linus and I are just sitting in different places with different perspectives. Yeah. He has a responsibility as someone managing a huge project to enforce rules as he sees best. Well, I have a responsibility to support users with working code and to do that to the best of my abilities.

I think they just need to be, you know, on the same page. Linus, though, gets the vote, right? So he's the he's he's Linus. But it's like when you see that he regrets merging, like, oh, don't give up hope. Linus, one day it'll be your file system. I know it will be. Don't give up hope. You have it running actively still anywhere?

Yeah, it's on this laptop right here. Didn't I make you deploy it on that big machine last time I saw you? We do have it up on a big server too, yeah. So, you know, maybe somebody should tell Linus about crazy old us, I guess. While we're talking about the kernel, there is a lot of discussion about the amount of CVEs, the common vulnerability and exposures

that are coming out of the Linux kernel. And I kind of understand the upset because when I worked in IT at a bank, my CTO followed this closely. And in fact, I had to have a whole presentation to explain to him why Red Hat might be issuing CVEs about Firefox and it does not impact our servers. And because he was looking at headline numbers. He was just seriously doing, well, I see Windows NT has these many CVEs and I see Linux has these many CVEs. And then I had to explain to him, well, we're using 10% of what Red Hat has issued CVEs about, right? And explain the whole development process.

Well, you can imagine how much amplified this is for the Linux kernel. And so the Linux kernel in some weeks actually issues as many as 60 CVEs, these common vulnerability and exposures. You hear them referred to usually as like CVE, then the year, and a number. And it's now in part because Linux is its own CVE issuer, a CNA, as they call it. And the kernel team has this philosophy of all things are bugs. And I wanted to just kind of dig into this a little bit because the Linux security team, it's not a department in a company, right? There's no CEO that they're reporting to. It is, it's a group of people that have come together and gotten more organized over time.

And I found a semi-recent clip, and I'll link to the whole clip, of Greg KH, Linus's number two man, that really kind of was clearly, in plain speak, just explaining how they came to be and how there is a Linux security team and what they do. And I'll play that for you. We're reactive. We're not proactive. There are other groups and other kernel security teams and projects that are proactive. There's a security conference every year. There's the Linux kernel hardening project. There's lots and lots of good stuff going on.

But that's not what we do. We just react to problems. And that's good. Somebody has to do this type of stuff. And this all started back in 2005 when somebody on the kernel mailing list said, hey, I want to report a security bug. How do I do this? 2005 was an interesting year. 2005 had the first kernel security list, had the first stable kernels, had Git. Lots of things happened in 2005.

We kind of grew up. So they said, hey, we need to have a list. So some of us had been doing this on our own, just in an ad hoc, informal manner, got together. And Chris Wright published this. He's now the CTO of Red Hat. He's gone on to better things than being a kernel developer. But he made, here's the rules.

He submitted a patch. And one of the most interesting things in this, and Chris and Linus set this up, is at the end, he said, we don't do NDAs. Linux kernel security team is an informal body, and we can't sign a contract. And that actually was the best thing that could ever happen to us. That set the stage for us doing this in a way that is company and government neutral. And it saved us so much problem.

So kudos to Chris for getting this right. So how do you do this? We have an alias, securityatchronal.org. You think you found a security problem? Email us. Just an alias, not a mailing list, no archives anywhere. Explodes out to the individual members. It's a small group of us. I think we're 10 or 12 people now. And we don't represent our companies.

We can't tell our companies who we work for what is going on. We have had problems of sometimes members of the list do tell their companies, and then we have to remove them from the list. It's all kept quiet. In all the years since 2005, we have never had any leaks. Pretty good relationship. So companies trust us. All we do is triage. We triage the report. We send it to us.

We figure out what's wrong. We drag in the proper developers if they're not on the list already. We work to create the fix as soon as possible. And we get it merged into Linus' tree and then the stable trees that me and Sasha release. The goal is to get it fixed as soon as possible. That's it. So I watched probably three or so versions of this presentation and skimmed another two because Greg has given this presentation many times over the years to Google, to Red Hat,

at conferences. And so it's been fascinating to watch him kind of slowly update it. And the most recent I could find online, I'll have linked in the notes. But a couple of things really stand out to me there, boys. Number one, he clearly, clearly articulates that they are a reactionary team. And I think we all kind of intuitively knew that. But Brent, I'm curious if that jumped out at you or if any of the other language, like he also in there mentions the fact that they try really hard not to work with embargoes, that they kind of are, you know, a little tricky about that. They can't even tell the companies they work for what they're working on security wise. What jumped out to you? Oh, man, it jumped out that it seems like such a tricky balance and a deep responsibility. And

the fact that they've had such a good track record up to now is deeply impressive. Yeah. And, and like doing the work that we all benefit from. I think we, we often forget about just how hard some of this stuff is, especially when you're trying to balance responsibility and who needs to know certain things and how to disclose these things properly and how to deal with the bugs properly as well. So kudos to everyone who works on this stuff. Wes, does it surprise you and not surprise you at the same time that they are reactionary only? I mean, we kind of knew this, but it's amazing this works.

Yeah, right. Well, I mean, there's some structure, but by and large, it's different companies and different individuals contributing. So you could have proactive security, but that would just mean you would need people to do it. And there are some folks who have a mindset focusing on kernel security like Keith Cook and others. So there's some sort of efforts vaguely in that direction. There's not like a red team that's their whole thing. You could have that, but you'd need the community to see that need and to have the resources and self-organize it. In a weird way, the market kind of solves for this because there's so many companies out there trying to make their name and whatever. So if you discover some major branded flaw in the Linux kernel,

you probably just got yourself contracts for a year. Yeah, there's also that factor is there's a lot of folks out there deploying Linux and Linux makes a pretty nice target in a lot of areas. So in a lot of areas that make a lot of money, we do end up seeing plenty of reports about bugs and potential security issues in the kernel. So let's talk about that because the kernel team is famous for saying a bug is a bug.

And I've heard people in the community, the security community really don't like that. But I want to hear it from Greg's perspective, and then we'll discuss on the other side. Because the biggest issue is kernel is 30 million lines of code. You only use about one and a half, two million lines in a server, three and a half to four million in your phone, about one and a half in your TV. But we don't know what you're using of those things. We don't know what your use case is. Linux is in everything.

It's in your smart meters, it's in your cars, it's in the satellites, it's in cow milking machines, it's in stabilizers, mega super yachts. Linux is everywhere. It's in my washing machine. We don't know your use case. We don't know how you're using Linux. We don't know what the security model is.

So we don't know what we're doing. We don't know what part of the kernel you're using. We don't know what code you're using. Whether you use this file or that file. So just take all the fixes. And most importantly, we don't want to know. You don't have to tell us. I don't want to have to keep track of it.

But you know this stuff. You know what you're using. You know this stuff. So just take all the fixes. The Google Android security team for a number of years documented all known security problems that were found in the Linux kernel and compared it to the stable kernels that we released. Every single one of them for two years were fixed weeks, if not months, before they were reported to the world. They have documented proof that taking the stable kernels always works. Your systems will be secure. And because of that, Android now requires this.

They require the stable kernels to be updated at a longer time. But we're trying to shorten that, make it a little better. But it's documented proof that we're fixing things before people know. I want to push back a little bit because what he says is obviously true. They don't know what part of the kernel you're using. Like he said, if you're using Linux on a television, maybe you're using a million lines of code. If you're using it on a server, maybe you're using a totally different portion. So we don't really know what you're using, so we don't really want to say what a bug is or isn't.

That feels kind of like a half truth, though. Like it obviously is true. They don't know. At the same time, you know, if something is like a remote exploit flaw, you know, if something's like a memory corruption bug that could be used to escalate privileges. So it doesn't feel like 100% of the truth. It feels like 80% of the truth to me. I think I'd maybe frame it because so I'd seen some folks when this was, you know, as

this has been announced as the process is ongoing, you've seen some pushbacks from folks. And to some extent, I think it's ultimately a question of who who's making those those gradations. And I think the case you bring up is true. But I also think if that gets surfaced, it's just going to be recognized as that and right away. And where really the issue ends up is being all the little stuff that you decide, like, does it report or not? Is it is it a you know, is this just a bug or is it a security issue? Some things are obviously going to be wide security issues or big problems, but some stuff's in that gray area. And, you know, the CVE process itself isn't super great for that. No. And they're just kind of putting on blast and they're saying, all right, this is all CVE process itself isn't super great for that.

No, and so they're just kind of putting it on blast and they're saying, all right, this is all CVEs. Yeah, and with the idea being that they could try to do the filtering. You could ask that they do that, right? But then what they're saying is, okay, we can filter the obvious stuff, but there's going to be a whole bunch of other things that you'll never know about that could end up being security issues. And we don't have enough context, we think, to make those calls.

So we would prefer to give you that information. Yeah. But it is, I mean, practically, it may then be that you're, that means downstream processes are forced to update, right? If your old method was looking for 10 CVEs, and now you have to handle, you know, order of magnitude more, I can get why you would, could be upset. Yeah, if you're trying to track all this, and you're looking at the numbers like my old CTO was,

you're like, oh my God. But here's the other thing I have to wonder, if it doesn't get a little awkward at times, because what if you have Red Hat saying this is a serious CVE, and you've got the kernel team saying, oh no, it's just a CVE, it's just a regular old, you know, normal CVE, it's no big deal. Is Red Hat going to make a, you know, are they going to have a big public disagreement? Like, how does that process work itself out? Is Red Hat more inclined to try to respect the Colonel team's reputation and just not escalate a CVE? Like, does it cause those weird, awkward pressures, do you suppose? I don't know. We have to look at what goes down in practice. See some case studies. I can answer a little bit about this. So part of the thing is that when

you're a CNA, no one else is allowed to disagree with you. Oh, okay. So the Linux kernel becoming a CNA means that no one else is allowed to create CVEs or to create their own judgments or whatever for security issues other than the authority that is the Linux kernel. So Red Hat is a CNA for some stuff. The Linux kernel is a CNA for its own projects, which is notably the Linux kernel. What about the situation where Red Hat patches the kernel, essentially forks it, and then they have the Red Hat kernel? Is that their own thing that they can issue their own CVE score for? They can issue their own advisories. And the way Red Hat slices this is that they have the meter and the upstream ones, and then they have a Red Hat CVSS score,

which is their own judgment based on the factors of their configuration and whatnot. And they present all of them at the same time. So that way they get to kind of avoid having a conflict here. Because, for example, a CDE could be critical when you don't know what kind of configuration it is, and you have to assume that no security features are turned on. But then it could be downgraded to maybe important or moderate, even. Because in a Red Hat system, by default, SELinux is on, and SELinux mitigates a chunk of the vulnerability or all of it yeah we've seen that a couple of times recently or with container escapes or similar i will say too

the reason why this is coming up now really is because the linux kernel team became their own cna in february of this year 2024 so we're kind of still just watching how this plays out and kind of you know observing it and plays out and kind of observing it. And I think so far it's been pretty unnoteworthy, except for these couple of moments where it's just been like, whoa, there's really been a blast this week. But when you dig into how this function works and what they classify as a CVE, it starts to look like business as normal. Nothing really too particular there. LinuxUnplugged.com slash membership. Support the show directly. Put your support on autopilot and for a limited time. Linux Unplugged.com content here that you get with a dollar off forever. You support it directly and you get access to the bootleg and to the fully cleaned up, no ad version of the show. Both are available to you as RSS feeds, private to you. And it's just something we say as a little extra thank you to

everybody who supports us. And with the bootleg, you get like double the content too, if you're looking for a longer podcast. So support the show, put it on autopilot and get access to a little extra content. Go to linuxunplugged.com slash membership. Use the promo code SUMMER. Take a dollar off a month forever. Well, this week we got in one of our favorite types of feedback. It is partly called pull request number three into our Nix configs GitHub repo. And there it is. There's the first mention in the show. We made it this far. Cheers, gents. quest number three into our Nix configs GitHub repo.

And there it is. There's the first mention in the show. We made it this far. Cheers, gents. You know, this time, it's really well worth it, just based on, you know, I took a sneak peek here at the feedback and I'm happy to drink for this. Now, TD Back writes in saying, after over a year of listening to the show, I finally gave in and started learning Nix.

Let's just say I now run NixOS on nearly every single machine in my house, and I even use it at work to create reproducible builds for employee devices. Oh, that's wonderful. I'd love to hear more about that. It has been a wonderful journey so far, and I can't help but get the same feeling I had when I first started learning about Linux. Yep. I mean, is that not exactly what we've been saying behind the scenes? Yeah, exactly. Also, Scouts Honor, you guys

100% disagree if I'm wrong here. We get this feedback a lot, like, I heard you talk about it, I didn't want to do it, I didn't want to do it, I finally did it, now it's on everything. Right? That is true. We get that a ton. We get other feedback too, but we do get that feedback. Yeah. They continue here. Recently, I wanted to start, quote, Nixifying some of my Docker Compose setup. I've created a simple module for spinning up a Podman container running PyHole as a systemd

service. So that way I can just stick it on my NixOS machine, any NixOS machine, and easily make it my DNS server. Given that a NixConfig repository was created for helpful NixConfigs at JB, I thought I'd share it with the community in case anyone else finds it useful. Thank you for an awesome show. Is that not value for value right there? Yes, thank you for sharing that back. And that also, in my opinion, is one of the super combos. Nix, Podman, SystemD is such an elegant combination that I am slowly just transitioning things to Podman if I'm going to have it in a container where I can. It's just it's so slick. I mean, do you want to speak to it at all?

Oh, no. Just, I mean, it's worth checking out, even if you don't nix, just to see what it looks like, because it's I don't know, under 50 lines here. And the config just enables Podman at the system level, and then defines an OCI container for PyHole, and then, you know, inside there, it really just looks like a Docker Compose file, right? You got

image, environment, ports, volumes, like, all the regular stuff is there. And then at the bottom, it tells Linux Firewall to open up ports so that PyHole can talk correctly. And, you know, this is one of those things that I just, I have been waiting for PyHole to get like properly Nixified. I don't know that that exists yet. So this is, I mean, just perfect in the interim. I have a pretty old, I have a three, I mean, for me, it's a three-year-old PyHole setup.

I think it's running on Debian. I mean, it's been a champ, but if I were to redo it in the future, like if the pie gives out or something, I think this is the route I'm going to go. That's nice. We'll have a link in the show notes if you want to check out the config. And now it is time for the boost. Oh, indeed. Martin DeBure comes in with 30,000 sats, and he got it on sale. He's our baller booster this week.

Hey, Rich Lobster! Martin DeBoost? Yeah, that's right. Thank you, Martin. Hi, Chris, Brent, and Wes. Hereby is my support for you taking a week off. Well, thank you, Martin. I'll just stop right there.

This has been on top of mind recently because I'm coming back from a trip soon, and I think we're going to have the details on our time off in the next episode. About the Hyperland topic. Yes, please. I recently watched some videos on the Linux cast titled 10 Hyperland Tips Under 10 Minutes. And is Hyperland good? A brief first look. I think it looks great, but it takes up a lot of time to properly set up. Best regards, Martin DeBoer. Yeah, I am feeling the itch to try out some of the new Wayland First experiences, and that's one of the ones on the top of my list.

How about you? Would you try it? Brent, would you try it if in a couple weeks we gave it a little go? Yeah, it's been a little while since I've actually run it on hardware, yeah let's check it out okay yeah we've been looking for a default uh you know distribution that has this set up by default and uh last episode we did uh get recommended one which i'm trying to find here i was trying to see if i could oh i don't see it in

our tags no way did. Do you got... Anybody remember? I recall we did get a suggestion. I'm blanking on... Yeah, I'm going through the links. Jeez, that's embarrassing. I thought I could pull that off, but I couldn't. But I bet we can find it and give that a go.

Yeah. And we're still open to suggestions, too. Space Nerd Moe boosts in with 26,000 cents. I hoard that which you all kind covet. Just sending some love for the great content you deliver to my ear holes week after week. Well, thank you, space. I really appreciate the value there. Thanks for the boost and hope you're enjoying Podverse.

Autobrain, who has one of my favorite usernames, boosted in a couple Spaceballs boosts. Two of them, to be exact. Not that. So the combination is one, two four five yes that's amazing i've got the same combination on my luggage enjoying the show on the way back home across northern new england nicks drink fellas oh enjoy boys nicks stable or unstable for personal use question currently running unstable on my pangolin without any incident in order to get the latest plasma. But is unstable okay?

Or is it better to override particular packages with stable? Any recommendations? I think just at the top, there's no universal okay. There's okay, does it work for you or not? And does it meet the expectations you have around that system and what you're doing? That's a very good point. Thank you, Wes. You know, this is one of the most commonly debated topics in the Nix Plasma community.

I see this question come up all the time. And what I generally see people fall down on is go with a stable system. And then if you want, for want for plasma you can go unstable the reality is as somebody now who's used all this setup for a while i know this sucks because i want the latest plasma 2 but if you just stick with the stable release then when they do ship like say plasma 6 they usually ship like 6002 or something right so you get a few of the fixes that smacked people in the face the first time if you just stick with stable. But if you're not going to do that,

because I wouldn't either, I would suggest just unstable for Plasma, which is, there's probably a lot of ways to do that, but it's a totally doable thing in Nix, is you can kind of just subscribe to just unstable for just these sets of packages. You know, you have all the flexibility of Nix too, right? So you can go unstable and then pin things if they do break or roll them back.

It doesn't have to be too stable. It could be to a previous commit on unstable or, you know, whatever. And you also, depending on how you change your config, you know, it doesn't have to be a huge commitment either, right? So like use unstable for a while. If it becomes a pain, you can just, you know, change where you're pulling your stuff and rebuild and you're back on, on, back on stable. Now I have a little bit of experience with this very simple first question, you know, next stable or unstable for personal use. I'm currently running both as

you do. Uh, so I have a, like only for day job machine, the, this little-link that i got recently and decided to run nix os stable there for i hope is obvious reasons uh and on my framework because as you do i'm running unstable and so i've not selected packages these are like the actual channels so i've dedicated to like running the entire OS as unstable on my framework. So I have this like side by side comparison for this very specific question for the last couple months. And I think it comes down to your personality. for plasma specifically is that it's kind of new and there's a lot changing. And so unstable, in my experience, will have, you know, a lot of bugs be showing up regularly, but also a lot of bugs being fixed regularly. So if you're okay with like, you know, for a day or two, something being a little unstable, as the label says, but then being fixed pretty quickly, then maybe that's okay. And if you got rollbacks, you got a little bit of additional insurance policy there.

Exactly. You can kind of think of it, right? Like the model is similar to what Debian does. The time frames are different. But the idea with the stable, right, is you have a period where you're not going to have breaking changes in there. So if you want that, you have stable. And then unstable, if you're comfortable

running a rolling release elsewhere, that's basically just what you're getting. The one other change that's not really related, there are a lot more updates on unstable, which means more rebuilds because of the way Nix works. And that can use a lot of disk space, so be careful. Disk space and network, so

there's other considerations. Alright, boys. Well, there you go. Thank you, Autobrain. Great to hear from you. Withers comes in with 23,456 sats. You're doing a good job. No message. Just wanted to send some support.

Thank you, Withers. So I'll take hybrid, too. Hybrid comes in with 13,345 sats. I am programmed in multiple techniques. He says, how many of the complainers are members or boosters? He's talking about Nix. He says, and I want to get into this for a moment. He says, how many folks are offering this criticism have skin in the game? Oh! Why did the immutable operating system go to therapy? Because it was feeling a little

unchangeable and needed to work through some of its persistent issues. But in the end, it just had to commit to being itself and roll back its anxiety. Oh, gosh. So I wanted to just talk a little bit about the next drinking game for a moment here in the show. We're going to dial it back a bit because some of the feedback we've gotten in Matrix and via the boost has been been really positive and constructive and one of the suggestions is is we're going to narrow the scope a little bit so it only will be applicable if one of us brings it up in the context of another segment where it doesn't really fit so for example when we were talking about the ubuntu kernel if we had made some sort of nix comment in there that would have been a drinking game moment however

if somebody sends in feedback or we get a boost about it or it's the intended segment for a reason, like we're intentionally talking about Nick's because it's a story or something like that, that would not apply to the drinking game anymore because we're losing hardware right and left to this drinking game.

We can't sustain it. Nearly took out a laptop two weeks ago in the studio after the show and Brent nearly took out a hardware keyboard earlier today. Yep. Danger. Dangerous. So we're going to dial it back a little bit. That sort of hardware replacement budget's really already allocated to go to radio, is the thing.

SatStacker 7 boosts in with 5,000 cents. You supposed! Relatively new listener to your show and first-time booster. Hey! Welcome, thank you. As a longtime Bitcoin and Lightning enthusiast, I knew about Podcasting 2.0 for some time now. However, it took me until right now to actually set it up

and finally switch to Fountain and start supporting your amazing show. Yes! B-O-O-S-T! You guys really deserve it and take your listeners very seriously. Keep up the good work and please keep talking about the topics you feel most enthusiastic about. Thank you, SatStacker7. It's great to hear from you. Welcome aboard.

And thank you for taking the time to get set up. You're getting into Fountain at a really interesting time. Fountain 1.1 just came out and the identity system is going to be Nosturbase going forward. I saw that they got a big banner if you go check the app. Yep. And so I was able to just take my Nostr key that I already had and brought it into Fountain. And now I have one identity across like several apps and now Fountain is one of them. And it's what they're doing is they're parsing.

They've introduced a Nostr improvement proposal and they're parsing for audio files and audio comments. So you'll get a feed of what audio shows people are talking about in Noster. And of course the people you follow, and it's a discovery mechanism because if somebody says, Hey, I just listened to this Linux podcast and they talked about how the kernel issues 60 CVEs a week, that would show up in your feed inside fountain. And if you boost, it shows up now in Noster. So they're, they're trying to essentially offload all the social stuff to something that already exists and that is peer-to-peer nice to have

you aboard thank you sat stacker for taking the time to get it all set up well bhh32 used fountain to boost in 5000 sats with a little bit of a psa it's a boost if anyone wants to use cosmic alpha or you blue cosmic and has hidden networks to connect to you currently can't i have created though a cli tool that wraps nmcli commands for network manager to add any connection including a hidden one or remove one and it's written in rust of course very nice all right yeah he links us you know we will link to things that aren't written just to make it clear. Yeah, I was gonna say it's not a requirement. I am

intensely interested in following the UBlue version of Cosmic. I could really see this coming together as the ultimate, like my notes machine in the garage or honestly, family machines. You know, the things that I've just dealt with before.

I would love to see Universal Blue Solve and Cosmic. If you have a company like System76 that can invest five more years and such developing it routinely and they're building it with something that's as robust as Rust. It seems like it has real potential and something that's really worth keeping an eye on. So keep us posted, BHH, please, because you're kind of like my man on the ground right now with that. So keep us posted, BHH, please, because you're kind of like my man on the ground right now with that. Nav comes in with an interesting, and we don't see it a lot, 8,888 sats. Everything's under control. And they write, one way to self-host something public-facing without revealing your IP address is a Cloudflare tunnel.

All right, full stop. This is a great boost. This directly talk is addressing what we talked about last week, and it's a solid alternative to what we talked about and how to set it up. They continue. Cloudflare provides a daemon that opens a connection through your NAT router from the inside. So from your node, it's kind of like tail scale. But instead of bridging a connection to other devices you own, it bridges a specific address and port from your land to the internet through Cloudflare. You do need a domain and you must use Cloudflare's name servers. Yeah, we've actually considered tunnels in the past for things like this. In fact,

we almost use that as the way to do this. And I do use Cloudflare tunnels for my BlueBubble server. And I do use Cloudflare tunnels for my BlueBubble server. So I'm a fan-ish of them. But ultimately, we kind of wanted to go a more traditional route, I guess. Yeah, it was a simple option. We had the tools available and gave us a little more flexibility. But yeah, there are that or Tailscale Funnel might be an option for some of these use cases. So there are definitely, depending on what your parameters are,

it is nice if you can take advantage. Cloudflare obviously has a great network infrastructure, so if they can handle routing the packets in a way you like, then that could be nice. And there is some... It's like legitimacy to be on the Cloudflare IP block,

so you're not going to get blocked from Comcast like some folks were when they tried to connect to our node. And just as long as you architect it this way, you can make it so it's not going to get blocked from Comcast like some folks were when they tried to connect to our node. And just as long as you architect it this way, you can make it so it's not that big of a pain to change things out. You might have to change DNS or whatever, but

when you think about proprietary services or providers, as long as you can go migrate this from Cloudflare Tunnels to something else later if something goes wrong. Yeah, and if you own the domain and whatnot. Yeah, no big deal. Use the tools that make sense. Now Alex Gates also boosted in on the topic using Breeze, 10,000 sets. Put some macaroni and cheese on there, too.

I just did something similar a month ago using LKE. Traffic in Linode forwarding to Tailscale external services on LAN. Oh, I think LKE must be Linode's Kubernetes. Aha! Neat. Well, thank you, Alex. Nice to hear from you. Of course, Alex is the podcasting

2.0 consultant and knows his stuff. And it's interesting to see a lot of us converging on solutions that are very similar. That's a theme I'm noticing across the traditional feedback, the matrix feedback,

and the boost, is a lot of us are doing this VPS or Cloudflare tunnel, like external solution routing to a LAN node. And I wonder if this isn't a lot more common than we think, boys, just based on the feedback we're getting here. Yeah, VPSs are cheap and available. And five bucks a month, why not have someone else handle that part of it? Yeah. And you know what? Your ISP doesn't need to worry about it. Plus, I mean, Linux is so flexible, right? Like, we can just do these things, especially with tools like Tailscale these days.

Like, it's wonderful. Monty33 comes in with 4,444 sats, which is a double row of ducks. Ah, fuck! And he's responding to a discussion we had with our members asking for feedback on how we cover Nix. He says, I totally understand your self-reflection, but I personally listen to your shows for the latest news and greatest innovations. And if that involves Knicks, then I'm asking for more coverage on it, not less. If you're willing to compromise on your goals and desires for your show based on listener

feedback, then I encourage you to get a more complete window into the thoughts of the community. Woof. That's a good point. Monty, what I really like about this is this is some really good, clear feedback. Just boom, right to the root of the community. Woof! That's a good point. Monty, what I really like about this is this is some really good, clear feedback. Just boom, right to the root of the issue. He continues with that second boost. I think the annual member booster matrix survey

would give you a more accurate window into how your most engaged audience feels about the show rather than relying on incidental feedback. The choice on whether or not to read this boost out in the air, it's your choice. Maybe you should just read it on the members feed. No, I'm going to read it right now because we wanted to talk about this today. Thank you, Monty, for bringing this up.

So I would be really interested to know if people actually want to participate in a survey. It would be a survey if we did this. This is like literally I'm just thinking of this on the spot. But it would be something that we'd only use for ourselves. It's not like something we would share with some company or advertisers or something. But would people be into that? And what kind of survey? I mean, the more I think about it, the more I kind of like the idea is like every year we just do a little survey.

I could see that being really kind of valuable. We're already doing the boosties. This is just sort of a meta version of that. They really need to tie the two together, actually. Oh, true. Like I could be a killer episode. I,

I will give a lot more thought to that. Um, and I would appreciate anybody's thoughts that are listening. Um, if you'd be willing to participate in a survey, what we should cover, all that kind of stuff. That's honestly something I hadn't really thought about because I just

assumed people would not want to do that. I just, I just also want to say thank you to everyone sending in feedback that's such deep reflection on what we're doing here. I feel like it's such an honor to get that stuff. So thanks. Our pal Gene Bean boosts in with a row of notes. Responding to last episode, you were asking about where folks were hosting these kinds of things. Yeah. Gene Bean moved from Linode to Hetzner and had been super happy, especially with price.

I'd like more reports about Hetzner because that does seem like a pretty well-priced system. I can see doing this kind of VPS that redirects traffic hosted on Hetzetzner instead of Linode, potentially, or maybe DO or something. Yeah, you know, back in the day, they had, like, dedicated servers, but it was in Europe, and now they've really built out their cloud offering, which has U.S. data centers, which at least would matter for us.

Yeah, I haven't tried the cloud offering, but I've heard good things. So there's a... And you've got to trust GeneBean. Yeah, that's true. That's true. I will roll up some feedback that GeneBean was giving us in the Matrix here,

too, really quick. Just plus one to the narrowed scope of the next drinking game. And Gene Bean also wanted to point out that the drinking game does kind of reduce the family-friendly aspect of the show a bit. And the reason he's talking about that is because he loves sharing the show with his five-year-old. Oh, jeez. I hadn't thought about that. And Gene Bean would do a survey. Hmm.

Do you have a thing you hate to drink, Wes, that you'd be actually willing to drink on mic? That doesn't make you puke? Hate to drink. I don't know. I think this is the way to go. Is instead of booze. Well, it would be a fun challenge to just come up with things that you hate to drink. That's true.

We have to individualize it, though. We could change it up. Because I was going to say, like, I'm not a big tea guy. Yeah. I don't hate it, though. What about, like, if we had to do shots of V8? Like, that's kind of weird. Oh, yeah.

Not a big fan. Okay. I'm open to taking listener it, though. What about if we had to do shots of V8? That's kind of weird. Not a big fan. I'm open to taking listener suggestions, too. Yes! Boost in your suggestion or leave it in the Matrix chat. All right. This feels good. This feels right. Thank you, Gene Bean.

Appreciate that. That was a good boost in multiple ways. Now, Spectre has boosted in two sets of boosts, and they are rows of ducks. Things are looking up for old McDuck. Hey, I'm a first-time booster. Did I figure this lightning thing out correctly? Congratulations.

Well done. Now, just catching up. As for listening speed, 1.5 times for me with max silence trim in pocket casts. Can Fountain add that feature? I'd swap pretty quickly. I bet they will. I bet they will, yeah. In time.

Although 1.5 is maniac level, right? I feel like 1.2 is reasonable. 1.3, you're getting a little silly. 1.4, you're getting stressed out. And 1.5, you might be a psychopath. They say they need to compress their listening due to dog walks and young twins. See, this is actually why we talk about Nix so much. It's just because we know you're listening to 1.5x and we're hoping if we say it a couple

times, like you won't actually miss the reference. You know, I totally understand the new babies and the dog walk thing because that used to be when I got a lot of my podcast listening in. So I retract the psychopaths and new dads. They're also reporting their first Linux was probably Mandrake. Their grandfather retired and closed their rural dealership and gifted them an old, long, rectangular IBM machine. What a description. Yeah, what kind of systems were they working on with that dealership?

Old, long, rectangular ones. Probably like the old PS2 systems. Yeah, they were like pizza boxes, but taller. Also, side note, for a long time, I don't know, maybe never, the PS2 line didn't use ISA slots, which were the predecessor to PSI. MCA. Right. Yeah, you good? Yep, yep.

MCA was horrible. Yes. I was so happy they moved to PCI. Yeah. Yeah, so I'd love to know more details if you want to boost back in Spectris because I love those old systems. Yeah, and also, let's be honest with ourselves, URPMI that Mandrake shipped with was pretty great. Now, Spectris continues with their second boost here, just saying, from that old IBM machine, much experimentation later, using a Debian, Ubuntu, and a bunch of flavors of the month, CDs in the mail.

And it turns out now I have a career building things on Linux, from robots to networking appliances and everything in between. Oh, we're going to need more details on that. That old... We're going to need more details. You're just going to drop that? That old mini mainframe I cannot remember a thing about and free software sparked a 14-year career and growing. With the show, please keep up the good work. I always find something new to apply at home or in the office.

Also, take a well-deserved vacation. Thank you, Spectrus. It's great to hear from you, and thank you for taking the time to set up all of the plumbing there. I appreciate that boost. That's really great. Lucas Burlingham comes in with 10,000 sats, and boys, I don't know if you know this. It's over 9,000! You and your math! I know. They write, I wanted to send a few sats alongside this tool I found on Reddit.

Maybe it's a future distro challenge material? The comments suggest a file system nuke option that wipes and forcibly installs the distro on this the creator says this this is impossible uh and i have feeling someone could figure it out and he links us to distro hopper wheel and this is so good so wait what i'm understanding here is this has a spinner and it just randomly installs whatever distro it lands on the distro hopper wheel of fortune now let fate decide which linux distribution you'll use next and what he's suggesting suggesting geez the next challenge is getting to me is that uh we should spin this thing and then we should do a challenge based on that so i'm gonna spin it and we'll just

see what we got from it. You ready? Okay, it's still spinning. And it oh gosh, NixOS it landed on Alpine. It almost stopped on NixOS hilariously and it landed on Alpine. It says, you've won Alpine Linux. Alpine. It almost stopped on NixOS hilariously, and it landed on Alpine. It says, you've won Alpine Linux.

Alpine Linux is a security-oriented, lightweight Linux distribution. This one is really cool for containers and stuff. That's actually what it says. Here could be a little pixel art of each distro, but I haven't found a nice collection of a Linux distro of pixel equals square bracket, so enjoy the little tux instead. Okay. but I haven't found a nice collection of a Linux distro of pixel equals square bracket, so enjoy the little tux instead. Okay.

How the hell would we do an Alpine challenge? How would that even work? You can Alpine desktop. I've never done it. Okay. Remember, we do it, so you don't have to. I'm into it. I like this. Do you want to try it?

Yeah. Should we do an Alpine challenge next time? Why not? I mean, right? Everyone's running it in containers. Brent, can you do it? Look at your desktop. On the go? He's on the road. Well, I mean, you and I are going to be running it in containers. Brent, can you do it? On the go?

He's on the road. Well, I mean, you and I are going to be in the same city. Oh, I'm going to be on the road. Yeah. Hmm. You have time. Tonight you get your, you flash your VN or something. Yeah.

We will link to the DistroHopper wheel. This could be a fun tool. That's so great. Thank you, Lucas. That's fun. All right. You know, and I'll be honest with you, we never were going to pick Alpine. I was never going to try to do Alpine. I mean, maybe

if you guys would have suggested it, but I was never going to do it. Well, when you're skiing, it's a nice option. The Muso boosts in with 9,216 saps across two boosts. Make it so. The first boost here, I'm curious about one thing with your new

deployment of services via Tailscale. I assume that your VPS has an IPv6 address. How are you handling traffic from IPv6 clients? Are you tunneling back to IPv4 or is IPv6 used all the way through to the server? Are you sure you didn't send this boost? I did not. Okay. All right. Now, did I pay someone? I don't know. That's a separate question. Yeah. There was some grousing that we couldn't do this entire thing with IPv6.

That is true. You're right. But no, so far we've mostly ignored it, which I don't feel great about. But it's working all right. Yeah, unfortunately. I don't. Does Tailscale do IPv6? I assume we'd have to sort of.

Yeah. OK. Great. Yeah, then probably if it worked, we'd probably just route it all the way through. Yeah. Yeah. He also asked if we have any progress on annual memberships. I think maybe.

But that progress is like I have to do a completely separate plan. I'm going to look into that more soon, probably when I get back, and I'll announce further details as I can. Thank you for asking. Great question there, Muso. Yeah. On both, really. Thank you, Mus. Yeah, great question there, Muso. On both, really. Thank you, Muso. Nice to hear from you.

The dude boasted in with a little setup info, not to be confused with dude abides. 5,000 cents. This dude does not abide. You're supposed! This unabiding dude says

VPS with tail scale hosting NGINX pointed to various LAN hosts. Yes! I agree. Plus one that. Nord comes in with a Spaceballs boost, 12,345 sats.

Yes. That's amazing. I've got the same combination on my luggage. The hell was that? Spaceball one. They've gone to plan. Says, I love the Nick's content, or all the content, really.

Talk about what excites you. It creates the best show when the content comes from a happy place rather than an effort to please the audience. That is some insightful ass Nord. Nord's calling us. We can't fake it. I wonder if Nord's created content before

because that's the kind of thing you learn after a lot of years of painful content creation. You know, we got this boost in earlier this week and I've been thinking about it all week. So thank you. It hits six boosts in with $12,345. Smoke if you got them. Sending in some NixOS love as I finally manage to convince my colleagues to migrate some hosts to it.

Oh, look at you doing the good work over there. Yeah, look at you over there, yeah. Thanks for convincing me to try it out. Oh, and we're using the NixOS image generator. What? And then links to the NixOS wiki Proxmox virtual environment page. What? Yeah.

Okay, so they're using the NixOS image generator to generate ready-to-use, fully configured images instead of installing HVM manually. Wes, Wes, Wes, we should do this. I like it. Yeah, this is great to know. So the NixOS generators project was under the Nix community over on GitHub. Yeah, so they have one config. You can have all kinds of formats in there for various cloud providers.

And yes, looks like they've got something specific to Proxmox, which is going to be handy for us. Thank you, init6. I always run at init level 6. I can't believe this. We got another boost in here from JSC for 8,888 sets. The traders love the vol. It simply says, carry on, carry on. Well, thank you.

We will. Bitcryptic also boosted in 4,444 sats in total. That's two rows of ducks. Thanks for the pointer to Coder Radio. I've added that one to my favorite streams and will boost there with details about my referral code exchange. Love what you guys are doing and love the value for value model. Thank you.

In the spirit of listener participation, I've decided to take a drink with you at each mention of Nick's. It only seems fair as a show of support, right? As one should also a plus one Fountain with Noster and plus one for listening at 1.5 times speed. Oh jeez.

Wow. I just think every time we play a jingle or every time we play a transition you know stinger it's just like

oh man that must sound like crap but I'm glad you're listening I'd rather you listen than not so. Wow. All right. Well thank you very much

for that boost BitCryptic it's great to hear from you. DistroStute comes in with 12,345 sats. So the culmination is 1, 2, 3, 4, 5. I use KeePass for two-factor authentication. It's no nonsense, no services, and works on all my devices. If that still feels like one factor, then I use a separate KeePass database to put your passwords in. You know, I've been continuing to think about this. And I do kind of feel like for my password manager, you know, maybe for like a Bitcoin wallet, anything that I want to have for 10, 15 years, I really don't want to have to log into a service. I want to be able to use it even if the service goes away. You know, five years later, I can open it up. Yeah, there's something about archiving this stuff too that i've been thinking about like let's say you have a password manager uh you know in your backup database that's from well i've been using

keypass for like over 10 years uh you never know but it might be a good thing to reach back to you just can't predict that yeah so I don't know if that means, I mean, because I'm just thinking about my experience with Authy and how I knew, even when I first set up Authy, I knew this day would come where I'd have this problem. With our final boost this time around, Sam H. came in with 5,000 sats. Since the time you started talking about NixOS,

I began running it on my desktop, on laptop, and then three home servers. So I certainly like hearing about it. I've been a user of Ansible, mostly at work, for more than 10 years, even prior to the addition of roles to Ansible. So I have long tried to build declarative system configurations. And for that purposes, I find Nix just works better. Yeah. I love that journey, right? Like I tried it and now I see all these places where I can use it. And it's funny how many times we hear that. And I love keep sending it in. I want to

keep hearing it. Thank you, everybody who participated in the value for value process for this week's episode. We also had 37 folks streaming. And when you bring it all together with the folks that boosted and the folks that just streamed stats as they listened, we had 66 unique people that were either sending or boosting. Now, this show is heard by tens and tens and tens and tens of thousands and tens of thousands and tens of thousands of people. 66 people participated in this process. It really makes the difference, and it keeps the show rolling. We brought in a grand total of 325,219 sets. Thank you, everybody. If you'd like to boost in, go get Fountain or Podverse or Castomatic or one of the podcast apps over at podcastapps.com. You can load it up with Strike, which works in over 100 countries,

including the EU and the UK now, and get going really quick. And Strike's a great company, too. And send your boost in, either from the web or from one of these new apps, which gives you all kinds of features as well. Or put your support on autopilot at linuxunplugged.com. Thank you, everybody, who supports the show and keeps us going. It's really been incredible. Over the last year, when the show probably would have died thanks to the adpocalypse, our audience has stepped up, kept us going, and made our audience one of our largest customers, which I'm extremely proud of. Thank you, everybody.

Now, let's talk about distributed file systems for a moment, because it's something I dream about. I see a LAN that uses the spare disk space of so many different systems pooled in some kind of redundant storage means. You know, you got a box over there with maybe 500 gigs free and a box over there with a terabyte free and a box over there with two terabytes. And you add it all up and all of a sudden you got yourself like, I don't know, 40 terabytes of free space. If you think about all the machines you have

and the space that remains unutilized, wouldn't it be fantastic if you could say, slice off 25% of every spare disk space on every machine and pull it into one set of redundant storage? It's a dream of mine. And I was looking at SanaFS this week. It's a distributed file system. It's POSIX compliant. And they're trying to kind of do the Google file system thing in a way, and I'll link to that in the show notes, the way kind of the Google Drive stuff is built.

It's designed to run on commodity hardware. It's high performance. It's scalable. And they say they're aiming for reliability. And they have several, it's like three primary components. They try to have a resilient architecture. They say continuous assured data integrity with CRC verification. Yeah, right.

I mean, like distributed systems are hard on their own. And then file systems, also hard on their own. Yes. And then you got to put the two of those things together. Yeah. I mean, it's early days with SonarFS, but I think it should be on people's radar. It seems like a compelling project. They have pre-built binaries for the Ubuntu releases, more recent ones.

Or you can build it yourself. I took a quick look at this and at the time of recording their repo was actually offline. So that didn't work so well, but it does seem like maybe this could be a contender for this. I want to take all of my spare drive space, at least across a bunch of my systems and pool it as one redundant set of storage. I like that idea a lot. I don't know if SonFS is it. I'd love to hear some suggestions that people are using if you're using something in production.

I know you took a quick look at it. Yeah, I did get it to build. At the time, it was kind of a pain. I had to do that process of build, wait for it to complain about some library that I didn't know I needed yet, figure out which package corresponded to that library, install that, or make it available in the build environment or whatever, and then repeat. But I got it going. But yeah, so it seems like, I don't know if it's entirely

new. I think it's based on some other open source projects, and there's obviously some ideas that have been around for a while. But I think it is early days in the sense of a mature all of your questions is an admin answer I've been around for a while. Yeah, yeah. But I think there are, it is early days in the sense of like a mature, all of your questions is an admin answer sort of level. I did just notice, I didn't see this before, it does seem to have a Nix package. That might be one option if their repos are still down. All right.

Neat also, I think it's been moving pretty fast. There's a Windows client now, which for some deployments might be critical. Yeah, honestly. And an NFS client. That's pretty neat. I know there's other options out there. I'd love to hear from people out there that are trying different things in production.

Especially as we go to more self-hosting, you could see like, what if we had pooled storage across a bunch of systems here at the studio, and then you put something like Minnow in front of it, or some other S3 compatible like SeaweedFS, something in front of that, and then we use that as some sort of object storage. I'm just saying, Wes, there's something to it, and I just have all these machines sitting around with big old disks doing nothing. You just love a cluster. I do. It really does come back to that. You are 100% right. All right, well.

See you next week. Same bad time. Same bad state. I'll be back in the States, hopefully, unless they stop me at the border for next week's episode and maybe have some stories on how it all went. We'd love to hear from you. You can always join us live. We do this show on Sundays at noon Pacific, 3 p.m. Eastern. And we want to hear from you. Don't forget, we're asking you out there. What could they do to bring you back to Ubuntu Or what have they done if you've already moved back? We'd love to know. And honestly, if you've got any suggestions for distributed file systems to take advantage of the disks on your LAN, please send those in as well.

I'd love to kind of get that rolling. Now, as far as links and all the details for what we talked about today, you'll find that on our website at linuxunplugged.com. A bunch of great shows over at jupyterbroadcasting.com. See you next Sunday. Thank you. okay before we wrap up i know we kind of run along. Let's just take this offline here. The cone of silence. We got a little bit of show feedback. We'll have a little private discussion here. Dragon, you have an update on stickers. You want to give that to us?

Yeah. So I was finishing them up today and then you said you were leaving today. And I go, dang it, man. No, I'm leaving technically Tuesday at 3 a.m but as far as when the show gets published it'll be basically right then we may have to get together to see if i can send them to wherever you're you know the thing okay um but other than that i mean because they're sitting right in front of me i've just i was like well i heard how many people were going i was like i need to make some more and so all. All right. I'll follow up with you on Matrix.

I'm sure we can get some kind of address or a place to send packages that you pick it up at. All right. Well, I got the cone down. Gene Bean has a little bit of spicy feedback for us, I think. What is it, Gene? Me and a few others have not only suggested, but thrown our sats behind our opinions. And it seems like we need a little bit of love for Gen 2 before y'all go off digging into other stuff. I mean, to be fair, I am honestly surprised you guys haven't done a proper Gen 2 challenge in actually ever.

It's a coordination thing. We got a powwow. We got a plan. Of course, we're always like Brent's traveling right now. We want to do it right. Well, you know, step one is making that JB chat room that we talked about, I don't know, a month ago so that we can maybe coordinate some of this stuff. You're right. That's a fair call out, Gene Bean. You know, we get excited about these things because you can do an Alpine thing in a few days, you know, but the Gen 2 challenge, you want to sit with it like a good meal. You know, you want to really take your time and enjoy the appetizer, you know, like that's so we just want to treat Gen 2 right. That's the problem.

and enjoy the appetizer. You know, like, that's... So we just want to treat Gen 2 right. That's the problem. So what you're really saying is Alpine's throwaway, but Gen 2 is something that you're going to savor and learn from. Yeah, right. Yeah, Alpine's a fling, and Gen 2's something we're willing to bring home to the family.

This is a disturbing comparison on so many levels, but sure.