LINUX Unplugged - 556: The xz Backdoor Exposed 🚨

Episode Date: April 1, 2024

We're breaking down the attack: how it works, how it was hidden, and why time was running out for the attacker.Sponsored By:Tailscale: Tailscale is a programmable networking software that is private and secure by default - get it free on up to 100 devices!Kolide: Kolide is a device trust solution for companies with Okta, and they ensure that if a device isn't trusted and secure, it can't log into your cloud apps.Support LINUX UnpluggedLinks:💥 Gets Sats Quick and Easy with Strike📻 LINUX Unplugged on Fountain.FMoss-security mailing list — Backdoor in upstream xz/liblzma leading to ssh server compromise.Fedora AnnouncementDebian AnnouncementUbuntu AnnouncementKali Linux AnnouncementArch Linux AnnouncementGentoo AnnouncementopenSUSE Tumbleweeed AnnouncementNixOS Unstable DiscussionWhy does it take two weeks for NixOS to replace xz?Andres Freund on Mastodon — I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc....rwmj on Hacker News — Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of its "great new features"A Microcosm of the interactions in Open Source projects — Make no mistake. This is the way it works. It needs to change.Devuan GNU/Linux on X — Devuan is not affected by the latest vulnerability caused by systemd.systemd PR: Dynamically load compression librariesMatteo Croce on X — I'm the author of such PR. While I absolutely didn't know that libxz had a backdoor, I really think that libraries should be loaded on-demand when rarely used, hence my change :)Ryan C. Gordon on X — This is probably how the xz thing happened, right?Jan Wildeboer on the Fediverse — Again the FOSS world has proven to be vigilant and proactive in finding bugs and backdoors, IMHO.Unplugged Core MembershipTXLF is coming up! — April 12 - 13 in Austin, Texas.LFNW coming up! — April 26 - 28Mobile Game Ads Are Boosting Podcast Follower Counts — Wondery, iHeart and Lemonada Media are all using a non-public product from MowPod - which gives extra lives and game credits to gamers if they follow shows on Apple Podcasts from game apps.MowPod's podcast promotion tools: tales from the barfortydeux's NixOS ConfigsPrism Launcher — An Open Source Minecraft launcher with the ability to manage multiple instances, accounts and mods.World Backup Day — March 31st — One small accident or failure could destroy all the important stuff you care about.Updating Our Fiddly Bits | LINUX Unplugged 494

I think there's a silver lining to this whole XE vulnerability. It shows us that there is a universal constant. All software does indeed suck. That remains in effect. And sometimes we even benefit from it. OK, but how does that help us? That seems not correct. Well, I think about it.

Think about it. If this backdoor hadn't been buggy and crappy, it never would have been found. Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris. My name is Wes. And my name is Brent. Hello, gentlemen. Well, yes, coming up on the show today, we are diving deep into the XE backdoor, how it worked at a technical level, how it was found, and how the attackers abused the open

source trust model. And then a little detail we discovered this morning that suggests the attackers had a limited window of opportunity. We'll get into the details around that, then we'll round it out with some great boosts, some picks, and a lot more. So let me say, time-appropriate greetings to our virtual lug. Hello, Mumble Room.

Hola. Hello, Chris. Hey, Wilson. Hello. Hello. Thank you for joining us in there. Mumble is always going along with us. They get a low latency stream. We got the details on our website at jupyterbroadcasting.com. Also, check out tailscale.com slash linux unplugged.

That's where you go to get 100 devices. Try Tailscale for up to 100 devices. Not a limited time trial. You can do 100 devices for free when you go to

tailscale.com slash linux unplugged. Tailscale is the easiest way to connect your devices, services, applications directly to each other wherever they are,

whatever they are on. Secure remote access to your production systems, to your mobile devices, your V are, whatever they are on, secure remote access to your production systems, to your mobile devices, your VPS, your VMware's. I don't know what it is. Yeah. Maybe for some reason you're feeling like you don't want to have a public SSH server running. I don't know why, but.

Yeah. For some reason, you just don't want your box exposed to the internet. Have a nice private flat mesh network with zero config. Works great in the enterprise too. Tailscale.com slash Linux unplugged. We are going to get into this. I wanted to ask a question before we get started because we may be double recording next weekend. And so we wanted to get this out there early. We're asking what are the first five things you set up after you install a Linux desktop?

So please boost in your top five apps or things that you set up on a new install. We'll be recording next week, so we'd like them to come in as quick as possible. But it's something we've been kicking around internally. We thought, all right, let's make an episode about this, potentially. If we get some good ones, I think we'll make an episode. As usual, I'm sure the audience has way neater things than we do. Probably. Probably more interesting.

On Friday, March 29th, 2024, we discovered that the XZ project had been backdoored, that the actual upstream project itself had been compromised. m4 file that was only contained in the specific release tarballs that you would download it wasn't in the repo itself so you couldn't really easily see that you'd have to go looking for it in this like pre-packaged release tarball which actually makes it kind of tricky for just a researcher to drive by discover yeah you know it's not just there in the like gooey sort of repo viewer you had to go go take a look and see and then be familiar enough with what you know to expect in that file plus there there's often differences in the release files for like c projects that you don't the people building it don't have to do all the like automake and configure stuff so that might not even be necessarily super suspicious on its own but okay so you get this you get this modified

build to host not m4 file only in specific tarballs. That gets executed, and then malicious code gets injected into the configure script, which then manipulates linker and compiler flags within the make file. And then the make file finally runs as part of the build process, and that causes a symbol called rsa underscore public underscore decrypt to point to malicious code. Then later on in the process, SSH ends up pulling in xz through a process we'll get into and then has an overridden symbol. So it tries to call its normal rsa public decrypt function. And instead, it gets this modified malicious one coming from lib lcma.

And I guess in there, they could include code, which could execute something as whatever user is running SSH. So it's not just that they get onto your box, but it's also it's a remote code execution vulnerability. Yeah. It's kind of the nightmare scenario for folks with open SSH servers. Yeah. All right. So this is just kind of coming out this weekend. We've been digging through it all weekend long.

It does seem like only a few systems were impacted. There is, of course, always with these types of things, you just want to be as careful as possible and just update your system regardless, even if your OS wasn't one of the impacted ones. It does seem, though, because the way this is packaged, though, Wes, it wouldn't be like if you got it from the repo. It would be more likely if it was being distributed as a deb or an RPM where some of these scripts would actually get triggered.

Yeah, or if you're just going to, like, if you go on GitHub and you click on releases, there are auto-generated links that say source code, and those, like GitHub automatically, you know, it takes the tag corresponding to that release and packages it up into an archive for you. But then there's the other tar files, which are specifically built, you know, as part of in some repos it's an action

or by hand by the maintainer. And so as part of the release process, these custom crafted tar balls are then stuck up. So that's how you'd probably get it. Yeah. What's interesting is that this thing also goes through a series of system checks. So it makes sure that it's interesting is that this thing also goes through a series of system checks. So it makes sure that it's on a glibc box. It makes sure that it's an x86 system before it'll

run. It checks for several pre-existing conditions. It even makes sure that you're not in certain types of debug modes where you might actually discover this thing. And it checks for those before it'll actually execute the script. So the author or authors of this thing went to a lot of trouble to make it look like regular old files that should be included. And it has to go through a series of events before it actually does get injected. And even at that point,

it then does a sanity check of its environment. Yeah, right. I think some of this stuff was contained in what looked like just binary test files. You know, supposed to be files that the test suite would run with the XE utility. And of course, when you got binary files, well, that's a great place to hide things. And this test suite, you know, was one of the big new developments that they were so

proud of that the attacker had been working on. But let's get down to brass tacks for listeners. Likely their distro isn't impacted, but we do know a few distros that have been impacted. Yeah. One sort of nice thing about this is it was caught early on, relatively early on in the process. These new releases, 5.6.0 and 5.6.1, they just happened in the last month or so.

So for your distro to pull it in, it's got to be a fairly up-to-date, maybe rolling release or development version. So we've seen things like the development versions of Fedora being impacted. Obviously, you know... Debian. Yeah, the latest Debian. Ubuntu did. It's been pulled, but Ubuntu had some in the upcoming code

for the upcoming release. Kali Linux as well. Kali Linux? Yeah. Arch, technically, but technically not. I guess Arch doesn't patch OpenSSH to support the systemd notification stuff. So there's a few different levels to actually be vulnerable in this case because of all of the checks and some specifics about how SSH needs to be configured on your system. So there's one level of, did the distro pull in the code that had these malicious modifications?

And then there's the secondary level of, did that code change actually end up doing anything? Yeah. So the actual vulnerability is SSH relying on systemd to pull in this compression library, which is doing the actual lifting and where the vulnerability exists. Right. So yeah, in all the malicious stuff, they end up modifying this symbol. So when you have the XC library code loaded in memory,

there's this RSA decrypt function that's sitting there. But on its own, SSH doesn't need XC. It doesn't use XC. It's not pulling that in. So how does this work? Well, some distributions patch SSH with a patch that enables it to work with the systemd notify protocol so that you can

easily have stuff say like hey systemd i want you to start these things or restart these things or do something after open ssh has finished loading and you you know you have a little tiny patch that kind of shims into ssh and once it's all done getting set up and it's in ready mode uh it can go sort of just you know sets up a socket talks to systemd and says like, hey, I'm done. Now, you don't have to do it this way. And it looks like these days, the project advises folks, it's like a simple protocol. You don't need to pull in the systemd library to use it. You can just kind of roll it yourself with very little code. But that wasn't always necessarily the advice you'd find just looking at it. Some projects need to pull in lib systemd, you know,

like the systemd libraries. So a lot of these patches that were implementing this behavior just pulled in systemd via the library, and that pulls in XE. Yeah. It's funny because it, so it's not exactly a systemd problem.

It's not exactly an SSH problem, but because the combo together are pulling in the XE library, it's everybody's problem. And then when SSH goes to actually try to call this internal function to itself to decrypt RSA, it gets the version that the malicious XE supplied by way of the system B library. Yeah. And so this happens. It seems the attacker got this into the XE library at about early March.

So some systems like a small point range release of Alpine was also impacted. NixOS Unstable did pull in the vulnerable version of the library, but it doesn't build it in a way that actually triggered that script, right? Yeah. So as you were kind of alluding to, parts of the checks actually check to see if it's in a dev or RPM build system. Yeah.

And of course, the NixOS build system. Not DEV or RPM. No, definitely not. So not only did NixOS not apply these system denotifying patches in the first place, so there's no linkage there. But yeah, then also the build system is different enough not to trigger that kind of code. So let's talk about the mad lad who discovered this, Brent, because it's a pretty interesting story. Yeah, Andreas Frond seems to have singly discovered this, which seems pretty amazing to me. He says, though, I'm not a security researcher nor a reverse engineer. Now, you might believe that

until he starts describing how he went about finding this. I know he says he's not a security researcher. Andreas, I got news for you. You are now, buddy. Yeah, a pretty famous one at that. He says, I was doing some micro benchmarking at the time and needed to quiesce the system to reduce noise. Saw SSHD processors were using a surprising amount of CPU despite immediately failing because of wrong usernames and such. So I profiled SSHD, showing lots of CPU time in libLZMA, and with perf was unable to attribute it to a symbol. I got immediately suspicious. I recalled that I had seen an odd Valgrind complaint in automated testing of Postgres previously, about a few weeks later after some package updates. And he adds,

here's one more aspect that I think emphasizes the number of coincidences that I had to come together for me to find this. I ran a number of BuildFarm instances for automatic testing of Postgres, and among them was Valgrind. For some other test instances, I had used fnoemitframepointer. For some reason, I don't even remember anymore. A year or so ago, I moved all the test instances to a common base configuration instead of duplicate configurations. And I chose to make all of them use fnoemitframepointer. As far as I can tell, Valgrind would not have complained about the payload without FnoEmit frame pointer. It was because GetCPUID expected the stack frame to look a certain way. Additionally, I chose to use Debian Unstable to find possible portability problems early on. Without that, Valgrin would

have had nothing to complain about. Without having seen the odd complaints in Valgrind, I don't think I would have looked deep enough when seeing the high CPU in SSHD below get CPU ID. He just happened to be running tests, trying to get Postgres working right or whatever it was. He just happened to have his environment set up just right. And he just happened to be using Debian unstable. Although I will say, you know, major props to then even, you know, all all those things happen not everyone would continue to dig so much and do such a nice write up and like really you know try to handle this well and cooperate with everybody and do responsible disclosure like i don't think andreas is giving himself and i hope i'm saying his name right i don't think he's giving himself enough credit for like how clutch as my kids would say he came in here and then the information he shared he's even answered questions to help clarify stuff.

As if I needed one more reason to use Postgres. I mean, it's incredible devs like these that are making it. Collide.com slash unplugged. You've probably heard me talk about Collide before because I think it's such a great tool. It prevents things connecting and authenticating to your network before they've passed your checks. But have you heard that Collide was just recently acquired by 1Password? Now, that's big news. These two companies have really been focused at creating security solutions that put end users first.

That's fantastic because it reduces friction between end users and IT. And for over a year, Collide Device Trust has helped companies with octa ensure that only known secure devices can access their data they're still doing that but now they're doing it as one password you can imagine when events like today happen the xd vulnerability it's really nice to know that you have collide because collide can work on devices that even don't have mdms like your linux fleet contractor devices and every byod phone or laptop that ends up in your company. And Collide comes with a library of pre-built device posture checks, but you can also write your own custom checks for just about anything you can think of when you need it.

That's pretty great. Now that Collide is part of 1Password, things are only getting better. So go check it out and support the show. Go to collide.com slash unplug. That's K-o-l-i-d-e.com slash unplugged to learn more you can watch their demo it's a great way to support the show too better than ever check it out at collide.com slash unplugged all right well that's the high level of sort of the technical side of this vulnerability. But unfortunately, there's also the human side.

I mean, OK, we've said XE was back to work, but how? How did that happen? It sort of was a tale that happened over more than the last two years. This is maybe the most troubling part of this story, more so than a remote code execution flaw, is the way this attacker apparently with others or this attacker working under multiple accounts leveraged the burnout that the XE developer was experiencing to sort of push them out. One individual submitted a pretty complicated patch, and then another individual came along and gave the developer a hard time for not getting it, I don't know, I guess accepted. Yeah, I mean, it's quite intense.

It's, you know, shame on you. You're making this repository wither. You should find a replacement. Your users deserve better. And then Geotan just seems to be in the right place at the right time. Hey, I'm building this cool testing infrastructure. Look how great this is. I have the time. I could take this on.

Yeah, right. Suddenly, here's this helpful new contributor that maybe you start thinking, okay, they could handle some more things. And they've been around for a little while. They've been working. In fact, the core maintainer at the time almost referred to them as a co-maintainer already before it was official. It is scary. it is really scary because it suggests a persistent prolonged plan where they identified a piece low in the stack that they could go after in a sophisticated manner

then they identified the core contributor to that piece in the stack. And then they bullied that person to essentially convince them they were burned out, then got access to the project. And then after that, reached out to other projects like Fedora and tried to get them to incorporate it as fast as possible. There's a great, very honest comment on Hacker News by a Fedora contributor who says, it's very annoying. The apparent author of the backdoor was communicating with me for several weeks. They were urging me to get this in as fast as possible because it had, quote, great new

features. And they write, I even worked to help them fix the Valgrind issue, which turns out was caused by the backdoor it added. Gosh, that sucks. The open source community is trying to be helpful. Yeah, right. Well, and generally, I mean, you know, we often do want updates.

Many times there's security fixes in updates and, you know, we try to keep things up to date. I feel like this attacks the very trust model in open source. I mean, it's the idea that, I mean, because the quiet part that I'm about to say out loud now is it seems very possible if this is a state actor. If that's a state actor, that implies it's an intelligence agency. How the hell do we defend against that? And we know how significant of a problem developer burnout is. And if state actors or even just average everyday attackers have realized that developer burnout is a soft target, this feels like a pretty significant problem. And it feels like a problem that could be more persistent than we realize. I mean, there's a lot of, you know, FOSS software we rely on at home in production. And this time it wasn't a lack of maintenance of OpenSSL or SSH. It was a much more nefarious and clever little attack,

and that's even harder to guard against. They were patient. They introduced apparently themselves around March 2022. So this is March 2024. We're actually seeing the fruits of that labor. In January of 2023, they start announcing the XE releases. In March 2023, they take over signing the release tarballs. Yeah, kind of switch some of the domains over

instead of being on the self-hosted stuff running from the original developer. Now they're over on GitHub where Gia has full control. Yeah, and that's all combined with where Gia has full control. Yeah. And that's all combined with this campaign to burn the main developer out so they transition out, so they're no longer involved. On their own accord.

On their own accord. They hand it over to Gia on their own accord, which is really just so diabolical about this. The question really has to be was the maintainer of XE a target of an intelligence campaign? I really don't know. But I'd like to know if that's what we're dealing with here. I'm sure there will be criminal investigations into this and maybe we'll get answers. But we don't know as we're recording Sunday because this came out on Friday. It's amazing the amount of detail we do have at this point. It's an incredible amount of citizen journalism that's been going on. Yeah, I mean most of this drop in on a Friday and folks over the weekend voluntarily investigating, writing things up and sharing widely. I'd love to know people listening to this how they think this impacts open source and just their thoughts on this because this to me is the scariest part of this story.

And then we have the most frustrating part. And that's the dunking that you start seeing in our own community, as if we're not all affected by this, right? So you always see the anti-rolling folks come out of the woodwork to dunk on rolling when this kind of thing happens, as if this wouldn't have eventually hit them. And then they would have had it for years before they realized it, as if, right? So it's, again, instead of being thankful that there are users out there testing the bleeding edge and finding this stuff, they dunk on them, which I find to be classic Luddite behavior. So congratulations, everybody. Yeah, right. Especially in this case.

I mean, it kind of seems, especially just given what most servers run, it was kind of targeting really more the, you know, long-term LTS distros with the Debian RPM build system. That was obviously the goal. The goal was not to pwn a couple of boxes, like, that are running Debian SID, right? That wasn't the goal. Really, it kind of sucks to be open to this tumbleweed right now just because the combo of rolling release and RPM. Yeah, yeah.

Just bad luck. Yeah, right in the crosshairs. And then, of course, you get the system D haters coming out. You know, the Dev1 folks tweeted, because got to be classy. You know, let everybody know that Dev1 is not affected by the latest vulnerability. Quote, the malicious backdoor in the XE lib ZLMA is a vector for remote exploitation of the SSH daemon due to dependency on system D for notifications. And due to system D's, you know, blah, blah, blah, the way it pulls it in. And then they go on to dunk.

They go on to dunk. And, of course, first they put a really silly gif about how they're not affected. And they say this is another proof that SystemD is an anti-pattern for security. With its crawling, ever-extending web of dependencies, it extends the surface of vulnerability to orders of magnitude. And once embraced, not even a large distro community can defend you from that as they clutch their pearls now we've we've explained how this works with systemd and how systemd doesn't actually recommend you pull all of this in but doesn't

stop developers from doing it but the other wrinkle that i find fascinating about this is it appears about a month ago, a PR was created to change this behavior at a systemd level to essentially load these compression libraries in a better way that would have made this vulnerability likely not work. Yeah, it seems like there's interest here

in especially libraries that aren't necessarily always used like a particular compression library to have systemd be able to load these in a more dynamic way that I think maybe then would mean if you were kind of pulling it in via linking, trying to do the systemd notify stuff, it wouldn't automatically pull in XE. Which would have broken this vulnerability. So in a sense, it's very possible that the attacker or attackers was aware of this and

felt that they had a window of time and perhaps that's why they were pushing as hard as they were to get it included in the places now or never yeah and maybe maybe that's even why it ended up shipping with some bugs in it that ended up getting it detected and if they'd had a little more time perhaps they would have been able to clean that up because all their other behavior shows a pattern of you know very very accurate very very calculated behavior and we see this one slip up one wonders if they weren't under a particular time crunch because the systemd pull request looks like it's in the process of being approved looks like that change is going to be happening it's just a wild little wrinkle to the story that this

little window would have been closed so the systemd D folks, I think don't, you know, they don't really have a lot of blame in this. It's like you said, it's dependent on how you implement this, how it's been implemented. They also have a fix they've been working on already to kind of shore this up. I also got to imagine to some extent there's, you know, there are other situations like this in, you know, other relationships between pieces of software. Oh, yeah. You know, SystemD is successful,

and a lot of systems have this patch, so it sort of makes a natural target if you're trying to go after maximum spread. There is, I guess, I will concede to the DevWan folks that when you take software and you commonize it across multiple systems, you do, I guess, make it easier for malware

to work across multiple environments. That seems sort of obvious. But we've been doing that. I mean, look at the GNU utils. Look at SSH, OpenSSH. I mean, that is a line that has been a trend line going in one direction for a very long time. Can it also be argued that

SystemD has solved a lot of vulnerability issues just because it's a more modern way of, you know, handling the problems it handles? Yeah, perhaps. I mean, it does expose. It depends on the level you're talking about. But, yeah, I mean, there are a lot of security-related options it exposes for you out of the box that you don't have to roll yourself, at least if you choose to hop into them. It does raise some uncomfortable questions, though, Brent.

Yeah, well, I wrote down a bunch here, but the main one for me is, like, okay, this was, seems like it was pushed out maybe a little sooner than this attacker or attackers were hoping for because, you know, SystemD might have been changing things on them, so therefore was discovered, thankfully. But that raises the question for me of, well, how many of these style of attacks and vulnerabilities haven't been discovered because they didn't have this pressure cooker where they had to ship something sooner. And so doesn't that

make you wonder, like, look at every single piece of software and wonder what's that complex set of situations that got through and just nobody's discovered yet? We were in a weird window of time. Carl, right? There's several distros that are on the verge of freezing their bits and I suppose this attacker wanted to get this in as fast as possible. You know, Carl's in the chat room saying, look, it was probably,

they're probably hoping to get an Ubuntu, probably hoping to get it in the next version of CentOS. Now was your time. It's a good point. It's not just that system D was closing this and that inevitably would have eventually trickled down to downstream distros. Ironically, by the way, would have shipped on rolling distros first, just pointing that out. So it was more than just that, though. It was also getting it included in these other distributions. So that way, once it's deployed at scale,

you could probably just scan the internet and start attacking away. Yeah, give it a couple of years, those become the widely deployed versions. So yeah, how many other things like this have ever been done? We'll never know. You know, Ikyulys Ryan Gordon, he tweeted a picture that is pretty alarming in this category. And it's, he got an email from somebody named 37648282799. My favorite person. Yeah. And the subject is project collaboration opportunity, brief GitHub login request. And here is the body of the email. Hi, I'm participating in a project and I need to use an active GitHub account

to log in briefly. I only need to log in to the website without making any authorizations or changes. I noticed that your account meets the criteria. Now get ready for this. They write, could I borrow it for a few minutes? If the website verification is successful,

I'm willing to offer you a reward of $300. If it doesn't work, I'll still give you a thank you payment of $30. Seems totally legit. Why not? I can't imagine that Iculus is the only one getting that kind of email. There's probably people doing scans of GitHub looking at what your account history is and what you contribute to. Yeah. What I like about this one is just a blatant phishing attack,

and they're not even trying to hide that. Yeah. Well, they just pray and spray, I suppose. And there's got to be a few people out there that are going to say, okay, I'll take $300. I'm not even using my GitHub account anymore. You just got laid off at work. You haven't had time.

Yeah, you don't even contribute anymore. So we don't even know if these accounts were these attackers' accounts. The interesting thing is that these accounts that were involved don't seem to have much other history outside of this with a few couple of things here and there. So after this event, I don't think it's a great look for the open source community. I mean I don't think any of us have our faith shaken. I don't think any of us have our faith shaken. But when I think back to like me trying to pitch solutions that were free software and open source inside corporate environments, these type of events would bring doubt about the safety of open source software. And here's a headline that TechBeam ran with. Microsoft engineer accidentally found the malicious code in the versions of the XEUtils compression tool, likely preventing thousands of infections. Accidentally. He accidentally found it. That is going to strike fear into the CTO's heart. But I don't know. Do you guys think it was luck? I mean, it was luck for Andreas as an individual, but at scale, when you consider the scope of the open source community the diverse users and the

diverse amount of itches they're all scratching was it luck i mean probably a lot of these things it's hard to say for the particular but um yeah uh at scale it seems like at least we can look and you can poke around and you can look at the commit history. And, you know, if it was some proprietary compression library, what would you have to compare except the binary bits? I think this is too why user adoption of open source is still a very critical component because it does mean more eyes and shallower bugs. The reality is if we have a robust, diverse user base,

then you will have somebody who has this really esoteric setup that is testing on Debbie and Sid in just the right environment and happens to find things like this. And, I mean, would it have taken us longer? Perhaps. But the fact that we actually have that scale and capability is an inherent asset to open source that you can't even put on a pro and cons list with commercial software.

You'd have to consider with commercial software, we'd never know about this kind of thing. It may never even be discovered. But then on top of that, you have all of the information that's now available, like all of the reconstructing of the timelines and all the commit history, even though GitHub pulled it, like we can still put it together. And now as a community, this gets announced on Friday. And by Sunday, we have our hands around it. We have a pretty good idea of who's impacted, what's going on, how it works. Before people even return to work, we've got it figured out.

I think maybe it also shows there is the community that is really there. And it's not a scramble of disparate businesses who all happen to use the same proprietary thing and then have to find a way to communicate. I mean, we already have issues and forums and mailing lists set up. So when folks were ready to help troubleshoot and compare notes and debate who's impacted and how do we fix this, there were already mature systems in place for that. I wonder what would your answer be, Brent? I'd like the audience to maybe send in their thoughts too,

but what is your answer going to be if somebody does say, hey, isn't there some sort of backdoor that's in all of these Linux systems? Like how, you know, how these things can filter down? Like, how do you even respond to that? I am sometimes shocked. I'll get these questions. I'll be like, what? How did you hear that? And what would your answer be, Brent? I think my answer would be, how do you know there aren't backdoors in other commercial software? Like it's sometimes built in on purpose by design. And so at least maybe there are some that we haven't discovered yet. We just don't know. And, but that's true of every piece of software.

It seems if you're looking at these kinds of situations, but at least we can say that most of it doesn't. And we've, you know, had thousands and thousands of people now working on this stuff, having a look and and know that this software is legitimate. So I think you can't say that no software has none because it seems like, you know, this is a super sophisticated set of compromises here that we're seeing. But you can say that actually we've tried really, really hard to not have those be, you know, put in on purpose and get through on purpose. I think it's like the dichotomy of open source development, really well captured. We'll have a link in the show notes that really shows you how this developer was socially engineered to burn out

when they're already on the cusp. And that is an inherent vulnerability in free software, is we have so many burned out maintainers and developers that are just waiting for somebody to come take this burden off their shoulders. And that's an attack vector. But then at the same time, we have the many-eyes-shallow bugs situation. That means we catch it on the other end, luckily.

Well, in a very real sense, all software is garbage. It's all terrible. I mean, I don is garbage. Yeah, yeah. It's all terrible. I mean, I don't know what people are expecting. It really is. And here we just get to see it. Jan on Mastodon writes, Again, the FOSS world has proven to be vigilant and proactive in finding bugs in backdoors.

The level of transparency is stellar, especially compared to proprietary software companies. What the FOSS world has accomplished in 24 hours after detection of the backdoor code deserves a moment of humbleness. Yeah, we should, you know, I agree. Jan also says we could tame down the flame war as an armchair expert shouting at each other right now. And like we said, dunking on things like system D or rolling distros, we could do a little bit less of that.

But we should also take a moment and recognize it's moment and recognize it is also a big accomplishment how quickly. And to see the different distros work super hard and burn the midnight oil to get the patches ready because they get a little bit extra heads up, but not much, usually not much. They sometimes know like maybe 24 hours in advance before the general public or something like that. So they really have to move quick. Thankfully, in this case, it was mostly about rolling things back. You can tell, too. I mean, this is like a tricky one. I know for NixOS, XE is like pretty low down in the stack

and part of the bootstrapping thing. So kind of have to rebuild everything to include a new version or a patched version. Yeah, I think they were saying 220,000 packages need to be rebuilt because of this. And then you've seen other packages and other distros where they've had to say they'd have to make it look like an updated package

and then it has a dash. Really, this is version 5.4, though. Right. To try to trick the package manager into like, oh, no, this is actually an update, even though we're not trying to actually roll it back. That part is really wild to me. But you can tell, like, we're moving fast and just trying to figure out whatever we can do

to get things fixed right now and then hopefully clean it all up as we learn more. Yeah, we will. This is going to be one of those stories I think we'll be hearing more about. We'll be probably seeing some sort of legal investigation. And I've wondered – so when GitHub pulls this like they did, they pulled the repository for XE. Well, technically now Microsoft is the only company that has access to that entire history and everything. And I would presume they're probably doing their own very serious investigation, maybe trying to identify if this user worked in other repositories under another identity. But by having this project on GitHub, when this incident occurs, now this is proprietary Microsoft information.

We've been lucky that the public has been able to sleuth it all together and find mirrors and things like that. But I just had this kind of chilling moment where I realized this is all Microsoft's code now. And they're going to initiate some sort of research. They have a very big, quote unquote, cybersecurity arm. And this kind of stuff is going to funnel right into that. They're essentially just going to get free work from GitHub for years. And they're probably going to follow this process. Instead of marking it read-only or anything, which would have helped other researchers, they completely pulled it down.

They have it internally. And they're no doubt scrambling this weekend to go through it all, for good and for bad. But just a reminder, take advantage of the decentralized nature of getting out there, folks. Yeah, yeah. or take advantage of the decentralized nature of getting out there, folks. Yeah, yeah. Speaking of decentralized, I just want to take a moment and mention that this coming Friday, as you're listening to this, the week that this releases, April 5th,

we have a Nostr workshop. Yes, a live workshop, notes and other stuff transmitted by Relay. So if you've been Nostr curious, I've been on and off myself. The protocol seems to be very promising, and we're finding new use cases for it beyond just like Twitter replacements. It's based on really simple, flexible event objects, which are just passed around as plain JSON. And every user is identified by a public key, and every post is signed. And every client validates these signatures.

That's the Nostra network in a nutshell. That's what you need to know about it. Oh, and of course course it's open source so one of the more compelling use cases that fountain fm has been testing is decentralized real-time chat not to replace like you know your favorite messenger app or your blue bubbles or whatever the crap you want but for like live events on the internet in the app or in the web and because everything's just being passed around as plain json it means that we have the ability to create cross-platform tools and apps and services if we want around this. And you as a user can create one identity that's verified by a public key that you can use across different apps and websites or Nostra services.

And it's just as simple as bringing your key and you can use these different clients and these different apps, which I've been playing around with. It's pretty nice. And so we're putting this to the test. The Fountain FM dev team is building a new live experience for the podcasting 2.0 world, and we're helping them by testing it every Sunday. They've created an embeddable web chat that's going to also be available in the apps

or other apps that want it, just powered by Nostr. You don't really need to use Nostr to use it other than to have a public key that works. So we're going to help people get started. We're going to have a live workshop next Friday. Again, it is April 5th, 2024 at 2 p.m. Pacific, 5 p.m. Eastern. And we'll help you get your Nostra identity going. We'll answer questions. We're going to have some fun. We'll give some sats away, help you get started over there too if you want to do some boosting and help us test this thing. It's a new use case, really, for notes and other stuff transmitted by relays.

Don't know where it's going, but I think it's an opportunity for us to help develop a new open standard and test it, at least, and see if it's worth pursuing. Again, Friday, April 5th, 2 p.m. Pacific, 5 p.m. Eastern. Date, time, your local area, jupiterbroadcasting.com slash calendar. Should be good. We'll be in the LUP stream. We'll be just doing

a LUP live stream, so come over and join us at jblive.tv. And also, thank you to our members. Unplugged Core members have been keeping this show going for a couple of years now, really. And as we are down one sponsor this week, they're stepping up and they're helping us. So thank you to our Unplugged Core members as well. Oh, we are just days away from Texas Linux Fest, April 12th through the 13th at the Palmore Event Center in the Austin of Texas. I'm excited.

Yeah. I think we have our place booked. We've got to get all our travel booked, Wes. We've got to get all our travel booked. We do have a Texas Linux Festival Matrix chat room that we'll have linked in the show notes. You'll find us down there at Texas Linux Fest. We're going to be there at the Sineri booth who's helping us get there and do some live streaming,

say hi to people. Sineri is also helping Texas Linux Fest get going too. It's really great. Looking forward to meeting them. And then Linux Fest Northwest, April 26th through the 28th, just days after the one in Austin,

we're going to be up in Bellingham, Washington at the Bellingham Technical College. And we also have a Matrix chat room for that if you can make it. That should be a banger. We'll have a live Linux unplugged. There's going to be up in Bellingham, Washington at the Bellingham Technical College. And we also have a Matrix chat room for that if you can make it. That should be a banger. We'll have a live Linux unplugged. There's going to be food. Linux Fest is a hell of a party and we'd love it if you could make it to that. It's a lot going on. So soon. So quick.

So quick. So great. And then just shortly after that, we'll get you a date soon, but Wes and I are going to be in Denver for a Red Hat event. So it's just a lot coming up. And I don't have any meetups for these, but I think at Texas, we'll just get together with folks. It's a small venue, and we'll head out for lunch or something. I don't think we need to do a meetup.

Come find us. Yeah, we'll bump into each other. Or you can find us at the Signary booth. It's pretty easy. And then also, I wanted to mention something going on that just, I am so grateful when things like this happen.

So, you know, when these types of days come out where we have a horrible vulnerability, you know, something like the XE backdoor that is just you need the information, the technical details, some of the background. You don't need all the hype. I think it's really nice to know that shows like Linux Unplugged, our primary goal for this is just get you the best information we can as fast as we can do it in the time we have and as accurately as we can. And if this show was following a different path, a different model, I think you'd find these types of things get amped up. You're going to find this happen. This will happen now on a lot of YouTube videos, not all of them, but a lot of them are going to start really ramping up like the SSH vulnerability, SSH compromise. You're going to find this will happen now on a lot of YouTube videos. Not all of them, but a lot of them are going to start really ramping up like the SSH vulnerability, SSH compromise.

You're going to start seeing this kind of stuff and really kind of amp it up for clickbaiting. And an article came out that I'll link in the show notes. Wonderly, which is an advertising biz, and iHeart, and Lemon Media, and others, I don't know. They've all been using this service, this private service from this group called Mowpod, M-O-W-P-O-D. And Mowpod, they invite you to participate in this program. But once they invite you, they place ads in games like on iOS. And these ads, according to Bloomberg, direct users to go into Apple Podcasts and download the podcast that they've been told and then come back to the game,

and they're awarded like an in-game weapon or in-game currency. So I'm just playing my game, and it says, oh, hey, go download Linux or not Linux on Plot. But in this scenario, yeah. If we were contacted by Mopan, and we wanted to pay them tens of thousands of dollars or whatever it is, yeah. Wow. Yeah. Wow. Yeah.

So these in-game ads prompt the user to go download a particular podcast, like a particular episode even. It will direct them to a specific episode. Once that's done, they get rewarded. And here's the crazy thing. Bloomberg did the research. At least nine of the current top 50 shows in the Apple podcast directory have been promoted this way. I guess it works. Wow.

And we would never, like a Linux podcast that's talking about an XE vulnerability would never, ever make it to the top of those charts. How could we compete against that? Right. That signal would never get out there. But now, now that there are charts like on Fountain that are based on the value that's contributed back from our audience, we're always in the top 10, if not in the top three of that chart. And I think that shows you the difference of the path we're trying to take with the show

versus the direction the traditional mainstream advertising industry has gone. And it's not good for content, and we don't need this type of content to be playing these games. We do not need it. So thank you, everybody who is a member, and and thank you everybody who boosts into the show. We really appreciate it. And now it is time for Le Boost. Oh, and speaking of that, Oppie1984, he's our baller this week. Hey, Rich Lobster!

44,718 sats. He says it's a zip code boost. Personally, I prefer the live content to be in a separate feed. Answering a question we'd asked last week, I mostly listen while I'm at work. I have no problem paying attention to the studio shows at my work. I find live content distracting. I usually end up skipping it since I need to focus and, you know, earn those Fiat Fun coupons for my boosting habit. And my dog's milk bone addiction.

Yeah, I know, earn those Fiat Fun coupons for my boosting habit. And my dog's milk bone addiction. Yeah, I know that one. My dog has recently been addicted to, like, they're shaped like bones, but they're pumpkin. Oh, he loves them. They go pretty quick, though. They do go pretty quick, so you've got to watch out. Now, it's zip code boost, Mr. Westpain. 4-4-7-1-8. Okay, Stark County, Ohio.

Maybe a city like Canton or Monterey Heights? Well, hello, Stark County, Ohio. Maybe a city like Canton or Monterey Heights? Well, hello, Stark County, Ohio. Thank you for boosting in, Oppie. We always really appreciate hearing from you. And you nailed our baller booster spot. That's not easy. Indeed. Yeah.

Mr. Payne, our next boost, please. Hybrid Sarcasm is boosting in today with 42,001 Satoshis. I hoard that which your kind covet. Got my 13-inch, 13-gen Intel framework. Decided to christen it with none other than NixOS. There's something about test driving hardware that you just can't do with a VM. test driving hardware that you just can't do with a VM. This has actually replaced my MacBook in every way except home.app. Home Assistant just hasn't bridged that gap for me yet. What? This is crazy talk, Hybrid. This is crazy talk.

I think we need to know more. Definitely. I mean, I've had the exact opposite experience. I was shocked at how easy it was to integrate HomeKit accessories into Home Assistant and just stop using any of that stuff. I'd love to know where you got hung up. Maybe we can figure it out. Some sort of challenge. You try Home.app for a week. I don't think so. Sarcasm. Home Assistant more seriously. I don't know about that. I don't know. I would love to know what he ran into.

I'd also like to know if people use Home Assistant in the audience because, you know, it's a big topic of conversation that's self-hosted, but we don't talk about it much here. So I don't have a good read on what the Home Assistant usage is in the audience. Hybrid continues on just to say that I feel at home with configuration.nix. I think I'll give Flakes a try when I need something that the.nix files just can't do. Yeah. Yeah, I agree. Yeah, I think we're going to have to do more on that soon. Our third baller booster, Vamax, boosted in two boosts for a total of $24,690.

So the combination is 1, 2, 3, 4, 5. That's the stupidest combination I've ever heard in my life. That's right. Two Spaceballs boosts in a row. Number one. Plus one for having live shows in a separate feed. I have no problem with the quality of live shows being mixed with planned shows. I do appreciate the work that goes into the planned shows.

I think it sets JB apart from other tech shows. Aww. But fountain live notifications refuse to notify me, and having one show feed I can check on a periodic basis would be a blessing. from other tech shows. But Fountain Live notifications refused to notify me, and having one show feed I can check on a periodic basis would be a blessing. I was thinking of you folks and good old plain pod man last weekend as I spent it fixing my cluster networking. Good old plain pod man, huh? You know, just the other day I had occasion to try out Scopio for the first time,

which is part of the pod man and build a sort of toolkit really nice yeah i know i know pod man is one of the unsung pieces of software out there that really doesn't get enough attention but you know i'm kind of over containers i don't know if you've heard i'm i'm past containers now it was great though because it built right on my mac i didn't't have to go use a Linux box just to do the Docker interaction because I wanted to fetch an image and take a look at some of the innards. Yeah. And, you know, packaged in Nix, showed right up. I suppose if you're still on a Mac, you might need Docker containers and PodMap.

I don't know. I'm beyond containers, though. Now, VMAX did continue with another Spaceballs boost, 1, 2, 3, 4, 5. 1, 2, 3, 4, 5. 1, 2, 3, 4, 5. Yes. That's amazing. I've got the same combination on my luggage. P.S. I don't know if you folks have ever done episodes or pics focused on networking or have suggestions on podcasts or YouTube channels that focus on exactly that.

But when my Cluster's Metal Bees ARP setup decided to start stubbornly assisting. It had no idea who or what my router was. And I was trying to figure out BGP speakers. I realized how badly I needed to develop my networking knowledge. Okay. Now, I mean, I am absolutely down for doing some episodes on networking. That's like saying do an episode on computers, though. It is a big, I mean, it's as big of a topic as Linux.

So what exactly? One thing that we've always kind of floated internally is a dedicated episode on building a Linux firewall from scratch. You know, not that there's anything wrong with dedicated hardware firewalls, but it might just be fun to have a, you know, see how a simple but effective Linux machine that would be straightforward to configure and tweak when you need to. Maybe you don't need a full UI. Maybe it could be an MVP. I think our audience probably has a lot of ideas. I know I've seen some chatter in the Nixnerd room

about NixOS routers. Okay. All right. Yeah. Well, then perhaps. Also, you know, VMAX, it sounds like a great little home lab you set up there. I hope you figured it out and update us because I'm sure you're running into some interesting problems. That's a great point. That's a great point.

That's a great point. VT52 comes in with 20,202 SATs. Coming in hot with the boost! As an i3 user, I really want to play around with HyperLin, but Wayland doesn't work for me. I have two displays at my disposal and both are multi-panel widescreens. They present as two

panels per monitor. Whoa. Yeah. Most Xorg things handle it fine, but Wayland-based stuff sees it as two separate displays, meaning I can't maximize or full screen something over the entire display. Does anyone know of a workaround? Ouch. Yeah. Okay. I can see how that might be prohibitive. I can't imagine how that could ever be solved unless the desktop environment baked in support for that in its display management tools directly. Like I could see Wayland technically supporting it, but Wayland won't natively just do it unless something using the protocol tells it to. I mean this could be insert enlightenment joke, but that's – I've never even heard of a multi-panel widescreen that presents as two panels per monitor.

I've never even heard of that. That seems like quite the niche. Here's hoping someone out there knows. Yes, please. That's a good one. Okay, and here we've got three quick little boosts. One from Zenzilla last week for 5,000 sats. Just to say, listening live, you guys are crushing it.

Then, of course, our pal listener Jeff came in with 5,000 sats just to say thanks. Hello. And an anonymous booster came in with 3,000 sats also to say thanks. Boost! Thank you very much. I love the live boost. Now, following up, Moonanite sent in 5,000 sats to say, Can a NixOS config fit on a floppy disk?

Maybe you should make a config and send it out to subscribers once a year as a particular thank you. B-O-O-S-T! That is such a neat idea. Are you checking your Nix config right now? Okay, Wes is going to check to see how big his Nix config is right now. That's a really fun idea. I've honestly been trying to think of

ways, I don't know why, to incorporate floppy disks as swag. Alright, 6K maybe? Oh yeah, we could do that. With the config and the flake and the hardware config. I don't know why, to incorporate floppy disks as swag. All right. 6K, maybe? Oh, yeah, we could do that. With the config and the flake and the hardware config. You know, we could probably make it smaller. This was just a random example. Well, we've got 1.4 megabytes to work with.

Well, really, 1.2. Right. Yeah. Got to format the thing. I love that idea, Moon and I. I'm going to put that in the old back pocket. And if anybody can expand on that, please do. I mean, wouldn't the first swag item have to be USB floppy disks?

Like you'd have to send those out first. I just don't know. Yeah, Brent, what? Well, I saw a little tidbit come across the news feeds that like EXT2 this week was being deprecated. week was being deprecated and so given we want to do this a you need to like procure the hardware both the floppies and also the means of reading and writing such things but which format would you throw on this it seems like a multi-faceted challenging problem to overcome here what file system i don't think we'd want to use extended fat just because it's a linux show right um technically extended two will be

around because it'll be supported by the extended four but the extended two driver in the kernel is being deprecated uh come on bcash you're saying yes exactly right yes you guys got it yes why did i even ask oh bcash would be so much fun. Maybe. Maybe. All right. User 42 boosts in with a row of ducks. And they write, I think the best and most efficient way to launch and manage Minecraft on Linux is a combination of PortableMC and Furium. They both are command line programs,

and PortableMC installs and starts Minecraft, letting you choose a specific version as well as automatically installing Fabric, Forge, and other mod loaders. Oh, okay, that part is really nice. Ferium manages mods and supports having multiple profiles, too. You can even upgrade all mods with one simple command, and it supports mods from either ModRinth or CurseForge. Yes, these are all the things you have to learn as a dad that has kids.

I'm so lost. I'm glad one of us understands. Okay, so those are all the things you have to learn as a dad that has kids. I'm so lost. I'm glad one of us understands. Okay, so those are all really great suggestions. Maybe we can try to link some of those. I will also give a shout-out to Prism Launcher, a custom launcher for Minecraft that allows you to easily manage multiple installs and that kind of stuff. And I mentioned it before on the show, but I'll say it again this week, AT Launcher, which is what my kids are using right now to manage their mods on Linux. Those apps are really useful because

they're watching Windows YouTube creators who are doing all these different mods and they want to do it on their Linux box. And that makes it possible. 1000s. Fun will now commence. So you got me convinced to try NixOS. I had an HP ProBook that I wanted to dual boot with the existing Windows installation. So I loaded up the NixOS live image, configured some disk, loaded up the NixConfig and installed. Easy. Reboot and nothing.

Just booted into the existing Windows install with no option to select next. My bugger. Looks like more work to do. Hopefully there's some, you know, I don't know, bio settings you can tweak there and get stuff to show up. Might be doing, is it doing that fast boot thing where you don't have time to interrupt it?

I don't know. I like that you got some theories though. You're helping them out. Get them in the right direction. Good luck, Watsi. Good luck. No, wartime boosted in with a row of, well, it's three, three, three, three sats from Fountain. I feel like that's a lucky boost. You're doing a good job. First time listener here.

I love the shows, and please keep them coming. Well, thank you for boosting. I always really appreciate that first time boost. Well, thank4 deuce. What? 40 deuce comes in with 8,484 sats. I'm going with it. It's a hot boost.

Coming in hot with the boost. You asked for it. You got it. Here's a link to my daily driver, Nixflake. Oh, we're going to put this in the show notes. Oh, this is so great. It's built for Hyperland. Amazing.

Perfect. It also has Plasma 6. Wow. Wow. And I recently added configs for River, Sway, and Wayfire. Amazing. Perfect. It also has Plasma 6. Wow. Wow. And I recently added configs for River, Sway, and Wayfire. I aim to keep all these configs fairly simple and close to vanilla and with some basic QOL batteries included, quality of life batteries included. He links us to his GitHub where he's got the 40-deuce file on there. He says, I'm using this flake to manage three hosts so it definitely

read my notes before you dive in that's good to know that is good to know he continues on thanks for all your content and nix coverage i bounced off nix last february and i came back to it around october while i have a ton to learn since digging in i'm loving the journey in my day job i'm a nurse so i don't have a ton of time for focused nerd projects. I've often had to leave projects for weeks or months when work is busy, which makes progress tough. With Nix, I finally feel like I'm able to build forward since I always have the blueprints and the building blocks. Game changer. blueprints and the building blocks. Game changer. I completely agree. It's, and we've said it before, it's self-documented.

You come back and you're like, oh, that's right. That's how I solved that. It's so great. I'm totally nodding as well. I can really relate to this. And what an advantage. Yes. I just like that we all have our own, like, how I bounced off Nick's stories.

Yes, we do. Clearly all like, how did I not see this? Ours is embarrassing because the audience told us about it over and over. Many times. But we got it. We finally got the message. Our buddy Gene Bean boosts in with 5,294 cents across four boosts. I think these might be sort of stream of listening boosts. Here's an idea.

Use the extras feed for the live shows at conferences and stuff like that. I think officially that is a deciding boost. We are going to do that. So if we record live, if we have live content from Texas Linux Fest or Linux Fest Northwest, we will record it locally and we will publish

it in the extras feed and then probably link it in the show notes for the show. Here's a row of ducks to say that regarding Nixix con at somewhere beside scale a few of us were trying to get them to also come out to self uh that i've never been to self be a good excuse to make it down there though gene also says plus one more desktop preset flakes got it keep that trend coming check the show notes we got one in there this week a good, which we should really check out. And Gene's being real sweet saying, I totally agree. Y'all are the Linux magazines of today. I absolutely get from y'all what I got from Linux Journal in ye olde golden, ye olden printed days.

Ye olde printed days. The days of print. Yep. Thank you, Gene. That's really sweet. Appreciate that support. Galactic Starfish comes in with a couple of rows of ducks things are looking up for all the duck uh and they write uh join the dark side brent use auto tyler

okay also give a plus one another row of ducks for a plus one to prism launcher which is a really solid launcher now true grits came in with a Spaceballs boost. 1, 2, 3, 4, 5. So the combination is 1, 2, 3, 4, 5. That's the stupidest combination I ever heard in my life! I definitely would love to see those live streams from Scale published, and the place I'd like to see them is in the extras feed. Also, with a Linux magazine comparison and sending out disks or USBs, maybe we could see the return of JupyterOS, maybe based on Nix. There could be several curated config options, such as gaming, coding, and podcasting workstation.

Any updates on getting Boost working for the members feed, by the way? I wonder how many people listening know the story of JupyterOS. I actually – can you tell it, please? I don't know. I want to, I'll tell it next week. If somebody can boost it and tell their recollection of it, and then I'll tell my version of it. I want to see before I give it,

if anybody can remember Jupiter OS and its history in relation to Jupiter Broadcasting. I was going to say, I think, you know, we're probably unlikely to do it, but we could certainly host a repo somewhere or if like folks want to make some of these, you know, we're probably unlikely to do it, but we could certainly host a repo somewhere, or if, like, folks want to make some of these, you know. Maybe if there was something out there, like, that was really nice,

we would want to package it and ship it somehow. Or what about, what about, like, a Flake desktop repository? And, like, that's JupyterOS now. It's like you grab a Flake and you get a certain experience. I don't know. We should talk more about it. As far as boosting on the members feed, I have raised the issue with Fountain.

I think we've identified the problem is that it's an unpublished feed and they're looking up the value information, the split information from the podcast index API. And because it's an unpublished private member feed, the API returns nothing. And so we're going to get that worked out in time. Sorry. We're working on it, but we appreciate you. Patrick also sent in a row of ducks. Boosting from podcast guru on Android. Another great app, a great episode.

And he says that serious laptop looks pretty tempting. Sam Squanch comes in with 10,000 sets. Coming in hot with the boost. To say? Bloop. Bloop to you, Sam Squanch. Thank you for the support. Thank you for also being a nice contributor in our Tele the boost. To say, bloop. Bloop to you, SamSquatch. Thank you for the support. Thank you for also being a nice contributor in our

Telegram group. Our very own Otterbrain with 10,000 sats, boosted from Fountain. It's over 9,000! Sure is. I joined the scale coverage quite a bit. I'm currently giving my NixOS install a System76 laptop a good workout through writing up

a grant proposal with multiple web tabs open, LibreOffice, GIMP, and Inkscape all going at the same time. Thank you for that boost. And we got just a batch of live boosts coming in too. So I'll wrap us up here, boys. Autobrain, appreciate your support. And thank you for the kind words on the scale coverage. Good luck on the grant.

GooseGuyQ came in with a row of ducks. They're excited to meet you allall at Texas LinuxFest. No meetups planned yet? Not yet. We're just going to do it live, as they say. And then GooseGuyQ also said, I spent the morning figuring out how to boost. Congratulations, by the way. Not necessarily the easiest thing.

I ended up getting Kraken and moving some previous Bitcoin I had from Coinbase to Kraken. Wow. And then moving them over to Lightning to Fountain. Here's someone else having issues with Cash App on Graphene. I'm not alone. Yeah. Oh, yeah, right. So other people, and yet I use it regularly on Graphene.

What's the deal? What's the deal with that? Does anybody know? That's so weird. And we both checked our settings. Like you've got the memory exploit protection. I think you probably messed with that setting. Yeah.

We also got Flake Latim came in with 3,000 sats, boosting in live from Wellington, New Zealand. Scary story. Have a great Sunday, team. Hello to New Zealand out there. Thank you for boosting in. Really appreciate that. In a New Zealand meetup. We got to do it at some point. We also got a, quote, overly complicated flake that deploys to my Ubuntu machine and my personal profile and my NixOS machine from JC Dickerson.

We'll include that in the show notes as well. Thank you, everybody, who boosted in live. I would just like to express the value to the world of long weekends. It seems like, you know, the research that went into this vulnerability this weekend, but also just everything we heard about people tinkering on projects they've been wanting to get to. Like, this is a gift to the world. I guess so. I mean, usually we work Easter Sunday.

That's usually always, I mean, always has been for us. But it definitely felt like a lot of the rest of the tech industry was working this weekend with us. Indeed. We were actually commenting on it before we started the show, before we even sat down. It's like, it really feels like everybody else is working this Sunday, too. Thank you, everybody who boosted in.

We had more than 23 boosters because we had those live boosters. And we stacked just a little bit more than 226,357 sats. Thank you, everybody. It's a fantastic amount. Can I just express that we got a very small boost that I wanted to bring out? It's from this guy named NoblePain100Sats. It says, B-Zip for life. Very true. Very good.

Thank you, everybody, who boosted into this production. It really is something special we're doing here. I feel like we have the combination of the world's greatest community combined with some of hopefully the greatest content yet to come when we go to LinuxFest Northwest and Texas LinuxFest. I feel like this year is just going to be fantastic. And the fact that we were able

to cover scale completely supported by our members and that these productions, even when we're down a sponsor, we continue to get the support via boost and membership. It means a lot to us. And, you know, it also means

that we're here reporting for you. You are our biggest customer. And that means we're always just here to make you as happy as possible. Thank you, everybody. Special pick this week, Brentley, special pick. I don't know if it is a pick, but it's a PSA perhaps today as we record. Just so happens to be World Backup Day. So a little reminder that maybe this weekend is a good time to, I don't know, do some backups.

So I wrote down some ideas here since, well, I think we're tech literates and we have maybe basic backups figured out. But a few stats listed on World Backup Day's website, which you can check out, 21% of people have never, ever, not even once made a backup. Wes, is that you? No. Apparently, 113 phones are lost and stolen every minute. Whoa, every minute? Could be yours. 29% of data loss is caused by accidents.

I was going to push those changes when I got home, but... That delete button is just right there. Mm-hmm, mm-hmm. I thought that was a different drive. I thought I was formatting my test drive. It's a feature of DD. Also, I think it's worth pointing out too, a lot of us store data in the cloud now. Just because it's in the cloud, that doesn't mean it's a backup.

That's your one copy. So I tried to come up with some ideas on how those of us who maybe have backups can already do something a little different on a day like today that might help the world or yourself. So, number one, help someone else like help that 21 percent who doesn't even know what backups are. And maybe that's a family member, a friend or something like that. But also, and I've run into this, if you set up a backup system for someone previously, go check up on it. Maybe it's broken. Maybe it's not doing what you intended to. My father, he came to visit last month and I was like, oh yeah, how are your backups going? He's like, I don't know. It's been having this error message for

the last three months. So I think it's okay. I was like, geez. So some of these things break and you know, it's good to check up on them from time to time. I think another good idea is verify. You can't just do backups. You should verify them. Depends how you want to do that. But at least go check a file or something like that. Also, maybe you've stood up some recent services that you didn't include in your backup solution because you were just playing around.

It was going to be temporary. Chris, you got any of those? I bet you probably. It's funny you say that i just um thursday realized oh i'm not backing up any of my audiobooks because remember when we did audiobooks i didn't actually back up start backing up that directory and i didn't lose anything but it was a preemptive move but you're right brandon's like oh i just i was playing around for the show it stuck i never thought to go back and incorporate it so it's a. Well, and if you're a small number of those keeners who already did everything on our list so far, I think you could even think about

just upgrading your backup strategy. Maybe you get off of Google photos, if that's important to you, or upgrade some of your local storage, for instance. And like Chris said, just, just back up some of that stuff that you keep not on your computer. And that might even just be like, maybe you ran an S3 bucket and you threw some very temporary project files in there and you kind of forgot about it. Or maybe your website. I don't know. Maybe that's worth backing up. So just anything you take for granted on an everyday basis, today's a good day to think about that. Good reminder. We have done some backup episodes, but it's, I think,

something we should just touch on from time to time. I know I need to do more backup of some of our studio stuff. Like, I don't think I don't have anything

like of the studio configs really backed up anymore because it's all sort of changed semi-recently. There's a fun episode we did semi-recently, Linux Unplugged 494, which has one

of my favorite titles, Updating Your Fiddly Bits. And there's a little session about doing backups and backup upgrades right at the start of the episode. Our bits are getting fiddly again. Yep. Yeah, it's very true. Well, okay. It's really feeling like we're just at the very beginning of this entire XE, I don't want to call it drama, but story.

We'll keep an eye on it. I don't really know what more is going to develop between now and next week, but I have a sense we'll learn more. Yeah. I know there's what we've seen that the same developer had made contributions to lib archive. Maybe there's other things in the works out there. Yeah. Or parallel operators. And will Microsoft, you know, on Tuesday or Wednesday, maybe start sharing a little bit more information? Maybe we can learn more then.

I'm going to be really curious to see like some postmortems from this situation but from a variety of perspectives because this touches on – well, it touches a lot of people from project maintainers everywhere to like distribution. So it will be interesting to see some lessons learned communicated from this that we can apply in the future? Yeah. Will we see any companies actually step up and provide some sort of support to these developers? We often see that conversation flare up and some noise is made for a little bit, and then it just dies.

Even the U.S. government gets involved sometimes. It feels like SUSE and Red Hat, I'd love to hear something from them on kind of what they, how they feel like a way to address this is. Because they could have been massively impacted by this.

They got really lucky. And you'd hate to see it take something like this vulnerability gets widely deployed on AWS before anybody gives a crap. But that just might be

what it takes unless we just address it now. We're going to have to start patching the rolling releases without telling the lts that doesn't feel right what if we hadn't discovered this like it's being called a back door uh you know uh what could have happened i guess is maybe the scary place to go to but important to think about that is a good question what if we hadn't discovered it? What if Andreas hadn't been, you know, decided,

oh, I'm going to track down and figure out why SSH is using a little bit more memory than it should? What if we hadn't been so lucky? That's a great question. Maybe one that we'll chew on for the week.

Thank you everybody for tuning in. Hope you enjoyed the episode. Hope it was informational. We'll have lots of links and resources over at linuxunplugged.com slash 556. Lots of stuff to go read over there if you're curious, including diagrams of how the vulnerability works and everything like that. Now, we may be doing a double next week. We don't really have anything nailed down.

But the live stream, it's going to be kind of, well, a change in when we go on our trip to Texas Linux Fest. So I guess that's my way of saying we've got one more on the books for sure. And we'd really like to have you join us next Sunday at 12 p.m. Pacific and 3 p.m. Eastern. See you next week. Same bad time, same bad station. And don't forget, we'd love to know the first five things you set up on a new Linux install and also send in how you do a totally hidden Linux OS. What software? How would you compose it?

How would you hide it? A totally hidden Linux OS and the first five things you set up on a new Linux install. Linking to a Nix repository can count as an answer. Boost it and let us know. and then go get those notes. Just check back in next week with us. Thank you so much for tuning in to this week's episode of the Linux Unplugged program, and we'll see you right back here next Tuesday as Insight. សូវាប់បានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបា Thank you.