LINUX Unplugged - 545: 3,062 Days Later

Episode Date: January 15, 2024

Kent Overstreet, the creator of bcachefs, helps us understand where his new filesystem fits, what it's like to upstream a new filesystem, and how they've solved the RAID write hole. Special Guest: Kent Overstreet.

This kind of goes back to the reason for Bcache.js even existing. Going back to Bcache, I was at Google, and we're trying to use these really high-performance SSDs that Google is building internally. And this is when SSDs were new, so everyone was looking for applications, and caching was the natural solution. But it turned out to be that just producing a B-tree that

is fast enough for indexing all block IOs was a bit of a challenge. Binary search turns out to really suck if you're really pushing it really hard because binary search is the worst possible algorithm for the way CPU caches work. One of the things they don't tell you back in school. So I came up with Heitzinger arrays before it was even published. Heitzinger arrays are where you build up a binary search tree in an array, kind of the same way you would build up a standard binary heap. So this means prefetching actually works. So that plus a whole bunch of crazy optimizations to make the nodes as small as possible, plus some other math so we can convert back from Eitzinger indexes to in-order traversals to get back to the sorted order representation.

That made bcache's Beecher implementation one of the fastest around, actually the fastest ordered persistent key value store as far as I know. That was when I knew I and the other guys working on Bcache back then that we were onto something. We had something that might be a good basis for a file system. Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris. My name is Wes. And my name is Brent. Hello, gentlemen. Coming up on the show today, yes, BcashFS has shipped,

and it wants to sting like ZFS and float like XFS. We'll bring the value to you this week and share insights from our in-depth conversation with Kent Overstreet, the creator of BcashFS. From the problems it is built to solve to the process of getting it past Linus himself and a lot more. We'll get into all of that. Then we'll round it out with some great boosts and picks and a lot more. So let's say good morning to our friends over at Tailscale. Tailscale.com slash Linux Unplugged. Try it out for free on 100 devices.

Tailscale.com slash Linux Unplugged. It is really the easiest way to connect all your devices and services to each other. Wherever they're at, across, carry your great NAT across the world, and it's fast. Really fast. And it's powered and protected by... WireGuard. That's right. Tailscale.com slash Linux Unplugged.

Go say good morning and try it out for free on 100 devices. Tailscale.com slash Linux Unplugged. And, of course, time-appropriate greetings to our Mumble Room. Hello, Virtual Lug. Hello. Hello. Thank you very much for joining us

today, guys. Nice to see you there. We call that Mumble Room our low latency audio feed because it is. It's Opus. It's running in Mumble, all free software running on our Linux recording system, a free software stack. A great way to listen on Sunday while we do the show. It's almost like you're here in the studio.

It is. Almost. As good as it gets, really. NixCon North America and Scale are just around the corner, March 14th and 17th. Just a few episodes away, like 10, 9 episodes, something like that away. Yikes. I do it in luck.

That's smart. Just that way we can think about like from a content standpoint. And with this trip, we're trying to do something kind of ambitious. We're in the middle of an ad winter. And it's always kind of been a goal of mine to go cover these Linux events. And the longer I do it, the more I actually think they're kind of essential to what we call the quote unquote Linux community. What we see online is an echo of what actually takes place at these events. And that's where

the real Linux community is. And I've always tried my best and always failed. And I'm always going to try to do better to cover these events and convey why that matters in free software. And I've always felt like the most honest way to do that is to get there either by paying for it on our own or by the audience helping us get there. Because it's like I just want it to be a pure thing because I want it to be a pure contribution back to the free software community. And that's why this idea of trying to get to scale in NixCon was like, okay, this is maybe our opportunity. It is the adpocalypse. Maybe the audience will step up. And man, have they, because we've had a goal to raise 8 million sats via boosts.

Very ambitious, but we're trying to cover the gas in an Airbnb and whatnot. And it's been going really well. So far, we have raised 92% of our goal. 7,324,551 sats towards our trip to go to scale. 92% of the way there. Wow. That's so impressive. I'm really grateful. This has been not just a goal for this year, but doing something like this, not just a goal for this year, but doing something like this where JB could go cover it with no attachments to any commercial entity, just completely focused on the event,

not trying to get a return for a sponsor, that we could cover something like this powered by the audience has been a long-term goal of mine. And we are 92% of the way to raising 8 million sats. I'd love to tip it over even a little bit just because it's California and it's extremely expensive. Now, listener Jeff pointed out on the live stream today that if you have some on-chain funds and you don't have an

easy way to get them into Lightning so you can boost, well, there's a tool that the three of us have used in the past to do just that to support our favorite podcast, and it's called bolts.exchange, B-O-L-T-Z.exchange. You can go both ways. You can go from on-chain to lightning and lightning to on-chain. If you've got no idea what I'm talking about, don't worry. Unplugcore.com. You can become a member and support us that way. That'll help get us there as well.

But 92%, we're almost there. Help us close that gap and support the show at the same time. Thank you, everybody. All right, BcashFS, where does it fit? I think that's kind of everybody's question when I tell them BcashFS is shipped. In Linux 6.7, last Sunday as we were live, it came out. But why should I care? And why should you care?

And Brent did the math. And I actually already forgot what you said, Brent. But we've been talking about BcacheFS here on this show since, survey says. Linux unplugged 158, which was 2016 in August. Now, why have we been talking about a file system since August of 2016? And I would like to make the case, and you guys feel free to fact check or interrupt or whatever, but I feel like Linux still hasn't solved the problem of a completely robust competitive file system that is even semi-feature comparable to what an iPhone can do

and what Windows XP can do. Extended 4 is a great file system. It's performant. It's reliable. It's lean and mean. Modern, it is not. It's kind of impressive. It's worked so well for so long.

Yeah, and I'm grateful. You know what else worked so well for so long? HFS. Extended FAT. FAT32. Also, very fast file systems, low overhead, have served us well for a very long time. But they are inappropriate for a modern workstation or modern server workload. They're inappropriate.

I think Extended 4 is inappropriate as well. I love what it's done for us, but I think we as a community have fallen behind. Some distributions, like Fedora offer ButterFS, OpenSUSE, but before that, RyzerFS. Of course, Ubuntu has made strides by offering ZFS as an installation option. But fundamentally, Extended 4 lacks certain features that an iPhone has, that every Mac that ships has, that Windows has had since XP.

Things like shadow volume copies, you know, in other words, snapshots and copies and sub volumes, maybe even compression or encryption. Things that protect users, protect the hardware like their SSD and reduce write. Things that are actually useful in a workstation environment like subvolumes for home directories and things like that. Right. Things that can enable some of those workflows that we've talked about when we talked about like bulletproof Linux setups. Or like on a server, you know, having the ability to send your file system is an extremely useful functionality. And ButterFS and ZFS have this functionality, but Extended 4 does not. And XFS has been around a long time. In fact, XFS is older than Extended 4

and Extended 3 and others. It's a classic. It's been around since the 90s, but we still, as a community, haven't embraced it for whatever reason and made it the default file system, even though it continues to be developed. Extended 4 remains the champion.

And I think, as a result, basic functionality is now lacking in both the desktop experience and in the server tooling experience. How do you build a standardized way, say, in the Dolphin file manager, or maybe in SystemD, to recover files and restore files if 90% of the file systems that are deployed out there, by default, don't have that functionality? You can't. You're not going to build the ability to recover files in the Dolphin or on GNOME files unless there's a standardized API that maybe 60% or more of your user base could potentially have access to.

SystemD isn't going to have this built in at a level that is extremely accessible and deployable until 90% of the servers deployed have this file system. So like even though Linux has access to file systems with these advanced features because, I don't know, they're not tested or they're not compatible or they're not trusted, whatever reason they're not fit in as the new default. They're just not default, yeah. Yeah, the result is people just don't trust that we have those features available or willing to assume. Why would you put the time into developing, say, a file recovery mechanism like shadow file copy that's been in Windows XP since Windows XP?

And, you know, macOS, I used to, on this here show, give them a hard time for HFS+. Oh, yeah. And then they layered Time Machine on top of that. And then they lapped and shamed us. And they even revamped Time Machine now to take advantage of the snapshot capabilities of their file systems. They even modernized Time Machine. They rolled out a brand new file system to all of their devices in production and then rewrote the backup tools that use it,

rewrote the installer and fundamentally how macOS installation updates are done, while we've sat on our hands and done nothing. So it's embarrassing from a workstation standpoint, and it's kind of embarrassing on a server-side standpoint. And there must be some reason for it. There must be some reason for it, but I hope, and I don't know if it's it, but my hope is that perhaps BcacheFS will fit into this spot. What do you think, Wes? You mean the new default Linux file system? Someday, when it's ready.

When it's ready. We're not there yet. But BcacheFS is what we're taking to calling a third-generation file system. And I think you probably explained this the best, Wes. Well, I mean, if you just kind of think about cycles and technology, we had the generation of file systems that EXT4 and XFS kind of came from. And then ZFS came on the scene, and it was one of the first copy-on-write file systems

that was widely deployed and used and trusted and taught us the whole next set of features that you could expect from file systems. But, what, ZFS got started sometime in 2004-ish? I don't know, something like that. In the bowels of Sun Microsystems. Right, and ButterFS, I guess it was, I don't know exactly when development started,

but maybe 2008. It was merged anyway into the kernel in 2009. But BcacheFS has had time to learn from those things, right? Like, you can take a look at how modern file systems have been designed, the existing stuff, what's worked, what hasn't worked, what issues they've run into, because, I mean, CFS and Butterfest are both great, but, you know,

their designs have had to make compromises, and in today's era, maybe you want to sort of remake those compromises or take another whack at it, and you can have new simplifying assumptions, or at least look to see, like, if I was going to do this from scratch with these features not bolted on but designed in from the get-go, how might you do it? And that's at least some of what BcacheFS is trying to do. The challenge really is that

BcacheFS is moving quick, and I think, you know, the documentation can lag behind development, especially as things are getting added now that it's been mainlined. When you look at the feature set today, I think it's safe to say it's got copy on write. It's got extended ACL support. It's got subvolume support. But there are features that are missing that we would consider to be – when I'm sitting here going on about standard table stake features, there are some still from BcacheFS that are not yet merged in.

Yeah, right. Send receive, for instance. That's not there. Snapshots work. Snapshots are doing great seemingly, right. Send receive, for instance. That's not there. Snapshots work. Snapshots are doing great, seemingly, but not send receive quite yet. Erasure coding, a.k.a. RAID 5, 6 type things, that's not yet in, but coming soon.

I think also there's probably a cohort of ButterFS users that have never been super thrilled with ButterFS, but maybe they don't want to use ZFS because it doesn't have a kernel module. They're on a Raspberry Pi, maybe. Low-end hardware. While BcacheFS isn't yet optimized for low-end systems, I think there is still a cohort of ButterFS users that may want to migrate one day to Bcache. I think it'd be pretty easy, too, just in the sense that there's a lot of similarity between the file systems, right? They're both in kernel, so you don't have to deal with anything new there. Yes. There's also a lot of places with Bcache. They're both in kernel, so you don't have to deal with anything new there.

There's also a lot of places with BcacheFS where the user experience interface is quite similar if you're used to ButterFS. And the snapshot interface is modeled on the way ButterFS does it, so that'll be familiar to you. And then I think you might just think it's going to offer some things like better RAID if that is useful to you that your kernel file system can't really do. So we had a lot of questions like what's working, what's next. So we asked Kent to join us on the pod and he was out and about, but was kind enough to share some of his time with us. He's a thoughtful guy and truly an expert on the state of modern file systems from what I can tell. We want to share the highlights of that chat with you. And my biggest question going into this episode was, well, given ZFS and ButterFS and XFS, as we've been saying,

why do you really need yet another modern file system? Feature set of ButterFS and ZFS, but performance and reliability and scalability of XFS, if we can do that, then we'll have something. So his quick answer, his elevator pitch is the feature set and reliability of ZFS, but the scalability of XFS and speed. Kind of like I said, you know, he wants to

have something, and I think he's really close, that stings like ZFS and floats like XFS that you could have built into your Linux kernel. Yeah, I know there's some work, I'm not sure that's ready either, but work on like a no-cow path that turns off the copy on write and is aiming to be at least comparable to performance with XFS. I don't think it's there. There's a lot of optimizations Kent talks about that

are on the table that can be done that have not yet be done. So, you know, we'll see when do those actually land. But it seems like there's still plenty of low-hanging fruit out there. Yeah, I think XFS is really, in his mind, the one file system to beat right now. Not ZFS so much. XFS is going to be harder to dethrone. XFS performs and scales really well, and it's very robust. And those guys have been real professionals how they've been doing it. But it's also like a code base from the 90s.

It shows its age and areas. And if we can really get the performance and the scalability to compete with XFS, that's what's on my mind. So, Jens, I know you've used XFS in the past. I'm curious what your opinion is. Chris, you've probably used pretty much every file system. I mean, I still have XFS quite a bit upstairs on my big old scary RAID. That's what I use. I'm actually using LVM and then XFS on top of that.

I've used it in production too. It is a really good file system. It was one of the first ones that I used because it had extended attribute support, which was necessary for Windows shops for the clients that I was deploying at. Oh, right. It just has a good recovery tool set.

The thing that I think I really appreciate is I just has a good recovery tool set. Like, the thing that I think I really appreciate is I have been in situations where I've had to recover data and XFS tooling was there for me. It came through, the file system was solid. I think I agree with Kent. Performance wise, feature set wise, it is one to beat. Right, yeah, if you're not, like, relying

on some of these modern, you know, file system things, like snapshots or copy-on-write functionality, then yeah, XFS has always been the thing I've reached for. And now they've, you know, they've been trying to add some of these modern, you know, file system things, like snapshots or copy-on-write functionality, then yeah, XFS has always been the thing I've reached for. And now they've, you know, they've been trying to add some of those advanced features on, but it's an old design from the 90s that you kind of have to stretch and make work in this paradigm, whereas BcacheFS is just, you know, it's modern.

It's had time to learn from both file systems, but also from databases, which is something Kent spent a lot of time looking into. And it's actually kind of like a file system on top of a database. Yeah, that's where I put most of my effort in, for sure. And that was the dream. There's been multiple efforts to do that. Microsoft had a big one, ReFS. Like there's been multiple efforts to do that.

Microsoft had a big one, ReFS. But it's hard to make a like a generic database scale in all the ways that a file system can or a file system needs to. And so that was a pretty ambitious project. And compared like technology wise to ZFS or Butterfest, the other two main contenders feature set-wise, ZFS was able to do snapshots by giving up on extents. And it's the old indirect block scheme. And if you've done file system with snapshots, you understand why they did that.

Extents that overlap in arbitrary ways plus snaps, introduces lots of brain-melting problems. And considering it was the first mainstream, I mean, there was Waffle before, the first mainstream file system to support snapshots, it's pretty understandable. But performance-wise, it's never going to scale as well as something that's extant-based. I think, too, when you're launching a new file system post-ButterFS, there's some natural questions Linux users are going to have around, well, have you solved for this problem? Have you solved for this problem? So we talked to Ken about that. You also asked a great question about erasure coding.

erasure coding. I am really excited about erasure coding in, in B cache FS because the vast majority of it is done and the design did turn out really, really nicely. No rate hole. I'm excited to put it through more, more benchmarking functionality wise, like all the core algorithmic problems are done.

Uh, getting it to play nicely with copy GC. It was a big endeavor, but actually stabilizing a feature and getting it really ready for prime time, that's like as long as developing it was in the first place. Now, I need a little lesson here for those of us who are less file system initiated. He mentioned, well, you mentioned eraser coding. Can you give us an insight into what that is? And he mentioned also the RAID hole. I have guesses.

Yeah. So this is a classic problem that ButterFS has been struggling with for a while. Actually, Kent kind of goes into why it's been a big problem for ButterFS. And it's sort of a design issue that BcacheFS just simply isn't going to be subject to.

Like you mentioned, the RAID hole has always been the big problem in RAID systems. When you're doing RAID 5 and 6, this doesn't apply to be subject to. Like you mentioned, the RAID hole has always been the big problem in RAID systems. When you're doing RAID 5 and 6, this doesn't apply to RAID 1. And you've got these stripes of unrelated data, and you've got parity blocks that let you reconstruct

any of those fails. Well, if you do a partial overwrite to some of the data in a stripe, it's fine if you're writing a whole stripes worth at one time, but that doesn't usually happen because your writes are usually not aligned so nicely. And the stripes are also quite big. Then there's always a window in time where the data that you updated is inconsistent with the P and the Q block

because you can't do writes to different drives atomically. with the P and the Q block because you can't do writes to different drives atomically. There's always going to be a window where either your redundancy information got written and the new data didn't get written, or the new data did get written and the new redundancy information didn't get written. And you think, oh, that's not so bad. And the data that I wrote is possibly inconsistent if I crash, but that's just the data that i wrote is possibly inconsistent if i crash but that's just just the data that i wrote i mean most applications can can deal with stuff that they were in the in the progress of writing that wasn't f-sync being corrupt but actually no because this causes

if you then crash and lose a drive so you have to do reconstruct reads, it will cause you to reconstruct incorrect data for everything else in that stripe that shared the P and the Q with. So it's a really nasty issue. And this is the fundamental problem that ButterFS has been struggling with. They're close to a fix. I think they're testing it. But they, you know it. It's really burned

their reputation, I would add. Yeah, right, and it's one of the things that CFS has had working, and it's a reason to use CFS. Brent, to your question, this is when you want to use something like RAID 5, RAID 6, where you're not doing a full mirror type setup because you want to get more efficiency, you want to be able to store more

with less having to use fewer disks, and so you use these parity calculation where you can write these parity bits instead of more replicas. And you could do it with ButterFS, you could do it with mdRAID, but you're going to run into these write-hole problems depending on the setup, and ButterFS just didn't have an answer for it.

So Linux hasn't had a very good default answer for that problem unless you want to go with CFS. Which means, as far as file systems built into Linux, we haven't had a great answer. The reason why BcacheFS doesn't have this problem? Well, we're a copy-on-write file system. Our whole file system is based on not overriding existing data in place. So why would we even overwrite a stripe, a part of a stripe in place?

We just create big stripes, same as we normally write to buckets, and we'll create new stripes as needed, but we won't overwrite our existing stripes. It's kind of integrated with our garbage collection. The one trick to that is that as we're building up stripes, we'll always be building up full stripes, not small stripes like ZFS does. Data won't be redundant right away because we can't write out the P and the Q until we have a full stripes worth of data. So we just replicate the writes initially. And then as soon as we've built up a stripe, then we discard the extra replicas. And then the cool

thing about that is that if nothing in the system doesn't fsync, forces a flush, then we can overwrite those buckets where we wrote the extra replicas to for the next replicated write, and it'll only cost us bus bandwidth. There will be a very small performance hit to these extra replicated writes. Yeah. It's a neat idea where you basically start by doing an extra replication, so you do have to have more copies than you need, but only until you've sort of spooled enough writes where you can write out the P and Q parity bits, and then you fix everything up and the extra data goes away,

and you get a really nice on-disk layout. And if things go well in terms of F-Sync, you kind of get to use the same area on disk to just keep doing that over and over as you're writing to the disk so it can all be very quick. Collide.com slash unplugged. If you're in IT, if you deal with security, you have to hear this. You've noticed probably a reoccurring pattern, especially over the last few years, as BYOD has become more common. But often, by no fault of their own, employees or their machines, their devices are a common threat vector. Phished passwords can also be a huge problem.

Stolen credentials are just unfortunately more and more common. And of course, they don't even realize this happened most of the time. It's not their fault. I think in a lot of ways, the technology so far has really failed them and they have inadequate preventative measures. And perhaps you even as a corporation have policies and procedures in place to make sure that they have these tools. But there hasn't been a great way to enforce it that isn't a massive burden on IT. That's where Collide comes in. It is a solution to this challenge for those in security or in IT. Collide ensures that only secure devices can access your network and your apps. So you say goodbye to compromised credentials and phish credentials because Collide's checking

that stuff before they can connect. And don't worry about a diverse operating system fleet. Collide gives you one dashboard for Linux, Mac, and Windows. And Collide will help end users solve problems. So if they run into something where they're out of compliance, where they have phish credentials, maybe they don't have the right patches, maybe they don't have the right software installed, maybe they don't have whatever it is, Collide can help guide them through the process of fixing that without putting that burden on IT. By using your messaging system, by using your processes and your procedures, Collide can help

them figure it out on their own. They're smart people. They don't need to message IT for everything. It unburns IT. It gives you more management, and it helps end users solve their own problem. It swaps that whole dynamic between end users and IT as well. That's huge. I wish I would have had this when I was doing IT.

So go check it out and support the show. They got a demo over there. If you watch that, I'll kind of explain it to you a little bit further. It's a great way to see how it works and support the show by going to collide.com slash unplugged. That's K-O-L-I-D-E dot com slash unplugged. Collide.com slash unplugged. Well, all that theory sounds nice, but if you're a practical file system user out there, you might be wondering, what types of workloads is BcacheFS actually ready for

or even meant for? Workload-wise, people are throwing database workloads at it. There's a guy out in China who has been throwing really crazy MongoDB workloads at it and pushing snapshots really hard. Yeah, ideally, any workload. It should be a truly general-purpose file system. Scale is just fine. I know I've heard people say that ButterFS does not really scale past maybe 100 snapshots.

maybe 100 snapshots, the FSEKI algorithms do not handle the references between from keys to multiple keys and other snapshots very well. But BcacheFS should scale to as many snapshots as you can take. It's got writable snapshots. The compression is

really good. People have been really happy with that. Oh, I heard that and I said, oh, wait, wait, I'd like to know more about the compression. But yeah, the way BcacheFS does compression is extant-based instead of lock-based. So we're doing compression at 64 or 128K granularity, which is quite a bit better than is typical. And so our compression ratios are quite a bit better than other file systems that do compression. We follow the process of these projects getting upstreamed. And at times there are moments of drama.

And a lot of times it's when the developer has to interact with the kernel maintainer. Well, in this case, that means Kent has to interact with Linus and he got feedback from Linus, which he incorporated. So we wanted to know what that experience was like. Oh, that was, that was stressful. Um, yeah, the it's, it's stressful partly because no one really knows what the process is or should be. The process is always kind of ad hoc consensus based, but all the people need consensus from are kind of angry and irritable and don't want to be bothered. But that's also as it should be. The bar should be high. We do

want to have high standards, and we do want to be pushing people to write the best code that they can. And there's questions like, who's going to be on the hook for maintaining this thing down the line? All the other people in the VFS that have to deal with file systems. If you're hacking on VFS code, which I've done in the past, and then you have to update all the file systems or all the block layer drivers, then code that gets added,

it's overhead for you. And if that code maybe is a pain to work with, if the tests aren't there, then those are very real issues. I think another common concern that I've heard people bring up is, well, is this just one guy? Right. Just because it's merged in the kernel doesn't mean anyone else in the kernel knows how or wants to work on it. Yes, that's forefront of my mind. Now that I'm upstream, it looks like funding is going to be getting easier.

I had a decent amount of funding for for like six years from a company in germany there was enough for me to work on it full-time and then uh tech downturn happened actually video edit uh like the the strikes in la was what did it for them and had to cut pull back so now i'm just on my patreon funding but now that i'm upstream, I'm getting more attention from like the NixOS guys and foundations in Europe. And there might actually be money for spinning up an actual team and getting some younger guys involved. That would be great. Get some young blood into the kernel, you know, learn from somebody like

Kent who really knows his stuff. Did you catch it, Brent? Did you catch what he said in there? I heard that special N-word that we love around here. NixOS, I think. I certainly did, too. Yeah. I was like, oh, now what's going on with Nix? Also, just the NixOS people have been great to work with. They've been kind of at the forefront of

getting BcacheFS out there and getting integration stuff sorted and just doing stuff without B having to get involved at all. They've been great to work with. And now there's people from the foundation that funds NixOS that are talking about sending some funding my way. That'll be really cool if that happens.

Yeah, that would be. Wouldn't that be great? Two great things working together. And Nix has been a great way to experiment with BcacheFS along the way. Oh, yeah, it makes it really easy. So it's like, before we talked with before we spoke with Kent, we decided

well, let's take a really quick look at it. We didn't do much. We just looked at the tooling and looked at it, but Nix makes it really, really possible. He also was talking about how he wants to rebuild his test environment on Nix to make it a lot cleaner. He has a lot of good things to say environment on Nix to make it a lot cleaner. He has a lot of good things to say about the Nix folks.

We did not prod, and it just came up in discussion. You know what else? Well, actually, this one maybe had a slight prodding by Wes Payne, but this also came up in the conversation. What other reoccurring topic? Oh, definitely be an early one, yeah. I am all for Rust. I dipped into Rust years ago what eight or eight or so years ago and just fell in love with it pretty early on it's in my opinion the the biggest advance in

that we've had in systems programming for for decades the dream to me i like i i hate debugging i want to be able to finish a chunk of code and move on to the next thing and know that the code that I wrote is done, is correct, that I'm not going to have to come back to it. Rust with the borrow checker is able to prove the correctness of huge swaths of things that we had to analyze manually before. It makes huge classes of bugs go away. I'd like to know if people like it when we go this deep. If when we get this technical, maybe people are glazing over at this point. But I really enjoyed picking his brain about this

stuff. And he has a wealth of knowledge about other file systems as well. There's some pretty decent docs over at bcachefs.org including a user manual which goes into more detail if you want it about the architecture underneath the file talk, but I love hearing about the people working on these technologies as well. And so often we just concentrate on the technology itself, but there's a real human writing this stuff, it turns out, and someone who's passionate about some of the tools we are. Yeah, definitely.

Yeah, I feel like Kent is unplugged people. He would so fit in at a meetup. It's ridiculous. So when we got connected, I mean, first of all, he talked about NixOS, right? He's a Rust fan, and he's a file system developer. Obviously a Linux user. Yeah. So those are all

checking the boxes uh to be uh you know great to talk to at a meetup but when we got connected to him he was on video and uh he was hanging out chilling camping across the country in the back of his subaru uh when it got merged i was uh in badlands national park way down a dirt road camped out over just gorgeous scenery middle of nowhere he's just doing it you know it's like it's not like a it's not like a super uh glorious setup but he's just making it work he's changing the linux file system landscape from the back of his subaru while he's traveling I guess he was in Portland not too long ago, too, so we just missed him. Maybe if he ever makes it out again, we'll take him out, show him a great

time. Super, super grateful for Kent for taking some time to talk with us from the road. I think it was snowing behind him, too, so I hope he's staying warm out there. We should have asked him what he's got for heat. But it was a great chat, and it really made me excited for the future of BcashFS. I think it's, you know think it's in the keep an eye and test it stage.

Yeah, exactly, right? It only just got merged. I know like 6.8, there's already some performance improvements coming down the pipe. So probably be a few releases while things stabilize and check out. But yeah, try it out on some test workloads and see what you think. The on-disk format should be solid, solid is stable has been stable for years i mean he essentially had some of that the basics of it figured out in 2015 when he started it and some of that really hasn't changed yeah right kind of grew out of bcash underneath and then uh this

it's got the the b tree layer and the transactional database on top of the b tree and then you got the file system unpluggedcore.com use promo code 2024 and take three dollars off a month forever for your membership you get double the content you could get the live stream which uh as of right now is a two hour and 16 minute ish episode of the show uh you support each production directly if you use the promo code 2024, you, for a very limited time, get that deal. Now, I checked on this morning. Two spots are left.

And I thought, what are the chances? Somebody's going to use that maybe on the live stream. By the time this publishes, nobody will get to take advantage of it. Thank you, everybody, who did, by the way. So I'm going to add 10 more slots just for people that didn't get a chance to hear it last week. 10 more slots. If you use the promo code 2024, it'll take $3 off a month forever. We'll put a link directly to that. And then you get the live stream version,

the bootleg, or the ad-free version that Drew still cuts together so you get that nice tight production. But they're going quick. It applies to new purchases, existing members, or if you're reactivating an expired subscription, or if you want to upgrade to a network membership, it'll apply to that too, which then you support all the shows. Of course, you can always boost in as well. The nice thing about that, of course, is that immediately goes directly to the people involved in the production. We also kick back some to the Podfirst developers and the Podcast Index as well to help support those projects. And we're counting those towards our trip to scale. Thank you everybody who has become a member, though.

You can become a member at unpluggedcore.com and of course we have links on our website. Now we've been doing some crazy stuff here this year and we'd love to hear if you've been joining us, say, on the 32-bit challenge or what you think of these new file systems.

Linuxunplugged.com slash contact for that, or you can boost in. Boost to great. Thank you, everybody, who does boost in. And we have a manual boost this week. Our first ever on-chain boost, gentlemen. And it's probably a good thing because of the size of this whopper and this is helping us get to scale this got us a big way to our goal that we talked about earlier today came in via matrix from imappy on january 8th for 3 million 225 275 cents $75. Hey, Rich Lobster! Officially making iMappy the largest booster in the history of the Unplugged podcast.

Giant boost comes in a unique way. Yes, he writes, first time, long time, and long time Linux user as well. Well, thank you. Remember when kernel 2.0 came out and MUDs were still cool. Yes! All right. Heck yes. Oh, man. 2.0, come on,UDs were still cool? Yes. All right. Heck yes. Oh, man.

2.0. Come on, man. That's not that long ago, right? Anyways, he writes, I've been a listener of your podcast for about six years now. The insights and advice I've gained have been invaluable as my role as a CTO in a governmental institution and in managing my own small business. I love that he's a CTO in a government institution and he's boosting us. That is a good sign. That's making me feel good. He says, this boost is a token of appreciation for the tremendous work you do. If it is possible, it would be great to have the show focus on crucial topics from the

perspective of a large institution, maybe central identity like free IPA and administration. Enterprise things like identity are an interesting topic for sure. Anyways, he says, keep up the excellent work, and I look forward to the return of Linux Action News one day. If I remember correctly, this boost will be the largest ever. Correct, you are, sir. And I challenge everyone to beat me. Challenge on.

As you should. Thank you so much for that tremendous support. That's very, very, very much appreciated. And that played a big role in our over 90% milestone this week. So very much, very much. And then, geez, wow, very much also appreciate him deleted coming in as our second baller for 623,456 sats. Hey, Rich Lobster! He says third place is just a fancy word for losing.

When I did my 32-bit challenge, I allowed myself a, quote, unlimited budget. If there was a problem that could be solved with money, I didn't hesitate to spend it. I like this approach. That meant, you know, big upgrades like on SSDs that use PATA and upgrading from one gig to two gigs. The difficulty of sourcing these upgrades was immense, and I had to buy from some very shady vendors, one of which still hasn't delivered my order. Oh, no. Systems that are focused to continue to run 32 but only are going to become extremely difficult to support. Systems that are forced, he says, to continue.

I know. And maybe you could become a specialist. I felt like, and I still feel like my takeaway was, and I'd be curious to know if you guys feel this way, we could not probably do this challenge at the end of this year. Like, that was maybe the doors closing. Did get that sense too oh yeah i mean like the software support is limited or a hassle oh yeah and then the hardware support is just gonna keep degrading even if people elect to continue software support and it sounds like where there was support you often had to build it yourself and the hardware couldn't really handle that in a reasonable way. So it's a loose-loose. Chris, I did notice that was multiple boosts,

and one of those was a 1, 2, 3, 4, 5, 6. So the combination is 1, 2, 3, 4, 5. That's the stupidest combination I ever heard in my life. Thank you, thank you. Nice catch. Rotted Mood came in with a live boost for 100,000 cents. Eight lines of shimmering cement running from here to Pasadena. Just says scale boost. And thank you. Scale boost. We appreciate the help.

We do. If you do boost in with a scale boost, too, let us know if we're going to see you there. I'd love to put a face to the boost name. Let us know if we're going to see you there. I'd love to put a face to the boost name. VT-52 boosted in with a total of two boosts, 96,459 sats, and one of those was a row of ducks. BSD Bake Off with Brentley. When?

Hmm. Hmm. I mean, I don't know if the audience appreciates the scale to which I find BSD annoying. This would be one of the hardest challenges. I also proposed a challenge to the boys over the break and neither one of them bit even a little bit. I'm just curious when's the last time you actually gave BSD like a good run? In my nightmare last night? In my nightmares. In my nightmares.

Reasons? Like if you had to give top three reasons? In my nightmares. In my nightmares. Reasons? Like if you had to give top three reasons? It's like if you were born on Earth and then you tried to live on Mars. It would be just possible, but constant friction.

Constant friction. See, I was enjoying the – I liked that challenge. I was down to do it. I was just having such an easy time with my part of the 32-bit challenge. Yeah, Mr. Mangia over there. So I think a bake-off sounds great. I mean, I got to be convinced. I don't know.

I really. You know, that free BSD kernel has some neat stuff in it. OpenBSD does some neat stuff too. We'll give you better hardware. If somebody gives me something I could do and some ideas and maybe what to try. Make a router challenge. I like my idea a lot better than the bsd challenge but it's fine it's fine thank you for the support i don't know about a bsd bake off though now vt's second boost due to some weather related isp issues i haven't been able to

finish customizing it but i've got a bbs up and running if folks are interested. You can tell that pebcac.lol, which is a great URL. I don't see it. It's not responding. Is it up? Is it up right now? Due to some weather-related ISP issues, I haven't been able to finish customizing it. So maybe it moves. Oh, I'm getting something.

Look at this. Synchronet BBS for Linux version 3.19. All right. Awesome. pebcac.lol, yeah. finish customizing it so maybe oh i'm getting something look at this are you synchronet bbs for linux version 319 all right awesome yeah i think this is our first boosted in uh claim of tell uh so it's pebcac.lol p e b k a c dot l o l some lovely ascii are in the banner here that's great nice job vt you crushed it i think that means we now have a crowning official BBS for JB, don't we? You know what's wild is VT boosted that on the 8th. And the episode went out on the 7th. So he did that.

Wow. That's impressive. He did that. Wasted no time. What's even more wild about it is the boost came in at least our time at 8 a.m. And the episode had only been out probably for like 12 hours at most. So he got this PBS. Maybe he already had it.

I don't know. But well done. Well done. Bola Para comes in with 60,000 sats using the index. Here's a small amount to get the gang a little closer to scale. And if you end up at Texas Linux Fest, I hope to come say hi to y'all. Great shows. Mmm, California.

Beautiful. Thank you. We're going to get there. Helping us ticket one episode at a time. We're going to be showing up like bosses. People will be like, oh yeah, how'd you get here? We'll be like, sats.

I paid for that Linux limo. That's what people say, right? When you show up at a place, they'll be like, hey, how'd you get here? That's a pretty standard question. Well, you'll be wearing your Bitcoin shirt they'll be like, hey, how'd you get here? That's a pretty standard question. Well, you'll be wearing your Bitcoin shirt. Yeah. Well, no, you know, it's not great OPSEC, actually.

Not the best. Not great. Space nerd Mo comes in with $55,555. I love that name. Coming in hot with the boost. Kicking in to help with the journey to scale. Thank you very much. Eight lines of shimmering cement

running from here to Pasadena. Mixie Beep boosted in with 50,000 sats from Castomatic, simply saying, scale boost. Mmm, California. Beautiful. That's very much appreciated. We're getting there. We're going to get there

and then I'm going to miss all these boosts. It's going to be a thing. I still love hearing from you guys. You have to keep boosting in. We're going to get there, and then I'm going to miss all these boosts. It's going to be a thing. But still love hearing from you guys. You have to keep boosting. But we really appreciate the support. Knights62 came in with 50,000 sats and says, this boost is to help contribute to getting to scale. Yes.

Eight lines of shimmering cement running from here to Pasadena. I wish I could meet you all there, but March is packed already. I understand. Maybe next time. Thanks for all the awesome content. By the way, I joined the Jupiter.party so I could enjoy the Fiat Fund coupons too. Knights, I appreciate the double support and helping us get to scale. We'll try to do the best we can, conveying the experience so you can kind of get a sense of what it's like.

I always feel like we don't quite nail it, but we're going to try our best. I feel very inspired by all the support. Got any ideas of how we can document it for you? Boost in. What would you like to know? Yeah, definitely let us know. Hybrid sarcasm boosting in with four boosts for a total of 41,000.

But I think it's important here to note that there are three empty boosts of 12,345, or one, two, 3, 4, 5. And then a final boost for 3,965 sats that just says, they've gone to plaid. He sent us the entire Spaceballs boost stack in one boost and unlocked it. The hell was that? Spaceball 1. They've gone to plaid. Our first gone to plaid boost. Well done, Hybrid Sarcasm.

Well done. The whole stack right there in one go. Jagbert boosted in again this episode. 30,066 Satoshis. Time to do my zip code boost. Also, glad Brent got the pronunciation of my username right last episode. Jagbert. Well done, Brent. Well done. I just really, I guessed and I happened of my username right last episode. Jigbert. Well done, Brent.

Well done. I just really, I guessed and I happened to get it right. Well, if this is a U.S. zip code, it looks like it's a postal code in Cobb County, Georgia, with cities including Marietta. Well, hello, Cobb County. Thanks for boosting in. Yes, thank you for boosting in. Let us know if we got that right or totally off base. Sometimes we got to go to the whole globe and I get confused.

It's a big map. It's a big map. I'm always impressed how quick you can do it. And you always hand it to me upside down. Yeah. Remaking Eden comes in with 12,191 sats. Okay, I've been waiting for my time to do a postcode boost. We also have two letters here.

The first two numbers are a letter and the second two numbers are a letter. Okay. The first half of my postcode, it's the first half of my postcode they write. I do the whole thing, but we need to take out a considerable loan. Understandable, understandable. Really enjoyed the 30-minute challenge. Keep up the great work. Remaking that.

That just made me laugh. So, Wes, i see you're doing the math over wow look at you i don't know if this is right um check my math here boys uh but if if it's like saying 12 is a letter 19 is a letter and then we just got this this one here at the end from 12 uh 191 okay 12 could be l and then that would make 19S. Yes. So we'd have LS1, which seems to be a postal code in Leeds, England. But then, I don't know, is that the full code? Because that could be wrong, because, you know, Remaking Eden notes that it's half the postcode.

Southern Fried Sassafras comes in with 1, 2, 3, 4, 5 Spaceballs boost. 1, 2, 3, 4, 5. Yes. That's amazing. I've got the same combination on my luggage. Using the podcast index, and they write, one negative of catching up while driving is that I almost forget the topic I wanted to boost in on. Barring that, keep up the good work, and may the Schwartz be with you.

Yeah, I understand. I get it. I get you. Our dear friend Cessna Mike has a little trick for you that Chris really loves. Oh, I do love this. And I don't know why I haven't actually adopted it, but when Cessna Mike is flying, and I imagine other pilots as well, he gets a notepad and it's got a strap. And it straps around his thigh and it's right there on his leg while he's flying. Ready to go.

And he can just note stuff down. I imagine the pen attaches in some clever way. I didn't really analyze his general crotch region, but that's where it's at. And I think Southern Fried Sassafras and myself would both benefit from such a thing. Indeed. If we remember, we'll look it up in the post show and I might get one. But thank you for the boost. Now, you've got two thighs, Chris.

What else would you strap to the other thigh? Well, I've always got to have my gun, you know. Maybe an O-Droid? Yeah. In a gun case, though, right? Yeah. Okay. All right.

Cool. Yeah, he's got to be cool. AutoBrain boosts in with 2,000 cents. First-time booster. Thanks for the show and the community. My first Linux install was a dual-boot Yellow Dog Linux slash Mac OS 9 on my wife's Apple PowerPC home graphic design production machine that she graciously let me play on outside of work hours. Wow. I sympathize with that.

Auto brain. Well done. Well, it does feel a little risky, too. Me too. And you know what I like to think, because it makes me feel good, is that users like you and I that were pioneering Yellow Dog Linux on these machines, we validated Yum, which was then brought to Fedora and then later became, you know, DNF or inspired DNF. So, like, thanks to basically Otterbrain and me, we have DNF is what I'm saying. Wow. Yeah. Yeah. Right? That checks out. That checks out. Kira Sia comes in

with one, two, three, four, five sats. So the combination is one, two, three, four, five. That's the stupidest combination I ever heard in my life! And this is a great suggestion. They write, I'd say for the scale sound effect, maybe get Lady Joop's Accelerating.

Oh. She has the quietest engine for something that big. And if you get down to like the pipe, you can hear the lug, lug, lug, lug, lug, lug. Like there is like an old school lug, lug, lug, lug, lug, lug, lug, lug, lug, lug, lug, lug engine in there. But man has Ford muffled that thing. Rightfully so. It's very quiet and when you're so you can like crawl through the campground and not make a noise unlike the diesel pushers which man they just wake up the whole neighborhood but i would love to get a sound over accelerating

because still when that when all when that v10 gets going and the that thing's designed to rev so it gets up to like five thousand six thousand know, just for just getting on the freeway and it holds it there. It sounds like a beast. I love it. Now, SWAT came in with a row of ducks. So, hey, since you like Winamp. Wait, Chris, you like Winamp, right? Why do you like it?

Yeah, I do. I feel like it. I like Winamp. I feel like it's, you know, taking care of certain things in the behind region. Lavas. You got to keep, you know. Yeah, you do. You know what I'm saying. Don't mess that up.

Winamp, which also brings back fond memories for me as well. I wanted to share a find. Jordan Elredge made a web re-implementation of Winamp so we can enjoy the original version on all OSs, preferably via Firefox, of course. Although XMMS was not so bad either. That's pretty nice. And it's up on GitHub, too, if you want to check it out.

And it even has like a windowed mode and a little desktop environment that kind of looks like old school Windows. Very impressed. I love the name, too, WebAmp. WebAmp, right. Yeah, that's probably good not to actually use the real name. They've also got some themes right here. There's a Winamp Skin Museum. There's skins right on the desktop that you can go with.

This is nice. Yeah, this is well done. I would say, indeed, that is taking care of the llamas behind region. D3Bot comes in with some space balls. One, two, three, four, five. Yes. That's amazing. I've got the same combination on my luggage.

And they're using Podverse, the GPL cross-platform podcasting app. And they say, I had to change my luggage because of that soundbite. I know, right? It's embarrassing. It happens. Just go the opposite way. Yeah. Who's never done that?

Yeah. For real, though, thanks for the membership discount code. I've been Occasional Booster, but I wanted to get the ad-free feeds and occasionally the wacky unedited live feed. Occasionally wacky. That's the new brand name. It's a wacky one this week. It is. Thank you again, Dexbot.

And I hope you enjoy the live feed this episode. I definitely recommend checking it out. Thank you to everybody who boosts in. We had 16 boosters this week. Now, across the standard boost system, we still brought in a remarkable 1.1 million sats. Thank you, everybody. That's great by any standard. It is absolutely just a huge, huge remarkable thing that we're actually going to get to this milestone.

I just didn't know if we could do it, and it is really great to see it. So thank you, everybody who does boost in or has been streaming in. I see you all in Han or streaming us right now. Really? That's great. Somebody's listening. They don't even know we're talking about them, dude. And they're streaming us sats. That's so cool.

So thank you, everybody, who supports us either with a boost or by membership or by streaming those sats. Now, we did get that on-chain boost. So before we get there, I just want to say thank you to all the traditional boosters. You're doing a good job. And then, thanks to IMAPI, we had that 3 million sat boost. That put us to a grand total this week of 4,186,259 sats raised. That is really remarkable, and it does indeed

win ass. It really whips the llama's ass. Thank you, everyone. At this point, I think it's a lock. I would love to get to our milestone just in case the price slides a bit, just in case gas goes up a bit. I would love to make it to that milestone, but I think at this point, one way or another, we're going to make it happen. Even if we're personally thrown in or whatever, like we're going to get there at that point. So it's a walk. Yeah, exactly.

I'd love to complete that milestone so that way we have that insurance policy. We have that safety. We're not taking that risk. But, man, this has been really great to see. And we are coming up with another solution for Texas Linux Fest. We don't want to come and ask for every single thing. We're trying to be really respectful about that. And this is a value for value production.

It's a fantastic way. It's not only are you supporting the show, but you're kind of helping us get to that next big content thing that we can then turn around and make a show out of for you. So it's kind of a great investment in future entertainment

and hopefully information, if you will. Now we have all kinds of links in the show notes. So go over to linuxunplugged.com slash 545. No special pick for you this week, but we do have a question. Either write this in on the contact form or please boost in. Do you agree? Is it time to

replace Extended 4? Practically everywhere. Not necessarily everywhere in every use case. But is it time for Extended 4 to retire and for something else to come in its place? Or maybe I'm wrong. I have a sense I could be way off on this just because I don't hear a lot of people talking about this issue.

So I'd love to know your feedback either via Boost or via the contact page. Let us know and that'll help shape our future. Are you ready to move on? Now, you know, we are live. We do this here show on Sundays at noon Pacific, 3 p.m. Eastern. See you next week. Same bad time, same bat station. And we now have the new

Jupiter Station feed. Go search in a podcasting 2.0. We have a lit feed, Jupiter Station, and you can listen live in your podcast app. You don't have to switch apps. I've always thought that was so weird. Right? Why can't we just be live right there where you're already looking at the past shows that maybe you're catching up on? Yeah. Podverse, Fountain,

I'm going to toss some recordings and clips from the live streams into the feed too so when we're not live there's still some stuff and there's jupiter station brand new just launching it this week links to everything we talked about on the website appreciate you very much for listening or sharing the show and of course for always tuning in thank you to our mumble room for helping us as well they're always great and our live stream got lots of things i'll see you right back here next Sunday. Thank you. you