LINUX Unplugged - 284: Free as in Get Out

Episode Date: January 16, 2019

ZFS on Linux is becoming the official upstream project of all major ZFS implementations, even the BSDs. But recent kernel changes prevent ZFS from even building on Linux. Neal Gompa joins us to discuss why it all matters. Plus some surprising community news, and a few great picks! Special Guests: Dalton Durst and Neal Gompa.

Running Linux on a toaster? Sure, that's cool. But what about a computer mouse that's a computer mouse? No, I mean it. Hey guys, how's it going? So what you're looking at is something I ingeniously named the computer mouse. This is a completely functional computer inside of a totally 100% completely normal sized mouse. Basically, I designed a mouse in Fusion 360, printed it out, and then added a Raspberry Pi Zero, a 1.5-inch screen, a battery, and a keyboard.

Yeah, he said a screen, Wes. This mouse has a fold-out screen, and this thing runs Linux. How cool is this? I really think what you're going to really use this for? Winning nerd battles. Sure, you're having a great LAN party. No one else has a computer inside their computer mouse. Right. You show up, they're like, hey, where's your computer? I'm like, this is my computer. It runs Linux. This is Linux Unplugged, episode 284 for January 15th, 2019. Welcome to Linux Unplugged. It's currently freezing.

It's you-know-what-off down in the heart of Tejas. My name is Chris. My name is Wes. Hello, Mr. Payne. Now, we have a heck of a show. It's good to be back. After getting all married up and traveling down to Texas, we put together a show packed full of community news,

including some really surprising stories this week. Dalton will stop by from the Ubuntu Touch Project to tell us about their new release. And then Neil joins us to explain and really put into context all of the hubbub going on around ZFS on Linux. How the BSDs are rebasing their implementations on the Linux version and the changes the kernel maintainers made this week to block the ZFS module from working with Linux 5.0. All right, it's amazing, and there's a lot going on, and there's a ton to get into.

We'll play that for you in a little bit in the show. And then, if we don't go too long, we have a big batch of picks. I'm going to tell you about my trip down here a little bit in the show. And then, if we don't go too long, we have a big batch of pics. I'm going to tell you about my trip down here a little bit, and then we'll put it all, wrap it up in a bow, and send it off to the internet. But the only thing missing from what would otherwise be a perfect recipe would be participation from the audience.

Some crazy idea where a group of Linux-loving people could just join an open Mumble server and participate in our show. In fact, why don't we even do that? Time-appropriate greetings, Mumble Room. Hello. Hello, everybody. Hello.

Hello. Hello. Thank you for making it even on Brexit Day over there, on Brexit Vote Day. Oh, it's always Brexit Day over there. We promise this is the last time we'll talk about it today. I don't know about that. Oh, yeah, today, yes. Oh, Wes.

So thank you to you and Joe for covering the show last week and talking about Fish Shell, my favorite shell ever, as I made clear in the interview. You sure did. You guys did great. I got a lot of good positive feedback on that, and I appreciate it because it gave me a chance to, well, I head down to Texas, you know, as I said two weeks ago in the RV.

And on day two, we hit some mechanical troubles. Oh, no. Of course you did. Yeah. It was a couple of, like, a couple of different issues. But it all kind of came together to really cause, like, a series of unfortunate events. I guess I'll put it that way, including a popped water heater and all kinds of things. But we persevered and got to our spot

where we had our marriage ceremony, and it turned out really, really nice. We had a great marriage ceremony under the stars, and then it completed just as the sun rose in the Tucson desert, and it was right between storms. A couple of days ahead of time, it had snowed. And then a couple days after we left, it was windy and rainy and we nailed it. Just perfect weather.

It was wonderful. Had great cellular connectivity too in the desert. I had 30 megabits down and like- Only you would mention that in the same sentences. Oh yeah, you know, I drove through this desert on an amazing trip got married and the connectivity was great these things matter that puts it back on the we'll go we'll return to list you know what i mean like if it's got good connectivity all right now now it's a contender it's a candidate now you're right though i am such a nerd but i meant it meant really good connectivity which meant we

could keep up to date with what was going on in the world, and we could put together a great show today. So made it down to Texas, and it's been in the low 20s every night since I've been here. So it's been nice and cold. That's why you go to Texas, right, for that refreshing winter coldness. Yes, yeah. It's my luck. It's my luck.

I was down here earlier this year when it was the hottest day of the year, 117 degrees in Dallas, and now it's in the 20s. During the day, it gets up to the 40s. But what's fun is there's a huge crew here. We're doing live streams for a bunch of content releases, so a bunch of people are in. And it's just a good atmosphere down here right now. There's lots of temporary desks set up,

and I get to check out everybody's rigs, which is always a lot of fun. So I've really enjoyed my time down here so far. And I'll be down here for a couple of weeks doing the shows and whatnot. And so far, enjoying the heck out of it. It's really cool to have the RV parked right next to the office because I can just, in 30 seconds, I'm working from home. That is such an easy commute. Right. And like I need to go to a meeting or do something in the office, 30 seconds later, I'm in the office.

It's phenomenal. You've got your own private conference room. People kill for that kind of space. I do. People actually a couple of times now have stopped by the RV just to chill for a bit. So Lady Jeeves is getting a rep for the place to chill. And anyways, just really loving it, enjoying it a bunch. So I really appreciate you guys covering the show while I was gone.

I think it made it just, gave me just enough time to pull it all off, especially since we were compressed with that one lost day due to mechanical issues. Well, congratulations and welcome back. Thank you, sir. But that's not what we're here to talk about today. Let's get into the community news. And we're going to start off the news with a story about Apple, which might be the first time we've ever done that in the history of this show.

But they're releasing a pretty significant open source project this week that's caught a lot of attention. Now, Apple originally acquired a project called FoundationDB in 2015. And last April, it announced that it was making the cloud database open source. But that's not what we're talking about today. That was sort of done and, you know, everybody's moved on. But things are taking a big leap forward this week, as FoundationDB has announced it's open sourcing the FoundationDB record layer. And along with this, Apple is outlining how they are using their backend services with this open source software, which is something they've never done before.

It's the first public confirmation that Apple is using this open-source database to power file storage on iCloud. So we're talking about an open-source project here that's being's being battle tested by millions of iOS and Mac users. The core part of it was released for free originally. But now we have this record layer. And Wes, can you help explain what this record layer is and maybe even how Apple might be using it? Yeah, well, okay. So they call their whole back end here cloud kit because, well, of course they do. Everything's a kit, Wes.

Everything is a kit. And I mean, you want one for the cloud. Sure, that makes sense. The record layer, it's basically their own version of a traditional record-oriented database, things like the standard relational SQL databases you're probably used to, but built on top of FoundationDB. And one of the things that's kind of unique to FoundationDB is it provides something of a low-level key value store, and then they've made it easy to extend. So FoundationDB is totally separate. They've added this record layer,

which is actually a Java program that interfaces with FoundationDB and adds all these extra features. So they only have to concentrate on running, you know, all the complicated distributed system part of that. That's the base FoundationDB. And now they can add stuff like something like a MongoDB or adding, in this case, something like SQL.

Now, the other part of this that is sort of the elephant in the room is MongoDB and AWS are at odds right now as Amazon has announced AWS DocumentDB, which is their own proprietary software that implements a MongoDB-compatible API. The reason why this matters is what Amazon can essentially now do is offer MongoDB as a service,

but it's not MongoDB under the hood, but it can talk MongoDB, the version right before they re-licensed. And this is a huge, huge blow to the MongoDB project because they sell their own hosted service with very few customers, unfortunately. And so then a few days later after that, Apple makes this announcement.

The timing is pretty coincidental, I guess. It's interesting, too. I think it kind of compares and contrasts the different types of open source out there, right? Mongo has a venture-backed company behind it. They're trying to make open source software, yes, which is great, but also running and making that software is the core of their

business. Not so with Foundation, to be sure. It started that way, but now it's just an open source thing Apple's using. Downsides is you don't know, you know, maybe Apple could pivot databases at any time. The upside, they're not really relying on it. This is not how they're making money, except as an implementation detail. Really, so in summary,

what we have here is Apple releasing two more open source projects in total. You have the original FoundationDB, and now this layer on top of it, which makes it more like a relational database, which people are very familiar with, but has the file extensibility and the state support and everything that they need for remote mobile apps. And I wonder if this isn't going to be pretty successful, because A, it's being used by millions of users already. B, when you're trying

to justify it to the CEO or whoever, you can say Apple uses it. And C, it's addressing a real need that application developers have. Yeah, I mean, it's got a nice license, and they've clearly learned. It's a more modern database. If you go back and look at when it was first, after they acquired the company and open-sourced it, people are raving about it being open-sourced just because it has such a new, modern, sophisticated take on concurrency and transactions, so you get a lot of nice guarantees, and it's easy to administer. Yeah, yeah.

That's really all I have. I don't feel like I need to talk about Apple anymore in the show. I think that's probably good. Let's move on. What do you say? Let's talk about the future of GNOME MPV, which is my favorite, favorite, favorite by a mile video player for desktop Linux.

I mean, LVLC, you're great, I love you, but nothing beats MPV. And GNOME MPV was a way to wrap it up in a nice minimal GUI. But I'm not so sure about this new name. Yeah, well, okay, so do you agree with this? Good app names should really be a single noun that's related to the app's domain. Something like fragments for a torrent app or peak for a screen recorder. Because if you do, you're going to like this new name. No.

I mean, I still agree with all that. Don't really like the name. Okay, I'm going to try to pronounce it. You ready? What do you think I should? Let's see. We should all take bets. Everybody bet if I'm going to get this right.

Celluloid. Yeah. I got it. Yep. I think that's good enough. You're telling me I got it? Celluloid. Oh, I got it.

I can't believe it. I can't believe it. I don't know, man. I guess for some reason it makes me think of cellulite, which is not what I want to think of when I'm thinking of watching a video. But Noam MPV is clunky AF. And I do like

the one word thing. And at least I can pronounce it, so it's got that going for it. And it's also the name of the transparent flammable plastic made in sheets from Camerphore and nitrocellulose formerly used in filmmaking.

Yeah, right. So if we were making video podcasts in the film era, we would know all about celluloid. Yeah. Yeah, I've heard of like celluloid cylinders, and I've heard the term used before. So this is going to happen, this switch-o, will happen around the time of GNOME 3.32's release in March. like this Switcho, will happen around the time of GNOME 3.32's release in March. So if you just got the most recent version, or I should say the latest version, which was 0.1.5, then it's still using GNOME MPV. You're not going to see this yet. What do you think? So they've got a new icon, too, and that does sort of fit with the name

because it looks like a piece of film. I say we go to the mumble room. What do you guys think of this name, Celluloid? Is it Ditch It or Keep It? It's a Keep It from me. Yeah? How come? It's a somewhat memorable name and it's pretty easy to identify. And the new icon is

pretty standout from video player icons. Yes, the new icon is the bee's knees. That is really nice. You're right. They should get credit for a great icon. Okay. The Mumble Room in its entirety. No, I'm kidding.

Actually, we've got a good show. We've got Conan in there. We've got Mini Mac, Sean, Tech Mav, Tyler, Universal Superbox, a.k.a. Dalton, all hanging out in there today. And I'm going to roll with it. I bet you it'll grow on me. I thought iPad was the dumbest name I'd ever heard when they first, and now people drop it. It doesn't even affect me. It's funny how that, you know, it's kind of like music. Once it sinks in, you have a different perspective on me. I thought iPad was the dumbest name I'd ever heard when they first, and now people drop by. It doesn't even affect me.

It's funny how that, you know, it's kind of like music. Once it sinks in, you have a different perspective on it. Yeah. Yeah. So Joe and I have been grousing on Linux Action News recently that Chrome OS is going to become the de facto desktop Linux. Like the one that average consumers use, the one that's

going to be used by millions of users, the one that's sold in Best Buy and electronic stores, it's Chrome OS, right? It's Chrome OS. And we all probably should sit with that for a bit and come to our own conclusions. I waffle constantly if that's a good thing or a bad thing. But I am really happy to report on the steady progress

of Linux application integration into Chrome OS. And this week, thanks to some sleuthing, based on a code commit uncovered by Chrome Story, users of the Linux beta for Chromebooks will soon be able to search and install Linux apps from the Chrome OS launcher itself. That's a big deal. Yeah, right.

I mean, it reads, add apt search into the Chrome OS app launcher so that uninstalled Linux packages and apps can be searched for and installed via the app launcher. I mean, that's a pretty plain purpose. And it's a nice workflow too. Yeah, not only is it convenient, but it actually is going to put it in front of the face of actual real users. That's, to me, the biggest thing.

Because it's in the launcher, they don't have to go to a command prompt, they don't have to go through all these different less treaded paths. They are just using something they use all the time. And we may even get to a point where, on some Chromebooks down the road, Linux supports turn on by default. That basic stuff's already installed.

I mean, this could legitimately one day lead to the adoption of more open source desktop applications, which could hopefully mean an increase in bug reports, an increase in development time, an increase in funding. It could mean an increase in bug reports an increase in development time an increase in funding it could mean an increase in quality but these are all maybes but with a broader

market share right exposure in a way where you don't have to go install your own distribution of Linux or even know what that is but you can still run GIMP yeah

and people have asked me like why do you care Chris why do you care if Adobe Premiere comes to Linux and it's like this it may not be something I would use maybe it would be I Premiere comes to Linux? And it's like this, it may not be something I would use. Maybe it would be, I'd have to evaluate it. And it's not that I want all this commercial software coming to Linux. I think we need it all. And I think we need all of it here

because we need more users. We simply need more users because we need more representation and we need better software. And the only way we're going to get there is if there's software to consume by users and there's only, and if there's not users to consume that software, we won't get there. So we've been stuck in this chicken or the egg situation now for years and years. It just hasn't gone anywhere. And now maybe, maybe Google is shifting that a bit. And they're going to take a platform that has a ton of users and they're going to turn this on. And now all of a sudden there'll be a whole new audience

for these applications. So Minimac, I want to get your take on this. Good news, this Linux adoption, and now this continued integration further and further of Linux applications? Very good news, but the problem is they don't support older Chromebooks. That's the major problem.

I have this Intel Acer C720, which is perfectly capable of running Linux and everything. But on Chromebook, they have not yet opened these new features like Android apps, Linux apps. And Chromebook, the hardware is perfectly capable of doing that, of handling that. Good point. Yeah, it's not like these applications.

You know what? In probably almost all cases, most of these applications that users would be using are lighter weight than its equivalent web app running in a big old browser like Chrome. Probably, yeah, definitely. Hmm, that's interesting. So Dalton, do you think we're seeing a beginning of a path here

of Fuchsia for the mobile and Chrome os for the desktop so what i think back to here is google's g visor that we talked about oh that was a while ago which is their container runtime that re-implements all of the linux kernels interfaces if they had that running on top of fuchsia then they could use their Apache-licensed kernel, and they wouldn't even need Linux at that point. Jeez, Dalton. Way to bring it down, man. Good point, though.

You would end up with the core that they can have, and with GVisor, too, right? They only will implement the stuff that they know they can contain and actually run, so they don't have to be exposed to that whole API surface. Right. It's a safe user experience, they'll say. And kind of just really quickly, just to tangentially

mention this, we did cover this in more detail on Linux Action News, I think 88. But Chrome OS is also going to start letting managed company Chromebooks choose their own Linux distro for this layer. Right now, I think we've talked about it before,

it's like a Debian derivative. Essentially, it's Debian. But now, you get to specify a URL of where your ISO image is at. And when the user chooses the option to enable Linux support, it'll pull it down from that customly specified URL and install the version or the image

of Linux that you've set up. Well, that's the rub, right, Chris? Because maybe it's you, but really, it's the device administrators, whoever they might be. Yeah, the IT department. Because I don't even, I think people are digging around in the source code, and they didn't see a way for individuals to turn this on yet. Yeah, it really seems targeted at the enterprise use case, which does make sense.

Chrome OS has made huge inroads there. Yeah, and I could definitely see certain enterprises that would want their own custom environment. So that way, maybe their own internal development team could be developing on that environment, which then gets uploaded to their production servers. So I think it could be a hit. I don't know if I buy the premise, but if the premise is true, I think it could be a hit. And you've got to assume that once the underlying tech is there, someone somewhere is going to release a tool so you can do it yourself. That's what I'm counting on.

When that tool comes out, I'll probably grab a Chromebook. I think I'm going to do it. I think I'm going to do it. Unless those chaps over at UbiPorts can get me a tablet running Ubuntu Touch that just sings, and they're getting closer and closer with every dang release. to touch that just sings, and they're getting closer and closer with every dang release. And Dalton's there in the mumble room to talk about OTA 6, which is now shipping. Oh, wait, I'm sorry.

No, I'm sorry. OTA 7. OTA 6 was the one that came out last month. Holy smokes, Dalton. You guys are really hitting that monthly stride now. We did. That's one of the big stories for this release, where we finally got our release cadence down. So we've got proper testing and a lot of work done in between the two releases, but we're still hitting that one month cycle. Congratulations. That probably took a lot of work as a project to get that, you know,

to get your heads around that and get that actually working. So that's a big accomplishment in itself. But what are some of the new goodies in OTA 7 that you're the most excited about? OTA 7's big headline feature for the user face, for the users, is keyboard theming. So that might sound like a small deal, but someone in our community contributed the support. So now you can change the colors of your keyboard. So instead of being blinded by this really bright white keyboard, you can have a darker experience. And there are screenshots of this in our blog post, which you can find at ueports.com slash blog. Excellent. Yeah, nice. So when you're using the phone late at night, you don't get that blast of light in the face. It looks like Morph has

gotten some nice improvements too. It's a steady improvement from now that Morph is in. It is. The Morph browser is our cute web engine-based browser, which is based on Chromium Blink. And it's been getting more stable with every release. So in this release, we've got fixes for multiple browser windows, which is always an interesting experience when you have a five-inch screen, as well as fixes to tab switching and a bunch of other things that everyone will find nice. And we've heard some really great feedback on. Good. That's really nice. The other thing I learned when I was reading through your release notes, and I hadn't thought about it, and then when I saw this, I was surprised that the project

is still active. I thought maybe it was a canonical project. But LibHybris, which is that magic translation layer that lives between the Android drivers and the LibC-based user space, right? So LibHybris does this really, really important translation that lets you use all kinds of different hardware. And that's still an active project getting updated, yeah? Yes. LibHybris was originally maintained by the Mamo team when Nokia originally brought out their Linux. And that has moved into the Mer project, which is the base of Sailfish OS, which is supported by Sailfish. It's turtles all the way down,

but it means that there's a great, well-supported project that other mobile platforms are also using. So it kind of works out perfect, doesn't it? It is perfect, and it's nice to have that upstream there. Yeah. Well, seriously, congratulations on another great release. Good to see that new cadence, that new keyboard does look sharp. I'm looking at the screenshots in your blog post right now.

And for you Nexus 4 and Nexus 7 users with the Wi-Fi only version, also a release that I think you're going to be particularly excited about. So go check it out. We will have a link in the show notes to linuxunplugged.com slash 284. Yes. Dalton, thanks for stopping by to tell us anything else you want to mention about this release or maybe the next release people to keep a lookout for? Well, with this release, we brought in a bunch of changes to help with porting. So I think people are going to be seeing some new devices pretty soon. In fact, I think that there might be one of those devices from that company that starts with X that Americans can't seem to pronounce about to hit our devices page at

devices.ubuntu-touch.io. It's in testing right now. Very cool. That is awesome to hear. Well, good work to you and the whole team. Tell everybody to keep it up and we'll keep watching. Keep watching. Don, I don't suppose there's any chance you're going to make it out to LinuxFest this year? I'm not sure. I just booked all my tickets for FOSDEM, so I might need to be recovering at that point yet. Yeah, yeah, yeah. Well, good. I'm glad you're going to Fosdum. So let's just do a bit of housekeeping here for a moment. I don't really have a lot to cover,

but LinuxFest Northwest is our next big public event. We're also going to be at scale. So that's now, we're locking that in. I think I'll be there. Wes is going to be there. And a new member of our team, which we haven't had a chance yet to introduce you all to, is going to be there as well. Oh, I'm so excited already.

Just all the good ways to hang out with our wonderful audience. Yeah. But LinuxFest is going to be the party. The list of people showing up, I'm so thrilled about. I'm looking forward to hanging out with everybody. LinuxFest Northwest. If you can make it, please go. April 26th through the 28th, 2019. That's all I really had for the housekeeping section, though, because there is some stuff, there is some stuff that we have to talk about.

You heard me mention it during the intro segment. ZFS on Linux is going to be the new upstream project that FreeBSD and Lumos and other operating systems that ship ZFS will base their implementation on. So the FreeBSD project will likely still have their own project repository for their implementation of ZFS, but it will be downstream of the ZFS on Linux project now. This is a huge shift. This is a massive shift. And this was announced a few weeks ago.

We covered it when it happened. Today, I came across what I thought was the end of days at first. At first, my first pass of this story, I was really kind of in panic mode. And I PMed Wes. I'm like, Wes, talk me down. Tell me what this is about. Because the headline and the ramifications are pretty big. ZFS on Linux has run into a snag with version 5.0. ZFS on Linux currently fails to build

against Linux 5.0 kernel sources. This isn't due to just some trivial API change. In fact, actually, it's just because there are some kernel symbols, kernel FPU begin and end, respectively. The kernel module for ZFS on Linux relied on those in part to basically have faster performance while doing some file system checksums. Those are no longer exported and the module can't build. Right. So these FPU symbols that allowed ZFS on Linux to do fancy fast checksums just disappeared in 5.0. And there doesn't

seem to be a simple solution for this immediately, especially one that doesn't involve using GPL symbols due to license compatibility issues that, I mean, we talk about this with ZFS all the time with Linux. And Greg KH chimed in on the mailing list, a thread about this on the

kernel mailing list. He doesn't seem to be too inspired to make any concessions. Quote, Greg writes, my tolerance for ZFS is pretty non-existent. Sun explicitly did not want their code to work on Linux, so why would we do extra work to get their code

to work properly? It's unfortunate, you know, this one might not be the biggest. It looks like they'll probably just not have vectorized floating point implementations for these checksums in the future, at least for the short term. But at a time when suddenly the ZOL tree is the place where new development is happening, it just kind of seems like a downturn or not a great relationship between the Linux community and the large and the ZOL project in particular. And it wasn't, I mean, obviously those weren't great, right? I mean, it's an out of tree driver.

And we should also note that those symbols begin with a double underscore. And so really they weren't super officially supported. They could have been removed really at any time. So here they are, they're gone. But just knowing that people as high up as Greg H have no interest in bending even a little bit to help our favorite little file system is pretty disappointing. Yeah. Do you think, Wes, it's a power play a little bit? Like they're looking at the situation, they're going, ZFS needs Linux more than Linux needs ZFS. And now you all just admitted that the ZFS needs Linux more than Linux needs ZFS.

And now you all just admitted that the ZFS on Linux is the best implementation, the one with the encryption you want. So why don't you just come play ball? Why don't you just come play ball? Is that the message they're trying to send here? That's one reading. I would assume that Greg in particular is probably a little more pragmatic than that and just can't be bothered to impose future support burdens,

even if they might be minimal or seem to be minimal now, on an out-of-tree, incompatibly licensed module. Yeah, in fact, I think I agree with you because reading into it, didn't it look like there was sort of iffy support for these symbols for quite a while now and they've kind of been on the chopping block? Yeah, I think what we're really going to have to try to pay attention to is the longer-term story. How many more symbols disappear? How many symbols that we just can't rely on are hard-coded into the current ZOL implementation, and how big of a deal will these be? Because if it's just sort of

like, this was going to be deprecated anyway, now's the time where we have to go do the backporting and the updates, so be it. If it signals a real degradation in the relationship, that'll be disappointing. Well, and it also seems like 5.0 is where you make some breaking changes. Like if there's something you've been thinking about and it's been on the chopping block and you know it's going to break a couple of things

because ZFS on Linux probably isn't the only software project to use this vectorized checksum support. There's probably other ones out there. But at the same time, like if you decided as a team, we're kind of done with this and it doesn't really fit with our overall licensing scheme anyways and what our intentions are and what the spirit of it all is, 5.0 would be the time to make that change. It would be time to just rip that band-aid off, I would think. We've got some good questions too in the IRC just asking about what about

NVIDIA, right? Because that's the elephant in the room. It's the other really popular driver. That's why I was freaking out earlier. Now, it doesn't seem like this. Obviously, you know, when you have a GPU, you probably don't need these particular symbols, but the same practices could be applied. And does it mean there's a different relationship? Does the kernel value the NVIDIA driver

more than, say, the relationship to the ZFS project? Yeah. I don't know, but it doesn't seem fair. Yeah. I'm glad that question came up, because depending on where you read this story, particularly on Reddit, if you read the comments, people are equating this to mean

the NVIDIA driver won't work anymore. And that's not the case. That's not what's happening here. It's really just this one vectorized GP, I can't remember what it's called. Yeah, the vectorized floating point operation. And Conan Kudo in the IRC has a great point. It was removed basically because they did a bunch of changes in the kernel

and entry modules no longer needed these symbols. So they were unwilling to support stuff that their code base wasn't using. I mean, there you go. There you go. So I thought what we would do here is we had an opportunity to sit down with an individual who contributes to the Fedora project, Mangea, OpenSUSE, and he's on the ZFS leadership council as part of his day job at Datto. And I've known Neil for a long time. And so when this story came up a couple of weeks ago, he was the first guy I

thought about calling up and chatting. And he gives us some great context about the situation. Well, with all this news recently about all the major projects that use ZFS, rebasing upstream on ZFS for Linux, I had to bring somebody on to process this. So Neil's joining us. And Neil, you and I have chatted in the past, and I always manage to mess up the last name. So help me with the pronunciations, so that way on the record, I don't get it wrong. Neil Gompa. All right. See, there you go. Hey, Neil, thanks for joining us on Linux Unplugged. And why don't we start a little bit with your background? Because I know you're involved with the Fedora project, OpenSUSE, you have a day job at Datto, and obviously work there with storage. Tell

me a little bit about your background. Sure. Currently, I am a DevOps engineer at Datto, which is a disaster recovery business continuity services company that's focused on managed service providers and people in that and all that kind of space. And we use ZFS or ZFS, however you'd like to pronounce it, for a lot of our infrastructure and, of course, our backup appliances and stuff like that on Linux. And part of that has also made it so that I'm involved with the ZFS on Linux project upstream. What kind of amounts of storage, roughly, you don't have to give me specifics, but just kind of roughly are you working with?

I think the last time we talked about this publicly as a company, I think we were at almost 200 or so petabytes of data. Now, it's not like it's an enormous cluster of ZFS because, for one, that's not a good idea. But it is 200 petabytes of computers that run ZFS as the data backing file system. bytes of computers that run ZFSS, the data backing file system. And the model that our company uses is that basically we map a person, a customer's equipment to what we call a cloud storage node. And that node receives all of that customer's data. So it's not like you have this massive fabric or something that along the lines of, like Ceph or Gluster. It doesn't work that way.

It's more like we have a large set of computers that run ZFS to store data. And so the servers may have, you know, they'll have, I think, on average somewhere like 200 terabytes or so, maybe more per server. So it pushes the limits of a lot of things. And the workloads that we're doing on it are quite varied. So this Datto's use of ZFS so extensively, how does this work? Because I know you were involved in the ZFS leadership and you were in the leadership meeting when they were discussing rebasing, like free BSDs implementation on ZFS for Linux instead of the Illumos version.

How does that part of it work? So is it a lot of companies that have some sort of financial interest in the future of ZFS? Is it based on their contributions? How does the leadership team aspect of all of this work? Because I think that's an interesting element to the ZFS project that, I mean, I don't think ButterFS has anything like that. So historically, the leadership is more or less the companies that worked on the Illumos tree. And those were considered the people that kind of led the open ZFS umbrella.

And would it be fair to say that a common conception was that the Illumos version was the... Mainline? Yeah. That was pretty much the historical basis. But a number of things have changed over the past three, four years. Like Datto, as a company, has contributed the ZFS native encryption to ZOL, and that has not been fully ported to other operating systems. Oh, I see. So that's sort of a big differentiator that the Linux version has that the other projects probably want. The library interfaces for ZFS are

different across all three operating systems. Even the ZFS command works differently between the different operating systems, excluding the fact that native encryption only exists on ZOL right now. It's just the implementations are not the same. They are not at the same level. They don't exactly have the same feature sets. And in some cases, their compatibility is a little bit wonky. So there has been this open question for a while of what we would consider the mainline. And historically, it has been the Illumos tree. But with the changes that have happened over the

last couple of years, that came into question again. And now it's the Linux tree is the one that's actually moving ahead and doing more. And the other trees are not doing as much. I think the most notable exception I am aware of is the ZStandard support that came to FreeBSD first because Alan Jude wrote it. It's not because it didn't go to Illumos first and then go to FreeBSD. There wasn't already a single mainline. It was sort of just following where development is naturally taking place. It sounds like it was starting to get worse too, potentially. Yeah. And so nowadays, if we wanted to follow in terms of like,

okay, where's the features going? Then we have to consider ZOL as the main line because that's where a lot of the feature development is happening. That's where the bulk of the usage of the ZFS file system is now happening, even though it is the redheaded, ugly stepchild of the OpenZFS family. Well, that's what I was going to ask you. That seems like a really awkward, contentious position. If the version of ZFS that perhaps has the largest user deployment

is the Linux version, but a large portion of the ZFS community themselves considers the Linux version to be sort of second class, that's a bad position to be in because then in theory, the project's most passionate members aren't putting their best foot forward to the marketplace. And that has definitely been the case until very recently. What changed is that Delphix is moving from Illumos to Linux for their platform. platform. And so that means that all the members of the, the main members of the leadership, like Matt Ahrens and, and company are moving from the Solaris tree to the Linux tree. And so that means that for all intents and purposes, the majority is now Linux developers. What distributions are they using? Is it all Ubuntu?

No. So, well, Delphix, I think, will be using Ubuntu, which I'm sure. Oh, wow. I thought, you know, I mean, at least they're shipping it, man. That's true. And Datto does as well. But the ZOL primary developers, which are the Lawrence Livermore National Institute, they use RHEL. And actually Red Hat Enterprise Linux and CentOS and Fedora are the primary test engines and the things that we test against because Fedora gives us the latest kernels and easy to test against along with the newer ABIs and stuff like that to make sure we're following the future well. That just feels like there's a bit of tasty irony in there, don't you think? In fact, right now, the leaders of those projects are actively working on Stratus to compete

with ZFS. Ah, so Stratus is not about competing with ZFS. That is... You don't think so? No, it's definitely not. Look. So just a quick recap to the audience. Stratus is sort of a collection of existing tools as well as new APIs to use existing Linux management tools along with XFS to create a ZFS-like competitor.

Is what, you know, a lot of people consider it, like myself and the people I've talked to. But what do you think? You think it's more of a stopgap or a ButterFS competitor? I don't consider it a competitor to ButterFS or ZFS in any reasonable way because the problem with it is that the layers don't know anything about each other.

So there's going to be weaknesses and lack of optimization on certain parts of it. There's a lot you can do to make... This is more about making the user experience simpler rather than anything else. I mean, it almost seems like that's where it's competing, right? It's not about the technical competition, it's about their customers want to do this X,

and if the performance is good enough, even if there are limitations, well, that's the same thing. The main impetus for the development of Stratus is that they want to have boot to snapshot, the stuff that Sousa did. And boot to snapshot is only functional on Linux properly with BTRFS. And if you try to use the other file systems or the other technologies, there's a whole lot of gotchas and quirks and problems.

And even to be fair with ButterFS, you can't even do this with mainline grub because the grub people are bad about reviewing patches for this stuff yeah oh it is i laugh but it's frustrating yeah no suza has i think seven or eight patches to butter fs that they wrote four or five years ago to make this work i integrated them into fedora with peter jones's help uh and peter jones and dusty mabe who are peter jones is the guy at red hat who works on the boot stuff and dusty mabe um is uh currently the guy that works on the red hat atomic and core os stuff and back then he was just kind of the guy who just worked on all kinds of stuff related to whatever the guy with the tie yeah yeah that's

that's more accurate but um dusty had had been doing this for a while on his own just like integrating the patches and then putting them into and then writing up instructions on how to set up the suza style boot to snapshot on fedora i wanted i've been doing it myself but like one of the downsides is if you're building grub yourself and all this stuff you lose a lot of secure boot support so uh i worked with peter jones and dusty mabe to integrate the sousa patches into fedora's grub and so now the boot to snapshot stuff can work if you like actually set it up um on fedora it's just not set up by default there's some work to be done but like my main annoyance at this point is that Grub is so bad about integrating these things. I have no idea what the hell happened

the first time around when they submitted the patches, assuming they did. I mean, again, I can't even prove that because navigating the Grub patches mailing list is horrible because it's that horrible mailing interface that the GNU project uses for everything. So it's that horrible mailing interface that the GNU project uses for everything, so it's just terrible to search. Thank you, thank you.

I always felt like the jerk for thinking that. It does seem a little bit like, we have such better technology. Well, like, the Fedora mailing list with HyperKitty and Mailman3 has a bloody search engine that can actually search properly across all the topics.

You can tag things. You can favorite threads. That is pretty legit. Let's bring this back to ZFS. So now moving forward, we've got certain features that are only available on the ZFS for Linux implementation that the market wants, like encryption. You've got Delphix, which is switching their product from Illumos over to Linux. You've got Delphix, which is switching their product from Illumos over to Linux.

But what seems to be the elephant in the room now is looking over at FreeBSD and the BSD projects that ship ZFS and thinking, after kind of a couple of years of really, really hounding you've got to ship BSD if you want the best ZFS, they're now shipping or will be shipping the Linux version of CFS. Is that an awkward position for them, do you think? Or is that just in the eyes and minds of the internet commentators? So ironically, I think they're going to have an easier time once they complete their rebase to ZOL. So one of the differences between the ZOL tree and the Illumos tree is that the ZOL tree is actually separated from the rest of the operating system. So it's easier for them to pick it out and plug in the BSD to Linux compatibility interfaces and plug the ZFS driver in. I think that from a community point of view, like the broader Linux community, ZFS on Linux will remain an ugly stepchild. And it's probably a very fair remark to consider it that way,

given how antagonistic both sides have been to each other for various reasons. It's not worth getting into right now, but that is not going to change anytime soon, unless Oracle waves a magic wand, which I don't think they'll ever do. But they could, and then the problems will start going away. And then maybe ZFS would be integrated into Linux tree

and everything will kind of get better and ponies and unicorns. But while it might be an ugly stepchild in the broader Linux community, within the open ZFS community has gradually gotten a bigger sphere of influence. The current major developers of the ZFS file system, as in like the features and stuff like that,

have increasingly been on the Linux side rather than the other sides. And for FreeBSD, they're going to find that it's actually going to be easier for them to integrate. Once they've rebased from the Illumos version to the Linux version, I suspect they'll find that it'll be much easier to integrate new versions faster. Now, that being said, the FreeBSD development process doesn't allow them to move very fast, very easily. They have a lot of problems in terms of their development process, the size of the number of people contributing to the FreeBSD operating system in terms of the kernel and the integration across the different

layers. But I think that this will improve their ability to integrate things from ZFS overall. I think that's a different take than I've heard, but I think that syncs up. Knowing what I know from chatting with Alan about this too, behind the scenes, asking him what he thought of it, and I think his views kind of sync up with yours. And I wonder if then we can move beyond sort of who has the best ZFS implementation and we just sort of achieve finally, which I think has been one of the project's goals, maybe from the beginning. You could correct me if I'm wrong, Neil, but we finally kind of achieve one common upstream across all operating systems. That's the holy grail, right? Is it possible? Is that possible? The main thing that hampers it is that ZFS, each ZFS implementation is too tightly wound

into the kernel that it is associated with. So today it is not, we don't have a modern open ZFS implementation that can operate entirely in user space, for example. If you remember from like five or six years ago, or even seven years ago even, like there was the ZFS fuse project. And that was the first one of the first implementations of it. And while it didn't

last for very long, it was a fairly complete implementation, if not a bit slow. These days, fuse is much faster and much more efficient. And if we did it again, it would probably be actually pretty good. But the problem today is that without a way to operate the ZFS file system entirely in user space with an agnostic API that can be used across different operating systems, there is no way we can have a unified file system tree. Because we can't have an implementation that we can say, okay, this is the behavior that we expect and test all the kernel integrations against it. That's just not a thing we can do right now. How would you pull something like that off? Would there be additions?

Would there, I mean, would it be something like a set of super feature flags? Is there a way to have a master branch and then have all of these considerations or compromises or whatever you want to call them for the different platforms this thing's going to run on? So, sure. I mean, most modern file systems are actually built in a relatively self-contained way

because they have to be able to understand their full context and state. ZFS is no exception to this. The only difference is, because of the way that ZFS on Linux has been developed and all the other variants, it's been they took out the common code and re-implemented everything inside the common code outwards into the kernel. So ZFS on Linux, for example, operates in like, I think it's like eight or nine kernel modules. Like it's a bunch of different kernel modules that make up ZFS. ZFS.ko, ICP, ZCommons, there's a whole bunch. And while we're talking about those kernel modules, let's talk to Neil live in real time.

So we recorded that a couple of weeks ago, and he's joining us in the mumble room. Neil, welcome to Linux Unplugged again. And what do you think of this latest news with Linux 5.0? Hey, Chris, it's nice to be back again, this time live. It's really not that surprising. I mean, we actually kind of knew it was coming to some extent, because as I mentioned to you, the last time we talked, it was more about having the, we often test against with Fedora. We even test against Fedora Rawhide, so as snapshots are coming in of Linux kernel releases, we are actively

testing that and we detect these kinds of problems basically like that. And because of that, it was no surprise to most of us in ZOL, the kind of the kick in the teeth was that there were no other consumers left in the Linux kernel itself, and they re-architected things. So it burned us in that regard.

But, you know, this is not all that different from what happens every other, like, seventh or eighth kernel release or something like that. I mean, it's just going to make it so that configuring ZFS when you're building it from source is just going to take two or three more minutes longer than it did before. I mean, at this point, I think we have 10 minutes worth of configure steps for detecting which functions we can use going all the way back to 2632. Oh, wow. God, I'm looking forward to when RHEL 6 goes away because when RHEL 6 drops off, we can move the bar up to 310, and that lets us drop a huge chunk of checks. Holy smokes. So is there going to be, as far as you know, like a checksum performance impact if it can't use this vectorized checksum methodology?

I don't know this for certain because, like, I haven't dug too much into it, but I suspect that this is really only going to impact encryption, really, because the FPU stuff, I think, is more about accelerating floating. FPU is a floating point unit, so it's accelerating floating point calculations. That's primarily used in cryptography. So I suspect that's the only thing that's really going to harm it there's not a lot of accelerated code paths in zfs that's part of the reason why it's kind of a heavy file system compared to most um and the other part of it is just that it has a lot of stuff and it has to re-implement um large aspects of what the kernel normally does because of the licensing in compat so i mean sure it is

what it is uh it'll move on like there's already a little bit of there if you look at the cfs on linux repository right now and look at the github issues they've already got the issues filed for like the different functions that have been uh basically they're blocked off from starting with 5.0 and you know they're just we'll just poke at and prod it and fix it as we go. I feel like we should maybe just more explicitly assuage Chris's fears. Are there changes to symbols or, you know, other upstream changes that would seem likely that could really endanger the future of ZFS on Linux? Look, the only way that ZFS on Linux will have a guaranteed future

is if it was mainlined into the Linux kernel tree. That is not going to happen until Oracle waves its magic wand. So nothing in this life is certain when it comes to ZFS on Linux or ZFS on any platform. Just take it as it can come, and for now I don't foresee a reason for it to disappear. Yeah. It does seem like, and I think I even put that in my episode where we all had our magic wands and we had our wish list.

If they dual licensed it even or just, let's just say they win GPL, just for a hot second. Let's just fantasize. It would enshrine that file system in the world's largest free operating system. It would essentially make ZFS a file system that could live on forever. Not that it can't on these other platforms, but as we've all seen with the GPL, it has that viral nature and that has worked out very good for Linux. And if ZFS could be included into that, it would be part of something larger. I mean, not to get all meta about this, but being inside the Linux kernel

is sort of like being in a time capsule. And the other thing about this that a lot of people don't realize is that we, in the ZOL implementation of ZFS, we re-implement quite a bit of what the kernel does. If we were actually allowed to be part of the kernel, that would probably simplify quite a bit of CFS's implementation on Linux. And the magic wand here would be an arrangement similar to what we have for

the GPU stack, the graphics stack in Linux. In the kernel side graphics stack, the GPU stack, the graphics stack in Linux. In the kernel side graphics stack, most of the code is dual licensed and expat MIT X11 and GPL v2. And so that allows large swaths of that code to be imported into operating systems like FreeBSD that prefer to have a more permissive pushover set of license terms versus the copyleft terms that the Linux kernel has. The thing to consider, though, is that if such an arrangement were to occur, there's a whole lot of extra considerations around how ZFSfs is uh how it's going to be maintained how will people make sure that the licensing providence remains you know sane stuff like that these are things that have been worked out as arrangements before it's been done with the linux graphics uh drivers it'll it can be done for zfs as well if we could get Oracle to wave its magic wand. Because if they make, say for example, a new version of the CDDL, that new version could basically allow you to have the terms that would be compatible with it.

That could be a way to do it via the CDDL. Uh-huh. Because there's no restriction currently in the CDDL license that specifically says, the way that it's termed inside of the ZFS tree, that says that newer versions of that license cannot be used. And the license steward, which was Sun Microsystems, now Oracle Corporation, is allowed to make a new version, and we are allowed to use that if we want to. Just put that out there.

Just going to put that out there, right? Just maybe send that out in waves out in the universe. Maybe something will come back. It's the same path that Fedora, Red Hat, Fedora, and SGI did to fix the XOR licensing problem back in the early 2000s. So it's not that. Oh, my gosh. And the Free Software Foundation did the same thing for the GFDL so that Wikipedia could move to Creative Commons licensing instead of remaining on the GNU free documentation license.

It's not without precedent. It can be done. It's just a matter of getting the right ear with the right people to make it happen. And if there's enough people who want it and somebody knows the right person, then it can happen. I'm holding out my hope. I like that. I think that's a, you know what, that's going to be the that'll be like the little little string of hope that I hold on to. Neil, thanks for coming on and sharing your thoughts on this story as it developed today. I appreciate you stopping by. That was great timing. And I have a feeling we'll be talking more in the future. Yeah, I enjoy chatting with

you, Chris. So happy to be on whenever you'd like me to. Thank you, sir. Thank you, sir. Well, before we scoot out of here, I wanted to just mention a couple of quick picks. Both are really neat. We've talked about dot file managers before on the show. And I thought what I do instead of me pronounce this, I'm going to make Wes pronounce it because he went and looked up the pronunciation. And I say there's a 50-50 shot you remember how to actually say it, Wes. Oh, betrayal. I see how it goes, sir. Chinois.

Okay. All right. All right. Good job. It's not that I didn't have faith. It's just that it was a while ago. It was a while ago. Chinois? Is it Chinois? This is on you now. You got the one reading, and that was it. So when you first linked this to me, I was a little skeptical. There's a ton of different ways to manage.files, from a simple git directory to a ton of custom tools that exist there. Thankfully, this guy, they've got a great readme. And a big section there, I already have a system to manage my.files. Why should I use this?

And there's actually some solid answers. If you haven't ever tried one or you're not happy with what you got, this should be a serious contender. Yeah. I heard good things. I heard good things. That's why I tossed it in here because I heard a lot of good things. And I'm thinking about it.

There's also a bunch of other ones. Probably the most famous is GNU Stow or Stow, however you want to pronounce it. Yeah, I've used that for a long time. Yeah, yeah. That's, I think, probably the most common one. I think this is a little easier to start with myself, so we'll have a link to that in the show notes. And then our last pick this week is a new web browser called Ephemeral. And I think this is a really cool idea.

It's the always incognito web browser, and it's by our buddy Cassidy over at the Elementary Project. And he writes, it's an interesting idea, Firefox Focus, just making privacy a habit by default instead of a separate mode you manually enable. And from that, he was kind of inspired to make a really straightforward, nice-looking,

as you would expect, I suppose, web browser for elementary OS. And it's a Linux app, so there's probably other ways you can get your hands on it but if you are on elementary OS go in the app center and check out ephemeral I hope this maybe gets around a little bit I think this would be good publicity for elementary as a project as you said it's beautiful it's streamlined it's simple and I think Cassidy hit it on the head with, you know, privacy as a default. Sure, you might need sessions for some stuff, for your work account, whatever.

But if you're just trying to, you know, buy something on Amazon or look up plates, this seems perfect. Yeah, I actually just like it just for the idea of having like a, you know, like an extra browser that's sort of like, just go log into this for a little bit. I do this all the time. I'll fire up an incognito window to maybe open up a Google account that's for one of the shows

that I don't log into too often and I just want a one-off. And I just could always use this browser. But speaking of publicity for elementary OS, our buddy Jason, who was on the show last week, I think when I was gone, and talked about Adobe Premiere coming, possibly or possibly not coming to Linux and also about their plans with Rush. Well, Jason has launched an elementary OS community challenge.

And the gist of it is his challenge is use elementary OS for a certain period of time. It's not super, super long. The whole idea is, especially if you're on Mac or Windows today. So if you're listening to this show and you've been Linux curious for a while, this is perfect for you. And I would like you to consider this.

The basic premise of the Elementary OS Challenge is simple. You ditch Windows or Mac OS for maybe Elementary OS. Maybe you choose your own. But, you know, Elementary OS would be a great choice. Oh, come on, come on. It's not called choose whatever you want, Chris own. But, you know, elementary OS would be a great choice. Oh, come on. Come on.

It's not called choose whatever you want, Chris. I know. I know. But I just want anybody to switch Linux. But his suggestion, I think it's a good one, is you all are going to go in on elementary OS for two weeks. Think about that. You'll have a community of people that are all trying this out. Like there could be some really good results from that, especially if Jason is able to stay in contact with that community.

And he says, since we'll be on this road together, he has a piece over on Forbes about this. He says, since we'll be on this road together, I'll be posting a bunch of content here to capture my ongoing experiences with the challenge. One day, may see a stream of consciousness journal. Other days, I'll be publishing short guides, software spotlights,

interviews with elementary OS app developers and team members, and my own discoveries and hurdles, if any. I think it's a great idea. He's also at killyourfm on Twitter if you want to follow him there because he'll probably be posting about it there as well. And I'll have a link to their challenge,

his piece on Forbes, if you want to maybe get into that. I know a lot of people listening, Wes, that are probably on their own distro of choice, but you know, you know because you see the stats, actually, so you actually do know, we have a very large Mac and Windows audience that are likely listening

because they're Linux curious. This could be great for them. Yeah, it really could. And elementary, I mean, it just keeps making leaps and bounds. So maybe you're like me and you hadn't used it since the last time around. It's really worth it. Two weeks is nothing. I mean, just give it a shot. Yeah, give it a shot. And if you do it, let us know too. I'd love to know your progress too. You can, you could at Chris LAS me, he's at Westpain, or you could just send us a contact form if you go to linuxunplugged.com slash contact. It'll go right into our inbox there or wherever you like to do that. We would love to know how that journey goes too. And I encourage you, if you've been thinking

about it, with Jason doing this, he has a pretty decent following. Like this is going to be a little community that crops up that can support each other. And you can follow him along as he goes through it too and writes about it. It just seems like if I was getting into Linux, I'll stop talking about it, but I just think this is such a cool idea. If I was getting into Linux today, that would be the thing that pushes me over, and it's just really exciting, and I'm just fired up about getting some new users, and I think elementary is a great distro for that potential audience too.

So with that, we're going to wrap up this week's episode. TechSnap has some great stuff in the works. Jim Salter from Ars Technica is joining Wes as the new co-host, and a new episode will be coming out soon, so go subscribe. Go to techsnap.systems slash shupshcribe and get that. It's going to be incredible. He's at Wes Payne. Thank you to the Mumble Room for joining us.

I'm at Chris LAS. incredible. He's at West Payne. Thank you to the Mumble Room for joining us. I'm at Chris LAS, and from a very chilly Texas, thank you for joining us, and consider joining us live next Tuesday. You can get it at jupiterbroadcasting.com slash calendar for your time zone, and jblive.tv for that stream. We'll see you next Tuesday! 🎵 Well, thank you, everybody. Really enjoy it. Mumbaroom, you were great. Neil, thank you so much for making it. Man, that worked out perfect.

I'll tell you what. It really did. Thank you. Sometimes it just comes together, even when I'm doing live shows. Oh, man, we had a great live show today, but there was a moment. You know, and I don't, I mean, I guess I am a professional. You are, yeah. I don't consider myself one, but I guess.

What was the last 10 years? I know. I know. It's so funny. But, you know, I totally, if I was a professional, I'm not anymore because I embarrassed myself so bad on the live stream. I don't know if I should say it or if I should just make people go watch it. But there was a moment where maybe because I wasn't directly on camera, something happened and I lost it. I laughed so hard.

Like, you ever see those outtakes when a cast just loses it and they can't keep it together anymore? That was me. Yeah, but normally that's like multiple people. And, you know, you were definitely the loudest. I was the only one laughing. There was like one other ha, but I was crying off camera, crying off camera. Just because of one simple mistake.