LINUX Unplugged - 324: RAMburglars

Episode Date: October 23, 2019

Is the ZFS tax too high? We pit ZFS on root against ext4 in our laptop pressure cooker and see how they perform when RAM gets tight. Plus we take a look at Pop!_OS 19.10, complete our Ubuntu 19.10 review, cover community news, and lots more. Special Guest: Alex Kretzschmar.

In celebration of Unix turning 50 today, we open with a special clip. Back in 1969, a couple of computer scientists here at Bell Labs started to develop some programs they needed for their own use. What Ken Thompson and Dennis Ritchie started developing then has evolved into the Unix operating system, which by now is widely used around the world. We were trying to make computing as simple as possible. In the late 1960s, Dennis Ritchie and I realized that the then current operating systems were much too complex.

We attempted to reverse this trend by building a small, simple operating system on a mini-computer. What we wanted to preserve was not just a good programming environment in which to do programming, but a system around which a community could form. Fellowship. Hello, friends, and welcome into the Unplugged program. My name is Chris. My name is Wes. Hello, Wes.

Hello. You know, we have something special planned for old 3.24. Oh, we've been hard at work. We have. We've been having a lot of fun this week. We wanted to put ZFS and Extended 4 to the test. The promise of ZFS is pretty high. Oh, yeah.

But is there a performance bottleneck for that data security? What's this going to be like using it for real day to day? Yeah, maybe you're on a laptop. You've got an SSD, but it's nothing super fancy. In our case, this week we used a Samsung Evo 850, and we wanted to see how Extended 4 and ZFS performed against each other in both normal conditions and lower memory conditions. Maybe you've got a few Electron apps running, you've got some tabs going. How then do things perform? And what kind of penalty do you take for that data security?

Or do you? So we wanted to find out. Plus, we've got some more thoughts on 1910. And we've also been running Pop!OS as just our daily driver, which has been pretty great. And we have some thoughts on that. And I had a quick chat with Carl from System76 and have some good little insight nuggets from him on things we can talk about, including that upgrade process. Yeah, that sounds pretty smooth.

I want to talk about that, too. But before we get to that and the community news and all the other goodies, I got to say hello to Cheese and Alex. Hello, gentlemen. Hey there, Internet. Hello there. And of course, a big time appropriate greetings to that mumble room. Hello, gentlemen. Hey there, internet. Hello there. And of course, a big time appropriate greetings to that mumble room. Hello. Hello. Hello.

Happy Linux Tuesday. We've got a chipper bunch today. That's great. Everybody's been properly caffeinated. At least I have been. Now, I have to also admit that I've got a CPAP machine now. So not only am I sleeping,

but I'm also still caffeinating. So you better watch out, Wes. Superpower of regular sleep. Well, we have some community news that I wanted to start with. Besides the fact that Unix turned 50, they actually, it's a funny thing. I have some links in the show notes. They celebrated over like a two-day period. So I think it was either yesterday and today or today and tomorrow.

But it's technically the 50th anniversary of Unix. And I have a really great clip that when I was playing is part of a longer series. And I linked that in the show notes. So if you guys want to watch that, it's so great. So great. In fact, if I remember, I'll play part of a training video that they did. Some of these old clips are hilarious. But also, I mean, it's shocking both how far we've come in 50 years and also how much is pretty much the same.

Yeah. Well, it's also just shocking how far we've come in 15 years. This is a really great interview with Matthew Miller over at Tech Republic about Fedora at 15 years old. Wow. It doesn't feel like it. Considering I was using Red Hat before Fedora as my desktop, it shocks me. It's really something.

It's been a long time, but it doesn't feel that long. Anyways, good interview. We'll leave it to you to read most of it, but there's a couple of tidbits in here that I wanted to chat about with you guys. Throughout the process, he asked a series of questions, he being James Sanders, sorry, over at Tech Republic. And one of the topics that comes up is flat packs and snaps. And he asked why Fedora prefers flat packs as the portable application standard. And Matthew responds, there are definitely some technical differences. If you talk to our engineers who are working on this,

they're very, very passionate about the technical differences. One of the things with Snap is it's the answer to all things. We see OCI Docker containers as having won the server enterprise in IoT use cases. Docker doesn't necessarily fit well for the desktop, so Flatpak is a technology specifically meant for the desktop. I mean, it shows, right? We talk about that property of Flatpak all the time.

It does put a different spin on it. I kind of thought of it always as like a limitation, and I kind of mix command line and GUI apps, so this was weird. That does make some sense, though. I don't think I have heard that as clearly just enunciated. Like, it's just simply not trying to address that. I, like you, saw it more as an oversight, as a lack of, like, something. Wait a minute.

You know, Wes, now that I think about it, I think I've even seen developer statements to the extent saying they were planning to add server support. Like, that was something they were going to build in later on. I'm pretty sure I've read seen developer statements to the extent saying they were planning to add server support. Like that was something they were going to build in later on. I'm pretty sure I've read those statements. Perhaps things have changed because I also tend to agree with this. As I've been getting pretty excited about self-hosting and been spinning up more and more of my own infrastructure, I've had the option for a couple of them to use snaps. But the reality is it's containers all the way down, baby. Yeah, and mostly advice and workflows and styles of doing so.

Well, it's built around the container ecosystem, and that ecosystem has come a long way in having cross-compatibility on its own. I'm glad to hear that, Chris, by the way. I bet you are. I mean, you know, the thing is, is you know this very well. It's just that containers just kind of came in and became a default way for people to ship software on servers. Absolutely right.

For better or for worse in some cases, perhaps. But we're not here really to debate that today. There's a part in this interview that he touches on about improvements that have come along, or innovations, as the article puts it. And I was just waxing about this on the live stream, about how this has really changed the game for me. Yeah, right.

It is something, if you think about it, it makes sense. And Matthew Miller's answer is interesting. I think it's when the graphics driver people decided to go with open source. Intel buying in first on the network drivers and then on the graphics drivers was huge. Now, AMD coming along, I have an AMD desktop that I use for gaming. It's awesome

how it just works without having to fiddle with anything. I love that. There's also obviously a subtext of, come on, NVIDIA, get with the program. I am blown away on my desktop workstation upstairs. away on my desktop workstation upstairs. I've got full 3D acceleration and I'm playing nice games and I'm using zero proprietary drivers, no binary driver for my video card. It's so great. And then of course the Intel side has been like that for quite a while. Right. We almost take it for granted, but over the 15 years, that's definitely changed. I wouldn't want to go back. TechRepublic also touches on a spot that seems to have really left a mark on the project. This was a while ago, in 2011, I think before Matthew Miller was even involved with the project. There was a shock to the ecosystem when Fedora 15 landed. Fedora 15 was the version that shipped Gnome Shell for the first time, Gnome Shell 3.0.

At the same time, SystemD. Mm-hmm. Yeah. What? Mailer says the release had an incredibly dramatic amount of change. Both Gnome Shell 3 and SystemD landed in that release, and basically we lost half our users doing that. Or at least, put another way, half of them did not upgrade to Fedora 14,

and they stopped growing. They say it was a pretty bad situation and that there were plenty of ways that blame was getting thrown around. But just absorbing all that change all at once is really hard on people. The lesson that they learned, if you look at the GNOME user interfaces, if you go back and install GNOME 3.0 and compare it to 3.32, it's a very different experience. Yeah, I mean, he's totally right. I don't think about it that way, but it is.32, it's a very different experience. Yeah, I mean, he's totally right. I don't think about it that way, but it is. It's radically different. It was almost a different vision now.

He says in here, I think that if GNOME Shell goes to 4.0, it's not going to be a radical redesign like that. It's going to be architectural shifts under the hood. It makes sense right i mean we need smooth and yes fedora moves quickly but at the end of the day i still want it to work 15 years old are you kidding me are you kidding me here's to the next 15 huh no kidding i'd love to know a little more chris what you mean when you say that version 3 is such a huge departure from 3.32 oh boy well first of all, first of all, it's a huge,

there's a lot of UI differences. It's very, very big, like big. It's big. Everything's big. Everything's touch-focused. Crashes like a maniac. Crashes like Richard Hammond, just frequently and big.

And also is very, very, very early. It's very rough. Let's just say a clunky doesn't feel well put together yet. Go look at a screenshot of it. It's really, it's remarkable. And if you remember back then too, touchscreens were really becoming a big thing. The iPad was a new product. The PC was a truck and iPads were a car and everybody was going to be on a car.

And what date was 9.3 released? I don't know. I mean, it was a long time ago. I'll look it up. I think I recall where I was when I heard about it. There's a few major moments. I remember where I was when Unity was announced. I remember where I was

when it was announced that Unity was ending. I was at LinuxFest Northwest when it was announced, in a room with Brian, and we were about to do Alas, and I was at Dell when they announced that it was ending, with Noah in a room with the Sputnik team, who also learned about it at the same time. And it was really something.

So GNOME 3 was April 2011. So it was a different world in terms of mobile and everything else, you know. Absolutely. Well, speaking of interviews, just a quick plug for me. I was interviewed by opensource.com,

Mr. Don Watkins, the gentleman over there who I've met in Denver. Really nice guy. Yeah, we met him over at System76. Don Watkins, the gentleman over there who I've met in Denver. Really nice guy. Yeah, we met him over at System76. Look at you with the memory. Just, dang, Wes is on fire today. And it's a Jupyter Broadcasting origin story

and sort of like the future where we're going under the wing of Linux Academy. And it published a couple of weeks ago. I just have not been good about mentioning it because I was traveling. It's worth a read, though. I mean, it's not too long. Oh, stop.

I even learned some things. You did? Oh, yeah. Really? Yeah. I mean, there's a lot of stuff in the, you know, kind of the longer term history that I just didn't know. Early days stuff. Sure.

Yeah. We'll put a link in the show notes if people are interested. I think, you know, that kind of stuff to me, it's like I'm so focused on what we're working on now that I actually kind of forget every now and then to look back at how far things have come. And it really is pretty remarkable. It's pretty – like I was just saying to Wes, it's like, you know, it's really – there's a lot that has changed. But in such like a way that I'm really proud of. Like it's really been for the benefit of like us taking our craft more seriously and trying to do a better job and trying to do better reporting and doing like research and like contacting sources. Like all these things, staffing, editors, like it's been pretty awesome.

It's just enabling us to do a better job of doing this. It's been iterative. Like if you've been a listener for a while, you know, you can really hear an iterative trend. And that's another nice way of saying our old stuff is really crappy. I mean, you tell me if the pull clips. Oh, boy. It's bad if the pull clips, oh boy. It's bad. We just cringe.

All right, well, we got a little interesting Valve news in the show. It's been a while. Did you hear about Steam Remote Play? It is a new Steam feature that lets you play on your couch or on your computer with your friends over the Internet. on your couch or on your computer with your friends over the internet. With Remote Play Together, they say, you can now invite Steam friends to join your local co-op or your local multiplayer with a shared or split-screen game. Yeah, only the host needs to own and install the game. The additional players just connect through Steam Remote Play.

It's kind of neat because it just kind of shows up as a menu entry item if everybody supports it. It's just a little button in the menu that says remote play together. It's really easy. And then like all your friends, their controllers just act like they are right there on your computer and then you've got a little menu you can control voice and volume and hear everybody. This is some magic. It sounds

so easy. And they right here say just some words on the screen. Supports Linux. It's going to be there. I mean, you know, they've done a lot of work on the streaming stuff in the back end. I remember playing with it, I don't know, four or five years ago now? Yeah.

And I haven't heard about it since. Obviously, it still works, so it's nice to see a new development. I have to tell you, man, I had a Valve developer in my ear like two years ago telling me we're doing big things with streaming. Just wait. And I was like, great, oh, I'm super excited. And then never heard anything and never saw anything. And I was like, oh, that project must have been killed.

I think this is what it was. Skunkworks. It was in the Skunkworks for quite a while and sometimes this stuff takes a little while to suss out, especially when you're supporting Windows, Mac, and Linux. Yeah, I mean, and over the network, that's always a little bit tricky. Well, I'm curious how it's doing the device stuff. Like, how is it taking my, like, say I got a controller over USB or Bluetooth

hooked up to my Linux box. Is Steam just taking those inputs and then forwarding it to the remote? box, is Steam just taking those inputs and then forwarding it to the remote? Yeah, is it just like raw, throwing it over the network? Is there something more sophisticated in the middle? The other thing that's fascinating about this, as you think about it,

is they're now first to streaming. They beat Stadia and they beat the Xbox. I mean, does this count? It is only in beta. They're at least right there up there. I'll tell you what I like about it is without a shadow of a doubt, it's going to have the best game catalog. Right.

The thing I'm thinking about Stadia is I'm going to get my Stadia package in late November and it's going to show up and I'm going to be able to play two games that are even worth it. I hope you like them. Yeah. Whereas this could be any game I've ever owned in Steam. And the fact that you can do local player stuff, that opens up to a whole range of games. It really does.

And you don't have enough FUTs with firewall stuff or anything. So got to give this a go. Got to try it. You know, I haven't really needed to do this or had an opportunity, but it's something that would be worth it. You know, the whole Steam machine thing, it's funny. Now it's almost a perfect time. I kind of want just a little Steam machine set up next to my TV now.

Oh, yeah, steam machines. Oh, yeah. Oh, yeah, that was a thing for a little bit. Aw. You know what you always say, Wes? And you always say this. It turns out if you try hard enough, you can. But sometimes not.

Hey, why don't we just take a moment and mention that the Gnome Foundation is full-on fighting this patent troll that is coming after them for this Shotwell photo transfer feature. Yeah, they're bringing it. It's nice to see. It is really nice to see. And in that effort, they are attempting to raise some funds. So, so far, they've done pretty good. They've got 108 donors. And they have raised $28,000.

They have a goal of $125,000, but my instinct is they could probably use triple that amount because it'd be really nice for them to be able to bring this fight. So we're going to have a link in the show notes. Linuxunplugged.com slash 324. If you can throw, you know, any

amount at them, they have some one times that are just click one buttons. They have some monthlies you can do. It looks like it's a really nice, clean, simple, not going to have to sign up with a whole bunch of accounts crap to do it. And it would definitely be a good cause. Because even if you're not a GNOME Shell user, this patent fight is really important.

Because if they can come out, well, I guess another way to put this is, Shotwell is damn lucky they're part of the GNOME Foundation. If they were an open source independent project that wasn't part of a foundation like this, they would be royally screwed right now. So it's important that

even if you're not a GNOME Shell user, that we help them fight this crap. Right. I mean, we want to send this tone that you don't get to be abusive and trolling these free and open source projects. You know, it gets me worked up and I'm already over caffeinated, Wes. And you know what happens when I get worked up as I start ranting? Let me get the soapbox.

Yeah, we left it outside. It's been a little while since we've gotten up on the soapbox. You never know. It could be just around the corner, though. You can never tell with that soapbox. It's sometimes lurking. Well, you're right. It could be just around the corner, though. You can never tell with that soapbox. It's sometimes lurking. Well, you're right. It was very easy.

I've just made a donation now. Really? Once I got past identifying the cars in the capture, of course. Oh, yeah. You've got to make sure you find the cars. Or buses. Somebody's got to train that machine. Let's do a little housekeeping.

Oh! So, we went down to Texas. Went to a cyber summit. And you may be wondering how that went. Well, my friends, I've got just the thing for you. Elle, Wes, Cheese, and myself did an extra on our trip to Texas Cyber Summit. Extras.show slash 24. Nice little wrap up of not just what it's like for some Linux people to go to a cyber summit,

but also good details on the great food that Carl took us to. Oh, boy. So don't listen if you're hungry. I can't help it. You know? Carl, you might want to listen to that one. I think you'd get name-dropped a couple of times in that one. And also, to go along with that,

Elle wrote up a fantastic write-up. Wrote up? Write up? Blogged? Blog post. Don't call it a blog. It's an article. Article post. That we'll link in the show notes. It has some descriptions, some pictures, some of the characters I was talking about in that extra.

And then if you want even more than that, cheese is popping off like crazy in that Jupiter gallery. Oh, it's so good. Jupiter.gallery. How about that? Jupiter.gallery. Chris, what's this? You say, well, here we go out to these things like these Linux fests. Or maybe you heard about Alex and I taking a road trip to go see Wendell recently.

Or Texas Linux Fest or Red Hat Summit or Linux Fest Northwestern. And you wonder, are there any pictures from this event? Well, friends, Cheese Bacon has the answer for you. That's right. And if you have any pictures from these events that you'd like to submit, you can do so by emailing me at cheese at jupiterbroadcasting.com. And that's C-H-Z at jupiterbroadcasting.com. And I'll be happy to put them on the gallery.

That's the best part is, as that happens, when I go back and check, there's more pictures. There's new stuff I haven't seen. Yeah. Yeah. jupiter.gallery. What else do I got to say? We'll just keep updating that throughout the next year and stuff as we go to events and whatnot.

We'll have that up there. And check out Extras. Extras.show slash 24, but not just that one. For goodness sake, go over there and listen to the What is a Container episode. It's just fundamentals. It's 2019. I mean, you don't want to be behind. You don't.

And it's 24 minutes. And Alex, you did a great job explaining this stuff. You and I talk about how we use these in practical setups that are approachable by people that are either beginners or using this in an enterprise. Because we've talked about these things a lot. We've taken all of the experience we have talking about these, and Alex talks about them all day for his day job. And we've put it into this episode. So I think it's helpful for people. And it's a great companion for the new self-hosted show.

Check that out. Extras.show. And also recently, some brunches just got posted. So, gotta go get those. Oh, so many good brunches. Especially that one there with Alan Jude. Mentioned that last week. You gotta go get that Alan Jude brunch. You know he's gonna be talking about BSD

and making you question your Linux faith. You don't want to miss that. And then come back to Linux Unplugged. We'll set you straight. Oh, that's true though. All right. That's the housekeeping for this week. I mean, speaking of Linux, right? Let's get into this 1910 in more detail. There is a lot to talk about with this release, but we're going to actually put some of it aside and just wait for 2004 and just focus on a few details for you this week that drew our attention. And one that I know Cheese and I have been talking about,

and I think Wes, you and I talked about a little bit, is I am noticing more than I thought I would that there's a lot of apps in the software center that are actually snaps. It seems to be the way it works is if there is a snap version of an application, the software center prefers that and installs that first. Now, maybe we should be clear.

How are you noticing this, right? Because. Oh, they clearly label it. Right. But would you have noticed, right? Isn't that the test? You know, is it detrimental if you're a regular user? Do you care?

Because you're investigating, you know, we're doing a review here. The only two ways in which I notice it is startup time. And when I sometimes go to edit something on my local file system, it's like a couple extra like – Right, there's the confinement comes into play. Yeah. And so there's a couple of instances where I'm like, well, the job of this application is to be like all over my file system. So I don't actually want the Snap version. And the way I've just worked around that is I just didn't go to the terminal.

I apt install and I get the depth. Right. But to your point, for the most part, no, not really actually an issue. And in the case of VS Code, seems to have been a net benefit because it just like self-updated and I got the latest VS Code and it was good to go. Right. I mean, I'm running the Chromium Snap right now as we're doing the show. It's working just fine.

Yeah. Yeah. That's so you can. And it does seem like they've gotten better, you know, I mean, the theming, the system integration, that stuff is a lot better than the last time I used these. Yes, yes. That has gotten a lot better. So, Cheese, you had a chance to not just only use stock 1910, but also Mate 1910 over the week, trying out different systems.

And what was your extra week with 1910 like? Overall, it was great. So, I started with the beta, then the dailies, and then the final. I tried it on three different machines. I tried it on my desktop, which is a Ryzen 2600, 16 gigs of RAM, 256 gigabyte NVMe, and an NVIDIA 1060, an older Dell Inspiron, and this new ThinkPad 440S, the T440S that I've recently picked up. I will say that out of the box, whenever I tried Ubuntu, I installed Steam, immediately went to CSGO, dropped in and was playing some games, some Deathmatch there in CSGO. And I was averaging around, you know,

200 frames per second, something like that. What? know, 200 frames per second, something like that. Yeah, 200 frames per second. What do you normally get on that box? Under Windows, it was right around the same. So, I mean, there was really no difference. It would peak up to like 230 frames a second, which was about average for what Windows was doing as well. Oh, so this was the last Windows box, the gaming rig. uh the last windows box the gaming rig yes so this was the gaming rig that i ditched um i had an extra nvme tossed in there and i ditched the ssd are you telling me 1910 killed windows for you uh yeah i mean i really the desk this desktop was

only ever really used for gaming anyway so um i decided just to make that make that jump. I do have the fallback SSD if I want to toss it back in the box. I physically removed it, though, so it would take some effort. You go take this victory lap. Jeez, this is a big moment. And Wes and I are proud of you. Thank you, guys. So good. So it sounds like performance is comparable. You haven't taken like a Linux tax for switching. No, not at all. And I was using the NVIDIA, which one thing that's awesome about 1910 is it comes bundled with the NVIDIA driver. So I was using the 435 NVIDIA driver. Worked great. Whenever I flipped over to Mate, though, I did notice a performance hit. I was averaging about 100 frames per second.

I'm not sure exactly what's going on there. Again, this was just kind of out of the box testing. I didn't do any additional stuff. So you're saying you had 100 frames per second switch that there must be something there. That's got to be a bug or compass issue or something. It's probably worth troubleshooting. Or the fact that Linux and NVIDIA aren't best friends could be a thing, too. I don't know. Well, so I had some Pi problems in my week with 1910. I mean, we primarily, the rest of the episode, we're going to talk about the ZFS and Extended 4 and comparing that in low memory and regular memory. But I just want to take a moment and say that they sneaky included support for the Pi 4 in 1910. It was sneaky. Mm-hmm.

They did list it in their official, like, canonical.com or whatever. Not much before. Yeah. Pi 4 support's not easy, and it's something that distributions are slowly getting to. There are architectural changes that are pretty significant. I heard it described as the Pi 4 is in appearance in similarity only to the previous Pis.

Like there's total differences for the USB stack, for the way that the PCI bus is addressed, for the video card. Like there's just major changes. Isn't it great that we don't have to care about that at all? I mean, once it's fixed. And if you're using Raspbian, you don't even think about it. They just did it. Which, you know. And if you're using Raspbian, you don't even think about it. They just did it.

Which, you know, that's the point of Raspbian. But they got like the bulk of everything in 1910, to their credit, which is great because people like to use Raspberry Pis for like Kubernetes training and things like that. And I threw it on there and they spun two images, a 32-bit

and a 64-bit. As we record, although this is very rapidly in progress, the 64-bit image has an issue where USB devices don't work. And there's a couple other limitations of it. It's an issue once you go over like three gigs of RAM-ish, like 348, I think, is the magical number. And the USB quits working. Uh-oh. And so if you use the 32-bit image,

it's totally fine. And no issues. You get to play around with 1910 on a Raspberry Pi if you're okay with 32. But you've been hoping to escape 32. I mean, come on, 32-bit?

Everybody knows it's going away. I mean, you're using all the bits. I will mention, too, the 64-bit and 32-bit images have SSH turned on by default. So even if you can't use the USB to plug in a keyboard, if you got the Ethernet plugged in, you can still SSH it. Perfect. They already have it in progress, though.

I did my part in the bug report because I was a good boy this time and went in there and said, here's my system information. Here's my, you know, I can reproduce the bug. And they seem to have identified all the issues and fixes are in progress right now as we record. So they'll probably, I don't know, maybe they'll spin a new image, I would hope. Yeah, hopefully so. Hopefully it just works in the future. Yeah.

Now, what about you? How's your week been on 1910? Any updates to your initial impressions last week? Honestly, I mean, it's just been pretty smooth. I had to, you know, make a few changes here and there, and there's an app armor that was still enabled on this system, which, I mean, I left it enabled, but, you know, I did some tweaks, got all my regular tooling working. Really, it's kind of just nice. I mean, I've been enjoying Neon, and I do miss Plasma

sometimes. There's little pieces of my workflow that I just haven't really adapted over. But I have surprisingly few complaints. Like there's no weird oddities. I haven't had any glitches. There's been no surprising reboots or programs not working. And I was doing a bunch of testing, sort of messing around with some of the bootloader stuff and modifying the system. It's just resilient. My past experiences with Gnome Shell tell me that this may be subject to change,

but right now I haven't had any shell stability issues. It's all been very solid, very reliable, and I'm very happy to report that. I had a chance to try out Pop as well. I think we all did, most of us. Boy, did I see a lot of excitement online about this. I wasn't even positive I was going to try it out, but then I was getting so much like, hey, Chris,

you got to try this. Hey, Chris, have you tried pop? It's really good. A lot of people, yeah, talking about the release. Yes.

So I did some digging into what, why are people so happy about this? Like what's going on? And there's a few things that I think System76 has done here, but the one that like

a lot of people were talking about on day one and two, for obvious reasons, was how buttery smooth the upgrade process was from the previous version of Pop. And they appear to have done some custom engineering around this. And so I emailed Carl, the CEO of System76, and asked him, what have you guys done to change the upgrade process and why is everybody saying it's so much better? Carl wrote back and he said, the Ubuntu updates weren't well integrated into GNOME and we found that live upgrades while in the user session can be unreliable.

We also had no way to communicate the upgrade and changes to the user through the product. We also had no way to communicate the upgrade and changes to the user through the product. Instead, users learned about the available upgrades through news or social media or like an email campaign or whatever. Now, customers receive a notification when there's a new OS upgrade and they click a button to start the process. The POP upgrade first prepares the system for upgrade, ensuring that all the required dependencies and upgrades are in place, and then downloads the new packages. If there are conflicts, the upgrade is automatically rolled back. The customer can go off and do other work while the upgrade is being prepared and downloaded. They'll receive a second notification when the upgrade is ready to be installed.

Clicking upgrade prompts to reboot, where the offline upgrade will then take place. And with the pop upgrade, customers will be able to upgrade from an LTS to any release. So, for example, if you decide you don't want to be stuck on POP 1804 until 2004, you could go to 1910. You'll be able to jump directly from 1804 to 1910. That's coming in

early November. Oh, nice. So, they've done some really interesting engineering there. Also, I love some of the other things. He says to him, the most striking change in 1910 is the new aesthetic. The team, Kate, himself, Ian, they started working together on a new color palette and contrast. He says they measured the contrast ratios between elements and text to achieve a high contrast by default rather than requiring

a separate theme. He says the difference is noticeable throughout, but especially in the shell menus. Yeah, he's right about that. And he says, I love the new dark color palette. It's incredible. It's like candy, like a luxurious kind of candy. I think they're very proud of it. It does look very nice.

I mean, I enjoyed using it. Yeah. Also worth mentioning their work around TensorFlow. I didn't really appreciate what a pain in the butt apparently TensorFlow is to set up. I love this. Packaging TensorFlow for Linux distributions is notoriously difficult, if not impossible.

That's the first line of the GitHub. Yeah, and I guess even like using the Docker container is still a huge pain in the butt. Well, right, you've got all kinds of different versions depending on quite what you're doing or how you need TensorFlow configured. And, I mean, if you're running these just off the command line in Docker, it's not a lot of fun. So, TensorFlow, which is a little Rust daemon, of course,

it just sits there and makes this easy for you and kind of orchestrates on top of Docker to get you all the stuff you need and smooth out the process. That's pretty great. And it seems to me like they've keyed in on a good customer demo there because what do they do? They sell powerful systems

and like laptops and desktops that people buy to use TensorFlow. Like that must be a decent customer base for them. Right. I mean, that could be pretty useful. And if it just gets you out of the gate with the standard technology you want to use on Linux faster, that's superb. The thing that I have to acknowledge before we go any further is I was skeptical of Pop! OS. In fact, I was pretty vocal about thinking it was a bad idea that they had limited resources and it could be better spent in other places.

We're several releases in now. Thankfully, I left myself a little bit of wiggle room on this. And I said, you know, if they add value and they make it something that's a true differentiator, then I'll reconsider.

They've done that. They've done that with the installer. They've done that with the upgrade process. They've done that with the theme. It's truly one of the best themes on Linux. And now they've done it. There's a couple other things

that are in there, like the pop shop that are really nice. Like they've done it. There's a couple other things that are in there, like the Pop Shop that are really nice. Like they've opted to keep Chromium as a dev. You know, they can make that choice as they do with their platform.

They've integrated their system firmware updater into GNOME Shell settings now. Just built right in. You don't have to go find it. And even on a non-system 76 machine, I appreciate that. That's, I think,

what has stood out to me, is it hasn't felt like Pop was some walled off garden, hidden away, totally different, right? It is its own thing, but a lot of the stuff you can still take, right? If I want to use TensorFlow, I can just go get that from GitHub and use it. Or if I want to use the theme, but not all of Pop!OS,

that's possible too. I think Pop!OS, because it's not tied to System76's hardware, you get benefits by using it, as I experienced in my recent laptop review. Yeah, right. But you don't really have any downsides. There's like not any negatives to it.

And it's not so derivative of Ubuntu that it's totally different. Like, Mint is kind of its own thing. Like, you can still skirt by by following some Ubuntu tutorials or getting a PPA and it's probably going to be fine. But with POP, there's no like probably about it. Yeah, it's going to be fine. It just works.

It's 1910. They want it to work. Exactly. So there's not a penalty really for using it. There is an upside if you like their aesthetic and the value add that they've contributed. And you're not running something that's so foreign that you can't just treat it like an Ubuntu system.

They've really nailed a sweet spot. And because of that, when I do one more reload for some testing upstairs, when I'm on my final build for my daily driver workstation, I'm going with POP 19.10. And the reason for me is I'm going to take Carl at

his word about that upgrade system. Because this is my got to get shit done workstation. And I was, after all of this, going to just reload it back to an LTS. I don't got time to play around on that thing. Yeah. When we reinstall all the other machines in the studio, you got to have one, you know, is working. Right. And I honestly was not looking forward to it because the new GNOME shell has kind of broken me. It's very fast. It's very nice. And I hate to go back to slower. Cannot stand to downgrade.

Right. And I was really hemming and hawing putting it off. And this gives me an opportunity here. I was really hemming and hawing putting it off, and this gives me an opportunity here. I can get the latest and greatest and have a pretty solid chance it's going to upgrade as smoothly as, say, a Fedora system will. I mean, I'm already looking forward to finding out how that goes.

So I'm going to put it on my main workstation upstairs. Hooey, Wes, here I go. You know, my dad used to have a saying, I used to be indecisive, but now I'm not so sure. I think that applies to you in desktop environments. You know, hey, if they keep making it better, I'm not going to ignore it. I'm no fool. I'm going to acknowledge it.

Plus, I mean, you'll already be set up for making all those deep fakes. You know that's my new hobby. I'm going to set up a side hustle where i just do deep fakes for hire i probably shouldn't sit that on there damn it damn it well i can i can attest to pop os being pretty solid man i mean i've been using it since march the upgrade process was super i mean it was just painless um it did in fact pop up say that there's a new version available. See what I did there? It popped up. It did say there was a new version available. It pulled everything down, asked me to reboot, installed the update perfectly, right back into it. The new contrast, I mean, immediately, I could notice a difference. and it was overall a really great experience for me.

I'm installing this on an older 2015 MacBook Pro and I really have no reason to go with another distro on that machine because very much like you, it's just the get shit done type machine, you know, and it works great. I really have no reason to move anywhere else on it. It was almost good enough to get me to give up Fedora on the ThinkPad. I got close. And what I ended up doing to compromise is I went through and just beautied up my Fedora install a little bit. It does look pretty nice. I got my Dastodok more dialed in. I got my theme dialed in.

I went and got the Pop theme. You know, something to note here, too, there is no ZFS on root in the installer for Pop!OS. So let's talk about ZFS on root. The promise is huge. Eventually, one day, you have a file system that protects you from bit rot. It can do incremental snapshots that are elegant. You could potentially have boot environments. You could move data around. You could overcommit storage. It integrates with container platforms. I mean, it's truly worth all of the hype.

I mean, the advantages are clear. Now, is it worth it on root? Especially if maybe you don't have an MV&E drive, or maybe you don't have 32 gigs of RAM. Right. Oftentimes we're thinking about big systems, maybe custom server builds that you've designed to be running this file system with a lot of data. The old myth was you can't use CFS without ECC RAM.

Right. And what about a laptop, let alone? What about kind of not the newest laptop with run-of-the-mill hardware? Yeah, maybe you got a Samsung Evo in there. Maybe you got 16 gigs of RAM and you got a few apps you're using every single day. And you don't have a bunch of available memory. Is this still a good idea? Genuinely did not know the answer to this question, and now I'm feeling like we have figured it out. So Wes and I, I'll give you a little bit of basics. So Wes and I took the Librem 15 with an Evo 850 drive, 512 gigabyte

capacity, and 16 gigabytes of RAM, and I think it's a Skylake processor. And we installed Ubuntu 19.10 fresh, latest spun ISOs, and did one install with Extended 4, and one install with ZFS on route. Ran the same exact software setup, same exact steps, same exact benchmarks. Then we did it all over again, but this time we robbed the system of RAM to create and simulate a very exact amount on both of them. We took out a chunk of RAM, so both of them had to operate in a low memory environment. And then we started a user session with a terminal and a web browser with some complicated, but not overly complicated work in there that was reproducible in both environments. Right. You know, model something like maybe you're running a bunch of virtual machines or doing other work or possibly just have a, you know, more low-end computer available.

Right. And so we did things like we threw some compile jobs at the hard disk. We threw some SQL work, creating lots of little files, creating large files, creating small files. We did it with as much RAM as the system could eat, and we did it in a more memory pressure environment.

So we essentially operated the system as if we were using about 60% of that 16 gigs when we did the low memory test. Because then the way we got to this number is we took several of our systems here in the studio, and we loaded them up with our daily driver applications. A couple of Electron apps, some Qt apps, some GTK apps, a web browser, a couple of terminals, you know, that kind

of stuff. And then we came to an average memory usage after they've been running for a while and determined, all right, this is what it costs us in memory to run our workstations. and determine, all right, this is what it costs us in memory to run our workstations. And then we allocated that for this test and ate that up virtually by just slicing it off from the kernel, saying you can't have it, and let it go. And what we found, in some ways, really surprised the heck out of me. The answer on the broad perspective is, it depends on your workload. Of course. Of course it does. But there's some areas where

even in low memory situations, ZFS pulled ahead. Not in all areas. No, certainly not. I mean, there's many categories where 64 is just going to be faster. It's doing less, right? If you were to attribute performance failures in a high memory pressure environment, it was actually extended for that demonstrated most of the variance.

Yeah, that's what kind of surprised us the most is even tests where maybe it was already slower or already faster. The change between having more memory available and it was like nothing in ZFS. I mean, you know, it got a little bit worse, but not much. There was some degradation, but not much, yeah. And then much more for EXE4. Yeah, you got like nothing in ZFS. I mean, you know, it got a little bit worse, but not much. There was some degradation, but not much, yeah. And then much more for EXE4. Yeah. You got like an entirely different box in some benchmarks with Extended 4. Now, in some situations, Extended 4 may have still performed better than ZFS.

I'll give you an example. Say you're unzipping a project and you're assembling it and getting it ready to compile. In that situation, in a low memory or a regular memory situation, Extended 4 is simply faster. ZFS, though, remains consistent. And then once you begin to actually build the project, and this could be attributed to how ZFS caches files, ZFS is actually significantly faster than Extended 4.

Yeah, you know, it was a pretty healthy competition here between what EXT4 was better at and what ZFS. Certainly not one-sided in favor of EXT4. No. MicroLarble did some of his own benchmarks over at Pharonix, which we'll have linked. On some higher-end hardware. Yeah, and our results are pretty consistent.

Now, he also had a bunch of other more like server applications, stuff like various databases, and of course those were faster with EXT4, and I mean, that's kind of expected. I was just surprised how similar things were just in the everyday use case and how resilient ZFS was to memory pressure. It doesn't really make sense to me. If I didn't have the data here, and we will publish all of this,

and you can compare it to your own systems, if you are familiar with how to use the Foronix test suite, actually comparing your system to our benchmark is one of the easiest things to do with the test suite. If you're just curious to see how your system would perform. So we'll publish all of our data with the charts in the show notes. But by going with this, I have to say, I would advise you're perfectly fine using ZFS. When you add the benefits. Right. Yes. You're probably going to ZFS. When you add the benefits. Right.

Yes. You're probably going to need to be taking care of the minutes. And things may still change here in how Ubuntu is using it. And one time we did run into a little snag where Grub wasn't quite updating properly after, you know, and that only happened on the ZFS side. So totally worth doing it. It seems like it's very usable day to day. Be aware you may need to learn a few things about ZFS said. So, totally worth doing it. It seems like it's very usable day to day. Be aware you may need to learn a few things about ZFS administration. That is a good red flag to bring up because I'm only talking about it from a performance penalty standpoint, not yet from an implementation standpoint.

Right, right. That's still in flux, and it's, as they label, experimental. And you need to consider it as such. ZFS won't protect you from, like, a total different schema layout change that's incompatible with what you've got set up. Like, there's just nothing that's going to... Or, like, we ran into grub not updating properly

when we were trying to do a couple of small little things to reduce memory and increase pressure. Thankfully, we were just going to reinstall anyway. Yeah, yeah, true. So I do want to make sure it's clear. At this point in time as we record, we are talking about performance and not necessarily implementation. But in a performance sense, when you look at all of the crap ZFS does from snapshots to write verification to ZFS send, which once you wrap your head around what you can do with ZFS send, Extended Forward is going to look like a child's file system.

you can do with ZFS Send, Extended Forward is going to look like a child's file system. And when you think about all of this that just comes with ZFS, and it's so big, and it's got all this stuff, the fact that it is competitive on a three-year-old laptop

where we intentionally drained the thing of RAM to give it a real world, heavily used system condition. I mean, we were monitoring, too. I mean we were monitoring too. I mean it was busy. It was working hard. It was working really hard. It was great. And I'm just damn impressed with what the ZFS

project has managed. ZFS on Linux, OpenCFS in general, it's just fantastic. And having it available, ready to go without the messing around with DKMS which by and large is pretty reliable, but I still find myself running into issues and not having that. It just feels so much more reliable and professional. I thought what we'd do is I thought we'd come on the show

and say, if you just need all out performance and you don't care about data integrity, snapshots, or any of the other features, compression, encryption, which, by the way, we did not have compression turned on. Compression may have significantly changed the game here. Right, we were just using defaults on both. If we had used compression, ZFS may have kicked ass on this. It would be really interesting to run it again, actually, now that I think about it. Oh boy, there's always more benchmarks.

But anyways, I honestly thought at the end of all this, we come here and say, if you just want full, nothing but performance, go with Extended 4. But the reality is, depending on your use case, you may actually get higher performance

even in low memory conditions with CFS, depending on your use case. We have a few links in the show notes. The one that I think would be really interesting for you to check out

are the low memory versus regular memory tests and the low memory extended four versus low memory ZFS. And then it's really easy to compare all of them actually, but we'll have the links in the show notes. It's some interesting data. I walk away from this thinking it's safe to, it's safe from a performance standpoint to use ZFS. And even if it's on a system that you want to have high performance from, and then there's additional benefits as you go up the hardware scale. Like when you start going to MV and E drives, there's additional benefits to going with ZFS. It just gets better. And when you start wanting to use multiple disks and pool them together, there's a lot more benefits to going with ZFS

that Extended 4 can't offer without having to strap on LVM or other management tools, whereas ZFS brings it all in one suite of tools. You're ready to go. So this is, like, really fascinating for me because what the data says is even in a worst-case ZFS scenario, old SSD, three-year-old system, limited memory, it's still very competitive. And then when you put it in a worst-case ZFS scenario, old SSD, three-year-old system, limited memory, it's still very competitive.

And then when you put it in a situation where you have a lot more hardware to throw at it and you have a lot more disk to throw at it, the benefits just really get bigger. All right, well then, I think it's prediction time. What do you think? Does this pave the way to ZFS on route by default in a future Ubuntu release?

Can you see it? I mean, it would make a lot of sense, especially on the server side when you want to protect from bogus updates or configuration mistakes. Right, and it would be a big advertising point if you had that integrated into that Ubuntu tooling, right? Ubuntu is a great choice

because, look, you can just roll back. If we could properly get boot environment support at Grub, it'd be worth it just to annoy the BSD crowd that already has it. You know what I mean? Look, we caught up. Yeah, but we talk about it like it's the first time. Oh, we've got to pretend we've never heard that they've done it. They hate that.

Because they hate that. They hate that. And so that would really make it perfect. That's what I hope for. Do you think that they would ever shift and do ZFS on the Ubuntu server while EXT4 on the desktop? Could do. You know, on my machine upstairs when I reload it, I'm not going with ZFS on my route. I'm still waiting.

Again, because that's an implementation thing. Also likely go pop and they don't have it in the installer. However, I am totally, I have three disks that I pool together and I am absolutely going to move them over from XFS to ZFS. Right. That's the other nice thing here is even if you don't do it on root, well, if you run a bunch of business at hand. I may be near the end of ever really using LVM again, although LVM has been great and solid and I've got nothing against it. I like the ZFS tools better. And as I learn them more,

I think they seem to be more complete. And the whole idea of bringing it all together where the file system is aware of the hardware and aware of the RAID and aware of what the user space is doing brings so many advantages.

So it's a bright future either way because you can still pick whichever file system floats your boat, but you could also go with something new and fresher. Chris, if you say it in and out of louder you know the bsd folks might hear you are they listening have you done any zfs experimentation over there i have yeah so i i moved from debian to ubuntu for my primary server simply to get the zfs support oh really

why why what was so like gotta have because had, what, various solutions in the past? I went to a talk by Jim Salter. That'll do it. At LinuxFest Northwest. And then Alan Jude gave me one of his books for free. And that was that. That's having two devils on your shoulder in that case, telling you. It's really compelling, you know, all the checksumming stuff. So the use case I use it for is I just have a pair of drives mirrored that store all of the irreplaceable data. So my drone footage, my photos and stuff like that, you know, the stuff that is, you know, irreplaceable,

genuinely irreplaceable. And then I use ZFS send to send that to England and then Duplicati does a bunch of other stuff with it as well. But I've really been enjoying it. And yes, there's a lot to learn. And there's some monitoring stuff with how like DF handles monitoring free space. And there are some things to learn,

but I don't mind learning stuff. That's part of the fun for me. Yeah, it does have a learning overhead. Yeah, but that's true for, you know, anytime you kind of step out of the box, right? Even with BunderFS. True. Okay.

Yeah, very fair. Brent, you had a question about CFS capabilities. Yeah, well, I often tend to like default to using encryption in many places as a way of forcing myself to learn it. But I wondered about ZFS and encryption. Does anybody have any experience with that or has done any research? I haven't played with it too much yet, but they do now have it natively inside ZFS. I think before you usually would kind of layer it on, but that would sidestep some of the direct give ZFS direct access to the disk benefits.

I haven't played too much with it, though. Yeah. So I think just recently landed in the last couple of point releases for ZFS on Linux. FreeBSD has had some solutions for a little while, but it's pretty nice. But it's pretty nice. One of the things that I'll probably look at doing in the future is, in CFS, you can have these concepts of pools of disks. So you can have like a media pool and like a documents pool and stuff like that.

And you can do partitions across them and whatnot. But you can also just say, encrypt everything in this pool. So if you want to have like a documents folder in your home folder, that could actually be mapped to an entire storage pool. That one folder could be mapped to the documents pool. And you could just say anything I put, which just looks like a folder to you, but is actually an entire pool of disks. Anything I put in this folder, encrypt it. And then once you set it, it's done.

You don't worry about it. You're exciting me. I think, you know, it's not quite there. It's not daily driver yet. You're exciting me. I think, you know, it's not quite there. It's not daily driver yet, but we've got such great potential where the desktop environments and the toolkits around them,

because like we haven't given a lot of attention to Plasma on this release, but also just- Still doing great. Still doing great. Very solid. And working away at their Wayland support, making it a top priority for KWin.

It's really gotten to a good spot. And this is just another part of it. This is a part of the stack that we get to enjoy as free software users. We just get this stuff for free. And a whole team of people worked on packaging this all up for us, and then one day, an arbitrary date, they decide you now get this thing,

and it can change your life. It can save your precious memories. It can change how you back up and move data around the world. It can provide features that you've never thought possible with your computer before. It makes your computer more capable.

Even if you choose not to use it, that's awesome. That's it, right? Like, I mean, there's no going back if you're like, oh, I didn't think I'd want those pictures, but I left them on this old drive here. I mean, if you have ZFS underneath you,

better safe than sorry. Well, if you just look at it from like an old school, like imagine your favorite Linux distro as a boxed product on the shelf. And, you know, every year they got to have new bullet points on that box. Well, your favorite distro box, if it's Ubuntu based, just got some unbelievable features that just would, if you're looking at it from features on a box standpoint, it's just incredible. And you combine that with everything else we can do on Linux these days,

it just blows my mind. Just blows my mind. So check out the links in the show notes for our benchmarks, including the comparison of low memory versus high memory, them with just, you know, having at the system as much as they can, and then compare it to your system. Yeah, we'd love to see that.

Maybe let us know if you get some interesting results. I have a pick for you this week that I think is going to be perfect for you, Mr. Bacon. It's called ThinkPad Tools. This is going to be applicable to everybody, but it's pretty nice for those of you that have a ThinkPad out there, and it sounds like you just got one. Yeah. So I just recently picked one up. I picked up the

older T440S. After doing some research, I probably should have got the T440P, but, you know, it is what it is. $170, 12 gigs of RAM, 256 gig SSD. I think he just wanted to brag. that's why he brought this up this is the biggest humble brag well i mean you could find the same deal for a little bit cheaper if you did some digging out there and i know that um after after watching popey you know go on his spiel about it uh popey's to blame he makes me want to buy used thinkpads too well and then i just given my older ThinkPad to Elle, so I was jonesing for that track point again, you know?

Well, welcome back to the club. You know, like, having a ThinkPad, I hadn't experienced this before having one. It kind of is like a little club. There's something special about Linux users and ThinkPads, and that's why we get all the nice toys, like the pick this week ThinkPad tool. So sorry if you don't have a ThinkPad, although maybe take Jesus' advice and go find one near you. I can't believe that.

That's a great value. And that'll be a great test machine. You know, we've got a nice range now. We've got a good range of super high-end machines that are ridiculous in scale. And then we've got machines that are real practical daily drivers that people might be picking up either secondhand or just own for a couple of years and don't need to replace it. And so I think we're going to get a good, with the whole team now, we're getting a good representative sample of people's machines. The other thing I'd like to add about ThinkPads is that they're super easy to repair. I swapped out the keyboard on my wife's ThinkPad in about an hour, you know, so even if you pour water over

the thing, it's cheap and easy to repair. We don't recommend you do that. We do not recommend you do that. Anyways, if you're thinking, well, what does it do? It'll adjust the track point speed and sensitivity. It'll help you get better battery information, and it can help you troubleshoot that undervolting CPU issue we've talked about before.

Yeah, I mean, I didn't know that my sensitivity was 128, but now I do. Now you do, Wes. Actually, it is like, I mean, the little battery status. It's nicely displayed in the terminal, and it's just in Python, so it's easy to install. Yeah, so check that out. Link in the notes for that, and all of it's at linuxunplugged.com slash 324. While you're there, you can subscribe to get the show directly every single week at linuxunplugged.com slash subscribe.

But more importantly, if you want to give us your thoughts, your feedback, your take on the new release of 1910 or anything else we've talked about, linuxunplugged.com slash contact. That's so easy. There's just like a form there and it sends an email for you. Easy. Easy. You could also tweet us at linuxunplugged on the Twitter. Yeah. And the whole network's at Jupiter Signal.

That guy right there. At Wes Payne. I'm at Chris LAS. Cheesy. I'm at Cheezbacon. C-H-C-B-A-C-O-N. That guy right there. At Wes Payne. I'm at Chris LAS. Cheesy. I'm at Cheese Bacon. C-H-C-B-A-C-O-N. Go get some

Alex. At Alex is a KTZ on the Twitter. Ironic Badger. Oh, that's right. Ironic Badger. Yes, Ironic

Badger. How can I forget Ironic Badger? So go get some Alex too. It's so ironic that I forgot that except that's not

ironic. That's not ironic at all. I know. I know. I'm sorry. Anyways, and did I mention mine? Probably. You did, yeah, but you didn't mention that you should go get more of you and Alex over at that

self-hosted show. Oh, we should mention that. Yeah, and while we're just doing plug skis, I should mention techsnap.systems slash 414. Wes and Jim have such a great discussion on ZFS in general, especially like around the snapshot stuff.

Like there's so many good points, but Jim's explanation of snapshots truly made me understand it and appreciate why it could happen pretty quick. Why the snapshots don't take a ton of disk space and why it's kind of a big deal that this is landing on such a large Linux distro. Put it all into perspective. Techsnap.systems slash 414 and join us next Tuesday

over at JBLive.tv. We do this show at 2pm Pacific at JBLive.tv. Got a chat room, the mumble. See you back here next Tuesday! Thank you. Now we've got to title this thing. Potato Computing. I like that. It's as slow as a potato. You know what?

Potatoes are great. So let's not throw potatoes under the bus. Especially them free range taters. It's the only ethical kind of tater. Of course. It's a free range tater. Be kind to the potatoes. So let's go get this thing named.

I'm going to vote for Ram Burglars. I'm going to try to slam this install on my system and then be done. Because in the last three weeks, I got a different phone. I've been loading and reloading systems, and so, like, nothing's set up. Like, I gotta log into everything, you know what I'm saying? Like, it's just been weeks of setting up software.

So, like, I'm gonna, after this show's over... Should we take away your root privileges on there? I'll set a BIOS password. Right, yes. No more. After this episode, I'm going upstairs, I'm reloading that box, I'm ordering food, and I'm not leaving the studio until my basics are logged into, and I'm calling it good for It's nice to have that ready to go, though, right?

So, like, tomorrow you can actually just get work done.