LINUX Unplugged - 314: Bigger. Faster. Harder to Maintain.

Episode Date: August 14, 2019

It's huge, and it's getting bigger every month. How do you test the Linux Kernel? Major Hayden from Red Hat joins us to discuss their efforts to automate Kernel bug hunting. Plus our honest conversation about which Linux works best for us. Special Guests: Alan Pope, Alex Kretzschmar, Brent Gervais, Drew DeVore, Ell Marquez, Major Hayden, and Neal Gompa.

I can't believe I'm saying this, but I've switched away from systemd. What? I don't believe it. I'm just kidding. Hello, friends, and welcome into a very special Linux Unplugged. This is episode 314. My name is Chris. My name is Wes.

Hello, Wes. Hello. How are you doing? Oh, I'm doing great because we've got a full house today. We do. It's a special week here at the Jupiter Broadcasting Studios. We have the full core team in-house. And I guess that's my quick way of saying people that are all involved in creating a brand new podcast for the network. What? Are in-house this week. And that means that we have a whole cast of characters here helping us get various jobs done. So I want to say a very happy hello to Mr. Cheese Bacon. Hello, Cheesy.

Hello, hello, hello. Hello. Good to have you here. It's good to have everyone here. And hello to Alex. Nice to have you in studio, Alex. Howdy, y'all. Howdy, Alex.

Oh, you're sounding good, sir. That sounded nearly real. Thank you. Nearly. Nearly. And it's always good to have Brent back in studio. Hello, Brent. Hello, everyone.

How are you doing? I'm great. It's good to be connected via real, like, cords and stuff to the studio. That's nice. Brent made me breakfast. What? Mm-hmm. Brent made me breakfast today.

He's such a gentleman. It's so good. And Drew barbecued lunch, so. Pays to have the team around. Now, of course, this wouldn't be a Linux Unplugged without our virtual log. Time appropriate greetings there, Mumble Room. Hello. Hello. Hello. It's really good. Aloha. Aloha, Neil. Aloha. Welcome back. Neil will be giving us his flock report in a moment. But Emma, I just have to say it's super great to see you back. How are you doing? I'm doing good. I will be off crutches pretty soon, but I returned to system 76.

I've returned to the factory, so I'm moving around pretty good these days. Man, if people don't follow Emma's social happiness on Twitter, she recently had a serious medical situation. Just, I mean, unbelievable to watch from afar, Emma. It was a safari accident, to be specific. That is very cool. We should make that a disclaimer because, you know, other people get injured, but Emma got injured on a safari. So that's true. But it's really good to see you back. I've been following your journey on Twitter.

Nice to see you back at work. We have some cool stuff you and I are going to be talking about in a little bit today too. So I'm glad you're here. But let's kick things off with a wee bit of community news. Did you see all of this hoopla around this horrible bug that was discovered in KDE that could potentially make it vulnerable to attack? And then the KDE project responded by just ripping the feature out of Plasma, I understand. What happened, Wes?

So it's not, you can call it a bug, or really it's a design decision, because it turns out you can execute scripts in your.desktop or.directory files. And why would you want to do that? You know, the funny part is when they removed the functionality to do that, the words accompanying the removal were, we didn't really have a use case. But you can imagine, right, it's just like a little hook you can run. You can have a script.

You can have a script to set up an environment ahead of time. I mean, I could kind of. I never wanted to. Yeah, yeah. That's funny. So they just ripped it out. Yeah. You know, there was a little kerfuffle here.

DEF CON was going on, so the researcher disclosed it in a way that the KDE project was not too pleased about because it just showed up on Twitter. And the researcher actually sort of stated that, well, DEF CON was coming up, and I just wanted to get this out there. Man, that shit really pisses me off. The one thing is when you're doing this is be responsible. Be responsible and work with the project. And going on Twitter just to build your personal

brand has got to be one of the worst human being things to do. Because not only do you make a project look a little foolish because you don't give them the opportunity to work with a heads up that just about everybody else gets, but you're putting real users at risk to make yourself look good. Yeah, there's just no point. It's disgusting. We could have got this solved and pushed out before anyone knew. Now, I doubt that this was being abused in the wild. It's not a super important vulnerability, but still. How do you think I got that keylogger on your laptop?

What? Yeah. Dang it. Yeah, we'll have links, though. You can go read the, there is a little white paper about the vulnerability, and you can go see that, well, that feature, it's gone. Finally, ladies and gentlemen, we have perhaps the first modern core boot server platform. Supermicro's X11-SSH-TF, I love the name because it's got SSH in it,

is a platform that supports Core Boot. This platform is the first modern upstream Core Boot Server platform on the market with an Intel Xeon E3-1200 V6 processor, which is commonly known as the Kaby Lake DT. Now, Wes, you did a little sleuthing into why Supermicro would be interested in doing this

and sort of what the motivations are behind this development. It kind of checks out, kind of makes sense. Right. So this was over at the 9ESEC site. They've worked with MoVAD, which is a VPN provider, and it's all part of the System Transparency Project. And basically the idea is to, you know, get as much of this stuff open sourced and transparent as you can,

especially if you're interested as they are as a VPN provider of, you know, being able to guarantee things to their customers about how their systems run. And it's all well and good to have an OS that you can, you know, has audit logs and shows that you're not keeping other logs or exactly what you are keeping.

But when you can't trust the firmware, you can't trust anything else. So true. And you can see from a, from like a marketing standpoint, this is a great thing to have on your site. Fully audible platform from the VPN software all the way down to the core boot hardware of this system. And I mean, as you've seen, if you've ever tried to use core boot, most of the time, anything that supports it is way out of date. But that's just not going to work if you're trying to run a business. Speaking of core boot, our friends over at System76 and Emma have a little news around core boot. It appears System76 is preparing to roll out their first core boot enabled laptop, which is I think this news is coming out just ahead of this year's open source firmware conference.

I bet those two things are not unrelated, Emma. It is exciting. And people that pre-order with the open firmware will get to pick up their darters at the conference. Whoa, cool. What kind of work has this taken behind the scenes to make this possible? Is there like some sort of crazy mad scientist lab where people are hacking away on firmwares? Has there been some sort of collaboration with another group? Like what has been the engineering side of this?

This has been in-house led by our lead engineer, Jeremy. He's very big into open firmware and developing and in Rust. And he's been putting it on all the laptops. They've taken them apart if needed. They've had to work with Thunderbolt to get proper licensing to include in our core boot. We've had to work on suspend issues. Right now, the Darter and the Gazelle both are doing pretty good supporting core boot fully. So it's these two systems. Is it possible for

existing purchasers, like people who have a darter that maybe they got six months ago, would it be possible for them to switch over to core boot and Linux boot? Or is this only going to be the new systems? This should be anyone that currently has a darter should be able to do it. And I'm guessing we'll have people reaching out and support if they want to implement that or test it out. And we'll be we'll have people reaching out and support if they want to implement that or test it out. And we'll be happy to give them instructions. So what's the sales pitch now? Is it user security and privacy? How does System76 look at this as like a, what is the value this adds to the product? Well, we want to make sure the entire product is open. So this puts us a step closer to that, moving away from as much proprietary software as we can. And this also lets us get to the lowest level of the hardware so we can maximize performance and just tailor the performance towards users' needs. I also noticed the brand new shiny 4K OLED display

Adder Workstation was announced. And this thing is such a monster that we were speculating last week about what its capabilities were. And sure enough, you guys said, well, why don't we send you out a test rig? So in a couple of weeks, in a few weeks, we may have a review here of the new Adder workstation. What's your take on this new one? I bet you must be pretty impressed.

You've seen a lot of laptops come out the doors there, but this one's 4K OLED, RTX 2070 graphics, 8-core Intel i9, 64 gigs of RAM in a laptop? Yeah, she's a beauty. If anyone picks her up, I'm guessing if you're a laptop namer, it's going to have some like sexy woman name. Yeah, it seems like a lady with that display. Beautiful eyes, you know. It's a beautiful eyes.

Yeah. The contrast ratio is actually 100,000 to 1. Oh. So the contrast is pretty extreme. And it also has the nits. It's 400 nits, so it's pretty bright as well. That sounds wonderful. I think we've all been waiting for a laptop like that.

I have a couple of different use cases. I want to bring it in in-house and see how it processes the drone footage because that's brought just about every machine down to its knees. And then set it up as a VM machine, maybe attempt to do some PCI pass-through with it. You're talking specs like this. Yeah, we haven't tried the GPU pass-through yet, but that is on the slate for this week for sure.

Ooh, well, I'll let you know. Yeah, definitely one of the things I would like to test it out and kind of run it through its paces would be just, you know, rendering out video, blender work, that sort of stuff. And, I mean, they depict it here on the actual landing page for the site. So with that 2070 in there, you should really be able, in the 4K display, you should really be able to get some high-quality graphic and design work done off of this machine. This will also be one of my first longer-term hands-on with Pop! OS.

Right. Looking forward, because I've been waiting. And then the last thing that I know, I've been looking at, you know, I've been looking at the specs of this machine.

The one thing that I really like so far, just from the outset, is a lot of the big, bulky ports are on the back of the laptop.

The power, the Ethernet, the HDMI, the display port, a USB-A, NC. That works so well when you're just

setting it up on a mobile desk. It's so clean. Yeah. So I'm really looking forward to it, Emma, and I'm really glad you're able to make it back and chat with us about the news. It's great to see the core boot systems. So what's kind of the timeline

for that to be just the default when people order one of those machines? I don't have information on that yet. It's really hard to predict with NVIDIA models. It's an ongoing project. But I do need to mention one other thing.

The adder is currently on sale for about another month. So if anyone is interested, we do have a 30-day return period. So now is the right time to purchase one. Also, kind of just saw this slip through the news, but it's probably worth mentioning too, is that it looks like System76 has been granted a Thunderbolt license. That seems huge. Yeah, that's for the core boot, so we can implement Thunderbolt functionality in core boot systems. That's fantastic.

That was one of the lingering questions I had. That's great to see. Well, Emma, that looks like an awesome release. I can't wait to try it out and kick the tires and see if I can't make it suffer a little bit. Awesome. We look forward to seeing what you think. Well, Emma, do keep checking in as you're feeling well and joining us because it's good to hear from you. We love to have you.

Will do. Miss you guys. We miss you too. Glad to see you back. All right. Just a couple of items in the old housekeeping this week. This Friday stream will be the final Friday stream. Say it ain't so. As we say goodbye to a crazy experiment that will result in a couple of new things that we're doing. So we looked at the Friday stream and said, how

could we do this a little differently? And we decided to break it up a little bit. We'll have more information about that soon. Livestream, community, Q&As, things like that that'll be coming down the pipe. But the first takeaway is come see us on Friday because we have the crew in-house this Friday. If you're listening the week this came out, we'll also release it, of course, over at FridayStream.com. It's going to just be a fun show with the crew in-house hanging out and we're launching a brand new feed. Well, we'll tell you all about this later on, but you can find it over at extras.show. Extras.show. All kinds of stuff will be ending up in, you know, stuff that maybe didn't fit in here.

An interview here and there, a random minimum viable podcast. We're trying extra content, perhaps from today's episode of Linux Unplugged, things like that. That complete keynote from Thomas Cameron from a couple of weeks back, that's in the extras feed. So we've created a new feed of extra content that doesn't quite fit any other shows, but we wanted something that you could subscribe

to and just get it easily. Right. I mean, you know when you run out of JB shows, you're already caught up on everything for the week. What are you going to do? Now we've got a feed for you. We don't want you to get the shakes. Extras.show, that's with an S, extras.show, and it's extras.show slash subscribe. It's just going to be fun, quick stuff in there that gives us an excuse to kind of play around sometimes,

including some stuff that we'll do on the live stream with the community very soon. So we'll tell you all about that. And a nice, nice bit of news on the Linux Academy core side, transcripts are nearly at 100% now. These are click-along transcripts that make it easy to jump around in the video, and there's a whole new batch of free courses that are available for community members if you sign up for Linux Academy Community,

including some Cloud Fundamentals, AWS Security Essentials, and a Kali deep dive. That's my favorite. That's worth going and making the free account just for that right there. Tell you what. So check that out. That's all over at linuxacademy.com. You can sign up and get a free community account. L right here.

And our team curates that stuff. So that's why we mention it from time to time, because it's a good deal. So go check that out, won't you? And much, much more. And keep your ears out on that extras feed, because there'll be a lot of cool stuff coming down that feed. Extras.show slash subscribe for that feed. Now, Mr. Payne, there is a problem out there, and it's getting bigger and bigger by the day.

That's the only way it gets, yeah. Yeah, and thankfully too. Otherwise, Linux would probably be dead. We'd probably be in a world of hurt. And a bunch of different communities are trying to solve this big problem that everyone listening to this show knows about. And that's the kernel is a huge project.

It grows in size and contributors every single month. On average, most releases have somewhere between 1,000 and 1,500 developers making contributions. Yeah, that's Major Hayden. He's a principal software engineer at Red Hat. And something on the order of 200 to 300 of those are brand new. Like, they've never made a contribution to the kernel before. He's going to be joining us in just a moment. And he's going to talk about a big project they're working on to try to solve

this issue. Major's here to give us some background on this. And it's the Continuous Integration Project. It's a really cool idea to kind of try to catch the low-hanging fruit with how fast the Linux kernel is moving. There are some challenges there. And I would say Greg Crowe-Hartman's got a couple of really good talks on this particular topic. One of the challenges, I guess the biggest challenge is that it moves incredibly fast. Like the project is large and moves fast. Last I looked, it was the fastest moving open source project that was out there by a good amount, even way past Kubernetes. I think it does like, if you take all the commits that have gone into Kubernetes, that's about two and a half versions of kernel development. So that's about

two and a half releases worth. So we're talking fast. Yeah, and so their idea here is, let's come up with something that can monitor this and send it out across multiple systems, architectures. Right, I mean, that's just a wide test base. The Linux kernel runs everywhere, as we say

all the time, and that means you've got to test it everywhere. And you have tons of different options when you're building it. Yeah, can you imagine doing this by hand? All right, I make a couple changes. I'm going to go compile that kernel. I'm going to go distribute it on six different machines and architectures that I have to keep in my office to test this. And then I'm going to test it, come back, wait, see if anything's changed,

and then repeat over and over and over again. That statistic, the Kubernetes entire product is two and a half kernel versions. Yeah. That's amazing. Yeah. And you think of Kubernetes as this massive project, right? Yeah, there was just that review done of it this week, some security thing, and it took them months. Yeah, and then you have new contributors, you have subsystems that are on esoteric hardware.

It's really a challenge. That's where this continuous integration project comes in. This project came about because there's a lot of low-hanging fruit in the kernel. They can easily be tested. There's quite a few things that you can test in a VM or you can test on a physical host, let's say, that are very easy to test like internally with just that one machine. And so the question was asked, why don't we make automation that just runs these tests

to catch the low-hanging fruit? Because the idea is we want to take Red Hat kernel developers and not have them go deal with these small, easy-to-fix problems that slow them down and make them have to go find a solution. But instead, let's catch these problems before the patch gets merged in the first place. So if someone says, you know, hey, I'm bringing in these 10 patches. And then we find that when they do that, all of a sudden memory allocation issues pop up, like you can't allocate memory like you could before, or, you know, it's just not structured in the way that you want to, then you can call it out.

Someone could say, you know, oh, there's a bug in there. The nice thing about that is you can find the bug right then when the patch is being proposed, instead of later on when quality engineering is taking a look at a lot of changes, you know, not just the kernel, but a whole lot of things that are changing and trying to work backwards and figure out where the problem was. In our chat with Major, one of the things that he pointed out was that, okay, we can solve this from a technology standpoint, but there's the human factor as well. And there's a couple different projects, but the one that the Continuous Integration

Project is working with to solve for the human factor is called, was it Patchwork, Wes? Yeah, Patchwork. I mean, if you think about it, Linux kernel development is done in its own style. It's kind of an older style. You're on the mailing list. And while, of course, there are Git mirrors everywhere, it's kind of secondary. So you need to get the culture to understand

how to interact with that, and you need a bridge to actually help the tooling work. There's a few software projects out that help with that. There's a big one called Patchwork, and it essentially reads through a mailing list and watches the emails as they come in and tries to figure out which of these patches go together. So a kernel developer might submit a patch and say, you know, okay, I want to change the way this works. It takes 10 patches to make this change. And so they

submit all 10 patches to the mailing list. And then what Patchwork will do is go and say, okay, wait a minute, this was one contribution from one developer. It has 10 pieces. These go together. And so then inside the web interface, you can go and look at all these patches together in one place. And of course there's an API as well. So if you have automation, you can actually just pull down the patches or what we call a patch series when Patchwork sticks them together into one patch series. So that way you can test them. That way you can report back status on them and that kind of thing. But it's a software project completely over on the side from the mailing list.

Now, think about this for a moment. moment. They thought they could replace all of this automated stuff, and they ended up running into this human problem where people have been doing it a certain way for a long time, have this that works for them. They got software that archives it. It's plain text. It's searchable. They got filters set up that have been working for them for years. It's part of a workflow. And you can't deny the kernel is successful, so obviously it must work. Yeah. And, I mean, you're not going to change that, right? I mean, that's established, so if you're going to build something to work with, that's your only option. Patchwork is an

incredible effort to sort of solve a human problem and automate it at the same time. We'll have a link in the show notes if you want to know more about it. And if you'd like us to dig into those kinds of projects, go to linuxunplugged.com slash contact. If you'd like us to deep dive into some of that stuff, you definitely can.

Now what's great is the way they're doing this, and even though it requires an incredible amount of infrastructure, they need system 390s, they need every various ARM system that's in production, they need lots of different Intel and AMD systems, the end result, hopefully, is improvements upstream. And so as part of this, we ask the question,

well, if we do this really well, and we keep things from getting into the RHEL kernel that's not supposed to be there, what if we could do this for upstream? So what if we could do this for an upstream kernel repository of some sort and not only be able to catch it earlier, but then catch it in a way such that we can help upstream development and help test and help maintainers actually know what's okay to go in the kernel and what's not.

Okay, so how many software testers does it need to change a light bulb? None, because software testers don't make changes. They will just report that there's darkness. There are many, many different continuous integration projects. And OpenQA is probably another very popular one. We'll have a link to a talk in the show notes. This talk is about OpenQA, the heart of OpenSUSE's automated testing. I'm Ludwig.

I'm employed by SUSE, and I was, and still I am, one of the engineers behind OpenQA, where it is now. OpenSUSE is a huge project, and so they're one of many that take advantage of OpenQA. Yeah, I mean, think about it. It's just too complex to test by hand, right? There's too many little things that can change. You have a giant collection of software to handle.

So solving that problem, they introduced OpenQA. One thing that's kind of neat about OpenQA is it replicates things the way a human might interact with, you know, simulated keyboard and mouse inputs. So you can test a huge range of software that might not be amenable to other testing methods. Yeah, and Fedora, as we've talked about before on the show, considers OpenQA a, quote, significant part of their release validation process. That's one of the great things about open source, right? I mean, one distribution solves a problem, and another one can just take advantage of it.

No, it is. It is. And I wonder, ladies and gentlemen, Mr. Popey is joining us now, mid-show. Popey, welcome to the Unplugged program. I know you've messed around with Jenkins a bit. Is there any kind of continuous integration going on over at Canonical for some of the projects there

that you're aware of? Oh, my God, tons of it. Really? Yeah, it's all over the place. Yeah. We use a little bit of Jenkins, and we also use something called Spread, which Spread does all the testing for SnapD

across a whole bunch of different distros. So we know that SnapD and Snaps work across different distros before we ship it. Of course, and it makes perfect sense in the snap scenario. And don't we appreciate it? So you have essentially VMs that are running the different distros that snap could potentially run on, and then when a new builds out, it deploys to those and tests to see if everything's working?

Yep. It does it for every pull request as well. So the more active they are, the more these machines get spun up. Yeah, they get busy. That's great. I noticed we had a poll request for the Caster Soundboard project on our Jupyter Broadcasting GitHub from the app image maintainer. And he said, you know, apply this patch to your project,

and every time you do a build now, I'll just pull it down and I'll automatically build an app image of Caster Soundboard. Sweet. Gave us instructions to get it hooked up and then using Travis CI and we'll just sort of build it for us. Yeah, it's a Travis CI pipeline. And it's a good example of how you can solve a problem like developing the Linux kernel with this unbelievable amount of problem domain. Or it can come all the way down to a simple open source soundboard that we use here in the network. Yeah, the same techniques work all over the place.

Yeah, it's really, you got to solve these problems at machine speed. It's really what it is. We use Travis a lot for a lot of the snaps we build. And at the recent Snapcraft Summit in Montreal, we had some guys from Travis CI come along because they want to streamline how they deliver software to the machines that are actually doing the builds because obviously they're spinning up thousands of these machines all day every day and you know it's very noticeable for them when Silicon Valley wakes up and everyone starts committing the code that they've made and all these Travis CI jobs start spinning up that's a lot of workload.

So they're constantly trying to optimize these machines to start as fast as possible and have all the compilers and all the other build tools, the whole build chain in those machines as fast as possible. So they spin up and spin down super fast. It's super fascinating stuff. It really is.

Yeah. I would imagine it'd be the kind of thing that if I was still in IT, I'd probably be building. Probably the kind of things I'd be standing up now. Totally a tangent thing, but is it just me or do you canonical folk have a lot more of these sprints these days? It seems like they went from a couple of times a year to all the time. I mean, one a quarter almost. Yeah. Well, yeah, there is a lot of them, but there's different types we have engineering sprints um which we tend to have every six months and roadmap sprints which are every three months and um then the snapcraft summits which are like ad hoc um so yeah there's a there's a fair number

of events going on at the moment there's a lot of air travel and getting around yeah it's good fun oh good i'm glad you don't I'm glad you don't mind. It can be fun. You guys do mind. You tend to do them in pretty nice locations. I've been following along on Telecast for Popi recently. It's fun. It is fun.

Is there like a URL you give out for that that people, if we should give it out here on the show? Oh, that's very kind of you. Yeah. You have to be a Telegram user. A Telegram user. but what i did was i exported all of the episodes that i made out and i've put them on my my website so there's a recent tweet flow from me about this uh but if you if you just go to t.me slash telecast with

poppy you'll get you'll get to the channel where all the Telegram episodes are. Learn everything about beer and curry you ever wanted to know and more. You had the most English telecast the other day where you were walking from home to the pub to get picked up by your friends to go get curry. And I just was listening. It's so fun because it's like, what, at most like seven, ten minutes ever? And sometimes much less. So it's like, who doesn't have time minutes ever and sometimes much less so it's like

who doesn't have time for that that's what i wanted it to be it's just like ad hoc just me having a little chat whatever's on my mind at the time so i don't have to go through the whole episode effort of creating an episode of something and publishing it somewhere and rss feeds and all that nonsense i just press record in telegram and you know the thoughts come out of my brain out of my mouth and into the phone and i give it a preview list and then hit send and that's it. I read nothing more than that. I think you're way ahead of us. Yeah. Well, so now my, my, my, my last two questions are, are you, uh, are you going to be back on Ubuntu podcast now that you're done traveling or is more traveling ahead? So we just recorded two episodes before I dropped in here. So yeah, I'm back on Martin's out cause he's on vacation.

So I'm back in and Martin's out. And I think in two weeks time, we'll have our first one with all three of us back, uh, for a while. And then last but not least, um, how,

how great is user air turning out? Did you have any idea how great this would be? I see it. It's just like three guys chatting and i have no idea like i don't know when you say when you say nice things chris is when we get nice comments we also get horrible comments of course mostly on mostly on youtube from people are listening can i just say not only have i been making podcasts for 13 14 years now but i've been listening to them for just as long and this is one of my favorite podcasts of all time.

That's very kind of you to say. Joe and Daniel put a lot of effort in. It's always you I'm nodding my head to. I'm always like, that's right, Popey. Because he's so reasonable. I've got kids like he does too. As Joe calls me, centrist dad. My wife

referred to you the other day, Popey, as that guy who said something about bed sheets oh my god it's also it's like the only show i get to listen to with my wife too there's that and i think a big part of what makes it work is daniel foray from elementary is willing to take a bit of as you would say a piss like he'll just sit there and he'll just dig himself in deeper he'll take the piss what deeper. Take the piss, Chris. He'll take the piss, not a piss. What is it? I'm sorry.

Hold on, hold on. Joe came in studio just to correct me. Yes, take the piss, not take a piss. Okay, thank you, Joe. Hello, Joe. Hello, Popey. I'm going to go and wait for my pizza now. Joe's here.

It's a whole family reunion right now. We're working on a new super secret show, you know. Yeah, very secret. so we've got we've got i'm so secret i even flew drew uh drew and uh and uh l and joe who haven't been on show they're here too so we've got in studio we've got cheese and we've got brent and we've got alex and it's a whole jb party west and myself and we're working on some new secret stuff it's very exciting so one thing i would have to say is if any of the listeners want us to answer any particular question,

they just have to let us know with hashtag ask error. Yeah. Ping us in the JB telegram or in IRC. You know where to find us. Just ping any one of us with your crazy questions for us to ask about life, the universe and everything. They get some great questions. So whatever medium you like to talk to these guys, hashtag

ask error, and it'll show up on their radar, and then they ponder them in the show, and it leads to some of the best conversation and discussions. Well, Popey, it's good to have you here, and thanks for giving us the canonical side of that. It's an interesting perspective on how they use continuous

integration. It makes total sense. Total sense for snaps. Well, while we are talking about other great podcasts, you notice we don't do quite as many distro reviews. We still do a few. No. We do like the big benchmark ones, right? Yeah, the ones we can't not do.

But if you want a review from folks who just can't seem to scratch that distro hop itch, check out Choose Linux. Here's a clip from each of them on the most recent distro they randomly tried, which is one I haven't given a go yet, Endless OS. I know that we start almost every single distro hop segment with, well, L, I see you had some issues, but not this time. This time, I'm in love. This is what I thought an operating system was supposed to be like. When people tell me about how they use Windows,

and it's just simple and you go and you use it. I would say that this is the prettiest GNOME retrofit that I've ever seen, too. It is almost on par with how good Deepin looks. I can see what you're getting at, Al, but it's just too simple for me. Hmm, endless OS. Since the man who uses XFCE. what you're getting at, El, but it's just too simple for me. Hmm. Endless OS. Says the man who uses XFCE. Nice.

Nice. Also, shout out to XFCE. Project had a release this week. Shout out. At Choose Linux. They're also throwing in a few picks here and there. I don't know. I might just stop doing the show and just listen to them.

Yeah, we can learn everything we need. Mm-hmm. Let's listen to them. All right, well, let we need. Mm-hmm. Let's listen to them. All right, well, let's do our pick this week. Now, everybody's always talking about their tiling window managers. And I'm like, yeah, I know about your i3. Yeah, I know about your awesome. But I like me a traditional desktop environment.

That's always been my go-to line. Well, Wes, today we are going to have our cake, and we are going to have it with a delicious side of ice cream. That sounds great. Yeah, right? It's called QuickTile. Keyboard-driven window tiling for your existing window manager. I'm sorry, what did you say?

Existing window manager. Hmm. Well, existing X11 window manager. So everybody's existing window manager. Hey, hey now. Yeah, I don't know, I don't know. I don't know. Like in Choose Linux, they were reviewing OBS.

And now it's like the reviews are getting to the point where you've got to start making the OBS disclaimer. I mean, I'm sorry. When you're reviewing OBS, you've got to start making the Wayland disclaimer. Oh, yeah. Like a lot of this stuff doesn't work. Yeah, or on Wayland. It's just going to be rough. Hey, we're getting there.

We're all very hopeful about Pipewire. So what my plan is is to just hang out on XFCE on X11 for a long time, and I'll just add a little window tiling key binding magic to my XFCE desktop. With QuickTile. Yeah, so that's where QuickTile comes in. It's an analog also to WindSplit. And if you remember that, that was an old school. There was WindSplit and WindSplit Revolution, which were plug-ins for Compass.

Now, if you don't know what I'm talking about, you're— You got nothing. You don't know what I'm talking about? Really? Oh, my God. Oh, Wes, show them the wobbly windows real quick. Oh, yeah, and I still got the— So, take a look.

Tell me what you think about these wobbly windows. What do you think of that? Is that fluff or is that fun? What do you think right there? Or watch what happens when I close the window. Yeah, watch this. Watch this. Come on.

Now, okay. Honest, your first impression. Okay. The wobbly windows, I think, is super fun. It sparks joy. Oh, yeah. I can't stop smiling. You're seeing it.

I can't stop smiling. So, I will say I'm a fan of that. The exploding windows, I don't know. Yeah. I can't stop smiling. So I will say I'm a fan of that. The exploding windows, I don't know. I felt a little cheesy. But imagine like you're angry and you want to close something. And you just explode it. You know what was even better is back in the day, they had burn down windows. You close it and it would fire effect and it would burn down. I'd be okay with like the turning the old CRT TVs off. Yeah. Oh, that would be great. Inasma, one of the fun things you can do is

you can set it so that your windows gray out when the application crashes, which is kind of a neat effect. I think I have that in Plasma. Yeah, yeah. You set that up? Oh, yeah. No, I didn't set it up. You can turn on the wobbly windows. Oh. Yeah, it's just under desktop effects.

I mean, you've got that GPU sitting there doing nothing, right? Most of the time. All those engineers work to build that GPU for something. Might as well put it to the... I don't even notice anymore. I feel bad showing it because the wobble doesn't seem like it's very wobbly anymore. You could up the wobble if you want. Of course, it's Plasma, so you could increase the wobble factor

if you want to increase the wobble factor. Popey will not be increasing the wobble factor because he did something that is so 2019 Linux personality that I just love and also fully appreciate. So on Twitter, which God only knows why he chose to do this, he announced that he was switching away from Plasma. And then was it six successive tweets explaining how it's okay that you're switching away from Plasma and that everybody was it six successive tweets explaining how it's okay that you're

switching away from Plasma and that everybody else's choices are okay too. Yeah. I don't like this whole, you know, internet Z-list celebrity has switched from distro A to distro B. The sky is falling rubbish. So I just had to super clarify everything in a complete succession of tweets. I liked it though, because you referenced back the first tweet of when you actually installed KDE Neon. And what are you on now? You just on mainline Ubuntu? Yeah, Ubuntu 19.04 with GNOME. What was the, was it just time for just trying something else or what was the impetus? Do you know the actual trigger to

this was last week's lup oh really when you uh i i i came in and my audio was all messed up and i've tried so many things and i couldn't figure out what the hell it was so i thought okay there's got to be something fundamental on my system i don't know what it is and i don't know who to file a bug against or what projects projects to file a bug against. So I decided, right, let's just upgrade to the latest of everything, and if it still happens, then I can start looking for where the bug is. And it hasn't happened yet, I don't think, which is partly why I'm here to test that this actually works. I'm wondering if your impression, not even necessarily from like a technical level, but just my impression is the Gnome audio stack

just seems a little simpler. Like it just seems like it's sitting right on top of Pulse and whatever Pulse is doing, that's what Gnome's doing. Well, I think part of it is that KDE gives you knobs and buttons to switch. So Plasma gives you like things you can tweak. And because the Gnome desktop doesn't uh it's it's harder to shoot yourself in the foot and i think it's entirely possible that i've

twiddled something in kde at some point in the last 18 months which has made this go bad and i don't know what that is that's possible but there are lots of other possibilities as well hardware failure it could be postOS audio is rubbish. There's so many things. I can't point the finger at any one thing. But yeah, the audio subsystem on GNOME seems to be simpler. You sound great today. Awesome.

Thank you. That's what matters. I can't argue with that. Yeah. I find that I hop about every nine months now. It's about my, yeah. It's the problem when all the options are so good. Mm-hmm. Mm-hmm.

It's all good. We're going to do some reloading here in the studio. In fact, we were thinking about having an on-air debate about what we reload to. I'm becoming less and less attracted to nuking and paving a system. Mm-hmm. So I built an Arch system a year ago. It turned one year old on July 27 27th and i've got it set up just the way i like it and i don't know if this is just because i've now been using linux for long

enough that i can actually troubleshoot my problem sufficiently that nuking and paving is no longer a requirement to get a stable system or whether it's just because I've become so comfortable in this pair of old, worn-out, comfy shoes that now it's like I just don't feel the desire to nuke and pave like I used to. I mean, I have a Proxmox server in my basement that I can just spin stuff up on. And my laptop and my desktop are work horses now, and they just need to work.

True, there is an element of, you know, you have to know enough about a system. Because half the re-kicks that happen on our team are, we did something weird, and we've got to start again. And if you keep doing it every time, you don't ever quite learn what it was that broke it. But you always have to do that math. Like, ah, my time is worth X, Y, Z. Exactly. I know it'll take me two hours to re-kick this thing. I know it'll take me five hours to troubleshoot it-ish.

Or more. Exactly, exactly, exactly. But Popey said something in one of his recent telecasts with Popey that I had to respond back because he said something. Popey, you were talking about upgrades and how people are always, like we almost have like a wipe and pave or a nuke and pave culture that just sort of is the default answer. And it's not necessarily always the best way to go. Sometimes just upgrading for a while actually is the best way. Yeah, I've long lamented the fact that people will say don't upgrade because upgrades are broken. And therefore you should always do a clean install. And I appreciate, should always do a clean install and i appreciate you know sometimes they are broken and part of the the the problem here is that people do all kinds of crazy things to their machines and then forget

they've ever done that and then six months later they go to upgrade and something they've done is so fundamental that the upgrade breaks now part of that is they then blame us, the distro maintainer, because we make a shonky distro that can't upgrade properly, yet they ripped out core components of the desktop six months ago, but they don't remember

because this was just a command they copy and pasted off of a blog somewhere in order to get a new theme or to get a new splash screen in Grub. Disable system D or something stupid. Or yeah, something ludicrous like that. But they don't know they did it. Did you hear that I've switched from system D?

It's great. Troll. Yeah. But part of the problem is us in that when we leave people in a really horrible state, when an upgrade breaks, you're left with often a black screen or no way to log in. Or if you do log in, your desktop doesn't load properly or something. And that's horrible. And when you're

in that state as a user, what are you going to do? You're going to nuke and pave, and then you're going to get it in your head that that's the right way to upgrade. And so it becomes ingrained behavior. And see, for me, this is one of the big advantages of building an Archbox is I know every command that went into building that thing, and I had to go and learn why I'm running that particular command. Whereas, as you say, copying and pasting something and piping it to shell is very easy and tempting, but it can lead to problems down the road that you just don't foresee

because you don't understand what it is you did. I mean, I love Arch, so don't get me wrong. But here's the context for the rest of the audience. Before the show, we were discussing what OS we're going to deploy in the studio after today's show. We have a

couple of things we just need to address. Some of these machines were either set up as Kubuntu or Neon and now they're running XFCE and we just want to do a clean install so it's just a base system. Start fresh. Start fresh. And we want to do it before there's a lot of issues.

There's a couple of niggly things we've noticed. And then, of course, because it's a group full of Linux geeks, the debate of what distro we should use has been the number one. I can't make this up. Oh, yeah. No, it really is the serious point of contention. So Alex is advocating for Arch. Drew is essentially advocating for Ubuntu LTS.

And I'm on the fence leaning towards Fedora. You say that, but you want Fedora. You'll be happy to see Fedora chosen. Yeah, I would be. I would be. More Ubuntu. So I'm going to make Yeah, I would be. I would be. More Ubuntu. So I'm going to make my – I'll start.

I'll make my – so Alex just made his case for Arch. I will counter that with we started with Arch. We started with Arch and now here we are. May I add that one of the additional problems that I doubt Alex has that we would have in the studio is you like to do upgrades immediately before live shows. Because that's when you're here in the studio working on the machines. There is a little bit of that. And it's fine. When you do the upgrades, when you have time and a safe space, Arch is fine. It's very stable.

I've stopped doing that. Okay. If you say so. Well. You got it. Okay. If you say so. Well. You got it. Okay. Maybe I haven't completely 100% stopped but I've reduced. I want this to be a soundboard clip.

I've moderated it. That three seconds was amazing. No, I think for me it's like I've always felt like a little bit of spitting into the wind but it is also like me saying my operating system should be able to handle this. Like, so you probably saw this week that Canonical announced that the absolute latest and greatest version of Ubuntu LTS now has like a nice GUI way to sign up for the live patch service. Yeah.

Get yourself an Ubuntu 1 account, and then you can patch three machines for free, and you can do it all through the GUI, and then you're secure through reboots for some of the more important ones. That's super appealing to me in a studio environment. And so initially, I envisioned like deploying Ubuntu LTS everywhere, bringing it all into landscape, using live patch to manage them. And then I started having sound problems, which we've documented. And we've resolved for the most case by switching up to Jack. It's pretty much solved our problems. But now we're sitting here with these systems that are sort of Frankensteined. And I've had a lot of experience in the last nine months. I don't know. How long has it been? Six, nine months.

This year. Yeah. Messing with Fedora. And it seems like a very clean, reliable system that has good, solid upgrades. And Drew himself has some good experience with it. Yeah, it doesn't hurt that our editor loves it. And between the two of you, we can pretty much pull anything off with Jack Audio. So I kind of feel like maybe it's time to just switch to Fedora, and then we have Fedora on the server and on the systems. Maybe it's time to just switch to Fedora. And then we have Fedora on the server and on the systems.

And that's the part that appeals to me the most is one common OS, one common environment. The server's running Fedora. My laptop's running Fedora. The studio systems are running Fedora. Everything is the same environment. And you are, I mean, you're kind of the primary admin. You're here in the studio the most. You're using the machines the most. So that's not for nothing.

Right. using the machines and so that's not for nothing. Right. But I don't want to do it at the expense of software availability or reduced production quality. And so that's where I wonder if Drew has a case for Ubuntu LTS. So I do. But before I get to that case, I will say I am already maintaining a group of packages that do what we need it to do for my own use, because I use Fedora on my workstations as well. I know, that's why I think it's funny that you're going to come in here and advocate for Ubuntu. So I would be comfortable with the studio having Fedora. To me, it does make sense.

What I like about Ubuntu LTS is the longstanding nature of it. Fedora 30 is going to be supported until, what, two years after release, right? And we'd probably bump up to 31 after that arrives. And it's a lot of upgrading, and it is targeting a use case that is more agile than a production machine like a recording studio, to me, should be. I will grant you, going Fedora is conceding that we are getting on an upgrade Tread machine. Yes. Yes. So that is why I think the Ubuntu LTS route is probably the better route for a production machine. Workstations, those can of the, to me, it's the right way to go. And the Ubuntu Studio repositories are fantastic. They have everything that I put in my Fedora repositories already in there. How do people find that repo? If you just do a search on the Fedora Copper for Drew of Doom or Linux Pro Audio, they'll find it.

Because it's actually really nice. Some of those tools have not been on Fedora before this. So it's actually really nice that you're doing that. So, Wes, here's my argument. My counter-argument, because everything Drew said, can't argue with that. Absolutely solid, especially when you add the fact that we could use LivePatch, so we could actually even keep the system secure

without having to reboot them. That's super attractive. And I would only need, like, these three systems. It doesn't need to be more than that. But you're working on Jack in the Box, which is a project to take a lot of the audio routing and put it inside a Docker container and sort of normalize the desktop and the OS.

And if we were to roll out something like that, it wouldn't really matter. Yeah, I mean, that is kind of the flip side of we could adopt Ubuntu LTS, but eventually we will have to upgrade. And that's a long, it's going to be a long time, possibly. That's good and bad. If we do go with Fedora,

we're going to get a lot of practice following the upgrade path. Well, and then also I think we could probably take advantage more quickly of things like Pipewire as well. Right, so I think it's kind of the difference between what Drew's suggesting and making them really appliances, like things we'd be, you know, you're finding a rack you don't futz with.

Right. And as much as we do use these as a production machine, they're also content sometimes. And, you know, we are still adapting our pipeline, how we do the shows, and we probably are going to continue doing that for a while. Part of me wants to use these shows as a way to push forward the most modern way to develop audio. Right. And like we're doing some super freaking cool stuff with Linux audio right now. Alex, I know you want to jump in. I want to give you a chance. Yeah, I do. So I've been looking in the Fedora project

dot org website whilst you've been talking and the average lifespan of a Fedora release hovers around, the shortest is 330 days, so just under a year. The longest is 420 days, and that was Fedora 22. But on average, it seems to hover around that 380 mark. So you are signing up to get on this train of upgrading at least every nine months, by the looks of it. That's what I estimate, too.

And that's my biggest concern really is that having to upgrade, it could be a really inconvenient time because these releases come, they say every six months with Fedora, but they also have a big disclaimer underneath that says it'll be approximately six months. It'll be done when it's done is the disclaimer.

So it could end up shifting slightly. So it won't be like, okay, it's May, LinuxFest Northwest is out of the way, we can safely upgrade this box now. You might end up having to wait a bit. So, I mean, for me, a rolling distro, you never really have to worry about that kind of stuff. I can also see where Drew's coming from, though,

with the LTS being a minimum to probably four years' worth of updates and stuff. I mean, for me, I'd probably go that route, right? I mean... Yeah. 18.04 is really in that sweet spot, right? Mm-hmm. It's really good. And, you know, one other option that hasn't been floated, CentOS.

If you're doing everything in a container, it doesn't make it... You're not giving up, you know, the thing Chris said. He doesn't want to give up software availability. If you're doing it on CentOS and everything's in a container, then... Part of me feels like I'm not participating. I'm watching Linux from the benches if I'm using CentOS. You have a laptop for that.

I know. These are production machines that just need to work. I have a proposition for you, Chris. How about you do whatever it is you're going to do now, and then you'll get it out of your system and in nine months time switch to 2004 lts where you can have zfs on route and then you can do snapshots before you do your upgrades and then you won't have to worry about it there you go he has a point there it's very. It's very, that is, as the good Mr. Spock would say, logical.

I guess I'm struggling to come up with a reason now. I think we've made a good case for the LTS, especially since the package availability is there from the other flavors. I guess what would you propose is our solution then for the variance between production and workstations? I mean, we work for a company called Linux Academy, right? I think it's... You think we can solve that problem?

I think it's a good thing to be familiar with multiple different flavors of Linux. You're right. You're right. You know, I mean, as host of Choose Linux, I ought to be, right? But yeah, no, it's... I don't see any reason to pigeonhole ourselves into saying, oh,

we run Fedora only or we run Ubuntu only or any, I, to me, I like being familiar with multiple different systems and I feel comfortable

administrating them all. All right. So, let's do it then. Let's do, I think we should do Zubuntu. Yeah.

1804 LTS right here. That'll be what we reload the system to. We just got to make sure we make sure it's decent because, like, there's some XFCE going on right now. And some of the window management, it's just, it's kind of bare bones. What, are you proposing something else? No, not necessarily. I'm just saying, you know, we want to make sure it's easy to use.

He wants those wobbly windows. Wow, that wouldn't hurt. Can I tell you about our app? Okay, we got to make sure we install a dock. What about QuickTile? What about using QuickTile? Yeah, you know, it is also Python, so we could probably script it too. Yeah.

Oh, yeah, you can. Oh, absolutely. Yeah, you've whipped together a couple of hacks using X-Render, which has been helpful. We could expand on that. But you're right. XFCE does lack some modern window rules and stuff like that that are super nice in Plasma. I'm just saying, we should think about those things

as we go forward. Thankfully, on the LTS it really doesn't matter what base distribution we can mix it up and install. Wesley, you're coming at me with questions, not solutions right now. To me, you're just, I don't know what you mean. Do you mean we can't use Ubuntu? My head's spinning now, like my whole world is falling apart. No, I just, I don't know what you mean. You mean we can't use Ubuntu? Like, my head's spinning now.

Like, my whole world is falling apart. No, I just want you to trick it out for me. Adopting a distro like a religion is stupid. Thank you, Thomas. Very well said.

So what are your feelings on the current iteration of Ubuntu with GNOME as the default? Like, how do you feel about GNOME these days? Because I know you've had a... Actually, I think GNOME

looks great, you know. I don't want to steal anyone's thunder, but one of our team members here is trying it out, and I was looking over his or her shoulder, and gosh, it looks so good. You know, it just looks so good. It's just not what I need for production appliance systems.

I actually would prefer nothing has a... No compositor, no special 3D effects, no nothing. i3, then. All right. Oof. Chase, did you hear what he effects, no nothing. i3 then. I got it. All right. Oof. Chase, did you hear what he just said? He said i3 on a production machine.

I'm glad you guys, I just was just listening to all you guys hash it out. The reality is, is that Drew is right, is that you should use the LTS because long-term support, fellas. Hey, what do you think? Maybe should we give I3 a go Yeah if you're down Let's do it I mean I3 is great Let's give it a go

And then we'll say We'll give it a review In next week You want to put a cheat sheet On the wall next to you Yeah okay Good call Cheat sheet

Alright okay Alright well we'll see How that goes So make sure you tune back Next Tuesday You can join us live Over at jblive.tv Get it converted

At jupiterbroadcasting.com Slash calendar And see you next Tuesday. It's for people who like to mess with computers. And you know who you are. If you're somebody who doesn't want to mess with a computer, I just want to buy something on Amazon, send an email to my kids look at some websites if you if that's you you don't want to mess with it probably not a good choice what was that hesitation before look at some you know what's funny is leo actually now is he's a pretty happy desktop

linux user that's what i hear ne Neil, how was Flock? It was awesome. Aside from being in Hungary, which I've never been to before, which was an experience in itself, it was great meeting all these people again for another year and seeing a bunch of new faces too. Now, this is the old Fedora Developers Conference. And did you get anything accomplished? Did you achieve any kind of kumbaya on anything that you set out for? How was that aspect of it? From my point of view, the big thing was I wanted to see how the relationship between the new RHEL 8

and Fedora is going to work out and talk with some of the people about the extra packages for Enterprise Linux for RHEL 8, as well as the Fedora modularity thing and its downstream RHEL cousin, the application stream stuff. So that was a big chunk of what I was looking toward. But there was also some great things about how we're going to handle some of the changes in the Python ecosystem. You already know from my previous times being on Linux Unplugged, talking about

murdering Python 2 in its sleep. So there was that, as well as looking forward to the future of where the Python ecosystem is evolving, the way that software is developed and shipped and things like that. And stuff like getting you know, getting into weeds of it, things like distutils and setup tools, if you're familiar with Python development, have been the staple or nearly the requirement for everything. That's going away as a requirement. And that means that a lot of things about how we assume Python software is shipped

has to change. And, wait, the most important part. Did you get sick? No! Yay! Hey, ladies and gentlemen, congratulations! Congratulations! Moving on up, Neal. Next, your

next level conferencing now, Neal. That's what that means.