LINUX Unplugged - 293: Netflix's Gift to Linux

Episode Date: March 20, 2019

Developers at Netflix are creating the next set of super powers for Linux, we'll get the details straight from the source. Plus some good Debian news, our tips for better battery life, and we play a little Hot SUSE Potato. Special Guests: Brent Gervais and Ell Marquez.

This is CNN Breaking News. Just three hours ago, no, two hours ago, as we started recording, Google has made their big announcement about Stadia, the streaming service that all runs on top of Linux and Vulkan. Have you had a chance to see this, Wes? I know we've been really busy today. Only just a little bit. I saw some headlines and I said, I'm going to have to dig into this later. So this is the evolution of that streaming beta, I guess.

You could trial run, maybe. Yeah, which I have been using, so I'm curious to see just how much better it is. That already works on Linux. But the thing that was crazy is in this keynote, and there's a big slide up on the screen that says Linux and Vulkan. If you want your games to work on this platform,

write your game for Linux. Wow. That's different. I'm very excited. I mean, I'm also skeptical about streaming games. I've had some experience. Some have been great. Some have been flawed. So there is definitely room for mistakes to be made, for this not to be that great. However, Google's kind of making the case, look, we can throw essentially unlimited GPUs at your problem. That sounds pretty good, doesn't it? It's like, oh, it's a game developer. I'm like, hmm.

The other thing they demoed, this is really where I could see this working, is play now buttons, like right next to a YouTube video about a game, and they show it up on screen within five seconds. The game is streaming to your PC or your whatever it is, and you're just playing the game immediately. And in many ways, many games are already operated

as like an online service anyway. So now you're just not running their proprietary client. You're just getting shipped some frames. The browser is the platform. But what I'm going to watch with a lot of interest, I mean, it's the gaming service. That's kind of neat. I'll watch that too. But what I'm going to watch with a lot of interest, I mean, it's the gaming service, that's kind of neat. I'll watch that too. But what I'm really going to watch with interest is does this lead to more Linux games using Vulkan? That's going to be a massive win for

Linux users. Right, not just like the first order of games that run on Linux, but just that being a standard people are trained to do, that makes it more, you know, there'll be more documentation, more developer know-how, it will benefit the community at large. Here's what I'm thinking. Even if, like, desktop Linux isn't enough of a reason for developers to be incentivized to target Linux, somebody like Google coming along and saying,

here's Stadia and, or Stadia, however you say it, and it has 5 million users thanks to us tying it in with YouTube. So write your games for that, because that's going to get eyes. That seems like there's potential there, doesn't it? This is Linux Unplugged, episode 293, for March 19th, 2019. Well, hello there, and welcome into Linux Unplugged,

your weekly Linux talk show that's hitting the road and barely surviving. My name is Chris. My name is Wes, and oh my gosh, I am so happy you're alive. Well, I'll tell you just in a brief moment about how I was almost taken hostage yesterday. But first, I'll tell you what's coming up on this week's episode. If you're a Debian user, we've got some great news for you. If you're a SUSE user, we've got good news for you. And a brand new Linux desktop just got a great new release. And we'll talk about the troubles that our favorite open source project for Plasma and Android users, KDE Connect, we'll tell you about the problems that they're running into with the old Google Play Store this

week. But later on in the show, once we really get in our groove, we're going to talk about eBPF in practice. There is some amazing performance, monitoring, hacking, testing tools that have come to Linux. And one of the best things that happened to us at scale is Wes and I got to attend Brandon Gregg's talk on performance hacking. And we watched in real time as he diagnosed all kinds of performance issues with Minecraft. Well, we took away a few of those essentials, a couple of practical examples, and we're going to cover them, what it means for us Linux users, and why the kernel devs are so, so excited about this new feature. It's big stuff. And if we had a little bit of time left,

we've got some feedback too. But before we get any further, you guys know what time it is. It's time to bring in that mumble-roo. Time-appropriate greetings, Virtual Lug. Hello. Hey. Good afternoon. Hello. Nice to have you all here. Most of the web runs on servers powered by Linux. You know, I have to say, most of the web runs on servers powered by Linux. I just came up with that. Now, what do you say, guys?

Before we get into the news, can we have a little story time with Chris? I had to tell Wes about this. Wes, Al, and Cheese already know about this because this was crazy. I've never had anything like this happen to me in my life. But yesterday, my wife, my dog, and possibly my home were all at risk of being taken hostage or worse. It was an insane situation that thankfully I can now sit here and laugh about. However, about this time, 24 hours ago, I could not. So I'll start with, you have this thing called BLM land and it's a beautiful gift from the federal government. The Bureau of Land Management has all of these areas where you

can go take your RV and you can park for free for 14 days. Isn't that great? 14 days you can go take your RV and you can park for free for 14 days. Isn't that great? 14 days you can go somewhere. You can just park there. It's just fantastic. And as we were driving along, going back up to Seattle from Texas, we found this great spot in Arizona where we could pull off of the I-10 highway for a little while and get fantastic cell signal. That way we could still get some work done.

And we've got these new lithium ion batteries that we had installed. So I knew we had like a couple of days of working time. We go there, we park, beautiful cell signal. That way we could still get some work done. And we've got these new lithium ion batteries that we had installed. So I knew we had like a couple of days of working time. We go there, we park, beautiful cell signal. Nobody is around, gorgeous view of the mountains. So we pull in late at night and we get set up and sleep great because it's under the stars. It's silent. It's just a gorgeous location. And about midday after working for most of the day and attending meetings and listening to a great episode of Coda radio, somebody comes knocking on my door. I open up the door and it's,

um, it's a rough looking fella. There's somebody who's, who's, who's seen, um, who's seen the inside of a jail cell a few times. And,

uh, um, just rough looking guy. And he's got a weapon on his side. Oh no. And it didn't take me more than, I mean, I mean,

I immediately kind of assessed what was going on and I was a little drunk at the time. So really not on, not on your a game tired from a long day of driving. I'm sure preparing to relax. It was the end of the day. It was about 3.30 and we had been driving forever.

It had been working since 4 a.m. So it had been a very long day at this point. And Hadiyah was about to start making dinner, so I poured myself a glass of whiskey, turned the old TV on, and that's when the door got a knock on. Boom, boom, boom. Open it up. Hey, man, can I have some gas?

We're all out. And I'm looking around and it says, he's got this, um, he's got an RV that he says he got for free. I don't know if he stole it or what, but it's really bad shape. Like there's parts on the side that are missing. It's got, it's the suspension has collapsed on the rear, right? It's in bad shape. He says, we're all out of gas and I could use a little gas. And I'm looking around, I'm thinking to myself, well, what are you doing? What are you doing this far off the highway up in the mountains? Why? Like, that's not where you go when you run out of gas, right? You pull off on the side of the road. You don't go down a long windy road where there's absolutely nobody. Right. And no,

no gas stations, no gas stations. And so, uh, you know, half drunk me, I'm like, well, all right, man, let's do this. Like, uh, I'm just thinking we could just throw a hose down my, uh, you know, half drunk me, I'm like, well, all right, man, let's do this. Like, uh, I'm just thinking we could just throw a hose down my, um, you know, my, where we, where you put gas in the gas cap and I'll just siphon off a couple of gallons and they'll be on their way. So the guy goes back to his RV for a couple of minutes and he walks up and he's got these small little like two foot hoses and a, and a couple of like connectors. And I look at it and I can see, I can see it's something that they've used a lot. Like they're going around to different rigs and they're

disconnecting the gas line and they're hooking up to that and they're filling their rig up with gas. And that's how they're getting down the road. And this all starts clicking because I can realize, oh, he wants to disconnect my gas line and take my gas, but they don't have any way of fixing it. So then I would be stranded. And then a couple other guys step out of the RV. They also have weapons. And then three mastiff dogs, you know, the big dogs, mastiffs, and one husky dog get out of this tiny RV, and they all start walking towards us. This is insane. It's getting a little tense, right? And so I'm trying to just be nice, trying to empathize with their situation a little bit, figuring like, if I resist, then, you know, they're desperate. They're going to take what they need. But if I can, if I can

cooperate a little bit, maybe they'll just, you know, they'll be cool. And so I'm looking at different ways, like we can disconnect the gas line because I don't have a lot of options here. And I'm like, there's just no way I'm comfortable with this so i'm talking to him some more and trying to get an idea of what they're looking for and um i walk him back over to their rv to kind of look at what their different options are and uh so i suggest when i say hey well why don't we do this why don't one of you guys just come with me and i'll just drive you to the gas station we'll fill up some gas i'll even pay for it okay yeah that'd great. So this one guy grabs a gun, grabs a knife and comes and gets in the RV. He's ready to go. The other two guys in the dog stay behind. So Hadiyah, Levi, and I take this guy down the road

to go fill up a couple of gas cans. And then when we bring him back, um, oh, and I should say, you know, I'm trying to be nice to the guy. So like we're at the gas station, I offer him a couple of beers, you know, just trying to loosen them up a little bit and, uh, get him, you know,

get, get a couple of gas cans filled up. And when I say gas cans, I mean like milk jugs, they had milk jugs and water jugs that we just filled with gasoline and then put caps on it. That's all we had. Good Lord. Yeah.

And then we got back and uh it was like this process of trying to get rid of them and not not like show our hand that much um it was one of those moments in life where you you had a sense this can go two ways i can either play ball and just kind of try to help these guys out a little bit or i can resist like you know when he walked up to my door, I could have told him to F off and that could have just started a whole thing. Right.

Cause they were, they were stuck and they needed to get going again. I have a sense they had somewhere to be and they had to get there fast or somewhere not to be. Yeah. Or somewhere not to be. And so, uh,

it took another couple of hours to get them to go. Hadia made them dinner even. And after they ate and filled their rig up with a little bit of gas that they got, we gave them like 10 bucks and quarters and they left. And man, it was a weird experience because like you start replaying like the 50 different decisions that you made along the way that probably made this go the right way. And how each in each one of those situations it could have gone really south like they could have been dismantling my rv

like best case they would have dismantled my rv worst case they would have stolen it maybe with my dog and my lady inside it like who knows what would have happened it was weird and i've never had an experience like that on the road before and And it was a damn shame, Wes, because it was a beautiful work spot with just bull cellular signal. No one around us. And you were chased off by these hooligans. Wow. Well, we thought about it. We're like, should we stay or should we go now? Because they left and we thought about it. And, you know, I had to sober up a little bit because I'd had some drinks. And so while I was sobering up, I was trying to decide what I wanted to do. And while they were here, I was drinking with them too,

trying to keep everybody social, keep everybody happy. So I'm sitting here, I'm drinking whiskey and ciders. That is impressive. You really just picked up right up on that and befriended them in ways, like you're saying, most people probably would not have. Yeah, I mean, it was like, so I had, but then like, so but like, I'm also like, there's a clock in the back of my head going like, you're diminishing your ability to escape.

Like I kept trying to arrange it. So that way Hedia and Levi and the awning would all be in, like, we'd have the awning in, the dog would be in the RV, Hedia would be in the RV. So if at any moment something went really south, I was going to try to run into my RV, start it up and just take off. But that's not a super fast process. Cause we have like these leveling jacks and shit. Oh no.

It wouldn't have been smooth. As they're firing. Yeah. And I just kept trying to like, I kept trying to arrange it so that way, like we had, we could escape if we had to. But a lot of times that just wasn't an option. And I just had to like, I just had to hang out with these guys and like drink with them and talk about their time in jail and talk about how one of them is trying to go get his kid.

And another guy was talking about how his wife died. Now he's now he's now he's calling himself a preacher who apparently makes money by selling tricks at gas stations from what I could pick up. And it was really like none of their stories checked out. Like each guy had a different story that didn't sync up with the other guy's story. They hadn't coordinated that at all. I gotta ask, they didn't happen to find out the name of this show, did they?

No loose stickers laying around, right? Yeah, that's why we gotta change all the logos now, because they know the show logos. So we gotta do new logos. It was weird and Hadiyah was really shaken up and you know

it's so I had to roll with it so fast that there was moments where she didn't know if I was just being blissfully unaware of how much danger

we were in right you couldn't communicate right because she thought maybe Chris just thinks he's making buddies with some RV guys

but there was a couple of times, like three times we were able to exchange a couple of looks that were like, yeah, I know what's going on, you know? And, and then afterwards we finally got a chance to talk about it. Like,

you know, just like, can you believe that just happened? Like it was huge. So we sat there decompressing for like a couple of hours sobering up. And then I thought, I got to go. I can't stay here.

I got to keep going because you know, those guys are going down the road, but if something goes wrong or they run out of gas, well, they know where to come back now. They know where to find me. So I got to keep going. But it,

it totally sucked because that's where I was going to record today's show. It's, I mean, it does, it did sound like a beautiful spot. Yeah. What a shame. I ended up getting down the road another three hours, and now we're just outside of L.A. I wonder if after they left, if for a moment there, like, I don't know, my mental image is just this group of, like, vigilantes or whatever, these vandals. And they're like, you know that Chris guy? He was a really nice guy.

You just like killed him with kindness, man. I did. I tried to just be accommodating and be a good host and kill him with kindness and empathize with their situation and try not to be judgmental about the fact that they're probably all outlaws on the run. You know, I just tried to roll with it and just try to, because everybody goes through a hard time.

I was trying to look at it from that perspective. Um, and the guy ended up like talking to me about his kid and how he's going to go try to get his kid from Georgia and problems he's having with his ex-wife. Like in a way we made like a genuine human connection. It's just that no moment was I under any false pretenses that these are my new best friends. It was weird. It was really something. And so then after it happened, I had to get in our group chat and I had to tell you guys all about it last night because it was just one of the craziest things that ever happened to me.

And I can only really tell you half the story, right? Because otherwise it'd take up the whole show. But man, am I glad to be here today doing the show right now. And I'm glad I found another quiet spot to record. Chris, I'm really looking forward to your memoir. I hope that comes out soon. No doubt, right? 100% agree with that, Brent.

This would have made a hell of a vlog episode. I'm just saying. But I don't know. It's just, whew. I did post a picture on my Instagram, which I don't use very often, but Instacrislast, where I posted a picture of the workspace. You can see the intro where Apollo in the picture too. And that was, it was just so nice.

And you could walk and, you know, didn't have to put Levi on leash. He could run around and it was so cool, but yeah, got to move on. Got to be safe. That looks like a great spot. But when you're so far out from society, you kind of forget that yeah there are different or fewer laws yeah and it was funny how they several times mentioned you know out here in the desert yeah okay i get it i get it i got it okay is that what

they sounded like a little bit actually they all had little, yeah, they all had a little drawl. But they were also, they seemed like they were on something that, I don't know what it would be, but they didn't, like they had glossy eyes and stuff. And they had a little bit of coordination issues sometimes. And so I don't know what was going on. Maybe they were drunk. Maybe it was something else. But I'll tell you what, that my uh that's my probably my craziest road adventure yet i thought getting hit by a semi truck but no that's that's this tops it were there a bunch more uh rvs out that like whenever you were heading out onto the land did

you like pass by a bunch more rigs and stuff or were you out there just kind of all desolate and these guys just roll up on you yep Yep, 100%. It was just us. That's why it's really weird. Like you're running out of gas. You don't come down this road to find gas unless you think there's a good chance there could be some RVers down there where you could go steal some gas. Because it is a,

like there's a few websites that list the spot that we were at. That's how we found it. So it's a known camping spot and it's a legal spot for you to go camp on federal land for up to 14 days. You think they were out there just because they wanted good cell phone signals as well?

Yeah, yeah. They got to make sure that, you know, their podcast has good signals. Everybody's hustling. I mean, if he's working at the gas station, you know, who knows, right? All right, well, let's talk a little Linux. I've never installed GNU slash Linux.

Last week on the show, we discussed the leadership challenges facing the Debian project. Since then, clearly because we covered it on the show, there is now at least five candidates and likely more that have emerged. And it's an interesting group of people. They all seem to have, all five on this first list here, all seem to have fairly long connections with the Debian project. I think one of the shortest one is from 2007. I guess that's a good crop of candidates. Why do you think it took so long for people to put their name forward? I don't know. I hate to speculate that it's because of some of the issues the project may be facing, but it's hard to say. The leadership position is also not necessarily

one that is purely technical, and it could be that people that are interested in the Debian project perhaps have more of a technical inclination. You know, they want to scratch that itch and not so much of a political itch, which the leadership position is a little more of politics than it is development. Yes, right. I mean, it is in many ways, you're performing a service for the project.

It takes time and commitment, and it may not be the little thing that is the most interesting in maintaining and being a part of the project. Each candidate in the list here is really fascinating. A couple that jumped out at me, Jonathan Carter has been a Debian developer

since 2017. And I believe he also maintains a fair few packages, including most of the Gnome Shell stuff in Debian. Another candidate, Sam Hartman, is interesting

because he's been a Debian developer since 2000. He's best known for the maintenance of the Debian Kerberos packages. And he's blind. I think that that's debbie and kerberos packages and he's blind i think that that's great i think inclusion is a great thing and uh even people with disabilities should have you know a prominent role in some of these projects absolutely that only brings awareness

to to those people with disabilities and and uh making that distro even more suitable for those people yeah and i wonder how that i wonder what that dynamic would uh how that would change the and making that distro even more suitable for those people. Yeah, and I wonder how that, I wonder what that dynamic would, how that would change the dynamic at events when, you know, it's one of the primary roles of the leadership position. I wonder how that would change. Several other interesting folks on the list as well, and I think we're going to see more show up.

So Debian won't have a leaderless problem. Now they have an issue of trying to pick just the best one. I think it's a pretty good spot for them to be in overall. It's in the campaign time now, right? And then after, is it another week from now? And then there'll be actual voting? That's my understanding, at least. I could have it wrong, but that does track my understanding. And we'll let you know what happens if anything interesting develops.

Why don't we, we only have a few community news stories. Let's keep going. What are the other ones that I just wanted to let everybody know about is that awesome Linux laptop battery optimization tool, TLP, has a new release. And I'll admit, I kind of thought this project was dormant.

If you're not familiar with TLP, it's a command line advanced Linux power management tool that helps save battery power for your laptop. You essentially install it and forget it. At least that's the hope. TLP is highly configurable though. So you could go in and modify the Etsy default TLP file to your liking in general. And there are third party GUIs like TLP UI, which you can install from your local repo and then manage some of those settings with a GUI. It really is just sort of install and forget. It runs in the background so it'll detect when you're on AC or battery and you can do all the normal things like apply different profiles,

more optimizations depending on what power profile that you want. And it just works. You can go use other tools like PowerTop. Don't use its auto-tune setting, but you can go use it to go check out and see exactly what's being tuned and you'll find it all runs very nicely. And in TLP 1.2, there's even better support for stuff like NVMe drives. I really like that. I've already upgraded because we've got those in those ThinkPads. There's also some neat stuff around Intel GPU frequency limits,

and you could already do this, but they've just integrated it in a nice little way to do it right in the tool, so you can tune whatever minimum or maximum boost frequency setting, depending on if you're on AC or battery. That's cool. I like that. Speaking of ThinkPad, there's a couple other things in there for ThinkPad users too. Oh yeah. A new native kernel API for battery features that are just specific to ThinkPad. So you don't need to be running a ThinkPad for this, but if you are, you get a few extras. Huh. I do really appreciate seeing the support for MV&E drives. That is all kinds of fantastic,

and it's pretty neat that you can tweak the Intel GPU frequency stuff. I didn't really realize that was a thing, and, I don't know, it kind of makes me even happier that I have an Intel card in my ThinkPad. So TLP is available in just about every Linux distribution under the sun, including Debian or Ubuntu

or Arch or Fedora or probably even OpenSUSE. I don't know. And the new version 1.2 is not available for most of those repositories yet. It is available though for Arch. Surprise, surprise.

And there is a PPA out there for Ubuntu users that is maintained by the TLP developer. So it's a legit PPA that you can go install if you're on a Ubuntu- that is maintained by the TLP developer. So it's a legit PPA that you can go install if you're on a Ubuntu-based distro. We'll have a link in the show notes. Link in the show notes for sure.

But you combine something like TLP with PowerTop and you can really get to a pretty good place in Linux battery performance. If you have been on a system in the past that you weren't quite sure what was hogging so much battery life, why was your battery draining so fast? Install PowerTab and PowerTab will show you what is hitting that CPU. What is waking up your IO? What is taking all of that power? That's what PowerTab will tell you. And then you combine that with something like TLP,

which helps tamp some of those things down when you're in battery mode. And it's intelligent about being on AC power versus battery power. And it's a nice combo that some distributions ship out of the box by default, but not most. And it's definitely worth an install. Not calling it an app pick, Wes, but it's kind of an app pick. You know what I mean? It's like a mini app pick.

It's a mini app pick. And it deserves it too. One thing that really impressed me when I was looking at this new release is just the quality of the documentation the project maintains. They've got all kinds of nice stuff, a really well-filled out FAQ, and some practical examples to get you started. Yeah, and I don't want to sit here and say,

man, this is going to fix all of your battery problems on Linux, but if you're having battery problems on Linux and you're not using PowerTop and TLP, do that, because it will definitely improve the situation. And in the case of my ThinkPad, when I have these two different components and if I don't use Chrome, I stick with Firefox, I can get eight and a half hours of battery life out of my T480. I think it's got like a 6L battery on there. I'm not quite sure what the battery size is, but eight and a half hours, that's an all-day battery life for me.

the battery sizes, but eight and a half hours is, that's an all-day battery life for me. And really, it's just a combination of tuning it with PowerTop, using Firefox instead of Chrome whenever possible, and then TLP underneath it to manage all that stuff. It's pretty nice. Brent, I know you're a mobile user. Have you tried out any of these things?

I learned about TLP maybe two or three years ago. And it's been awesome. And I love, like Wes said, it's a kind of a set it and forget it sort of thing. So I have it as a, on my main sort of install script and I just don't even think about it. But the ThinkPad stuff has been really nice. But one question I have around that is, how is a regular user,

you know, someone who's new to Linux and it's just a desktop user supposed to know about these things that they're available? It seems like it took me years to find it really. And it was almost by mistake. And so how do we spread this stuff around, right? Yeah, I agree.

Elle, I think you ran into something similar. Like earlier you were having battery drain issues and I mentioned to go try out PowerTop. Again, it's sort of like, how do you even know about PowerTop? Well, and one of the things that I ran into that I was joking around the IRC room that I'm raising my hand

is a lot of my battery drain issues were happening when my system was supposed to be idle. I'd shut it down or I'd put it in suspend mode and I'd come back. And a lot of these tools, I couldn't find the logs to go find out what was happening when I didn't actively have them running. Yeah. You know, the battery drain on idle is so frustrating. That is like, it's like, what are you doing, computer? You're just sitting here. What is wrong with you? And if you don't have the logs, or you don't have the tools to expose that information,

it's just a black box. You're trying to fix it. But I do want to hit the pause button here, because MiniMech is flagging me, and he's saying it's not always all roses. You've had some problems with TLP, Mini? Well, I had the same problem. I was trying to get down my power consumption

when in idling state. So I was tuning like hell and all of a sudden my computer freezed and it didn't wake up again and I was not knowing why. Until in the end I found out that I had a firmware bug in my SSD and once the SSD went to sleep,

it didn't wake up again. And it was really difficult to do an upgrade for the firmware because you need Windows to do the firmware upgrade of the SSD and you know what. So you can go too far when it comes to the configuration with TLP.

Yeah, it sounds like if you hit a weird firmware bug, there's not much you can do about that. And how would you even know until you try something like this? Well, it took me days. It took me days to find out. Oh, man. All right, that is a good disclaimer. That's good to know. Do you happen to recall what the brand of that SSD was, or maybe even the model, just so the audience has a heads up?

I still use it. It's Transcend MST400. It's used in a lot of Chromebooks. You can still buy it. And I guess with the new versions, the firmware has been upgraded, so there's no problem now. But when I bought the computer,

there was still the old firmware. Yeah, and if you got that old firmware, it only updates through Windows. That's a bit of a chore. Back to you, Brent. You were wondering why Firefox versus Chrome for me? Yeah, you mentioned it a few times, and I wondered what that's about.

I've noticed that the overall battery drain on Firefox seems to be significantly less than Chrome. I don't know what that fully is. The generous side of me would like to say, well, the Mozilla developers have really optimized Firefox, but then like the, the skeptic enemy is like,

well, that's because Chrome is just like using like Ram and CPU, like a madman or render pages as fast as possible. I don't know which one's better. Really. I don't know. I don't technically know the reason, but I've,

I've definitely experienced this now. Cheese. I know you wanted to jump in earlier. Have you had a chance to play with TLP at all? I haven't had a chance to play with it, but I did want to comment just reading through the article and some of the documentation. What's kind of interesting is that you can set up these configurations, but you can set these up all the way down to maybe even enabling or disabling integrated radios so maybe it could be a software software type switch for your bluetooth and wi-fi or whatever whenever you're at a conference or someplace you don't feel comfortable it could act as that software kind of switch that you see on some of these hardware devices that are

available and or coming down the pipeline. You know, now you say that, man, I could see like having like conference mode TLP config where it just shuts off all the radios. Just go dark and save battery power while you're at it. I like that. If you're out there and you've got a good config or you've got a war story like MiniMac did,

linuxunplugged.com slash contact. Send it in and let us know. Now, this next story, this is a hard one because it depends on your bias. When you read this story, you're either going to read it two different ways. I mean, it's all about money, really. Show me the money! What we have now is a company that is quasi-independent, calling themselves fully independent. And I'm talking about your good friends over at SUSE. They have announced that they are once again an independent company that is owned by an investment firm.

I'm not quite sure on that. It kind of depends on your definition of independent and if being owned by a private equity investor counts as independent. Yeah, and I mean, as long as we're just going to play the technicality game too, Wes, like there are big promotions all over their blog. It's in every press release. They are now the largest independent open source company. The only problem with that, I mean, again, we're just playing technicalities here, is the Red Hat IBM acquisition isn't done yet.

So technically, Red Hat's still the largest independent open source company. Well, that's what you get for reading blog posts and PR articles, Chris. That's so true. That is so true. And they've gone through a few hands over the years. Oh, yeah. I mean, I can't think of that many other companies that have kind of been passed around so much. Way back when, they were first acquired by Novell. Oh, yes,

Novell in 2004. And then, okay, that stuck around for a little while. In 2010, AttachMate bought up Novell, which then a little less time now, Microfocus in 2014 bought AttachMate. At that point, things got a little bit better for Zeus because they were turned into an independent division. But then, well, just last year in 2018, they were sold to EQT. Right. And I believe they are still the primary owner. EQT is what's called a, quote, growth investor. So it could be that SUSE is an independent corporation that is getting growth investment from EQT. So they're not actually owned so much by EQT,

but independent is still debatable. Oh, SUSE. We'll see what this means and if it actually, like, there have been some replacements of various leadership positions, so perhaps that will have long-term outcomes for the company, but only time will tell. Yeah, and everybody bear in mind, this doesn't really change anything for OpenSUSE, which is the distribution we often talk a lot more about. SUSE here is, this is the enterprise product. This is the product they're trying to position as a solid competitor to Red Hat Enterprise Linux. Hence, this now renewed focus on being an independent open source company.

It's kind of, it's a reverse jab saying we're not owned by IBM. But you got to ask yourself, like, you look at the market and you look at IBM's acquisition of Red Hat. It does kind of, and this is my opinion, but to me, it seems like it's only a matter of time until somebody else comes along and buys Seuss. Just like I would imagine that if Canonical went public or if Canonical wasn't in its current financial raising stage, it would likely get bought up too. It's just the reality of the marketplace right now. Yeah, it does say something about the value proposition of some of these companies. Well, we'll see how long they can hold out

and if anything actually happens. And I am surprised to see it is popular in some embedded spaces. I've seen it of the Linux I see at like various point of sale machines. I think SUSE is still the most common. Yeah, and they do have a pretty legitimate and large vendor ecosystem around

them. When you've been around for like 25 years like they have, you build out deep industry relationships, and those are things they can still leverage, and maybe even more so now. So I think at the end of the day, the takeaway from this story is SUSE is in a better spot than

they have been in a long time. They're independent again, and they've got the deep pockets of EQT should they need it. That's actually a pretty competitive spot for them to be in. I wonder how Seuss being passed around like that has affected their community and their community involvement. I mean, if you look at, say, the community around Ubuntu or something like that, or even the Red Hat CentOS community,

it just seems more vibrant, maybe? I do agree. It seems larger. However, I feel like OpenSUSE, being its own separate thing, has insulated a lot of the enthusiast community from these changes at Sousa. Like as an open Sousa user, it's almost always been sort of the footnote of all of these stories. Oh, by the way, if you use open Sousa, this doesn't affect you.

Like that's been the footnote for all of these purchases so far. And in a way, that's sort of been like a buffer between all of the corporate shenanigans and the end enthusiast users who are using the distribution. I think it's, I mean, you're right. It is hard to say if there's been a cost and their community doesn't seem to be as large as some of the other distributions. But it's still a pretty large, healthy community overall when taken in totality. And it survived a lot of change of hands, which is kind of impressive.

Oh yeah, absolutely impressive that they're able to forge along with, I wouldn't say that it's turmoil, but I remember way back in 2004, whenever the first acquisition happened and thinking that was going to be a real growing point for them. Their marketing is wonderful. But sometimes I'm curious if that marketing is directed where it

should be. You mean YouTube videos instead of something more directly beneficial and contributing to the community? Yeah. I mean, given a little something more back to the community, if you're going to spend that kind of money, I think it could better be better spent on other things, other projects, and just opening up and, and accepting it with open arms, the community and what, what they want to see.

People do love them though. They love those videos. I want to just, I didn't do it earlier. I want to mention, we've got Brent in the virtual lug today, Bruce cheese, who you're just hearing from L's in their mini Macch and uh thor thor hop hello mr thor nice to have

you join us that's that's new i think we've had thor in there before um and uh cheese is hard at work right now on refreshing the look of our logos for the shows. And man, I am so excited by this process, Wes. And it's awesome. Yeah, yeah. And I don't know, man. I just, geez, I just, we're not going to say much yet because it's still early days. It's, yeah, we don't want to spoil nothing.

But we've got a new fresh look that, geez, is hard at work on right now. And we've had some great, great input from the Linux Academy design team too. And I'm just really happy with the way things are going. And if, this is a big if, but if everything kind of syncs up and lines up, our current plans and intentions

are to reveal the new look, the new logos, new swag at LinuxFest Northwest this year. Yeah, that deserves a bell. It's a big undertaking. There's a lot of work that's already been done. There's a lot of work that's still going into it. And just the scope and scale of this is, I think, is pretty dramatic. Yeah, it's every show.

Plus all the different iterations you have for social media and for the websites and for the MP3 badges and all all these there's hundreds of different ways we use the logo that people don't even really realize because you know it's just all part of what we do and uh i so many so many times have i wished we've had the resources to uh clean these up and update them over the years so just super excited to have a here cheese working on that. So I know we can't say much because it's early days. Yeah. I will say that, uh,

I think people will enjoy a kind of unified look. Yeah. There'll be some familiarity, but it'll be a nice clean unified look across everything. Just like our new friends over at the Mate desktop version 1.2.2.0 is out this week. There's the new mate panel with a crazy ton of rework especially for wayland support and the number one absolutely no doubt about it most important feature wanda the fish has now been updated to work properly on high dpi displays it's about time

i know and never fear wes it's still pixelated on purpose because Wanda the fish is supposed to be pixelated. I actually don't know who Wanda is. I haven't met Wanda yet. Could someone? You got to go Google Wanda the fish. You got to do it. Yeah, it's maybe one of the best panel plugins, extensions,

whatever you want to call it ever. So, you know, GNOME extensions really have a legacy going back to gnome 2 where you could put these little applets up in the um up in the toolbar little eyes to follow your mouse around whatever yep x eyes was another one and so wand of the fish and you could go you go search for one of the fish gnome or mate and you'll see what i'm talking about uh i don't know why anybody wants to spend CPU cycles on something like that. And yet at the same time,

I love that it's, let's see, it's not the first feature, but it is one, two, it's the fourth feature that they list. This is what I noticed. All the most important stuff is buried at the bottom. Things like improving stability by closing memory leaks and fixing build warnings,

removing deprecated code, a whole bunch of updated documentation, and they put in a bunch of work to get their CI better, having clean builds on major distros. That's huge. And it's at the bottom. I know. Like that's going to improve every future release

from here on out. Like that's how big that feature is, but it's on the bottom. But Wanda the fish is at the top, man. And it's below the calculator now supports 15 characters of accuracy, which is great. And I like that a lot personally, but reorder this. Yeah. Yeah. It is nice to see that GTK cleanup and that Wayland support coming along.

There's something that just tickles me about this classic oldstyle interface paradigm that is modern, high DPI, GTK3, ready to go, running on Wayland. Like, it just tickles me. It just really does. I want to mention a story that's a little frustrating right now. And this is one of our last ones for the week. And it's KDE Connect. Just such a great tool.

I've met Albert. He is a fantastic, talented developer. And he tweeted today that Google has yanked KDE Connect, the Android side that you need to connect to your Plasma desktop. They've yanked it from the Play Store on grounds of violation of a new policy for apps that access your SMS. of violation of a new policy for apps that access your SMS. Now, this policy does have an explicit exception for companion apps,

like many others, but KDE Connect, for whatever reason, was not granted that explicit permission. Now, those scammer apps that are stealing your SMS contacts and uploading those to Chinese servers, still good to go. Now, it has been resubmitted to the Play Store, right? Yeah, I believe so. It's just so frustrating, though, because this is one of my favorite applications, and I am one of the things I'm the first thing to install whenever I reinstall my Android phone.

Yeah, and it had to get resubmitted without the SMS reading support. Yeah, that's it. They don't have that capability anymore, which I have other means around that, but it's kind of one of the prize features of the application, at least for users still stuck on SMS. It's an obvious, obvious sort of failure on the Play Store side

because this app has legitimate use for this data, and it's been using something that has been allowed up until this point. And then where it really falls down, though, is there's really no humans he can talk to. Yeah, and I think as Brent points out, and he can chime in more,

but there are competitive marketplaces. So if the Play Store wants to be seen as a place to get safe, legitimate software, they're doing exactly all the wrong things. So maybe we should be using F-Droid. Yeah, this is probably it, right, Brent? I mean, this is the opportunity for people to try something different.

Yeah, so true. I've been using F-Droid for a few years, just as it started out as a bit of an experiment just to see, oh, what's in there? But it's really come into its own in the last year. So I use it exclusively, actually. I just checked my phone since the intro where you mentioned KD Connect, and almost all of the apps I use daily are from F-Droid. mentioned KD Connect and almost all of the apps I use daily are from F-Droid. And so if anybody doesn't know about F-Droid, it's just basically a repository for open source focused apps that

you can get on Android. So you don't need any crazy logins or permissions to install it. You can go get it at f-droid.org. But for KD Connect, it's on there. They're updating it all the time. And I was really sad to hear what the news was here on KD Connect, but to hear, for me at least, that it was pulled from the Play Store doesn't really affect me any. So I'd encourage anybody who is interested in Linux Unplugged, they'd probably love F-Droid if you're not in it already. Yeah, you could always use both, too.

They're not mutually exclusive either, so if there's some things in the Play Store, you could still do that. Or you might be able to get rid of the Play Store, depending on what you really need, and just stick with F-Droid. Is that what you do, or are you mixing it up, Brent? Well, I've been mixing it up now

because I have thought that I needed to. And one of the last apps that I thought I needed to do that with was Signal Messenger, which I use quite a bit. Which I wonder if that got pulled too, because it definitely has access to your SMS, but you mentioned there were some exclusions. But I noticed that that was the last app and you can just get it from their own website. So I just checked and none of the Play Store apps I'm actually using on a daily basis. It's mostly

all the cruft that comes with the phone. I would say I'd be totally fine with running FDroid 100% of the time now. There's been a huge increase in the amount of usable apps in there and the quality of them, so it's been great. That's nice. That's really good to hear because it's probably been a year

since I've checked it out. So I think I will load it up on my Pixel and I will give it another go because I'm a big, big fan of KDE Connect. And I think this is the killer app right there to really make me give it another try. You might have some problems, but I'm not sure with the GNOME

experience, but on KDE the KDE Connect just gets better and better and better and better every single month. So the features are amazing. There you go. There's your unplugged app pick again. Not quite an app pick, but it is a bit of an app pick. After it. We're knocking out of the park team.

I am really happy to see that. And I hope that Google works with the KDE Connect folks. The real issue is, and their main complaint has been, we just haven't been able to talk to anybody. We just want to explain what our version of the story is and see if you'll work with us. But we don't even have that opportunity. And that's the classic complaint about Google, right? When their machines do it correctly, they're kind of great and streamlined. But when there's any kind of kink in the system that you

need a human to resolve, there are no humans around. I'm sure the game stream service, though, that's going to be great. All right. Well, I want to talk about eBPF. Don't call it eBPF, just call it BPF. No, it's not your Berkeley packet filter anymore. It's something much, much cooler built into the Linux kernel that kernel developers and performance engineers like Brandon Gregg at Netflix are absolutely excited about.

So just a brief moment of housekeeping. There's really only one thing I want to tell you about this week, and that is LinuxFest Northwest 2019 coming very soon. And we've set up a telegram group. Go to jupiterbroadcasting.com slash LinuxFest, and you can get in on our Linux Fest Northwest community telegram group. And in fact, I want to extend this invitation to even those of you who can't make it to Linux Fest. We'll likely be using this URL for Texas Linux Fest and other fests that go on around the country or the world. So I encourage you to join if you want to contribute positively to the

conversation. And the other reason is, as you heard, we're going to be updating the show logos and the network logo. So I thought one thing that might be kind of cool to do is do a big old series of giveaways in that Telegram group of the old retro sticker logos while we still got them. Did you say retro swag? Yeah, man. Well, I mean, it's current swag now, but it's soon to be retro. Yeah. So we'll start doing that as LinuxFest Northwest gets closer for folks in that room. And I don't know, we're kind of kicking around. Like

if you show up at our booth at LinuxFest and you show that you're in our telegram group, maybe we might have a little something extra for you. And it's not a bad place to go anyways, because that's where we coordinate things like the parking lot barbecue, which I got to say, Wes, Emma is getting me so hyped. Emma from System76 and I are chatting now like on a daily basis about like, hey, what about this? Hey, do you think we could do this? Like today we were discussing the table situation, you know?

She is a great organizer and really knows how to throw a fantastic party. So I'm sure it's going to be even bigger and better than last year, which was already amazing. I feel like we should get her on here sometime and have her maybe a week or two before and have her throw her hype game out there for it. That is a great idea.

I will suggest that to her. And really, I should mention, make sure on air that System76 is providing a ton of the food and drink too like everybody's bringing something but they're bringing i would say the bulk of the meat and that's not cheap and i just i just it's so cool because it's not necessarily like going to change a thousand lives but it might make you know maybe like you know 50 to 100 people really happy and there's and they're still invested in doing doing that. And that's one of the reasons I'm in LA right now.

I'm bringing the RV back up from Texas to get it up to Bellingham for LinuxFest Northwest. So like quite literally, the wheels are in motion to make this a huge event. We have all of the details at meetup.com slash Jupyter Broadcasting. That's where the barbecue meetup information is at. And that telegram

group, which we'll probably use for future Linux Fest 2, but right now it's for Linux Fest Northwest. That's at Jupiter Broadcasting dot com slash Linux Fest. And that is where I'll be giving away the soon-to-be retro stickers. So if you want to get in on that, or maybe a little extra swag at the booth, you go get in that telegram

group, you hang out, contribute positively to the conversation, and we'll all have a great time. All right, that's it. Just a real quick, you know, a little short housekeeping this week, because really I want to hear about EBPF. I'm not often like starstruck when I see people, but Brandon Gregg, he's got this, he's got this amazing look about him. It's, he's got a coif of hair that looks like it's always being blown in the wind. But there is no wind. You're like, well, how does this work? Right. And what did you say, Wes? It makes it look like he's always having a good time the way it is. I mean, giving a talk at scale or fighting performance problems in

production at Netflix or surfing on a beach. I imagine he looks exactly the same. That's exactly it. That is the perfect way to describe his look. He's got, you know, he had this great ensemble, a pink shirt, a super bright red belt and some red pants and just really caught the eye. And we talked about this a little bit last week. And we talked about this a little bit last week. He did some incredible takedowns of Minecraft performance using eBPF, which I think I'm doing it a disservice, Wes. I'm about to hand it over to you.

But I think maybe the closest parallel is it has a lot of same benefits a tool like DTrace might have, which has a lot of hype around it. Yeah, exactly. And it can be a little confusing because BPF is fundamentally a virtual machine. It's a bytecode that runs on a virtual machine inside the kernel. And so it has a huge number of use cases. BPF is obviously Berkeley Packet Filter and came from the use case of filtering packets. That's not what we're talking about today and not really what Brendan's doing.

He's talking about tracing. And one of the new tools on the block that is very similar to DTrace in some aspects is BPF Trace, which is kind of a command line playing around handy one-line scripting utility whose command language is modeled after DTrace. It kind of looks like awk. So you might be wondering, okay, what does BPF Trace do? Well, it allows you to write small programs that execute whenever an event occurs. All right, that's not really a useful explanation.

I know, what's an event? Well, an event could be a system call, a function call, or even something that happens inside a function call, like 50 milliseconds has elapsed since the last time something happened, or a page fault occurred. Once you get an event, this tooling lets you do something, right? Like log something, compile statistics about what's happening, and you have access to interesting information here, like the PID, stack trace, time, all the arguments on the call stack,

the return value, all these nitty little details that you might want to understand how your program is really running. Now, do I get those details because the BPF virtual machine is living in the kernel so it can learn what the kernel knows? I guess all of that sounds good, but why being a virtual machine in the kernel, why is that the important part? We're going to get into that. And basically it's for performance reasons because you don't have to incur a context switch or try to go dump a bunch of data from kernel memory into user space. Instead, you can compile statistics inside the kernel. Once you've computed those, just do one copy at the end of the final computed value instead of having to share all the intermediate data, which as Brendan will talk about, sometimes that can be like gigabytes a second. We should also note here that we're

talking about cool tools that you can use now. And that is one of the big takeaways is that this stuff has been around for a little while, but it's matured. And finally, most distros are going to have kernels and possibly even packages for you to use. So these tools exist. But to really understand some of the motivation and background, I thought we should talk about the history. And we can't not talk about DTrace. I mean, you already brought it up, Chris. And DTrace is a full system dynamic tracing framework originally developed by Sun. And dynamic tracing, it's all those things we've just talked about. And then the dynamic part is usually you don't have to recompile, right? You don't have

to have worked with the developers to go pick points that you want to specifically trace. On the fly, you can add new trace points so that you can understand what's actually happening on your production systems. And this is a huge capability and Linux hasn't had it. Yeah. And this is critical for very high performance workloads where you need to understand what components are getting slowed down at what times and what is the root cause. And imagine a Netflix scenario, for example, where you're serving up streaming video to millions of people, a few seconds, even a second of delay times a million is unacceptable. And so the requirements and the pressure for extremely high performance applications sort of necessitate this tracing ability where it is exactly what the word implies.

You can trace the steps that the application went through, what resources it accessed, and use that to sort of walk back a story of what it was doing and where it slowed down. Now, you may wonder, okay, DTrace was actually put under, re-licensed under the GPL just last year in 2018. Shouldn't we just be using that? Why is Linux reinventing the wheel here? But as you'll see, eBPF is nice both because it has, you know, this general virtual machine that is already used for other features in the kernel, and so is a simplification and a unification of code paths in that way.

But also, in some ways, it's more powerful. Before we get into that, though, let's try to sympathize, because for a long time, Brendan was the expert on t-trace, right? He made all kinds of great tooling around it, spent a lot of time working at Sun on these systems. And then he went to work at Netflix on Linux, where he didn't have any of these tooling. In fact, I think he was a bit of, I don't think a Linux new, but it wasn't his daily driver. And he, like you said, is a big fan of DTrace. And the reason why I want to underscore that is DTrace has become one of

those projects that has its fans. And it becomes a holy war when you start talking about different methods of tracing. And if it's not D-Trace, it's garbage to some people. And I'm not saying Brandon was one of those types of characters, but he was familiar with that camp, and he was kind of a D-Trace fan. And to

see him make this move and appreciate what BPF can do and how you can use it, not only is that fascinating, but the journey of joining Netflix that got him here is also an incredible story. Yeah, let's let him tell a little bit more of it. Okay, thanks for choosing this talk. My name is Brennan Gregg. I am a senior performance architect at Netflix. And as of tomorrow, I'll have been at Netflix for five years. Time flies. That means I'll have worked on Linux full-time for five years.

When I came to Netflix, I was a DTrace expert and performance analyst. And coming to work on Linux full-time, I knew that I'd have a challenge of lacking all of the tools that I created. But I knew this challenge would also motivate me to find out new ways to accomplish things. And I discovered early on that Linux had Ftrace.

And so early on at Netflix, I created a suite of tools called perf tools that were based on Ftrace, and they let me solve a lot of problems. I also met Alexey Storytoyov, who was working on a new virtual machine for the kernel called eBPF. And I saw how that could be used

to do more programmatic things. And so I've also created a lot of tools using eBPF. And that's the thing, is he's just trying to pick up whatever framework comes along that can actually help him solve his real-world problems on whatever operating system

he happens to be using. Very pragmatic. The nice part here too is that it's kind of been around long enough that it's easy to access and he was working on that problem right at scale. And so now five years on at Netflix, I can finally do what I want to get done in production using all of these new tools. It's really amazing and exciting. And also we're at the point with these technologies where early on we made sure they got into the kernel because that takes a long time to propagate and now we're getting them packaged. And so I was swapping emails with Canonical yesterday to make sure the BPF trace package appears for the Ubuntu distributions, which would be

really awesome. They've already created a snap for it. And we have BCC, which is the other front-end, has been packaged. So we're at the point where these superpowers, you can use them for analysis just by doing a package add to a system.

I like that point because he's trying to underscore that they are here or they're coming. Really, they're here. You have these tools. You can download scripts. You can install packages. I do note there that he's using package add, which is the FreeBSD command. But really the point is, this is

stuff you should learn. You don't have to be a programmer. You don't have to be able to write really complicated scripts because a lot of this stuff is pre-packaged for you. So let's learn more about what it can do. Enhanced BPF, also known as just BPF, can do lots of things. It's kind of crazy to explain. It began life by a company called Plumgrid. They were doing it for software-defined networking, and Alexey Sartoyev came up with it. And that was so that it could handle packets and then you could define some sort of proxy or router or whatever you want in software

and it would handle it. This is really good. This is really good for us because packets can be very frequent. Packets themselves can be millions of times a second. So since this was designed from the beginning to handle very high packet or very high event rates, it means it's been optimized. And so we can use it for things like off-CPU analysis and schedule analysis.

It's been used for lots of other things as well. Denial of service mitigation, intrusion detection, container security, observability, what I do, BP filter, even device drivers. There was an infrared device driver written in BPF. I mean, yeah, you can just use it for basically everything. And that means people have been speculating it might be taking over the kernel. There's been a number of discussions about BPF taking over everything to the point where it seems that the ability to write a user-defined

program and have the kernel run it, and that user-defined program to have more and more capabilities sounds a lot like a microkernel. So microkernels is where you write stuff in user space and you have a minimal kernel that runs it. And so it's a little bit scary to think about microkernels and Linux, but anyway.

I think it was Jonathan Corbett who first came up with that realization that we're heading towards a microkernel. I want to underscore here, too, there is a step where there's an analyzer that takes the BPF program, and the BPF bytecode itself has some nice properties, like there are no loops, so in some ways you can guarantee that it will halt. So there are some safety checks. And because there are disparate systems in the kernel using this stuff,

I know that the lot of I's thing is often a fallacy, but it's true, this is just being tested a lot and it means we're not having disparate code paths that are all doing slightly different things. They can all be tested in one unified architecture. The other thing I think that we should make clear, and I didn't really realize, is what's with the name? Because

maybe we're doing a couple stuff. There are networking applications, but that's not really what we're talking about. So why are we still calling it Berkeley Packet Filter? So the question was last year, we wanted to change the name of BPF, because BPF is kind of a terrible name. BPF, maybe I should have said it, it stands for Berkeley Packet Filter. But nowadays what we're doing has nothing to do with Berkeley,

little to do with packets, and little to do with filtering. And so a better name would be something like Virtual Kernel Instruction Set, VKIS. Virtual Machine... Anyway, Alexei and I are engineers and we're terrible at naming, and we couldn't come up with a better name. And Alexei has settled on BPF because he sees it as... And he wants it to be BPF and not eBPF because he sees it as equivalent and he wants it to be BPF and not eBPF

because he sees it as equivalent to x86 and ARM. Because it is, BPF is an instruction set. And so this is a virtual machine instruction set. So he wants it to be three letters like everything else, x86, ARM, BPF. So that's why we're stuck with BPF. All right, don't call it eBPF, it's BPF. I understand.

You know, it is. That three letters does make all the difference. Okay, so that's like a little bit of practical foundation. It is this, you know, machine language. You probably don't want to write that. And it can be a little bit complicated to understand what your next steps are because there are multiple front ends.

There's lots of front ends for BPF. Raw BPF is where you write instructions like this, BPF instruction level programming. There's some samples inside the kernel source code. I think very few people are ever going to need to do this. It's really hard. There's also, if you look in the kernel source code, there's C examples as well, which is a little bit higher level. But we've spent years trying to make it even easier. So we've come up with a BCC frontend which has a Python interface, a C++ interface

and a Lua interface so far. And I imagine people are going to add other interfaces to BCC as well, other languages. And so that lets you define your kernel program in C and then the frontend that manages it in Python or whatever that is. And that's really pretty handy. He'll talk about this a little bit more, but it means you can have one part of your program

actually do all the complicated little BPF stuff, and then rely on nice high-level Python libraries to do, if you're writing in a REST API on top of this monitoring tool, whatever you might be working on. That is what we should contrast with BPF trace, which is what we started talking about. And that is not really for standalone tools. It's for one-off scripting and exploration.

And there's another front end called BPF trace, which I was working on a lot last year. BPF trace is good for powerful one-liners and custom scripts. And so with BPF trace, I can say BPF trace, T is for trace point, syscall, sysenter open, print out the file name. Or do a

histogram of the read size distribution by process. And that sounds actually pretty darn handy. You do need to know how to do that. Thankfully there's extensive documentation. And maybe this is a good point to underscore. This kind of tracing is kind of opening up new bounds,

stuff that we haven't really seen before, not just in Linux, but anywhere. EPFF is solving new things. Off-CPU analysis was this cool thing we really wanted to do, but it was so prohibitive to trace scheduler events. Now EPFF is letting us do it. Another thing it's letting us do is I can save stack traces when a thread blocks.

I can save stack traces when a wake-up event happens and I can associate wake-up threads with blocked, wake-up stack traces with blocked stack traces. So I can take that off-CPU time example further with Minecraft because sometimes when you're looking at a lock and you're saying I'm blocked on this lock, your next question is, well, why? Why was I blocked for 10 milliseconds on this lock? Who was holding the lock for 10 milliseconds? I need to know them. Well, you can know them

if you trace the wake-up events. So the kernel has to manage this. So a thread releases a lock. The kernel then does a wake up from that thread to the blocked thread. So you grab the stack trace there and you associate it here. Doing that association was something we've never done before in kernel context. Dtrace could not do it,

but it's something that BPF can do. So you can kind of see why he's excited. This is genuinely neat stuff. I like that example because some of the like technicalities of the various cape robes and kernel fixtures that you're attaching to may be beyond a lot of people. But trying to understand what other part of the system is in contention with you or hogging resources, I think we've all been trying to diagnose those problems. But okay, maybe you don't want to write BPF trace scripts or anything. Is this still applicable and useful? And I got to

say, yes, a lot of these, you can just run on the command line and they will tell you stuff about your system that you might want to know. And you're already installing debugging tools, right? People are installing stuff to monitor what network connections are happening, stuff to monitor what disk space is using

or watching IO latency. These are some of the best tools, especially for end users. If you're an end user, the BCC repository has lots of canned tools, so you can just use them straight away. I only demoed two so far, Profile and OffCPUTime, but there's lots.

The tools have man pages, example files, documentation, and usually you can get BCC on a system nowadays just by doing apt-get install or yum install. Which, I mean, is super easy, and clearly he's used Linux a little bit there. Yeah, there was no doubt. Actually, watching him work, it was pretty clear he knew exactly what he was doing.

And even if, you know, so in this case, in this example, right, he was showing Minecraft. But, you know, in his day job, of course, it's this massive, massive scale stuff. And so it's fascinating that they've created something that a very, very small portion of people could write direct bytecode for all the way up to abstracted layers with tools that anybody can just pass some flags to, and it does all of the hard BPF interfacing for you. It's a whole range. Yeah, and the tools aren't hard to use. One of them is biolatency. As an example, just to have one in the slides, this is a BCC tool called biolatency. It's showing the disk IOLatency as a histogram. It's efficient. So the

count column is what's maintained in kernel context, and it just copies the count column to user space and user space prints this out. Prior implementations, I had to dump out every disk event and post-processing user space, and that's too expensive. So you can actually just get a really nice histogram right in your terminal, install it from your package repo, and you can see a histogram of block latency.

Or another cool example is using and monitoring whatever people are typing into the readline library. Let's hear about OpenSnoop, which is also really cool. If you want to go see whatever TCP connections are happening on your system, well, there's OpenSnoop about OpenSnoop, which is also really cool. If you want to go see whatever TCP connections are happening on your system, well, there's OpenSnoop. OpenSnoop shows all the files that are opened.

In fact, it's great for locating config files and log files and things that can help you troubleshoot or debug software. Yeah, so maybe you're running some new software you've never run before, and you're trying to wonder where are the configuration files? What files are touching or modifying on my system?

OpenSnoop tells you in real time. That's another sneaky app pick right there for the episode. OpenSnoop is another great tool that people should go grab. Yeah, another one right there with it is ExecSnoop, which is the same thing, but you can watch as users on your system are running applications. You can just watch whatever they're executing. ExecSnoop tells me what's happening when Minecraft starts up. So I can say, okay, so that ran, and then it ran this thing,

and then it ran these other things. Oh, he just has the best presentation tile, right? I love listening to Brendan Craig. There were so many cool tools, Wes, that he covered during this talk. A couple of times I leaned over to you and I'm like, man, we cannot tell people about this. Oh my gosh, yes, right? There's just so many that you might want to use.

EXT4 slower comes to mind, which is where you can just go watch slow EXT4 stuff. You can actually debug the file system separate from how it's interacting with the block device. So you can actually isolate where the part of the problem on your system is. You can also instrument things like TCP open or close calls so you can understand them. Now, unfortunately, there are some gaps. You may notice that there's no UDP

commands here, but this is all a project run under the IOvisor name. That's a bit weird, but it's actually under, you never would have guessed it, Chris, the Linux Foundation. And that's where BCC lives. That's where all these cool cool tools are you can just install them, start playing with them we'll have linked in the show notes this great graphic that just sort of instruments the stack and tells you like here's how you do application level debugging like there's one to go look for slow MySQL queries just like built right in

you don't have to do anything to specially tune it just run that command so it's super easy to find and play with and really the reason I wanted to talk about this today is because this is new technology, and I think Brendan says it best, but we should all be sharing the cool stuff you can do with it. So takeaways, you can easily explore systems

in powerful new and custom ways. You can also contribute. And also please, if you can, share war stories and blog posts and give talks about these tools. It's a lot of new stuff. And if you come up with a cool usage, it's going to help the BPF community to share it. We are finally living in the world that we've all really wanted,

where we have a functional D-trace for Linux or something even better. And the future is probably going to be GUIs and other tools, but it's not going to get there unless we build it. Yeah, I think that's the important thing for people listening to keep in mind is if you're an expert at this kind of stuff, it's totally ready for you to jump in right now. But in a year or two,

Brandon thinks that we'll probably see graphical applications where you check a few boxes, click a few buttons, and run it against the application, and it does a lot of this deep analysis for you in a few boxes, click a few buttons, and run it against the application. And it does a lot of this deep analysis for you in a very consumable way in a graphical session. And we're not there yet, but those tools are likely coming. And it's kind of exciting because this is a major new feature that's been retrofitted into our lovable Linux operating system. Yeah, really. I mean, eBPF is here to stay. Start playing with it.

You can use it for all kinds of better ways to understand your system. And in the future, hey, it might just replace your firewall. But that is for another show. Yeah, and a big thank you to Brandon for giving that talk. And he kind of inspired us,

you know, at the end there, and he was like, spread the word about this stuff. This is really cool things that we can do now. And it's a challenging topic sometimes to talk about because it's not necessarily applicable to everybody's day-to-day life. And it's a slightly technical thing, but at the same time, it's a massive kick-ass feature that our favorite operating system now has, and it is worth spreading the word on that stuff. So

thank you for grabbing those clips, Wes. All right. Well, we do have to kind of wrap it up there. We could keep going all day, but at some point I do have to hit the road and keep on moving on because I got to get back up to that studio. There's nothing like hanging out in the studio with Wes. Hey, I think we've learned it's not safe for you to stay in one place for too long.

You know what's funny too, man? You think about it is I drive around with a live tracker in my RV posted at jupiterbroadcasting.com slash rover. And I have never had a problem from an audience member. It's never been an issue. The people that I had issues with probably had, they would have no idea that I had a tracker in there. Although if they had stolen the RV, we would have known where they take it. The thing is too, is I've got two different trackers in this RV. So even if you saw the visible one, there's a hidden tracker as well.

So we totally would have been able to find it. And that would have been a hell of a story. Just not one I want to share. I'm glad it didn't go that way. Hey, go check out Wes on TechSnap. They just did a killer episode on machine learning and the ethics of AI. I learned so much from that episode. TechSnap.systems. Go check it out. He's at Wes Payne. I'm at Chris LAS. And if you're craving more Linux news, go check out LinuxActionNews.com.

A weekly Linux news podcast between Joe and I, about 30 minutes every week. But the most important thing you could do is join us live on a Tuesday over at jblive.tv. See you next Tuesday! The Unplugged Program. All right, guys. Let's title this thing, jbtitles.com. You know, I didn't think about doing this in the main show. Well, I kind of thought about it, but, Wes, I've decided to out you in the post-show. Ladies and gentlemen, it has come to my attention that Mr. Wes Payne has gotten hooked on the disco dingo sauce.

So what do you think so far, Wes, of the old 1904 beta? Or I guess I'm not even sure if it's beta yet, you maniac. I mean, I did grab it from the dailies, so I don't think so. That's what you got to do. You got to do that. It is what you got to do. Honestly, it's been a lot of fun. You just kept talking about it on that there Linux Action News program that I enjoy so much. And I was like, okay, well, I know Chris, and I know what he's been through with GNOME. So if he's talking about this, and then you mentioned some other stuff about Disco.

So I was like, that's going to be what I'm going to run. I was really happy. Honestly, Neon's been working just fine for me. Almost no issues. Super Rock Solid wasn't in a hurry. I mean, I've got other partitions for trying stuff out. I wasn't in a hurry to repave there. Something about Disco, though, it's getting me in that fever.

I don't think I'll wipe it just yet, but I have been spending almost all my time on the live install, yeah. Huh, you know, I'll tell you, the reason what's really drawn me in, and I don't know how long this is going to last, so this is kind of an interesting experiment in a sense, is it's not often, it's not very often that you install something and it is perceptibly much faster. That's what struck me. I just was like, this is super pleasant and I could

stay here. Yeah. I mean, it's, it's, that's fun. And you know, often you'll get that with just a fresh install, but I have installed GNOME a few times, and I'm familiar with what a fresh GNOME shell install is like. And this is different. This is a totally different beast. Rawhide's version's great, too. It really is, but DiscoDingo just seems like perhaps they're a little bit further along in their implementation

and a little more thoughtful. And, you know, I got to say, like, credit where credit is due. The Fedora camp and the Red Hat team has plugged away on GNOME Shell for a very long time. They've fixed some important things. But it is without a shadow of a doubt in my mind that Gnome is getting in a much, much better state now that Canonical is involved in the project. They're putting their efforts into the project, too. And we are seeing the collective results of more people, more talented people working on Gnome Shell, and it's just simply making it a better product.

And man, if that's not great, and if that isn't just one of the best damn things about open source, because these two companies are competitors, yet they're making it a better product. Right, we can find some collaborative shared resources and infrastructure that we can all invest in and we all get to benefit. It's so great. It's just so great. And that's not to take anything away from Plasma or Mate or any other great desktop environment. No, actually, I was wanting to go play with that new Mate release, but I was like, well,

I'm already kind of playing with a different desktop environment release, so maybe I'll wait a little bit. But I do always love Mate, too.