LINUX Unplugged - 329: Flat Network Truthers

Episode Date: November 27, 2019

Build one flat network across cloud providers, personal networks, with even thousands of nodes. We feature two amazing open source solutions, and the creators behind them. Plus community news, first impressions of Google Stadia, listener feedback, and some great picks. Special Guests: Alex Kretzschmar, Guus Sliepen, and Ryan Huber.

So I've been trying out Google Stadia this week. Oh, boy. Just a little bit. Just a little bit. You hipster, you. Well, I decided to get the, what do they call it, like the Founders Pack or whatever. My thinking was, is twofold. This is going to be somewhat good for Linux gaming, even if it's a little,

because it actually requires developers target Vulkan and they have to target Linux. Right, that's what's running on the backend. And Google requires that. So I like that idea. But for me, it's sort of like outsourcing the bad, outsourcing the proprietary. With Stadia, I can have all free Intel drivers on my laptop, great power, low heat, all the stuff I want, but then I can open up a browser and I can play actual, real, quality video games. That's the dream. And when you're done with it, you can forget about it and you don't have this hardware laying around.

Or like, maybe I get home and all of a sudden something changes in our plans and I find myself with a free evening. All I've got is systems that are incapable of really gaming. And that happens three times a year. So you're not really wanting to invest. Well, I mean, you've got to imagine too, right? As far as disk base goes, I mean, games these days are anywhere from, you know, 20 to 100

plus gigs worth of storage. And if you don't play them all that often, you're just sitting there with a disk that's starting to fill up with games you don't play that often. I had that problem. I had Steam installed. I was like, I haven't played for a while. I'm just going to delete all of this. And then you wanted to play with me and I didn't have any of it ready. Steam cash, bro. That's what you need to do. Yeah, I know. That's going to happen at the RV, but that's a topic for self-hosted. What were your first impressions? You got to try it a little bit. You had the controller wired into my PC because that's the only way to do it. I'm impressed. I mean, I'm definitely in the casual gamer style.

More than enough for my needs. And if I had not known about Stadia and I just walked up and you gave me the controller, I was thinking I was playing any console. Have you ever played Destiny 2 before? Not 2. I've played Destiny 1. Because you walked in and cleaned that entire room out with a shotgun. I mean, I may have played some video games in a past life.

I mean, like, you had some tactics, too. Like, you went around the corner. I mean, yeah. So response-wise, it's pretty dead on. Yeah. What I see it as is you trade. So you could do some of these games on a cheaper GPU if you didn't want to invest every year, every couple of years in a big GPU. Right.

But you'd have to turn down the textures, turn down the distance, turn down the reflections, the shadows, the lights. With Stadia, you can have mostly that stuff turned up pretty decently well, but you trade a little bit of fuzz when you got a lot of motion. Right, it was a little bit of blur. But honestly, I mean, in the middle of the firefight,

you don't notice any of that. You're not paying attention because you're just really focused on the thing you're shooting at. So I probably am going to stick with it. My next big commitment, though, would be to move up from the free games to, like, the pay games. And they're, like, full price, Wes. And they just live on Stadia.

And, of course, you're worried how long is Stadia going to be around. I mean, yeah. It is Google. Hello friends and welcome to episode 329 of your Unplugged program. My name is Chris. My name is Wes. Hello Mr. Payne. Hello.

Big show. Huge show. Hello Cheesy. Hello Alex. Hey girl. Hey there babes. Hey there. And hello virtual lug. Time for appropriate greetings, mumble room.

Good morning. Hello. Hello. Hello. We have about as many people in the quiet listening lobby as we do in the actual room. This new time has been interesting and well-attended. I really appreciate it. We're glad to see all of you.

So this week, we're talking about a problem I've been trying to solve. In the spirit of free and open source, I am scratching my own itch this week, and I am NAT busting with my VPNs. I have a situation where I'm behind carrier-grade NAT at home, and I want to create an overlay network across all my machines that are in, on a Linode data center, on a DigitalOcean data center, in my RV, at the studio, when I'm traveling on my laptop.

I want an overlay network that spans the entire internet, that puts them all on one LAN. Right. So you can just forget about it. It doesn't matter if you're here at the studio or you're down at Linux Academy headquarters, you can pop onto your home network, no problem. There's a plethora of options to do this. We talk a lot about WireGuard. WireGuard's not necessarily the best solution for that.

We'll tell you why, and we'll tell you what some of our favorite solutions are. And then we went and got interviews with the creators of them. So that's all coming up in today's episode. But first, we have some community news to get into. This one's nuts, Mr. Payne. Did you see this Android, which has its very own weird, strange kernel legacy, its own strange sets of patches, may, may, may end up going mainline. I actually think this is one of those initiatives that's going to die on the vine about a year in, but—

Right, it seems like there's a lot of pressure and momentum against this. Yes, well, people don't like proprietary crap in the kernel. But the current Android ecosystem is really kind of polluted with hundreds of different versions of Android, each running a different variant of their own Linux kernel. Each version is designed for a different phone and it's different configurations. It's always the way it's been.

And Google has been working to fix that problem by mainlining the Linux kernel, just going with a straight, like, upstream kernel one day. It's kind of the dream, right? I mean, we talk about all the great things about Linux and why upstream development is good, and of course, we don't get any of that on Android. And that's because before it reaches you, the Linux kernel on your Android phone goes

through three major steps. First, Google has to take the long-term support version of the kernel, whichever one it's going to use for that version of Android, and then add all those Android-specific changes. This is called the Android Common Kernel. Then this code goes to the company that creates the system on a chip that actually runs your phone, which is probably Qualcomm. And once the system on a chip maker finishes adding all that code to support the CPU and whatever other chips are on there,

the kernel is then passed on to the actual device maker, such as Samsung or Motorola, and then they add code to support the rest of the phone, like the display and the camera. Yeah, or maybe some new fancy feature that is their differentiator. Each of these steps takes a while to complete. That's how software and hardware development works. And it means that the end product, that kernel,

doesn't work with any other device. It also means that the end product, that kernel, doesn't work with any other device. It also means that the kernel's very old, usually about two years old on an Android device when it ships. The Pixel 4, which shipped last month, has a kernel from November 2017. Ouch. I don't even want to know what kernel my Pixel 3 is running. I know. Now, Google did announce plans last year to fix this mess. Yeah, yeah, that's right.

I know. Now, Google did announce plans last year to fix this mess. Yeah, yeah, that's right. They said, we know what it takes to run Android, but not necessarily on any given hardware. So our goal is to basically find all of that out, then upstream it, and try to be as close to mainline as possible. Kind of sounds like maybe we won't get fully mainline even in Google's vision. That might be difficult with Android's long history. It's a good goal, though. Yeah, they did show it off on a prototype up on stage.

I don't think it was perfect, though. No, no, I don't think so. Does seem like some things were not working, like the battery percentage, which was stuck at zero. That sounds like the Librem. So what's Google's plan to make all this work? Well, they plan to take a page from their Project Trouble playbook. Before Project Trouble, the low-level code that interacted with the device and Android itself was just a big old stack of mess. What Project Trouble did was it separated the two and made them modular. So Android updates could be shipped quicker and the low-level code could remain unchanged between these updates.

What Google wants to do is bring that same modularity to the kernel. Their plan, quote, involves stabilizing Linux's in-kernel ABI and having stable interfaces for the Linux kernel and hardware vendors to write to. Google wants to decouple the Linux kernel from its hardware support, end quote. So this means that Google would ship a kernel and then hardware drivers would just be loaded as regular old kernel modules. Now of course this is just a proposal and there are still quite a few technical problems that have yet to be solved.

I mean that does sound like a big change, right? That's a different way to approach things although I can see why some decoupling would be nice given the slower cycles outside the kernel development. Yes, I don't actually think it's necessarily a bad idea. It's just such a radical change to serve one group's needs. It is a large group, but it is a radical change. I think the Android problem is not a unique one.

There's probably thousands of hardware vendors at smaller scales in Google's shoes. They make a product that's an IoT device or something like that. It doesn't have to be Android. It could just be based on Linux. But it takes a long time for a group of people to build hardware and software. And so it's easy for a couple of years to go by, even if you're not a phone maker. Like phone makers probably have it down at this point. They probably have it about as efficient as they can

because profitability is driving that. But Fred's IoT shop that's making router boards and stuff that get shipped in 100 different Chinese types of knockoffs, like, you know, they don't have that kind of efficiency of scale. It's probably a much wider spread problem. So there could be some real logic in doing this. I'll be curious to see what the upstream kernel community thinks of these proposals. Yeah, it doesn't seem like it's got initially a very good reaction.

Right. And I mean, I wonder how much of it, do you lose some of those advantages, right? The things that the kernel community is always talking about, about getting, you know, proper upstream, traditional upstream support. Right. How much of that will actually apply here? Yeah, they really want to see people just make it open source, make it GPL, and then it's just included. We'll all maintain it together. And that's a pretty deep philosophical outlook, which is why it's been really fascinating to watch WireGuard's slow march into mainstreaming.

WireGuard is the VPN technology we traditionally talk about all the time on this show because we've been following it very closely. It's been held up by the adoption of its main crypto, which we thought may slide. We thought perhaps they were going to swap out the clever crypto that WireGuard uses for something already in the kernel. Well, this week, Linux.55 seems to support key elements of the Zinc crypto effort, which WireGuard depends on, which I thought wasn't going to happen. I think what's happening here is, you know, when WireGuard was sort of released, the crypto stuff kind of came fully formed. And the kernel community doesn't always love, oh, here's this giant chunk of code. Will you apply this patch, please?

So I think what needed to happen was the kernel community and working with the WireGuard community and sort of working out which pieces made the most sense and in what order are we going to start making some of these changes. Thankfully, what's landing in Linux 5.5 is enough to unblock the dependencies in WireGuard. Now, with that said, WireGuard

is not landing in 5.5, but should, emphasis on should, land in 5.6. Yeah, I'll believe it when I see it at this point. It's just like been crazy because it's been every couple of releases, we're like, well, the next one from this is going to have WireGuard.

I mean, the other good part here too is that we're just getting better crypto in the kernel too, right? I mean, good point. Zinc looks like it's super good. But I think the thing that you'd mentioned is really the key underlying here.

It's like just enough to unblock WireGuard. So we're not getting a complete implementation. Right. And there'll probably be continued discussions on more code changes

in the future. I do love it though. I think I could easily do an episode if we wanted on the open source projects that I'm loving this year because the list

is different than it was last year. And... There's just so much good stuff. WireGuard is definitely on that list. Game changer. There's just some tools that they're great, fundamental building blocks, and that means the things you can do with them

are almost endless. It's funny we're talking about 5.5. It's probably worth mentioning that 5.4 just came out. Right. So it's actually just shipped as been kernel 5.4. And one notable thing that we've mentioned in the past is it's coming with that lockdown feature as well as early extended FAT support. There's more details in Linux Headlines. If you're not listening to Linux Headlines, that's where we put that kind of stuff.

Linuxheadlines.show. I think that was the Monday episode because I think I did that one. That's right. And I think I covered it in Monday. So the details are in the Monday episode of Linux Headlines. But it's everything you need to know in Linux and the world of open source in less than three minutes. Three minutes or less? It's so great.

It's so great. Well, guess what, Mr. Payne? It's time for a little housekeeping. And it's about time. Last week, we talked about the System76 superfan event. Well, cheesy. You got updates on that and some Pinebook updates, too. Yeah, absolutely.

So, you know, we didn't get a lot of time to cover the Superfan event because we were pretty packed that show. But I've done a write-up there. And if you go to LinuxUnplugged.com and hit the blog tab, you can check out that write-up. A lot of cool pictures and just really a great synopsis of the event itself. It was wonderful. It's really interesting to be around Linux nerds that are actually producing hardware. So, you know, it was just an experience

that you'll have to go read for yourself. And I've also updated the jupiter.gallery page with some photos from that event as well and also included a Pinebook Pro review so if you caught last week's show and you want a little bit more information I suggest you hit the linuxunplugged.com blog

and check out those posts as well. I love it man what's neat about that gallery first of all I gotta say these posts you did a great job so thank you for writing these up but what I love about that gallery, first of all, I got to say these posts, you did a great job. So thank you for writing these up. But what I love about that gallery is you get to see some insides of the System76 warehouse. You got some in this post too, like, I love this picture of Carl. He should make that his Twitter profile.

That's amazing. That's really good. So you can find those at linuxunplugged.com slash articles. This is something we try to do. You know, we talk about this stuff in the show. com slash articles. This is something we try to do. You know, we talk about this stuff in the show, but we've recently, within like the last nine months or so, started doing these write-ups and posting these pictures. So that way you actually have a little more too than what we're just saying here. You can go see what we're talking about, get a better sense of the vibe.

And Cheesy has been doing a great job of updating the Jupiter gallery, Jupiter.gallery. So go check that stuff out. Also, did you mention your ThinkPads with Popey episode? Oh, dude, that was such a fun episode with Popey. Anybody that's looking at, like, a used ThinkPad and wants to kind of know

what might be the right one for them, but also what you can replace and fix after you bought the ThinkPad, that stuff blew my mind. I had no idea. No idea. No idea. So that was a great episode.

That's over at Extras. And while I'm talking about Extras, I'll mention it because we're on a roll right now. Extras is on YouTube. All of them are now posted on YouTube. If you're a YouTube connoisseur, I'll have a link to the channel. We don't have any subscribers yet because we just turned this thing on.

So we don't have a name. Go subscribe and you'll get little notifications when we get new stuff in there. How many subs do we got to get to name it? A hundred maybe? Maybe it's a thousand. Might be a thousand. Either way, we need your help. Yeah, so you'll find the link at linuxunplugged.com slash 329. Go subscribe so that way we can give it a name.

And we'll be posting all of the extras on YouTube as well as extras.show. It's all over there. Including that Popey on Thinkpads episode. That extra stuff is so good. I should also mention Brent isn't here today, but he did message me

earlier and said, a new Brunch with Brent is out with Jacob, and it's fantastic. He talks about how TechSnap specifically got him into the industry. It sounds like I've not had a chance to listen. Yeah. Because it just came out this morning. Jacob's a good guy, though.

I've had breakfast with Jacob. It's good. It's good. Brunch with Brenton. Breakfast with Jacob. All right. That's the housekeeping. Now, let's get into these net-busting VPNs.

So we're not talking about WireGuard this week. This week, we want to create a mesh network of sense or an overlay network. And the granddaddy from the 90s in this space is Tink. So how does Tink differentiate from WireGuard, Wes? Tink is quite different from WireGuard. WireGuard is very simple, which is great, right? That's one of the things we love about WireGuard. But when you're making a WireGuard connection, you're really just making, you know, one point-to-point link. You've got the machine you're connecting to and the machine you're connecting from.

Tink can do a whole lot more. Not only does it also have support for things like Layer 2, but it's a mesh network. So it has a concept of understanding all the different nodes you've got on the network. It manages that. The best routes to those nodes. And it can figure out the best routes to traverse that network.

So as long as two nodes that you've got on your Tink network can talk through somehow, they will find a way. So you can set up a public relay on a VPS, and you can have, like my systems, which are behind a massive carrier-grade NAT. It is punishingly awful. But Tink can accommodate that.

Yes, very much so. As long as one node's public? Yeah, you just need enough information, and usually having some sort of public node not behind NAT. That will make things a lot easier, but it's got a lot of sophisticated NAT busting integrated right inside of it. So we sat down with one of the original creators

of Tink, and we asked how did you get the idea for Tink to begin with? It started out when I was... I found this new feature in one of the very early linux kernels so it was called ether tap it was the predecessor to what's now called done and yeah i wrote a little program to experiment with it and then i yeah naturally i created some kind of tool to create a network connection between two computers and tunnel the packets back and forth.

And yeah, so that was already a working kind of VPN, except it didn't do encryption or it was very hard to actually set it up. I had to do a lot of things manually. And then a friend, Ivo Timmermans, he joined in and he actually took this a step further and he made a proper daemon out of it, introduced encryption. And then, yeah, also with some other people who joined in. But I think Ivo and me were the people who coded most in the beginning. So how long ago was this roughly? So we started in 1997. Wow, that was a whole other internet.

That is literally another internet ago. Yes. What's that like? What's it like, a project like this? I mean, there must have been moments of security panic. There must have been moments of trying to decide if it's worth modernizing as the kernel changes. I mean, there must have been all kinds of things to keep up with over that time period. Yes.

Well, you mentioned a few things that indeed happened. It started out as a kind of a hobby project. So we were all students back then. And we actually wanted to have our own private network so we could exchange files easily without having to, well, without being that visible to the rest of the network. Of course. And one thing was that, of course,

we were not the first one to invent a tool like this. There were already commercial solutions, or I think we had VTUN and SIPE in the Linux world that did something similar. Was the beauty, though, that it really sort of created a truly flat mesh network? I mean, you could have friends all over the world, essentially, that are in one flat network.

Exactly. So, for example, Evo was not in the same city as we were. So this was a good way to make sure that we could access his computer and run things like IRC and FTP. I like the idea you can use it for development resources of the project you're developing. Absolutely, yeah. So I noticed that the project for a while has been nearing 1.1.

So it doesn't seem like the project's moving at 100 miles per hour, but just as of a couple of months ago, there was an update on the news list. So things are still continuing. So development continues. How would you describe the pace and future velocity of that? stable and I'm not planning to add any features to that. There are still users who sometimes find bugs or say, well, okay, if this little change, it will be more useful for us. So then we try to implement that. And then of course, yeah, there's 1.1, which is a big improvement over 1.0. I just need to find time to finally finish this. Of course, that's always hard. I thought several years ago that I would have the time to do this, but unfortunately, I'm not a student anymore. So I have a proper job and a life. Real work comes first, yeah. What sort of features might we be looking forward to when you do find that time?

I think already a lot of the features that I want to have in Think 1.1 are there. But there are some rough edges that I want to have smoothed out.1 are there. But there are some rough edges that I want to have smoothed out. One of them is, for example, key management. So already it's a big improvement over 1.0. It's much easier to create keys, exchange keys with others, also in a secure way.

Oh, that's great. But one of the issues is still, I would say, handling key renewal. If, for example, your public key is compromised somehow, you want to update it. You want to make sure that all the other nodes in the network get that new key in a safe way. Yeah, absolutely. Also, if you don't trust someone anymore, at the moment, it's kind of a manual process to block this person or this node on all the other nodes. And it would also be great if that would be done in a much more automated way.

But that requires some changes. So there is no infrastructure at the moment to spread that information throughout the network. Right. That sounds like a lot of coordinating to coordinate. Yes. I'm curious what you might see on the larger side of a Tink network, about how many nodes have you seen out there? So myself, I've seen networks of over 500 nodes. One of the biggest ones that is kind of public is the Chaos VPN network.

So it's kind of the European vpn network so it's the kind of the european hacker spaces they are united using this network and there's actually a counterpart in the us dn42 where they use various technologies including also tink that's quite a large network but i've heard from other people that they have similar sized networks. And there are also businesses, companies that use Tink internally to connect offices together. And it's very hard to get information of exactly how many nodes they have and so on. Yeah, I would imagine people are in part doing this for security, so they're not keen to share all their details. No. But I'm just always fascinated because I can see it working well for myself with a dozen nodes. But I was just curious since I probably won't ever take it that far.

I was just kind of interested to see how far it could go. So it sounds like 1.1 is still out in the works, but not necessarily a timetable. But as far as features go, really everything that's in the release now is pretty solid. It's there. It's fairly feature complete. As we noted, Tink's been around for a while. I'm curious if there's any features about Tink, things you love that stand out as reasons to use Tink. I mean, there are more competitors these days, but Tink is still great software. Well, one thing that I think makes Tink stand out is it doesn't impose any kind of restrictions on the network topology you use. If you try to configure Tink, you will notice that there is

nothing in a configuration file that basically says, use this IP address or this subnet. Basically, it says, well, yeah, you can create a script yourself, which Tink will call for you. But in a script, you have to configure actually the virtual network interface that's created by Tink. Right, so you have all the control. You can modify that, make it configure whatever you need. Yeah.

That is great. Right, and it also strikes me as similar, you know, Tink supports Layer 2, which a lot of the newer VPN solutions just sort of skip, but that can make for very flexible implementations. Yeah. It seems particularly attractive for someone like myself who's behind a NAT, and it's a particularly challenging NAT. I've got a VPS.

I can throw Tink on that as well, and Tink can sort of navigate that for me and solve that problem, and that's what makes it really special for me. Well, Hus, thank you so much for joining us and just kind of giving us some background details on it. I find this stuff fascinating. And your project is one of those that's been around for such a long time. It's sort of an unsung hero to a lot of people where they find and it solves that perfect problem. But it's not a project that gets a ton of attention either. Really appreciate you taking a moment so we could shine a little light on it. Well, thank you for having me. So let's kind of define more

specifically how we're using the term overlay network and what we're using it for. I'll give you an example, a day-to-day example of myself, Drew, and Wes. We all have home studios and we'll sometimes need to connect into resources here for production purposes, or sometimes Chris forgets to export an episode of Linux Action News and somebody needs to remote in. Never happens. Never happens. And we all need to connect back to JP1.

I would like to connect to my home network as well, regardless of where I'm at from my laptop, and be able to connect to the JP1 infrastructure. And so I needed a solution that could kind of create an overlay network for all of this, one LAN. Right. There's sort of, it's a magical feeling when you have a flat network because everything's accessible. It just feels right next to you, right? And right at your fingertips. Yeah. Yeah. Things that like do auto discovery,

like old video games, like StarCraft, you can, you can all see each other. It's, it's a really unique experience because you can be on a Raspberry Pi, on a MiFi connection in an RV, and be pinging a box in the studio like it's just a local machine. Right, and you can simplify some things and work across regions. Maybe you've got different providers or different data centers.

You can make things flat that way, too. Unfortunately, I've used Tink, and we started evaluating Tink for our use case here, and we ran into some problems because really, we've used Tink, and we started evaluating Tink for our use case here, and we ran into some problems because, really, we've got a few different security domains. I don't need access to your home network. And really, there's some parts of the studio that we only need access to a few machines here at the studio. Tink doesn't really have any filtering capabilities.

There's nothing built in to help you manage multiple different sets of users. And when you want to add a new host, well, if I want to talk to a new machine that you've added to the studio, you got to go give me a new file that's got all the information about that. And that works okay for a few machines, but if Drew's adding a new machine

that he wants to share with us, it gets tedious fast. Wes and I were pretty excited when we saw Slack announce Nebula, an open source global overlay network. And we invited Ryan on to tell us about it. Ryan, this seems like something that's, A, I mean, it's pretty notable that it's a pretty big open source project from Slack. But B, it seems like it's pretty well developed. Like it's a complete ready to go solution.

Yeah, that's right. You know, it's, it's funny to have Nebula out in the wild and people seeing it for the first time because it's actually old news inside of Slack at this point. It's, it's three years old. Oh, wow. Okay. Yeah. So you've been using it in production for a while then? It's been in production for about two of those years. All right. So an overlay network, it seems like you could have accomplished this possibly using IPSec or maybe even WireGuard with some really sophisticated routing. I'm guessing that was tried and done and didn't work. Where did that kind of fall down for you guys? So our original network and are you going to provide a link to the blog post in the in the show notes possibly absolutely uh your medium post and uh also the official announcement too on the you bet

yeah we oh great we've been reading it voraciously great yeah so um so we actually did ipsec originally and we ran into uh we were so at first it was fine right we as you expand out you have one region in a cloud provider and everything's fine. And then you expand to another region and IPsec was kind of the obvious choice for connecting multiple regions. It was the most supported way. You know, there are now in the past year or so, they've added support for connecting remote regions that don't require VPN. But we can get into that in a minute. So we, you know, we used IPsec out of the gate, and it actually worked fine

in the beginning. But as we scaled up, the complexity grew. And I jokingly called our IPsec setup, our RAID zero network configuration, because, Because the problem was you had to do these kind of strange routing tricks because no one box could handle the traffic. And because of that, if you lost any of the boxes on one side, even one of them, you lost routing to an unknown number of nodes. And so, you know, it wasn't really viable to keep running that remote region that way. So we would have to offline an entire region, hence RAID 0 network configuration. No kidding. That was one of the major challenges. And so we were already many

thousands of servers at that point. And there were also folks that dissuaded us from using IPsec at scales that we were, you know, planning to hit. And I can't speak to whether they were correct or not. But, you know, that was a caution that even some folks that work with IPSec regularly, they said, in transport mode, which is sort of the node to node version of IPSec, they said, you might start to see performance issues when you're above 1000 nodes. And again, I, this is secondhand. So I'm not saying that we saw that. I'm just saying that was, you know, that was kind of a warning that I was given. And so when we thought about the problem also, we, we looked at, you

know, kind of traditional VPNs. And one of the things that we didn't care for was having to tell every node about every other node ahead of time. Right. That's a lot of overhead. You have to manage it somewhere, configuration management, or it's got to be somewhere. Yeah. And it's not just that. But think about when you get to many thousands of nodes, say you're running a dynamic environment where, you know, hosts come up and auto scale and that kind of thing. know, hosts come up and auto scale and that kind of thing, a host comes up and you have to ensure with some amount of, with some measure of certainty that, that every other host it might talk to has the proper key before it starts talking to that host. Right. And that becomes a complex

to manage problem when you're talking thousands of hosts. And so we went a different way with Nebula and we, did certificate-based authentication between nodes and another part that I I know to I mean certificate authentication is notable but the other thing that I noted about this that seems to stand out is groups play a large role in sort of the access control of this that was one of the the events that was really the moment we decided to write Nebula. And it's, so the primary authors are myself and Nate Brown, who also works at Slack and has worked there for four and a half years. By the way, a bit of history, I've worked there five years and was the first security employee. Nate has worked there four and a half years and was the second security employee.

So we've- That is fantastic. Yeah. So we've been there quite a while and we've actually done some really fun work outside of this. But so we were we were actually just chatting and we were actually I know you mentioned Tink before the show. We were talking about that briefly. I really like Tink. Tink's this great project from the 90s. And I think the guy's name is Gus that's been working on it. And he it's got some really, really cool routing tricks and, and, you know, sort of gets around NAT and that kind of thing. And so we actually very seriously looked at what it would take to use Tink or maybe contribute back to Tink and go that route. But what really led us to writing Nebula was this idea that we wanted to encapsulate security groups, as you mentioned. Hmm. This, at, I struggled to quite understand this, but when I started looking at the example

configuration files, and I saw an example in there where it was broken down by servers, laptops, it started to click a little bit more that you could start to manage a much larger set of machines by grouping them up. Yeah, maybe you could clue us in on how you guys are using groups. Yeah, so, you know, the kind of contrived example, I mean, we actually tie groups to chef roles. So if you're familiar with, yeah, so with config management, because the chef role really describes what, you know,

what a host or what a container's job is, then that kind of makes a natural grouping. And we also add in some special stuff like which region it's in, which availability zone it's in, that kind of thing, just so that we can do even more fine-grained. And we opportunistically add groups we might use, and even if we don't actually use them for filtering rules, but we add as much kind of metadata as we can about hosts.

And so, you know, the contrived example I give is, if you're a database server, and say, you know, in a very simple setup, you just allow web servers to talk to that database, then the only rule on your database servers is, you know, assume MySQL, the only rule on the database servers would be, I allow TCP 3306 from anything that's in the group web servers. And that's powerful because you no longer have to care about IP addresses or host names or anything. And this isn't new, right? This is something that, you know, cloud security groups have existed for a long time. The difference is Nebula security groups are, well, they're cross platform, which is pretty cool, but they're also cross-provider and cross-region right right that is a big so this i mean we're talking about as an overlaid network and you can really use this to stitch everything together i mean is this a lot of traffic then going through nebula yeah i i don't have the numbers in front of me but it's many billions of packets per second it's it's really neat to see yeah it's it's uh it's many billions of packets per second. It's really neat to see. Yeah, it's in heavy use.

I can only imagine what it must have been like to evangelize that inside a company and then roll it out and then have to work out the kinks. That must have been quite an experience. Yeah, there were some fun times, especially in the early days. One of the really great things about this project in general, and kind of following on some of the other projects I've worked on and Nate's worked on, is this is something we also just wanted for ourselves, right? So there's that selfish aspect, which is we want this to be great for Slack. We also want to use this for our home networks and everything else. Like, we really wrote it for ourselves and for Slack at the same time. So tell me the thoughts behind going open source. Yeah, there are a lot of good reasons to do it. I think the main one is, you know,

everyone says the give back to the community. And I very much agree. Like every company is built on so much open source software that if you've done something that can save someone else time and it doesn't, you know, degrade your security posture, there's not a, a business reason. I think it's really nice to open source software in general. An open source by default sort of thing. Yeah. And, and I think we've, we've been pretty good about that. Anytime we've had something that's sort of generally useful, we've, we've kind of put it up there, but, but the other reason is getting more eyes on it and, and just having more people use it. You know, the, we, so one of the things that I lobbied for internally was to add Nebula to our official Slack bug bounty program. And so, so that, you know, that means that when we put this

out there, if people find security related flaws in Ne in Nebula, we'll actually pay them for those through the Slack bug bounty process. That is great. Yeah, that was a big one for me because, again, this is a pretty key part of our infrastructure. And on top of it, you know, some amount of trust from a community when people are using it. from a community when people are using it. And so, you know, we wanted to make sure that we weren't just throwing this over the wall and then saying, good luck. I love that.

I think we're seriously considering using it for remote studio productions to connect back to the main studio. And looking through the configuration, there's a couple of things that I really love. Like the beacon service or whatever you want to call it is named Lighthouse. So we can put that on an external system. That's great. That's a couple of things that I really love, like the beacon service or whatever you want to call it is named Lighthouse. So we can put that on an external system. That's great. That's a great name. The server, I don't know what you call it, but the function to make sure that

the NAT mapping remains is called Punchy. It keeps punching a hole. That's so cute. My friend and colleague Nate hates the name Punchy, so I'm going to make sure and play this section, but I called it that. It made me smile. Yeah. And there's some other stuff that over time we really learned. I'm not speaking ill of GoGo, but one of the best debugging steps we had was whenever I would be on a flight using this. The latency over that really taught us a lot about some of the handshake issues we might run into in extreme cases. So that was pretty neat.

The core of Nebula has existed and been in production for two years. But we've added features, you know, sort of as we worked on it. And you mentioned Punchy. One of the features that we added not long ago, probably six or seven months ago, I don't have it in front of me, was something called Punch Back. And I don't know if you saw that in the config as well. I think I did, yeah. Yeah, so sometimes you're behind a really difficult NAT, and like a symmetric NAT, I won't go into the details of why, but a symmetric NAT can be extremely difficult. Oh, I live that life right now, actually. I have one of my systems.

Are you on mobile? Yes. Yeah. So you on mobile? Yes. Yeah. So you would really enjoy Punchback. And I actually did that because I use a hotspot quite a bit of the time as well. I really enjoy road trips. And so I have like an AT&T hotspot. So do I.

Kindred spirits. Oh, you're speaking my language. And so what Punchback does is when a host wants to connect to another host, it asks the lighthouse how to reach it. The lighthouse gives back an IP address and a UDP port to try. And when I query the lighthouse from a node, it also signals the node I'm about to connect to to punch outbound to me. Right? So as I query the lighthouse and ask about a node, the lighthouse is signaling that node

that I'm about to try and send a packet to it, and it should punch out. And by the way, a really important thing here is the lighthouses actually don't transit any of these packets. They are purely for querying and finding nodes. The handshake never goes through a lighthouse. Okay. And so the lighthouse signals that node to punch out and then we start sending packets and we try all the different ips that

that we're we know about or that we've learned about and the thing that punchback does is it says okay the lighthouse signaled me that somebody's trying to reach me i'm gonna give it five seconds and if they don't reach me i'm to try and reach them as though I had initiated the connection. And that as long as there aren't two boxes behind really terrible symmetric gnats, that that works almost almost every time. Oh, my gosh. Well, I think that just that just sealed the deal. I mean, it's clear that you've had to fight through some of these things and worked out a lot of those kinks. Yeah. Oh, man. That's so fantastic.

Ryan, this is one of those moments, you know, you always hear, it's almost cliche, like the thing about open source and free software is everybody's scratching their own itch. It sounds like in this particular case, you had exactly the same itch. And I just am so thrilled. I think it's great. you had exactly the same itch. And I just am so thrilled. I think it's great. I will link to the announcement and we're going to play with it. We're absolutely going to try it out.

Is there anything we need to know? I mean, we've talked about a lot of good things here, but is there anything as potential adopters we should be aware of? Depending on the platform you run, there's some stuff

that hasn't been merged yet that's going to make that easier, especially if you use any Macs or Windows devices. And so one of the difficulties in Windows that when I was sort of developing some of this with a friend. By the way, there's a gentleman that we work with named Wade who did every port. So we wrote this for Linux and he ported it. And Go makes porting just so nice, but he ported it to Windows and Mac.

And even in one single hack day, Wade is the person that got it running on iOS. So, yeah, it was really impressive stuff. And so, but one of the difficulties with Windows, you know, I haven't run Windows as a primary OS in many years, so I'm kind of out of date on it. But one of the issues was installing services and how services need to behave is kind of – it's very foreign if you're used to Unix-style services. And so there's a really nice Go library out there that can make a binary service aware.

And so I integrated that into Nebula and made a version of Nebula that's a self-installing service. And so in that case, you can actually copy nebula.exe and one config file to a Windows host, and then you can run nebula-service install. And it actually does the whole bit and installs it as a Windows service. And that library is actually cross-platform. It's really neat. So it even does that on Linux or OS X, whatever you're using. We have a bit of a pedants argument about doing that on Linux. So I don't think that'll be the default build on Linux because, you know, just use system D or upstart is kind of the,

the right way to go. But, but for the other OSs, you know, OS 10 or windows, I think that's going to be the default behavior very soon. And, and there's a branch that already works. That's one thing I, I mean, you can't miss about Nebula is that simplicity. It's, it just seems ready to integrate into all kinds of other systems, however you might do it. Yeah, we were brainstorming a couple of ideas already. Ryan, I hope you'll keep us in mind in the future if there's any developments you want to share with the community.

We'd love to pass it along. I think you can count us officially as fans of the project. Well, that's great to hear. I know people probably say this and it sounds cliche, but I was really surprised how many people were interested in this. Because before we put it out there, you know, we knew we were super into this, but you always think your thing is really neat and you hope other people do. But the response has been really great and we're really happy about that and we want to keep that going. So, we're going to do some things to make it even easier to stand up Nebula, especially for folks that maybe don't want to stand up a lighthouse and do all of that just to test it out.

So we're going to make that a bit easier going forward. And I can't wait to see what use cases people come up with. Very good. Well, thank you for joining us. Thank you. Something we didn't touch on, but Nebula has also undergone a paid security vulnerability assessment along with numerous internal Slack reviews as well. So it's looking like it's in pretty good shape. Yeah, they're certainly using it.

And we did give it a go. In fact, we have been really having quite a bit of fun playing around with Nebula. It's been one of those projects that just is like you feel like it's cutting edge, but it's doing something that solves such a cool problem. It's great. And it's simple. It's easy to get started with. I mean, it's Go. So you just go download the two binaries that you need.

And the configuration files are easy. It's just YAML. And because of the way it works, you don't need a whole bunch of configuration files. You basically just need the key pair for your machine. You need the CA certificate and then the config file for the daemon, and you're done. Yeah, it's a lot of kind of like conceptual stuff. You have to shift your head a little bit because it's all key-based,

so you got to get used to that, but that's not a big deal. It's not necessarily like a service. It's not going to be a package you install. It's a tarball. You download and extract, and you run it. But if you're okay with and you're comfortable with those kinds of things, with very minimal effort, you can create an overlay network that is secured. And it's like browsing a local LAN.

For fun, we went on two different MiFis. Wes did it from his place. I did it from the RV. I did it from the RV. We have systems at the studio. And they can just all immediately talk to each other. And it'll be smart about things, so it'll figure it out. You know, if I'm from my house, I can talk to one of the studio machines. But if I come into the studio, they figure out that that traffic just stays on the local LAN.

So we even got it going on my Raspberry Pis, to an extent. My Raspberry Pi 4s, I think, did we figure out they were in 64-bit mode? I can't remember now. But there's a 32-bit and a 64-bit version that will run on the Raspberry Pi. And we got it working really solid for a bit on one of my Raspberry Pis. But when we set it up on a second Raspberry Pi, we ran into some strange issues where it really kind of came into our limits of the ability to troubleshoot. And because this is new, there isn't a huge community around it to solve it. Yeah. And there are, you know, there are some downsides. It is

very simple. It just means that there's, you know, it only operates in the one mode. And right now, the NAT punching doesn't seem quite to be working. There are some extenuating circumstances, because as you said, you have two layers of net, which makes just about everything difficult. I have this fantastic, best ISP I've ever had in my life, local ISP. They're like one of the last in Washington. A little small outfit. Yeah. Like if you call them and they don't answer, they see it on their caller ID and they call you back. Hello, Mr. Fisher.

I saw I missed your call. Oh, yes. I see. I could lower your bill by $10. I'll get that taken care of for you, sir. Thank you for calling. Seriously. Like, yes, I see I could lower your bill by $10. I'll get that taken care of for you, sir. Thank you for calling. Seriously. Like, yeah, no problem.

Okay. Thanks. So they're a great ISP, except for you don't get it. I get like a 192.168.5.whatever. So I get some weird public IP address that is obviously NAT. And I don't know, I don't even know how far down the NAT chain I really am. But then, of course, I have a NAT of my own network. So I have a gateway device that's connected to that NAT,

which then does NAT behind it. And so, as you heard in the chat there, it is designed, Nebula, to punch through this. And it does to an extent. We can get one Raspberry Pi working for a bit. It can't talk to all the nodes, though. Yeah, we're having a few. We haven't got it quite working.

That said, we did just pick this up yesterday, and on 80% of the machines, it just worked. The ones that aren't behind super crazy NAT, even mild NAT, like MiFi NAT. Tether into my phone, no problem, can jump into the studio. It's really neat to just see it work, and Wes was

able to bang it out into a systemd unit file in like 30 seconds, so now it's a service that just runs on our boxes yeah no problem super simple so I have a question for Wes I use a tool called SSH shuttle to tunnel my DNS around from the US to Europe and stuff can you send with either Nebula or Tink, can you route DNS through those tunnels easily? Or where does the DNS actually happen? I mean, I don't see why not. You just need an endpoint at the other side. Yeah, so they both have provisions for it.

Specifically to Nebula, there is a configuration option to declare where the DNS is or even to act as a DNS forwarder. And so we have experimented a little bit with that, but you do kind of need a name server on either end to do it. You have to do that part. It kind of sounds hard if you don't try,

but once you try it, it really turns out if you try hard enough, you can. It's really not so bad. No, no, it's not. And it's just nice. You know, it's neat to have, we live in a world where you share things in the cloud, right? And everything is an HTTP service or HTTPS really these days.

Having a flat network sort of harkens back to the good old Unix days, right? So now we could have a Samba server running here at the studio, but if it's listening on that overlay network, you can get those files from anywhere. But especially because of the security group stuff built right into Nebula, which I love, you can do that securely and not have to worry about

exposing someone to the public internet, which you just shouldn't do. Yeah, listeners of our self-hosted program know that I have a mandate to never expose the RV to the public internet. So something like this, that's why this double net situation, I take it as a challenge because I really have a mandate to not publicly expose anything.

That's a good rule for life, Chris. I agree. And what I love about both Nebula and Tink, but specifically Nebula and WireGuard, they're really good about reconnecting once your connection comes back online, which is really useful for me for both laptop and the RV. Nebula also is, you know, it's not that old. They've been using it internally for a little bit, and it's certainly new to us. But you can tell it's a modern tool. I mean, not only is it

written in Go, but it's got great logging support built right in and metrics. So if you've got a Graphite server or a Prometheus server, that's built right in. Yeah, there's some, I could see Slack building some services around this thing in the future. I could absolutely see them productizing this if they wanted to. I don't know if they will. What Ryan told us is essentially it's enough for them already because they have a situation where they have systems on AWS, they have systems on Azure. I'm assuming, he didn't tell me this, but they have multi-provider systems and they

want a LAN where they can have secure MySQL things happen or secure Samba things happen. Right, where you can sort of abstract that away and not your individual hosts don't have to deal with that anymore. They can just receive traffic. I think that cross-provider thing is going to be what makes this really take off. If they keep talking about it like that, that's going to click with people. All this talk about hybrid cloud, I mean, one of the

providers could be your own LAN. Absolutely. I have a system on Linode, I have a system on DigitalOcean, and we have the Reaper box here. I would like to connect all three of them together. Both of these allow for that. And they can just be on one private network. Flat network. I haven't done extensive testing, but I did a little bit

last night, just being at my house in the studio, and performance is looking good. That was one of my concerns with Tink. It's older and not necessarily optimized for performance, but this is using new quick crypto and it did great. Yeah, I guess it's worth mentioning, neither one of them run in the kernel. Yeah, just using

the ton driver. Yeah. Which is perfectly fine. I think there might be probably a slightly higher performance story with Nebula than Tink. Yeah, I believe so. So that's something to consider when you have a lot of nodes, right? Because who's talked about 500 nodes

and Ryan talked about thousands of nodes. So that's something to consider too. Yeah, and because of the key-based and the certificate-based authentication, you don't have to do anything to add a new node. I mean, the certificate authority needs to sign the new keys for that node, but that's it because the lighthouse actually takes care of communicating those details. So if I want to go talk to Chris's laptop, I don't know where that is. And I might, you know, he might not trust that I should talk to them. All I have to do is

go share my information. And because it's signed with the certificate authority, it's all good. That's it. So if you have any questions, go to linuxunplugged.com slash contact because we're still tooling away with this. We're having a lot of fun building networks, and I think we're going to get my NAT situation resolved. I don't know exactly how, but I think we're going to get there. We'll figure it out. Right, Wes? Well, maybe we'll have to ask Ryan.

I think we might. Okay, let's do some feedback. Before we go, I said I want to do some more feedback and follow-up in this here show, and people took me up on it. LinuxUnplugged.com slash contact. Our first one, Matthias, who says he's a happy Manjaro user in Germany. He had to mention that, didn't he? Of course.

No, I love it. I actually love to know. He has a question about command line autocomplete. He says, short question, a while ago you guys mentioned a cool little tool for the command line that helps me remember my complicated commands and makes suggestions while I start typing, but I forgot the name of it. I think

he might be talking about Fischel. We do talk about Fischel basically all the time, and it's fantastic. Fischel is now the thing that I install almost immediately on a new Linux box, and I go into the Etsy password file, and at that right there, I replace my shell with fish. Now, another one we've, I think, mentioned before is FZF, the little command line tool, the fuzzy finder, and that also will search your command line history.

I was thinking it could be that one, too. So that's a good one. Do you want to toss a link? Absolutely. In the news. All right. Liam writes that one, too. So that's a good one. Do you want to toss a link? Absolutely. In the news. All right. Liam writes, hello, Chris. I've also been experiencing random lag or skipping on my workstation with an NVIDIA GTX 1060 after I upgraded to Fedora 31.

I was wondering if you ever got anywhere with the mouse lag or if it's an active bug somewhere I could contribute to. Oh, I like what you ended up doing. This is a good sort of come clean moment. I hadn't actually thought about. I was like, I'll do an update. Doop-a-doop-a-doop. Well, okay. So, you know, I really can't stand leg.

This is a particularly weak spot for me. So what I did was I burned the entire house down. I decided I wanted to switch to Plasma for a bit to see if there was an issue over there. So then I went down the rabbit hole of trying to find a great Plasma implementation. Really, I actually downloaded the Neon ISO. I was going to go just with Neon again because I had such a great experience last time. I'm imagining you standing in front of your computer, USB drive in hand, sort of waffling, considering, just about to plug it in. That is literally the situation. I'm not kidding. And I came across and I said,

self, you know, you don't want to always you need to challenge your assumptions about distro self, because I had a real kind of realization when we put Fedora on their server. It went so good, Wes. I was like, this is way better than I expected.

I came with all these assumptions about Fedora on the server, and I was schooled. So I thought, I need to give Manjaro a real go again. Because my early conceptions of Manjaro are not really, Manjaro's come so far. It really has. And it's really one of the great distros. And I thought, I got to try out one of the greats.

So I pulled down their Plasma edition of Manjaro, and I formatted my workstation yet again, and I put Manjaro Plasma Edition on there. I didn't think I'd ever see you remove Fedora. I really am not trying to be a distro hopper guy. I just want to get my work done, but somehow I have fallen into this trap of working on my tools again. Yeah, maybe that's the thing I should be doing at the end of the year,

get myself positioned for 2020. Yeah, maybe that's what I'll tell myself. Nice the end of the year. Get myself positioned for 2020. Yeah. Maybe that's what I'll tell myself. Nice new workstation, everything working ideally, no mouse lag. Here's the thing, Wes. Still had mouse lag. No.

Yep. On Plasma, using X instead of Wayland, KWin, AMD graphics, still had mouse lag. Have you considered just getting a faster mouse? Well, I just thought about locking myself in the bathroom.

Honestly, I was so upset. Can you switch to just the console directly? Because that's, you know. Yeah, yeah. So I went down, again, different threads. And a common thread I kept coming across was people were noticing this issue when they were experiencing high disk I.O. and GPU usage at the same time. Not gaming types of usage, but just like things that use the GPU on the Linux desktop. Which Chrome uses the hell out of the disk cache and the GPU when it's rendering a page.

And that is primarily when I would notice it. When I'd have a video or something playing, perhaps, or something going on. Maybe you've opened a bunch of new tabs. And of course, I've got multiple monitors. Even though I've got like a 5 an AMD 560 in there. Whatever. 570 actually. I'd have

the issue. I'd have lag. I'd still, even on Plasma, I'd have the damn mouse lag. So I thought, okay, well maybe I could reduce some of this. I can't really reduce the GPU usage. That's modern browsers. But can I reduce disk IO? And so I decided to load up Profile Sync Daemon, which syncs links and syncs your browser profile directories to RAM

using TempFS and RAM. Thus, it reduces the hard drive calls. It speeds up your browser. I want to make it clear here. I'm doing this on a really nice MV&E, okay? I've got... It's not some 5400 RPM. No, it's no slouch.

I legitimately noticed an improvement when I did this. I mean, have you tried RAM? It's the best. It is the best. Everything's faster. What's great about Profile Sync Demon, besides the fact that it could cause you to lose your entire profile, is that it supports like all of the browsers,

even Vivaldi and I'm not sure about Brave, but like all the different browsers. And it automatically detects them if you like. And it handles and manages the syncing to RAM and back. So you want to properly close and open your browser at the beginning and the end of your day to give it a chance to sync. You'll notice if you watch the cache usage from the kernel file system cache, you'll notice if you watch that indicator closely, you can see it blasts the cache as Profile Sync Daemon does its sync.

It just saturates. On my system, I think this box, it's either 32 or 64 gigs of RAM. I mean, it just uses a ton of cache for like 15 seconds. It's really fast because it's going from RAM to an MV&E. Right. But if you were to lock up and hit reboot, it would never have had a chance to write that profile back to disk.

And so any changes would be lost. Now, the nice thing is that's pretty much mitigated these days by browser sync for the most part. So it's constantly syncing to the Vivaldi cloud service. And yeah, I'm on Vivaldi too. A part of this, I switched over to Vivaldi. That's a whole nother story. But so thank you for asking

about the mouse leg. Hey, you know, it works out well too because of course, ProfileSyncDaemon is in the AUR. Of course it is. But it's pretty much available for anything. I think, my suspicion is, is that something is no longer

great on my box, or perhaps my original suspicion is that around kernel 5.2 time period, somewhere when I transition from the 4-series kernel, from an older distribution to the 5-series kernels on this box, that's when these problems manifested. I do suspect if I were willing to not have the latest goodies. So you're saying you're going to switch to neon. Yeah, because you're like, what, on 416 or something?

Well, let's find out. Because I never had leg on neon. Same hardware. 415, baby. Yeah. And when I used the GNOME implementation of Ubuntu, that's on 1804, I didn't have mouse leg. It wasn't until I went to 5- Kernels that I started getting mouse lag. And I wonder if this is never going to be fixed.

It's kind of getting, you know, three-year-old box now, something like that, getting there two and a half, something. People are not going to be releasing new drivers for, it's just, I think I'm doomed. So my solution is to just reduce the disk IO while using GPU for web rendering stuff.

It doesn't seem to affect games. It doesn't seem to affect games. It doesn't seem to affect that stuff. How well has this actually worked? I mean, do you still get occasional mouse lag, or is it pretty much gone? Very, very, very well. Nearly gone. Excellent. Maybe a glitch or two a day now, instead of every few minutes.

It's a huge difference. So you can use your nice computer now. Yeah. So give that a go. See if that works for you. I don't know if it will. Okay, now we've got to get out of here, but we'll do one more. Because I promise, towards the end of the week, we'll start doing this. And we'll do another batch, if you'd like, linuxunplugged.com slash contact.

But I think this is a good one to end on. Cameron writes in, regarding some requests for feedback, I'd love to have a form for the podcast, or better yet, for Jupyter Broadcasting as a whole, so the listeners and maybe you guys can all communicate and share info, news, et cetera. So that's something we've actually been talking about internally. We have been around for so long now that we have had about three different iterations of,

boy, if you count before Jupyter Broadcasting was formed, probably five different iterations of, boy, if you count before Jupyter Broadcasting was formed, probably five different iterations of community forms. Some of them were so large that that's actually how we ended up starting to work with Alan on Scale Engine, is we had to move on to infrastructure that could handle the amount of traffic we were getting. So we definitely have had some in the past,

but they generally, over time, kind of decline. And they become a pretty large source of work. And I want our interactions with the community to be excited, like we're excited to do it, not unsure. So our current outlets are not quite, they're not like something on the web. And that's what's so great about forum software because we do have the IRC.

IRC.geekshed.net pound Jupyter Broadcasting, hashtag. Octothorpe. Octothorpe. So of course we have that. That's a persistent chat going all the time. But we also have the Telegram group, jupyterbroadcasting.com slash Telegram. There's over a thousand people in there chatting, usually about a hundred online at any given time. And then last but not least, we actually have a Discord, too. We don't do much with it, but there is a Jupyter Broadcasting Discord, and I'll put a link in the show notes. I mention all of those because we do have some outlets of persistent chat where you can come and talk about the shows.

But the nice thing about forums is you've got topics. You can, you know, episode 329 of Linux Unplugged where they talked about flat networks. Right. It is a different thing. It's its own beast. So I could be convinced of it again. Probably need some feedback about it, though. And what would be the advantage of that, say, over the Jupyter Broadcasting subreddit,

which we've never really mentioned, but we have a Jupyter Broadcasting subreddit, which also kind of acts in that form capacity. But not everybody wants a Reddit account. But not everybody wants to go make a form account. So feedback, linuxunplugged.com slash contact probably won't read them all on air, but we will read them off air and help it inform our decision. So there you have it.

We did some feedback. Really quick before we run, also just a few picks to mention because I love these. Check out WeSlack. It's a WeChat plugin for Slack, and its developer was Ryan, our guest today. It has its own community now.

It's its own thing, but he started it all. Yeah, he no longer is even involved, but it's still a thing. So if you want to use Slack still in an IRC client, it's possible using WeSlack. And then I came across, I don't even know how I'm going to say this one, Grinds.

What do you say, Wes? Give it a shot. Yeah. Why do you do this to me? Gearens? Gearens. Gearens. This is a fairly slick GTK desktop front end to Plex.

We finally have one. Why would you want this, you say? I'll tell you why. Because browsers are monsters. They take everything, your disk, your disk, your CPU, your RAM. They make your damn mouse leg. They are the worst. They're huge. One tab takes up way more CPU and memory than it should. During some of our benchmarks, we have seen when a system gets

loaded, we have seen how a browser tab can bring a system down to its knees when it's under load. It's crazy. This does none of that. It's a native, clean client. When it's sitting there and you're not playing anything, it's not using any resources. It just sits there quietly. It doesn't peg your CPU constantly. And when you do want to play, it takes only the amount of resources it takes to just decode the video, which is probably pretty damn fast on your system. It's glorious. And it's a GTK application, so it looks great on your GNOME shell. What am I going to do with all this extra CPU time? I don't know.

I don't know. Run Slack, probably. But I was so elated when I found that because I like to have like, I have series on Plex that are like my barely kind of just background stuff that I play while working. Oh, yeah. So now throwing it up on a screen and not eating a bunch of resources while I'm working is fantastic. Trying to reduce resource usage, even though I have 12 cores.

It's ridiculous. Mr. Payne, you're over there on the TechSnaps. TechSnaps.Systems. TechSnaps.Systems. There's no S on the TechSnap, but I like saying it. Don't confuse everyone. I like saying it. I like confuse everyone. I like saying it. I like it. So, yeah, I've been really enjoying it.

Give my rounds of compliments to Mr. Salter, because I think he's really, both of you really hit your groove for a while now on that show. So it's been, it's like, when it's like good for a while, you know? I love that. Totally agree. Awesome show. And of course, Alex and I on Self Hosted next week.

Brand new episode of Self Hosted. Oh, I'm excited. Tomorrow, live stream, live self-hosted hack tomorrow, Wednesday, jblive.tv, 10.30 a.m. Jupiter Broadcasting Time. Alex is going to do a little live hacking on the stream. Oh.

Just a little self-hosted bobo. Bomo bobo for self-hosto. ESP 8266 hacking and LEDs and all that kind of stuff. Sounds like just what we need to start off the holidays. I know. I love it. Of course, all of that can be found at jupyterbroadcasting.com slash calendar. We're at letxunplugged.com and we'll see you right back here next Tuesday! අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි අපි Before we get out of here, jbtitles.com,

I have a question for the virtual lug. Does anybody own a USB stick that they could still find that is less than 16 gigabytes? Megabytes. Megabytes? Megabytes. Do we have one for... No.

Megabytes? Hey, I mean, Joe does not play. Yeah, Joe has a 16 megabyte SD card. No, not megabyte. Oh, wow. I can't beat that. You have a 256 megabyte SD card. 256 also here.

Oh, I thought I'd beat Mr. Bregsy here with my 4 gigabyte. No, he just keeps his favorite GIF, just one on there. Oh, sorry, GIF, is that what you prefer? Do you remember those early digital cameras that actually took an entire floppy disk? They took a 1.4 megabyte floppy disk, and then they saved

the horrible JPEGs to the flop? Yeah, I think it was a Canon camera, and I think you could shoot, you could get 12 photos on one floppy, I believe. You would have killed for 16 megabytes back then.