LINUX Unplugged - 452: Synapse Collapse

Episode Date: April 4, 2022

How we nearly crashed our Matrix server; what we did wrong and how we're fixing it. Plus an update on elementary OS, GNOME's next chapter, and we kick off the NixOS Challenge. NixOS Challenge Goals: ... Study the Nix Expression Language (https://nixos.wiki/wiki/Nix_Expression_Language) Setup at least one Nix/NixOS system (https://nixos.org/manual/nixos/stable/). Install htop (https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/system/htop/default.nix). Join the Nix Nerds Matrix channel (https://linuxunplugged.com/matrixinfo). Post a screenshot in the NixOS Challenge GitHub (https://github.com/JupiterBroadcasting/nixos-challenge/). Complete all the above before the end of April. Special Guest: Danielle Foré.

Transcript
Discussion (0)
Starting point is 00:00:00 All right, I know we're getting a little too excited about NixOS these days, but I'm really thinking we should switch to Linux from scratch. Whoa, like build a ground up, like really learn something kind of thing? Yeah, you know. That's going to take a lot of time, Wes. Oh, no. I've got something for you. Oh, yeah?
Starting point is 00:00:14 Yeah. On our Linux just today, a giant bash script to build Linux from scratch automatically. You know, it reminds me of that old saying, Wes. It's my pedestrian. It's how you use it. Hello, friends, and welcome back to your weekly Linux Talk Show. My name is Chris. My name is Wes. And my name is Brent. Hello, gentlemen. Well, coming up on the show today, we're going to tell you about how we nearly
Starting point is 00:00:44 blew up our matrix server. Don't worry. We saved it at the very last minute, right before the big stream last week. We'll share what we messed up, what we learned, and how we're going to try to build it back better than ever this time. You know, it's kind of one of those IT war story shows where we admit that maybe we made a mistake, plus an unexpected circumstance comes along and it blows up our whole setup. But we really do kind of have an idea of how to fix it. And then after we tell you about all that, we'll kick off the next challenge. Then we'll round out the show with some boosts and picks and a lot more.
Starting point is 00:01:16 So before I get any further, let's say time-appropriate greetings to that virtual lug. Hello, Mumble Room. Hello, Mumble Room. Hello, yay. Hello. Hello, Mumble Room. Hello, yay. Hello. Hello, hello, hello. Some old friends in there. We have a whole good group in there.
Starting point is 00:01:32 Powerful Sunday crew. Thank you for being here on a Sunday. Now, I do want to mention, we are so close. This is like your last minute notice. The April 9th, 2022 Raleigh north carolina meetup is happening next weekend as we record this uh meetup.com slash jupiter broadcasting for the deets cancel your plans book your flights and come with an empty belly because there's barbecue place there is also a brewery and do let us know if you're coming so we can warn the restaurant meetup.com slash jupiter
Starting point is 00:02:04 broadcasting it gets real close. And then you won't hear us talking about it anymore. It's going to be like this era of when we were talking about the East Coast meetups. I'm just getting kind of emotional about it. But yeah, I think I fly out like Wednesday midday. Land and record a self-hosted. Have you packed yet? No.
Starting point is 00:02:22 I did find the luggage, though. But I'm currently doing that whole like, could we get away with one carry-on for both of us? Whoa. Because the wife's coming. You know? And she's kind of an overpacker. Don't tell her I said that. I'm not.
Starting point is 00:02:35 She's one of those, you know, we might need. Well, we might need. You know what I'm saying? So we end up with, like, a bag of stuff that we might need. And, yeah. I don't think it's going to be one bag, but, uh,
Starting point is 00:02:46 we'll see. I'm, I'm just thinking like today's going to be a great show because we've got some good community news. We're going to kick off the next challenge. We got kind of a come to Jesus moment with our matrix server. And no matter how bad we screw up today, we got a double slow cooked pork shoulder going in the oven right now.
Starting point is 00:03:08 Wes brought like this amazing piece of prize meat. Did you win that like in a fight or something? Well, yeah, that's, that's also what explains the black guy, but I didn't, I didn't really want to tell that story. Okay. That's fair. That's fair. But I'm looking forward to snacking down on that.
Starting point is 00:03:18 That's going to be really great. Plus Wes brought us some ciders. So we're going to just, it's going to be, doesn't really, doesn't really matter how the show goes, but we'll do our best anyway. I suppose. I suppose. Well, you promised. All right. All right. Well, let's kick off the community news. Danny Foray joins us from Elementary with an update. Of course, we've kind of been wondering what's going on over there, Danny. So what is the latest? Yeah. Thanks for having me on. So just kind of in a nutshell, there's been some change-ups at Elementary Inc., which is the company that kind of supports and funds the elementary open source community. So these days, it's just me now.
Starting point is 00:03:56 I'm the only employee and owner of the company. So things are changing a little bit there. It's been a little bit chaotic, but we just want to make sure everybody knows that I'm here and things are okay and development is still going on. I have kind of more of a lengthy dive into what's going on and what's coming on our YouTube channel, which is linked from our website, elementary.io, for people who are curious to find out more. Excellent. We will also link to that in the show notes.
Starting point is 00:04:27 Wes and I got to catch a bit of that. And, you know, I think you just, you said a lot of stuff there. So I think a lot of the details are kind of pretty well covered. I was really impressed. You really kind of just laid it all out there and you were really honest about it all. So why don't we talk about like, you know, where things go now? Because I think that's what we're all kind of interested in is what the future is and kind of what you might
Starting point is 00:04:47 see as short-term changes and maybe some long-term stuff. Like what's next? Yeah, absolutely. I definitely want to focus on the future and what can be done better. And I think like something that's been huge for me is kind of reevaluating like what is the purpose of the company and its relationship to the community and not only just the elementary community, but the wider open source community. And something that I really want to get back to that I feel like we used to do a lot more of was more donating to upstream projects and more funding for developers in the community. I think GitHub Sponsors is a really big opportunity. So I think that's kind of the role that I
Starting point is 00:05:24 want the company to take is more like a support role. That's going to be quite the process, right? Because there's probably not like a ton of funds to go around, I would imagine. So how are you thinking that's going to work? Like, is there going to be some of the income that comes into Elementary Inc. that's then going to go out to different members that are working on projects that benefit the distribution? Yeah, I haven't figured out exactly how to split things up yet. And I definitely want to talk it over with our developer community and find out what they feel is important to fund and how things should be split up.
Starting point is 00:05:54 But I really want to make giving a bigger part of what the company does. And kind of with the downsizing that's happened, I think that we do have a bit more of an opportunity to make it more so that we're pushing money out into the community and not so much like just funneling money into a for-profit company, you know? Yeah, I follow you. I do, I think that's a really good goal. I wonder, are you going to,
Starting point is 00:06:24 is your life expenses going to be covered by that? Cause you have to eat too. Yeah. Um, I don't think really that I'm going to have to change anything about, um, my like employment status or anything with the company. We kind of already made some like, uh, budget cuts and stuff towards the end of the year, like as a response to COVID. So I think like where I'm at right now, like I'm comfortable and I'm good. And I think that there's room to do more. Great.
Starting point is 00:06:51 That's exciting. So keep us posted, right? Keep us informed on what's going on. Keep us up to date on what the new developments are and come on here and share with us when things are moving and shaking. Yeah, absolutely. And if people want to follow along in like more real time,
Starting point is 00:07:04 like elementary.io is always the place to go. We got a link to our blog there and social media accounts. Very good. Well, thank you, Danny. Thanks for taking a bit of time to update us and good luck on everything. We'll be checking in from time to time and trying out new releases as they land and all that kind of stuff and looking forward to the future. Thanks. All right. We will have links in the show notes at linuxunplugged.com slash 452. Why don't we stay in the ecosystem just for a minute? I noticed that Chris Davis did a blog post. Chris Davis, a.k.a. Brain Blasted in the GNOME community. Chris calls himself a GNOME developer who wants to improve the GNOME ecosystem as much as possible.
Starting point is 00:07:43 Got to appreciate that yeah currently works on fractal matrix app for gnome which i only laugh because we've been talking about it behind the scenes a lot recently but i don't know if any of us have tried it no yeah it's like next on our list uh and a few other great apps too like gnome weather and a couple others you've probably heard of but uh chris did a post about plans for GNOME 43 and beyond. And I really like some of the stuff that they're talking about here. So we've seen the arrival of LibidWada. I'm sorry, LibidWadia. We've got to make sure I get that right. And that really created a platform, you know, a library that developers could use to create GNOME looking applications. But it also makes it simple to implement some stuff we've been wanting for a long time, like customizable accent colors, right? Global accent colors that can be opt-in for app
Starting point is 00:08:31 developers. Yeah, this is looking really good. They have some screenshots here. And I think, Wes, the change here is, and you probably, you could probably tell me if I'm wrong, but I think the real shift here with this is like this stuff that was being done before with CSS changes is now getting done via an API. Yeah, I guess developers using the recoloring API will be able to programmatically change colors in applications and then have dependent colors update automatically. So you can create presets which can be used for, say, to recolor the window based on a text view's color scheme that you're using somewhere else, right? So the whole window can match whatever you're choosing for your text view. And I've also seen some suggestions, like maybe something's in an administrative state,
Starting point is 00:09:13 so you make the window red. And this API gives you, I think it gives you, I don't know about more control, I can't really speak to that, but I think it gives a better end result. And Chris Davis himself has been working on Loop, which is, get ready for this, Wes. It's an image viewer, but it's written in Rust. Hey-o! That's right.
Starting point is 00:09:34 And GTK4 using LibidWin. I don't want that dirty C code to touch my precious JPEGs. Oh, of course not. No, you need to render those JPEGs as fast as possible and with as much memory safety as possible. You know what, though? We only bring this up because doesn't it feel like we're seeing more and more stuff for GNOME written in Rust?
Starting point is 00:09:53 I mean, it's almost like a meme at this point. Yeah, I guess there's some getting to be a battle-tested Rust GTK sort of friendship, I guess. Yeah, yeah. I like the direction of this stuff, I have to say. One of the things that Chris also wants to work on
Starting point is 00:10:07 is the XDG desktop portal file pickers. Like, how do I say it? So, see, you got a Flatpak application. It's sandbox.
Starting point is 00:10:15 It's limited right now in how many files it can open that are associated with a file that you want to open. Another way to say that is, let's say you saved out
Starting point is 00:10:24 an HTML archive and you've got an HTML file with some images and CSS and stuff like that. And you open that with a Flatpak application. Well, you'll maybe be able to open the core document, but all of the associated files, like the CSS file and the images folder, none of that is going to be accessible by that Flatpak application right now. Right. Or, you know, often you have stuff that has, like, a project file, and then all the resources for that project are kept, you know, next to it in the same folder or something.
Starting point is 00:10:49 Right, I'm sure that's super common for IDEs, right? So Chris wants to fix that. He wants to kind of just change the way that works and make it possible, just tweak the portal so that way it's possible for that to actually not be the case. He also calls out some accessibility fixes. And the reason why I'm mentioning this is I feel like we have an opportunity, for that to actually not be the case. He also calls out some accessibility fixes. And the reason why I'm mentioning this is I feel like we have an opportunity here as a community to help Chris because he is currently looking for work.
Starting point is 00:11:15 If you know someone out there that maybe is looking for an employee that would fit these skills, we'll have a link in the show notes. Additionally, Chris has a link to Patreon, GitHub sponsors, and PayPal. And there he's just looking for some help to set time aside for these issues. And what Mr. Davis has done here is essentially zeroed in on what I think are issues that affect real people that use the GNOME desktop. These are sometimes things that we criticize Gnome about is focusing on things that
Starting point is 00:11:46 don't apply to the actual users that are using the desktop and maybe they're targeting that elusive new user or they're targeting
Starting point is 00:11:53 that elusive mobile Linux user. And I think, you know, you see some of that what Chris is talking about. He definitely wants to improve some
Starting point is 00:12:00 stuff to work better on mobile, but I really don't have issue with that. Make it work great on both. Okay, I'm on board with that. Yeah, right. That's a good compromise. So I just wanted to call attention stuff to work better on mobile. But I really don't have issue with that. Make it work great on both. Okay, I'm on board with that. Yeah, right.
Starting point is 00:12:06 That's a good compromise. So I just wanted to call attention to this work. It's a reminder to us that the GNOME community is made up of a lot of viewpoints. And, you know, I think as we covered this Libid Way to stuff additionally, we learned that a lot of times you'll see individuals speak on GNOME's behalf,
Starting point is 00:12:25 but it's actually made up of a lot of individuals with different opinions. And so we're starting to see a diversity in those opinions now. And I just think that's a great thing. And I like this goal set here. I'm getting kind of low-key excited about the slow progression GNOME's going in. There's always a couple of trade-offs we make, but ultimately, after they change something and then they iterate on it, I think the users end up for the better.
Starting point is 00:12:48 I mean, we've gotten to a pretty nice place, and I'm excited to see where it goes. And it's just interesting here to hear someone who clearly is making some changes that we like, who has shown skills in doing this work that's saying, if you can offer me some more support community, I'm willing to keep doing this and to do more of it. And so
Starting point is 00:13:03 it's time to reflect on that and what we want. And Danny, before we jump off of all of this, do you have any thoughts on LibidWadia, how it may affect elementary OS and that stuff? Because that's sort of your neck of the woods to the whole GTK area, even though you guys don't have GNOME Shell, you're pretty reliant on the GTK stack. And I got to figure you're looking at this. Yeah, I mean, so far in porting to GTK4, the changes that were made here have been extremely positive. Like it's a lot of GNOME design language stuff is no longer in GTK4 and it's in libabuida. So this is a huge win for desktops like us or for XFCE that have our own design language and we don't have to kind of be
Starting point is 00:13:43 pulled along behind the GNOME boat. But for the GNOME community, I think this is also a huge win for their app developers because GNOME has never really had an official supported theming API. And with Lib Adweighted, now it does. Yeah, I think it's an adjustment, right? It's not a downgrade. It's not a bad thing. And I think initially maybe was seen as that. It was seen as the end of theming. And it's just a change in how theming is done. And I feel like it is a new chapter in the GNOME desktop. We're just arriving
Starting point is 00:14:13 there. It's the very beginning. It's the very first chapter. But we're just there. We've just turned the page. It's exciting. 42 is a great release. 43 is going to be a great release. And I'm excited to see how distributions like elementary OS and others that are using all those tools embrace it. 42 is a great release 43 is going to be a great release and I'm excited to see how distributions like elementary OS and others
Starting point is 00:14:27 that are using all those tools embrace it Linode.com slash unplug go there partner and get yourself $100
Starting point is 00:14:36 in 60 day credit on your new account and you support this here show I don't know thank you I don't know I don't know
Starting point is 00:14:41 but maybe maybe that guy could do the ad reads from now on right he seems pretty excited. I'll tell you why he's excited about Linode. He's been using it for a while.
Starting point is 00:14:49 You know, he runs his business infrastructure on there. And so he knows you can trust your business infrastructure on there. But he's also got his kids game servers on there. So like you want to set yourself up like a safe Minecraft server or maybe you want to get yourself, you know, I don't know, a little CSGO action. I don't know what you do. I don't know. I little CSGO action. I don't know what you do. I don't know. I'm not going to pass judgment.
Starting point is 00:15:07 I'm just saying go to Linode.com slash unplugged, get $100 in credit and support the show. And they got 11 data centers around the world. And I'll tell you, I don't always pick the right data centers. And I'm going to own up to that in a little bit. But you can. You can learn from my mistake. I'm going to use Linode the wrong way so you don't have to. Also, Linode's constantly upgrading their
Starting point is 00:15:28 infrastructure. They got these super fast connections. It's embarrassingly fast coming in. 40 gigabit connections coming into their machines. And they are their own ISP. Then they got new MVME PCIe storage. And I just learned recently, Wes, a couple of our machines, we get free upgrades. What? I thought we were going to have to pay more.
Starting point is 00:15:44 Wait, free upgrades? You go in there, and if you qualify, you look at your disk, and they just got a little button. You press that button, and you just get it. Wow. Maybe I shouldn't be saying this. Maybe it's not true. I don't know. But it is online, and I'm pushing that button. Let me tell you. The button says upgrade, yeah. I'm pushing
Starting point is 00:16:00 that button, Wes. I love that button. And I love their object storage. This is my new, like, I don't know what I'm going to do with all this stuff, but I don't want to store it on this particular machine button. And I love their object storage. This is my new, like, I don't know what I'm going to do with all this stuff, but I don't want to store it on this particular machine. So I put it in object storage and that's been working so good. Everything gets its own addressable URL. It works super easy with their upload or their CLI client that I use to upload it because their CLI client's easy too. The whole thing's ridiculously simple. You don't even have to be like a server expert to figure this stuff out. But if you are, you're going to be impressed with the power that they let you get to if you need to.
Starting point is 00:16:28 I'm talking like they'll even give you guides to like take one of their rigs, wipe off their image, and put NixOS on there if you're that kind of maniac. They'll do it with you. I've done it before. Don't threaten me. I'll do it again. I'll do it. Don't challenge me on this. I'll tell you what.
Starting point is 00:16:43 Go try it for yourself and see what we're talking about. Go see why Wes spins up his stuff over there. He's spinning up servers all the time. You'd think he's like some sort of DJ spinning the tracks. Yeah. I don't listen to music. I just listen to that WebUI. That's right.
Starting point is 00:16:55 That's right. And that awesome, what do you have keyboard? The coder keyboard or whatever it is? Yeah. Code keyboard? Code keyboard. Yeah, that's right. I'm sitting there on calls with Wes.
Starting point is 00:17:04 Clack, clack, clack, clack, clack, as he's setting up the Linodes. It's like how I know he's doing something, actually. But, you know, it's a great thing is we all can use it, right? Wes can over there, you know, he's fine. He's using it. But, you know, I can stumble my way through it too
Starting point is 00:17:16 and act like I'm an expert as well because it's pretty straightforward and they got great documentation and just the best damn customer service in the world. That's what I hear all the time from the audience. So go try it out with $100 and support the best damn customer service in the world. That's what I hear all the time from the audience. So go try it out with $100 and support the show. Linode.com slash unplugged. That's Linode.com slash unplugged.
Starting point is 00:17:36 Just about two years ago, we launched the Jupyter Colony Matrix server as something of a living experiment, let's say, here on the show. And since we spent the last week pulling that same server back from the brink, we thought it best to share what we screwed up, what we've learned, and there's a lot, and how we're going to try to build things back a little better. That's right. We did mess up.
Starting point is 00:18:06 We messed up bad. I acknowledge it. Sometimes we get these wild ideas. Yeah, maybe a little over our head. And we think, what could go wrong? And then we realize we've stumbled into something that is old school complicated. Not new school complicated, old school complicated. That's all right.
Starting point is 00:18:24 That's my power zone. I can deal with that. I didn't realize I was signing up to manage an exchange server, but that's what I'm doing basically. If you're an old IT guy, you know what managing an exchange server is like. That's what managing a Synapse server is. So maybe some terminology, just so that way we're all on the same page keys right now.
Starting point is 00:18:41 Matrix is actually the protocol, like HTTP is, right? So you've got Matrix, which is a protocol. Element, the chat client, is like a front-end. It's like your web browser. There's multiple front-ends. Element's probably the most feature-rich. And then there's Synapse. Synapse is the back-end matrix server that we're using. Dendrite's another one. Is there other? I don't know. But Synapse is the one that's deployed at any kind of scale. Yeah, you know, and that's kind of been the one that the team behind Matrix have been using is like the sort of primary implementation development stuff. And, you know, Dendrite's a new implementation, but hasn't necessarily caught up on all the features, which kind of right now, after they go through the, you know, the RFC kind of process, then
Starting point is 00:19:20 they're kind of usually first implemented in Synapse. And I guess the sales pitch, the short sales pitch for Matrix, if you're not really familiar with it, is it's a decentralized federated chat system that supports direct encrypted communications and public chat rooms. And it features a lot of the same kind of nice-to-haves that your Mattermost, your Slacks, your Discords have. You can have a server with multiple channels. You can paste a Gif or a gif i don't care what you call it you can paste it in there it'll display it you know image previews kind of the rich like chat features you get from a modern chat system they're all in matrix a lot like how discord does it but one that you can self-host and one where you could say create an
Starting point is 00:20:02 account on matrix.org and then join our matrix server where you'd find all our rooms and stuff. It's great. Yeah, you know, I mean, instead of a sort of proprietary implementation or even just a particular implementation here, you've got something that has a generic protocol and has a spec and then can have a plethora of clients. And yeah, you've got that old school internet feeling like with email where you don't have to go through some gatekeeper.
Starting point is 00:20:24 You can just say, yeah, I want to have a home server. And then you can start participating with our home server, with the main one, with any of them. Yeah. And I want to make it clear. What we are about to tell you would not be a problem if it's you and like a handful of people using a matrix server. You could probably run it on a Raspberry Pi and you'd be happy, right? What we're about to tell you is a journey you go through when you start with like a hundred users and then you got nearly 3,000 users and just like all of the things that change when that happens. That's what we're going to talk about today. So our initial deployment was a bit anemic. It was basic. You know, we were just really wanting to get some hands-on experience so that way we could talk somewhat intelligently about it on the show.
Starting point is 00:21:08 I thought we had so much to learn, but we initially deployed a four-core Linode box with eight gigs of RAM and like, you know, a fair amount of, I don't know, 150 gigs of local SSD. It wasn't a lot. Yeah. Again, right? Like when we just did it kind of simply because we'd been talking about matrix, we obviously thought it was cool and it felt like, okay, well, if one of the things is about that, you know, that we highlight about this is that it's open source
Starting point is 00:21:31 and you can run it yourself, we better run it ourselves to just try it. So we just went with like a super simple Docker compose setup, which was just two containers, right? Like we got Postgres for the database and we've got the synapse container, just the, you know,
Starting point is 00:21:43 the one right from stock upstream running the main Python application. This is the good and bad about things in containers too it's not about the technology it's like it does enable you to quickly get up and get running but it's not necessarily configuring the most optimal situation for a high demand production use and you just have to kind of be aware of that uh and you know this was fine two years ago when we maybe, you know, after a year had 300 people on there. It was getting a little bit slow, but as long as Wes logged in from time to time and made sure the disk wasn't filling up, we were okay.
Starting point is 00:22:15 Yeah, you know, did our updates. I got to say that really like the upgrading process has always been nice. They've got great, you know, release notes that kind of call out if there are things that they're making breaking changes. And other than that, you just update the tag on the Docker image and away you go. So that part's been nice. They've got great release notes that kind of call out if there are things that they're making breaking changes, and other than that, you just update the tag on the Docker image and away you go. So that part's been nice. Yeah, and solid updates. They're almost always introducing
Starting point is 00:22:33 something that Wes and I are like, oh, that's worth an upgrade. They've been really good about that. And Dendrite's moving along really well, too, but Synapse legitimately has been making solid improvements nearly every time we want to do an update. So it really hadn't been too bad. But over the last 10 months or so, the Matrix colony has grown. In fact, I would say even kind of like at the beginning of the year or so, I wasn't really sure, like, are we going to keep running this Matrix thing?
Starting point is 00:23:00 It's kind of becoming a pain in the butt. I think I groused about it on air once. You know. You know, maybe, yeah. Well, we were just, that was the first steps of us realizing that we either need to decide to like commit to this or rethink our involvement with it because it is, with the size that it had become,
Starting point is 00:23:17 it did take some resources. I do like to take a think about it because we kind of haphazardly just implemented Discord a few years ago and i think we're still kind of dealing with the consequences of that so i thought you know we'll take it slow with matrix we'll see where this goes and then when it started to get to this pain in the ass stage i thought i really haven't made up my mind i really haven't committed to this yet you know where i actually think things really kind of changed to tell you the truth
Starting point is 00:23:41 is when the whole free node thing went down and then all of a sudden to tell you the truth, is when the whole Freenode thing went down. And then all of a sudden, a lot of the projects that we like to communicate with started popping up on Matrix. And I think that particular instance also made people realize how nice it was to have something that could be decentralized so projects can run their own instance
Starting point is 00:24:00 but still participate in the overall network. I'm on a Red Hat instance, I'm on a GNOME know, I'm on a Red Hat instance. I'm on a GNOME instance. I'm on a bunch of other projects' servers, but I do it from the JB server. And then it just kind of started accelerating. And in about three months or so,
Starting point is 00:24:18 it just really kind of started growing a lot. It picked up, and that's awesome. And there's a really cool decentralized community growing on there. And as a result, I've been spending a lot more time on there. And I realized this is there's a lot of really good conversations. People are kind because it's still in that kind of phase where it's not huge, you know, but it's also big enough that there's always some conversation going on. And then that room aspect of it kind of breaks things up. Yeah.
Starting point is 00:24:43 Yeah. You know, we've got there's different communities that can all kind of breaks things up. Yeah, yeah. You know, we've got, there's different communities that can all kind of flourish there, but it still has that, you have to opt-in kind of functionality because, you know, you might already have Slack or Telegram because of your friends or for work or something, but if you don't do an open source things,
Starting point is 00:24:56 you probably don't yet have elements. So you have to want to be there. And it's, you know, not a great process to get on board, especially if you want to onboard directly to be there. And it's, you know, not a great process to get on board, especially if you want to on board directly to our server. You have to like change defaults in the on the login screen. It's just not like as great. And I know it's a little confusing, right? Like if you're not familiar, you have to figure out what a home server is and then figure out which one might be right for you or where it makes sense to go do that. Right, right. Yeah. Do you want to host your own thing?
Starting point is 00:25:23 Do you want to just join matrix.org? Do you want to set up an account on the JB system? Right. Now, it's kind of like when you start with Linux, you got to pick a distro, and sometimes people just get stuck there. But over the last few months, it seemed like people were moving over. They were trying it out.
Starting point is 00:25:38 And then within the last, literally like the last, since the last time we did the show, we realized the load was getting way too damn high and then one morning i woke up to a telegram message that said i think the matrix server is down i'm not sure if the matrix server is up can you check on that and sure enough it was and i realized pretty quickly that we had used up all of the available disk on that system. Sure had. That basically kicked off a series of nights of Wes burning the midnight oil to try to rescue the matrix server and him and I doing like emergency preparations to like, like
Starting point is 00:26:17 maybe migrate a server or like spin something up entirely and all this kinds of stuff that we had to do. Some of it was fun. Some of it was a bit hectic. entirely and all these kinds of stuff that we had to do. Some of it was fun. Some of it was a bit hectic. But we realized, I think, probably what, by day two,
Starting point is 00:26:32 we're probably going to have to just totally rebuild the way we've implemented this if we really want to scale it to like maybe 10,000 plus users. Yeah, you know, we'd been kind of interested in utilizing the new worker system that Synapse has added so that you can kind of deconstruct it from just this one monolith Python application to have that work shared across different processes. But that takes some plumbing
Starting point is 00:26:52 because now you're taking that system apart and it needs abilities to be able to coordinate between the workers and you have to add Redis to the system, none of which is super hard, but it just has its own particular setup and we're not developing this and we haven't gone down that path before. So it was just a new, you know, several new docs pages to read and get familiar with while having it on a live system
Starting point is 00:27:13 that we were hoping to, you know, not totally break after we just got it back online. And of course this all starts happening. We really realized Tuesday, this is really screwed. The system's really screwed up now. The problem was, it was down for a while, and then when we brought it back up to sync with the rest of the federation, essentially maxed the server out. And then the other problem we discovered was that when a user would log in and sync all of their rooms, a single login session of a single user
Starting point is 00:27:42 on a 2,000-user system would peg the system to 100%. We also realized pretty quickly that while we had thrown more hardware at this because we thought that's going to solve the problem, we'll just throw hardware at it. So we upped it from four cores to six cores. We went from eight gigs to 16 gigs, and we sliced off another set of storage.
Starting point is 00:28:03 This is like the third time we'd sliced off more storage at this point and we figured there that'll fix it but we discovered by default synapse is kind of well it's sort of silly single threaded it's kind of amazing how single threaded it is and so you can throw six cores at it but that doesn't matter because it's using one core unless you do this worker thing. So that was really our realization is, OK. All right, let's get this thing to a stage. That way it survives the live event on Thursday.
Starting point is 00:28:37 Yeah, because, of course, we just started to push people towards using the matrix server, right? That was part of the live stream. Let's get people to sign up for matrix. And now here the matrix server has fallen over. So you banged out like a couple of generic workers is I think what they're called yeah you know they can kind of just just do anything and then there's some more support you can split things out further uh we haven't yet walked down that path probably something we want to try uh still still more reading to do really but this was just an attempt to like okay well yeah maybe we can split off some of the stuff you can do some
Starting point is 00:29:00 of the you know responding to the federation events and syncing there and then we can have some other things that are handling and interacting with the clients, just try to keep those a little more responsive. And it seems like it's worked. It's not, it's still not great. It's not where I think any of us want the server to be in terms of responsiveness, but I think it's been usable, at least. Yeah, it's still a little slow,
Starting point is 00:29:18 but it's usable, and we want to work on that. We have, and what we realized too is sort of our hodgepodge approach of adding disk sort of randomly at the machine, various like local disk and network disk and object store, all these random mixes of disks. Well, we were constrained by the lowest common denominator. So our disk throughput was essentially completely pegged. We were maxing out our disk IO. We were single threaded on the core.
Starting point is 00:29:42 And because we'd been down for a while because we had used up all the disk from storage, it was a nightmare while that thing synced back up. It just completely slammed the box as people were trying to sign on all of that. So we realized we got to do this workers thing. We got to readdress the disk situation. We kind of started making a list of stuff. And also I'm told Wes that version three of the sync protocol is going to be
Starting point is 00:30:04 lighter weight and faster and it's close. So we're going to get a little assist. That's the nice thing, too, here, is they are making improvements as we try to make improvements. To their credit, they really have. Like, I remember, you know, maybe six months or a year ago, we would often have issues with the memory use of Synapse. Just over time, it would slowly, between it and the way it was kind of using Postgres,
Starting point is 00:30:22 like, the system would just really eat RAM. That doesn't happen anymore. Yeah, that's true. That has gotten better. So that's nice to see. Also, I got to admit, because this was a this was a test that we were doing for the show. We deployed this system in California. It's quite close to us, but maybe maybe not the rest of you all.
Starting point is 00:30:42 Not for an international audience. Not the best location for it. Maybe not the worst of y'all. Not for an international audience. Not the best location for an, maybe not the worst, but not the best. So I think, you know, one of the things we would do differently now is we would deploy to a centralized data center and we would kind of look at some of our options here. And I think when you consider all the improvements
Starting point is 00:31:00 we could make for a large-scale implementation, one would be better utilize the workers, go even further with that. Each major, like, functionality aspect of the backend Synapse server can be handled by a worker. And some of them, it's like one worker to handle, like, presence.
Starting point is 00:31:15 That's a worker, right? But something like Federation Sync, that's like 10 different workers you can set up, each with their own individual config file and NGINX rules you may or may not have to set up. Yeah, right. Because you might want to start setting up NGINX load balancing or you use Regex to direct
Starting point is 00:31:31 some traffic to the worker that's going to handle that traffic. So there's just a little, lots of moving pieces to reconfigure. This is about the time I realized, you know, I should have just contacted the folks at EMS and just said, hey, would you like to sponsor a few episodes and in trade just host our matrix community? Because I'd like this to be faster for people. I'd like it to be a nice experience with Matrix. And we are not experienced Matrix admins
Starting point is 00:31:53 or Synapse admins by any stretch. We've just kind of realized we need to take it more seriously. And so we've thought, well, how would we build this better if we were going to redeploy it? And I think one of the things we're going to do is experiment with the NixOS build, kind of in theme right now, as a way to kind of learn the edges of this thing, to see if we could
Starting point is 00:32:11 build something with multiple workers out of the box and something that uses a centralized data center with NVMe storage, right? Just one pool of big NVMe storage. And maybe something that collects metrics out of the box, because you discovered there's actually quite a bit of information we could be getting from the system to better understand what it's doing. Yeah, you can have Synapse emit Prometheus metrics, which is great. So from that on, we just need
Starting point is 00:32:36 to start sucking it up and throw it in some dashboards. Yeah. So that's something we want to play around with, and maybe inevitably we decide, okay, NixOS, this isn't the way to go. Let's go with this community answerable solution because that's pretty popular. Yeah, definitely.
Starting point is 00:32:50 And I think even regardless if you use it or use pieces of it, it's an awesome store of community knowledge around how to admin a Synapse box. And a lot of the, you know, it's got pointers to tools of like, there's an admin web UI you can use. It's got notes on database maintenance,
Starting point is 00:33:15 which is another area I think we need to spend some time either deciding how much history we're going to keep for some of these rooms or, you know, spending some time compressing past state events. There's just one of the things we run into was our database was getting larger than we expected. And Synapse really needs it to be pretty, pretty fast if you want good latency. Yeah, that's tricky. That is something we'll have to figure out too. And we want everyone's input on all of this. We're going to collect your feedback. We do have a matrix ideas room on our matrix server, which will be linked in the show notes.
Starting point is 00:33:35 But if you go on there, if you go on the colony.jupyterbroadcasting.com matrix server, you'll find it. I'm picturing maybe a future build. Maybe it's NixOS based. Maybe it's not. It's NVMe storage. It's centralized data center.
Starting point is 00:33:47 And maybe it uses object storage for, like, the image files and video files that people upload to the Matrix chat. Yeah, that seems pretty reasonable. I could see something like that. Just something that we could build that we know could run for a while. I think six cores and 16 gigs of RAM so far has been sufficient. I feel like once we optimize the Synapse configuration, that's plenty of hardware to throw at this. Yeah, that was the other part too. You know, we kind of, we got
Starting point is 00:34:09 the default working Synapse. Like, oh, hey, look, it works. It's doing federation rate. Cool. And then we had not done performance tuning. So we've also turned off presence. Maybe we should never have had it on. I don't know. People have thoughts about that too, maybe. Yeah. Matrix Ideas Room. That'll be linked in the show notes.
Starting point is 00:34:25 Go over there and let us know what you think because we're kind of putting all this together right now. The system's up, it's running. It just doesn't perform at the level we'd like, but we're tweaking it and then we're going to eventually probably migrate it to some new ultimate matrix server. Right?
Starting point is 00:34:41 Doesn't seem like the plan. Yeah, I think so. And just another reason, maybe the other part we're kind of experimenting with here is maybe you want your own home server if you're into that kind of thing, rather than signing up for ours. That's true. And you won't run into any of these problems if it's just you and
Starting point is 00:34:56 a couple of friends or just yourself. You really would not have any of these issues. It's just when you have thousands of users and they're always on there 24-7. And then, like a jackass, you hold a live stream where you tell everyone to go sign up for your matrix server the very week that it's collapsing under its own weight. And then, of course, make us all have to work super overtime so that way we can at least get it working and functional so that way the live stream isn't a total disaster. Which it still was touch and go.
Starting point is 00:35:23 There were moments the server barely made it. Yeah. It was definitely lagging there for a while. But hey, conversations happened. And sats were given. And sats were given. Speaking of that, that live stream is posted at
Starting point is 00:35:39 xers.show slash 84. Brent did the yeoman's work of trying to quickly collect up everyone's lightning payment addresses and paste them did the yeoman's work of trying to quickly collect up everyone's lightning payment addresses and paste them in the doc. Nice work, sir. I think we did okay. I think we hit everybody, but hopefully crossing fingers.
Starting point is 00:35:56 And then Wes had to turn it into a battle when he tried to boost people before I could boost people. So the sats were flying. I love this. We gave away 60 unique lightning donations, usually around 1,000 or 2,000 sats each, and we gave them to 42 different individuals on the live stream.
Starting point is 00:36:14 32 new people, at least by our count, signed up for our Matrix chat, although the great thing is those numbers continue to roll over the weekend. They've gone up. We featured an interview with the Fountain FM developer, Oscar. That's in the live stream. We also covered
Starting point is 00:36:28 some JB projects that the three of us are working on. That's all in there. Extras.show slash 84 if you'd like to participate in that. And if you'd like to send us one of them boosts that we talk about all the time, go grab a new podcast app at newpodcastapps.com and then get yourself
Starting point is 00:36:43 set up with something like Blue Wallet or Strike. Grab a few sats, stuff them in your Lightning Wallet, boost away! That's how you do it. newpodcastapps.com All right, guys.
Starting point is 00:36:57 I think we've waited long enough. People want to know about this NixOS challenge. We're going to put the details in the show notes, so if you miss any of this, hopefully we'll have it
Starting point is 00:37:04 listed there. But it is time to officially kick off the next challenge with some goals. Number one, study the next expression language. Number two, set up at least one Nix or Nix OS system because Nix is also a package manager available for your current distro. Install HTOP using Nix and then join our Nix Nerds Matrix channel. Come in there, tell us how it's going, tell us how you're doing, share all that kind of stuff. And then last but not least,
Starting point is 00:37:32 post a screenshot of either HTOP or for extra nerd points, a full working graphical desktop over on our GitHub, which we will have linked in the show notes. The idea is, the challenge is to complete all of that by the end of April. At the end of the month, we'll do a review of how it went for everybody, the moments
Starting point is 00:37:51 that stand out to us and what our plans are as a team to keep using Nix or not and what we're going to deploy with all of that. I challenge thee. So you've been officially challenged and Brent's already all in. You've created a GitHub for people to use where they can post screenshots and stuff. Tell us a bit about that. Yeah, we kind of put together some ideas to post a bunch of screenshots there instead of necessarily the Matrix room because then we could just review them all. It's actually really fun.
Starting point is 00:38:19 And there are discussions happening with each screenshot as well, which I find really great. So we set up the GitHub's discussions, which is actually super useful for this. But bonus, and I might not have told you guys this, I also set up a section to share some config files and discuss them as well. Great idea. Excellent. And also some resources that people are finding really handy. So there's some activity happening there. I think, you know, if you're
Starting point is 00:38:45 familiar with GitHub, that's a really natural place to have these discussions. If you're not familiar with GitHub like me, great way to learn. This is just like the perfect opportunity if you've ever been tempted by the idea of trying out Nix. We're doing it here with you. The goal is pretty doable. You just got to get HTOP going. And we've got
Starting point is 00:39:01 the Matrix community and the GitHub as resources. And the people in the Matrix community have been so awesome about answering questions and helping people figure out and get their heads wrapped around this. Yeah, I've been really impressed by that. So good.
Starting point is 00:39:13 It's a real nice thing. And if you're not sure if you want to try out Nix, just ask yourself, would you like a totally deterministic Linux box? It's always this way. It's a static thing.
Starting point is 00:39:22 When an update changes, it rebuilds and it tests. And you can roll back if things don't work. Think about how powerful that could be in a production environment. Like I'm picturing studio machines running NixOS that's rock solid. If some update comes along and it screws something up before a damn show, we can just roll back and just kick it and do the show and not worry about it and fix the problem afterwards. I like that you're still not willing not to do updates right before the show. You're just trying to plan around that.
Starting point is 00:39:48 No, I'm doing updates. There's a red dot in the system tray. I got to do the update. It's got a number on it. It says there's a bunch of them, Wes. I can't not do the update. What am I, an animal? And so from time to time, we're going to do the check-in.
Starting point is 00:40:01 It is a month-long challenge. So really, I mean, come on. You can spin up a VM. You can go get a Linode going. Just try it out. Try it out. And then let us know how it goes. And we'll kind of check in from time to time.
Starting point is 00:40:11 Because we're going to be traveling next weekend. I mean, we'll probably be still talking about Nix. It's like all, if you look at, so we all collect links throughout the week for this show. And literally, the only thing we collected this week was Nix links. And some Matrix links, obviously, because we were dealing with Matrix. But it's like we are just obviously going to be thinking about this and talking about this, but we'll check in. You know, and just to point out, too,
Starting point is 00:40:33 if you don't want to go the full Nix OS yet, you don't have a spare system, you don't have a VPS or a VM around, you can install Nix on any old Linux box you got. Just start with that, and you can install Htop from there. Could just be a great way to manage software on a distribution that you already use. Maybe a new way to solve that problem. So the next challenge has officially begun.
Starting point is 00:40:54 Get the details in the show notes. We do have a bit of feedback, though. Mr. Brent, do you want to kick us off? Sure. I figured we could talk about NixOS feedback that we got. I've been sort of watching the chats go by, especially in our Matrix channel, and I thought I would grab a few that were interesting. So Nivoyu suggested, honestly, it's everything I hoped NixOS would feel like.
Starting point is 00:41:17 This is exactly how I want my systems configured, because it feels like one of those Arch bootstrap scripts, but I actually know exactly what's going on here. That was pretty funny and very good feedback. It feels like Arch in a lot of ways, but then when you tell the Nix package manager to go do its thing,
Starting point is 00:41:36 and you're like, I really haven't told it much. I just told it I want a system. And then it just goes and it builds a system. Or even with NixOS, you just say like, yeah, I need this service to be running. Here's a little config for the service. And then it figures out, oh, well, you're going to need that package. You're going to need a a system. Or even with NixOS, you just say like, yeah, right, I need this service to be running. Here's a little config for the service
Starting point is 00:41:46 and then it figures out, oh, well, you're going to need that package. We're going to need a startup system to you and we're going to... Dear NixOS
Starting point is 00:41:51 or dear Nix Package Manager, I want containers and then you tell it to go and you end up with a system that can run Docker and Podman, you know, or you could do just a Docker
Starting point is 00:41:59 or a Podman system. I also found with the package caching that it's really speedy whenever you make any changes, even quite large changes. So I thought that was a nice experience too. I see Josh wrote in. You want to take that one, Mr. Payne? Yeah, Josh
Starting point is 00:42:11 writes in, quote, I've already been hacking on this config and learning the ways of Nix because I've determined this is my dream distribution. Whoa! Strong words, Josh. I like it. We are in the new distro energy phase right now. Like, it is strong right now. Like, wow. Although, in the new distro energy phase right now. Like, it is strong right now.
Starting point is 00:42:26 Like, wow. Although, I got to say, we're like three weeks into it. So it's like sticking around. And, you know, part of the goal here, too, is to figure out where the limitations are, too, right? We get it. Like, when you adopt a weird system that's out of the mainstream that does things differently, that, you know,
Starting point is 00:42:41 needs some workarounds to make some things work, that's going to be part of the experience. And I'm interested in like where that falls and how much that is and how hard it is to kind of figure out those workarounds. How does it feel in practice day to day? And now, as the French say, it is time for Le Boost. Merci à tout le monde pour Le Boost. This week we got user 406, who got in early.
Starting point is 00:43:09 5000 Sats says, I can't make it live on Sundays, but wanted to send you a boost so you know I was out here. Had to get into Bitcoin for this. Here's hoping to see you next Sunday one day. Thank you for getting into Bitcoin. You know, what I feel like is there's going to be a lot of people who are just trying this out for the first time
Starting point is 00:43:32 to send us a boost and I'm all about it because I think we're leaving people with more options down the road. And I still recommend if you're in the States or a couple of other countries,
Starting point is 00:43:42 just grab the Strike app. Super easy to get started. Really painless. And you know what, user 406? Whenever you do get that Sunday day off, we'll be here waiting for you. That's right. Every Sunday.
Starting point is 00:43:54 Well, Toxel boosted in with 800 sets, writing in about LUP 451. I'm a fan of JB's show since mid-2012. Wow. I've never written any feedback, but Chris's speech in the last episode inspired me to try out the Fountain app and write up a boost.
Starting point is 00:44:13 Thank you for 10 years of the best tech podcast content out there. And by the way, the sets I'm sending you are from the OSM BTC.org donation fund. It's a nice way to earn some sats while contributing to the OpenStreetMap project. Wow. Wow. That is fascinating.
Starting point is 00:44:30 Thanks, Toxel. That's so great. You know, it's one of the fun things about these boosts, too, is I feel like we've been hearing from folks that we've never heard from. And we definitely hear from people we have heard from, too. It's an interesting new channel, by the way. Crashmaster18 sent in a thousand sats yesterday for 451.
Starting point is 00:44:45 Thanks for the Fountain FM recommendation. I got the Bitcoin Lightning set up. I like the boost idea. I use the Cash app to fund my Fountain wallet. I also joined the Matrix using Element Easy Peasy. Thank you. Oh, nice. I feel like that needs a bell. All right. You know what? One bell. Something shifted in my head at really the beginning of the year, and I really started thinking more about
Starting point is 00:45:09 pushing decentralized stuff and stop just no longer really trying to push centralized services onto the community. And I think that's what I like in part about the Lightning stuff, and that's part what I like
Starting point is 00:45:18 about Matrix. It's decentralized. It's self-hostable. I think it's the future. Cardinerd wrote in with a thousand sats, been using NixOS as my main workstation for about six months now, so
Starting point is 00:45:30 longer than we have. I'll never look back. I really look forward to seeing how y'all feel about it. Also, Raspberry Pi plus Nix works like a charm. I will say, I think ultimately, I am very, very excited about trying that for my home network setup. Now that I'm thinking about redoing the home setup and we're getting into Nix,
Starting point is 00:45:51 I'm just thinking, how great could that be for the home setup? Right. You know, Alex divulged to me that he has a few pies laying around that he's not using right now, so maybe we can do a bit of testing when we're out there next week. No name with 500 cents boosting in yesterday on LUP 451. Podfather recommended your show. And yeah, best Linux show I've heard.
Starting point is 00:46:12 There you go. We got a recommendation from the Podfather. That's gotta be serious. Check that off the list. Shout out to Adam Curry. Respect to the Podfather. And check out his podcast with Dave Jones, who we've had on the show before. It's Podcasting 2.0. That's the name of it. You can find it over at podcast.index. It's a great podcast if you are interested in maybe where podcasting could be going.
Starting point is 00:46:32 They do a weekly show. The tools that they are building, the open source tooling that they're building, and how they want to keep podcasting decentralized. It may be a little weird. That's all in their show. It's a good show. Limiting Factor wrote in with 13,500 sats yesterday. He says, what a pain in the ass this was to set up, lol. But much like everything we do in our community, it's overly complicated for its own various reasons and yet totally worth it. Amen.
Starting point is 00:47:04 I do appreciate it. That's kind of our style. It is early. Like this is one of the things we thought about with Matrix and The Boost and, you know, even when we launched the shows, guys, seriously, even when we started doing the shows 15 years ago, it was like, we're kind of appealing
Starting point is 00:47:18 to a lot of early adopters. We want to make shows for everybody, but we want to make sure we're also making content for early adopters because that's us. And we want content that's made for us too. And we don't want just like willy nilly, like hype men, early adopters. We want people that think about this stuff deeply, that try this stuff, that build this stuff, that work with this stuff. That's why we're running our own node. That's why we're running our own matrix server, right? Because we want to get hands-on experience with this stuff and truly take advantage of the fact that it can be self-hosted because if people like us won't do it, who will?
Starting point is 00:47:50 Really no one? Yeah. There's got to be an audience to use this self-hosted stuff or else they're just going to stop building it for folks. And that's part of my thought. And if we can encourage people out there to also have fun with the technology, because it is a lot of fun, and also create a bit of a demand for the self-hosted technology, for this peer-to-peer technology, for this decentralized technology, for open source technology. Well, and the more of us that are doing it, you know, as we've seen just with, you know, with the next challenge, with the self-hosted show, just the more of us that are interested in it, the better we can all help each other. I have a fun pick for us before we get out
Starting point is 00:48:23 of here. What sneaky last minute pick. This is wild, man. So I'm going through my notes because I heard from the developer of Userland. It is an app that's available for Android that lets you run Linux on Android. Like Ubuntu or Debian or Kali Linux. You don't have to root your device. You just essentially get a fast local VNC connection to theux that's running in a self-contained app vm it's so cool and i actually i tagged this for us to to feature this as a pick three years ago wow and then i never it's still going yeah
Starting point is 00:48:58 that's a great sign right like i went to check on it i'm like well surely it's dead now no no it's still going and i had a chance to talk with the developer it's'm like, well, surely it's dead now. No, no, it's still going. And I had a chance to talk with the developer. It's open source. It's GPL. It's up on GitHub. We'll have that link. It's available in FDroid as well, which we do like seeing that because there's just not enough stuff in FDroid.
Starting point is 00:49:15 There's good stuff in there, but not enough. And so it's really nice to see that. You just install this like a regular app and then you can run a full Linux distro on top of an Android device. regular app and then you can run a full Linux distro on top of an Android device. They also, if you check, once you go to that link, if you look at their Play Store apps, this isn't really a pick, but this guy's like a bonus pick. They kind of use the same trick for two other things. So they have the whole user land for Linux distros, but then they have Foxbox which is a desktop Firefox. Oh, like the desktop version of Firefox
Starting point is 00:49:44 on your phone. Yeah, so you know like when you go to Reddit or other sites, you're like, would you like to see the mobile version? No, I don't. You don't get that when you use the desktop version of Firefox, right? Plus you get all the Firefox extensions and stuff. Yeah, sure. But then the other one that actually feels like
Starting point is 00:50:00 maybe this should be a million-dollar product is called CraftBox. And it's a VM that fires up a Linux machine and just runs a hosted Minecraft server on an Android machine. So maybe you got a fancy phone, you got a tablet you're not doing anything with. You could actually turn that
Starting point is 00:50:19 into a local Java edition Minecraft server using this. And all of this is GPL3. So wanted to give a shout out to that. My Android phone can finally be useful. Great. Chris, it's so interesting that you brought this up and didn't tell Wes or I about it because literally yesterday I was playing with an application. I guess there's another bonus pick called Termux that you can install via FDroid.
Starting point is 00:50:42 And I wanted to simply do some rsync backups to my phone and play around with what I could do with some old phones. And this is so the same line of thinking, and it makes me smile that somehow we always seem to get to the same place at the same time. So, thanks for that. That's great. Of course, link over at
Starting point is 00:51:00 linuxunplugged.com slash 452. Alright, gentlemen, I think that's everything, yeah? I think we came to do what we wanted, and we got our self-support shoulder waiting for us. And Nick's expressions to write, I think. We have some Nick's work to do. We got a whole matrix system to rebuild,
Starting point is 00:51:15 and rebuild it better. You know, we're going to call it the million-dollar matrix system. Like the million-dollar, you know, you get the reference. Of course. Yeah, of course. We have the technology. If you'd like to join us live and participate in our mumble room or chat room or just listen live because it is a different show, well, you can do that on Sunday. We do it at noon Pacific over at jblive.tv.
Starting point is 00:51:33 And we do have the calendar linked at jupiterbroadcasting.com slash calendar where you can get it in your local time. See you next week. Same bad time, same bad station. Of course, don't miss Linux Action News. We did a roundup of a lot of things coming to Linux 5.18. Plus, we talked about the new Rolling Rhino, a 1.2 release. Mix, remix, unofficial. We got the details and more over at LinuxActionNews.com.
Starting point is 00:51:59 Be sure you go grab that. Of course, our members feed. You get the full live production or an ad-free. It's your choice. You can support the show at unpluggedcore.com or support all the shows on the network at jupiter.party and get all of the bonus content. That's how you do it. And links to everything we talked about on our website, including our contact page, linuxunplugged.com. And we'll see you right back here next Sunday. so i'm uh i'm trying to find a spot to like take my son camping for like his 13th birthday but i i'm out of ideas.
Starting point is 00:53:06 Like we've been everywhere. And I'm not saying that to brag. It's like I really can't think of anywhere to go. And I promised him like a trip. And now I'm just coming up blank. Has he been out to the Olympics? That was what I thought. I suggested the Olympics to him.
Starting point is 00:53:17 And he's like, oh, I don't know. I kind of like to be somewhere in the woods. That's like that's a rainforest. Yeah. That's like the most woods you can get. I need to find like a great spot and suggest it to him is what I got to do. I've been thinking about like, where would I want to go if I were his age? It'd probably be like swimming, hot tubs, you know, hiking.
Starting point is 00:53:35 He loves all that kind of stuff. Hey, he can have breakfast in a hot tub here if he wants. I want breakfast in a hot tub. A journey to visit Uncle Brent. There you go. I've got woods. And we show up and he's got breakfast ready for us. I mean, while you guys are going out east, you might want to swing down to Florida and catch a lunch.
Starting point is 00:53:52 There's quite a few these days. I really... Our initial plan, we were planning to swing down to Florida. That was initially the plan. I just couldn't make it work from like a budget standpoint. But the first draft did have us going to Florida for a bit because I've never been other than just like staying over at an airport or a hotel for a night to then fly out somewhere else. Like that's my extent of Florida travel. Yeah, but in super secret, huge Disney guy over there.
Starting point is 00:54:20 Yeah, it's a magic kingdom after all.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.