LINUX Unplugged - 452: Synapse Collapse
Episode Date: April 4, 2022How we nearly crashed our Matrix server; what we did wrong and how we're fixing it. Plus an update on elementary OS, GNOME's next chapter, and we kick off the NixOS Challenge. NixOS Challenge Goals: ... Study the Nix Expression Language (https://nixos.wiki/wiki/Nix_Expression_Language) Setup at least one Nix/NixOS system (https://nixos.org/manual/nixos/stable/). Install htop (https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/system/htop/default.nix). Join the Nix Nerds Matrix channel (https://linuxunplugged.com/matrixinfo). Post a screenshot in the NixOS Challenge GitHub (https://github.com/JupiterBroadcasting/nixos-challenge/). Complete all the above before the end of April. Special Guest: Danielle Foré.
Transcript
Discussion (0)
All right, I know we're getting a little too excited about NixOS these days,
but I'm really thinking we should switch to Linux from scratch.
Whoa, like build a ground up, like really learn something kind of thing?
Yeah, you know.
That's going to take a lot of time, Wes.
Oh, no.
I've got something for you.
Oh, yeah?
Yeah.
On our Linux just today, a giant bash script to build Linux from scratch automatically.
You know, it reminds me of that old saying, Wes.
It's my pedestrian.
It's how you use it.
Hello, friends, and welcome back to your weekly Linux Talk Show.
My name is Chris. My name is Wes. And my name is Brent.
Hello, gentlemen. Well, coming up on the show today, we're going to tell you about how we nearly
blew up our matrix server.
Don't worry. We saved it at the very last minute, right before the big stream last week.
We'll share what we messed up, what we learned, and how we're going to try to build it back better than ever this time.
You know, it's kind of one of those IT war story shows where we admit that maybe we made a mistake,
plus an unexpected circumstance comes along and it blows up our whole setup.
But we really do kind of have an idea of how to fix it.
And then after we tell you about all that, we'll kick off the next challenge.
Then we'll round out the show with some boosts and picks and a lot more.
So before I get any further, let's say time-appropriate greetings to that virtual lug.
Hello, Mumble Room.
Hello, Mumble Room.
Hello, yay. Hello. Hello, Mumble Room. Hello, yay.
Hello.
Hello, hello, hello.
Some old friends in there.
We have a whole good group in there.
Powerful Sunday crew.
Thank you for being here on a Sunday.
Now, I do want to mention, we are so close.
This is like your last minute notice.
The April 9th, 2022 Raleigh north carolina meetup is happening next weekend
as we record this uh meetup.com slash jupiter broadcasting for the deets cancel your plans
book your flights and come with an empty belly because there's barbecue place there is also a
brewery and do let us know if you're coming so we can warn the restaurant meetup.com slash jupiter
broadcasting it gets real close.
And then you won't hear us talking about it anymore.
It's going to be like this era of when we were talking about the East Coast meetups.
I'm just getting kind of emotional about it.
But yeah, I think I fly out like Wednesday midday.
Land and record a self-hosted.
Have you packed yet?
No.
I did find the luggage, though.
But I'm currently doing that whole like, could we get away with one carry-on for both of us?
Whoa.
Because the wife's coming.
You know?
And she's kind of an overpacker.
Don't tell her I said that.
I'm not.
She's one of those, you know, we might need.
Well, we might need.
You know what I'm saying?
So we end up with, like, a bag of stuff that we might need.
And, yeah.
I don't think it's going to be one bag,
but,
uh,
we'll see.
I'm,
I'm just thinking like today's going to be a great show because we've got
some good community news.
We're going to kick off the next challenge.
We got kind of a come to Jesus moment with our matrix server.
And no matter how bad we screw up today,
we got a double slow cooked pork shoulder going in the oven right now.
Wes brought like this amazing piece of prize meat.
Did you win that like in a fight or something?
Well, yeah, that's, that's also what explains the black guy, but I didn't,
I didn't really want to tell that story.
Okay.
That's fair.
That's fair.
But I'm looking forward to snacking down on that.
That's going to be really great.
Plus Wes brought us some ciders.
So we're going to just, it's going to be, doesn't really, doesn't really matter how the show goes, but we'll do our best anyway. I suppose. I suppose. Well, you promised.
All right. All right. Well, let's kick off the community news. Danny Foray joins us from
Elementary with an update. Of course, we've kind of been wondering what's going on over there,
Danny. So what is the latest? Yeah. Thanks for having me on. So just kind of in a nutshell, there's been some change-ups at Elementary Inc.,
which is the company that kind of supports and funds the elementary open source community.
So these days, it's just me now.
I'm the only employee and owner of the company.
So things are changing a little bit there.
It's been a little bit chaotic, but
we just want to make sure everybody knows that I'm here and things are okay and development is
still going on. I have kind of more of a lengthy dive into what's going on and what's coming on
our YouTube channel, which is linked from our website, elementary.io, for people who are curious
to find out more. Excellent.
We will also link to that in the show notes.
Wes and I got to catch a bit of that.
And, you know, I think you just, you said a lot of stuff there.
So I think a lot of the details are kind of pretty well covered.
I was really impressed.
You really kind of just laid it all out there
and you were really honest about it all.
So why don't we talk about like, you know, where things go now?
Because I think that's what we're all kind of interested in is what the future is and kind of what you might
see as short-term changes and maybe some long-term stuff. Like what's next? Yeah, absolutely. I
definitely want to focus on the future and what can be done better. And I think like something
that's been huge for me is kind of reevaluating like what is the purpose of the company and its
relationship to the community and not only just the elementary community, but the wider open source community.
And something that I really want to get back to that I feel like we used to do a lot more of
was more donating to upstream projects and more funding
for developers in the community. I think GitHub Sponsors is a really big
opportunity. So I think that's kind of the role that I
want the company to take is more
like a support role. That's going to be quite the process, right? Because there's probably not like
a ton of funds to go around, I would imagine. So how are you thinking that's going to work? Like,
is there going to be some of the income that comes into Elementary Inc. that's then going to go
out to different members that are working on projects that benefit the distribution?
Yeah, I haven't figured out exactly how to split things up yet.
And I definitely want to talk it over with our developer community and find out what
they feel is important to fund and how things should be split up.
But I really want to make giving a bigger part of what the company does.
And kind of with the downsizing that's happened, I think that we do have a bit more of an opportunity
to make it more so that we're pushing money out
into the community and not so much like
just funneling money into a for-profit company, you know?
Yeah, I follow you.
I do, I think that's a really good goal.
I wonder, are you going to,
is your life expenses going to be
covered by that? Cause you have to eat too. Yeah. Um, I don't think really that I'm going
to have to change anything about, um, my like employment status or anything with the company.
We kind of already made some like, uh, budget cuts and stuff towards the end of the year,
like as a response to COVID. So I think like where I'm at right now,
like I'm comfortable and I'm good.
And I think that there's room to do more.
Great.
That's exciting.
So keep us posted, right?
Keep us informed on what's going on.
Keep us up to date on what the new developments are and come on here and share with us
when things are moving and shaking.
Yeah, absolutely.
And if people want to follow along
in like more real time,
like elementary.io is always the place to go. We got a link to our blog there and social media accounts.
Very good. Well, thank you, Danny. Thanks for taking a bit of time to update us and good luck
on everything. We'll be checking in from time to time and trying out new releases as they land and
all that kind of stuff and looking forward to the future. Thanks. All right. We will have links in the show notes at linuxunplugged.com slash 452.
Why don't we stay in the ecosystem just for a minute?
I noticed that Chris Davis did a blog post.
Chris Davis, a.k.a. Brain Blasted in the GNOME community.
Chris calls himself a GNOME developer who wants to improve the GNOME ecosystem as much as possible.
Got to appreciate that yeah currently works on fractal matrix app for gnome which i only laugh because
we've been talking about it behind the scenes a lot recently but i don't know if any of us have
tried it no yeah it's like next on our list uh and a few other great apps too like gnome weather
and a couple others you've probably heard of but uh chris did a post about plans for GNOME 43 and beyond.
And I really like some of the stuff that they're talking about here. So we've seen the arrival of
LibidWada. I'm sorry, LibidWadia. We've got to make sure I get that right. And that really
created a platform, you know, a library that developers could use to create GNOME looking
applications. But it also makes it simple to implement some stuff we've been wanting for a long time, like customizable accent colors, right? Global accent colors that can be opt-in for app
developers. Yeah, this is looking really good. They have some screenshots here. And I think,
Wes, the change here is, and you probably, you could probably tell me if I'm wrong, but I think
the real shift here with this is like this stuff that was being done before with CSS changes is now getting done via an API.
Yeah, I guess developers using the recoloring API will be able to programmatically change colors in applications and then have dependent colors update automatically.
So you can create presets which can be used for, say, to recolor the window based on a text view's color scheme that you're using somewhere else, right?
So the whole window can match whatever you're choosing for your text view.
And I've also seen some suggestions,
like maybe something's in an administrative state,
so you make the window red.
And this API gives you, I think it gives you,
I don't know about more control, I can't really speak to that,
but I think it gives a better end result.
And Chris Davis himself has been working on Loop, which is, get ready for this, Wes.
It's an image viewer, but it's written in Rust.
Hey-o!
That's right.
And GTK4 using LibidWin.
I don't want that dirty C code to touch my precious JPEGs.
Oh, of course not.
No, you need to render those JPEGs as fast as possible and with as much memory safety
as possible.
You know what, though?
We only bring this up because doesn't it feel like
we're seeing more and more stuff for GNOME written in Rust?
I mean, it's almost like a meme at this point.
Yeah, I guess there's some getting to be a battle-tested Rust GTK
sort of friendship, I guess.
Yeah, yeah.
I like the direction of this stuff, I have to say.
One of the things
that Chris also wants
to work on
is the
XDG desktop portal
file pickers.
Like,
how do I say it?
So, see,
you got a Flatpak application.
It's sandbox.
It's limited right now
in how many files
it can open
that are associated
with a file
that you want to open.
Another way to say that is,
let's say you saved out
an HTML archive
and you've got an HTML
file with some images and CSS and stuff like that. And you open that with a Flatpak application.
Well, you'll maybe be able to open the core document, but all of the associated files,
like the CSS file and the images folder, none of that is going to be accessible by that Flatpak
application right now. Right. Or, you know, often you have stuff that has, like, a project file,
and then all the resources for that project are kept, you know,
next to it in the same folder or something.
Right, I'm sure that's super common for IDEs, right?
So Chris wants to fix that.
He wants to kind of just change the way that works and make it possible,
just tweak the portal so that way it's possible for that to actually not be the case.
He also calls out some accessibility fixes.
And the reason why I'm mentioning this is I feel like we have an opportunity, for that to actually not be the case. He also calls out some accessibility fixes.
And the reason why I'm mentioning this is I feel like we have an opportunity here as a community
to help Chris because he is currently looking for work.
If you know someone out there that maybe is looking for an employee
that would fit these skills,
we'll have a link in the show notes.
Additionally, Chris has a link to Patreon, GitHub sponsors, and PayPal.
And there he's just looking for some help to set time aside for these issues. And
what Mr. Davis has done here is essentially zeroed in on what I think are issues that affect
real people that use the GNOME desktop. These are sometimes things that we criticize Gnome about is focusing
on things that
don't apply to
the actual users
that are using
the desktop and
maybe they're
targeting that
elusive new user
or they're targeting
that elusive mobile
Linux user.
And I think, you
know, you see some
of that what Chris
is talking about.
He definitely wants
to improve some
stuff to work
better on mobile,
but I really don't
have issue with that.
Make it work great
on both.
Okay, I'm on board with that. Yeah, right. That's a good compromise. So I just wanted to call attention stuff to work better on mobile. But I really don't have issue with that. Make it work great on both. Okay, I'm on board with that.
Yeah, right.
That's a good compromise.
So I just wanted to call attention to this work.
It's a reminder to us that the GNOME community
is made up of a lot of viewpoints.
And, you know, I think as we covered
this Libid Way to stuff additionally,
we learned that a lot of times
you'll see individuals speak on GNOME's behalf,
but it's actually made up of a lot of individuals with different opinions.
And so we're starting to see a diversity in those opinions now.
And I just think that's a great thing.
And I like this goal set here.
I'm getting kind of low-key excited about the slow progression GNOME's going in.
There's always a couple of trade-offs we make,
but ultimately, after they change something and then they iterate on it,
I think the users end up for the better.
I mean, we've gotten to a pretty nice place, and
I'm excited to see where it goes. And it's just interesting here
to hear someone who clearly is making
some changes that we like, who has shown
skills in doing this work that's saying,
if you can offer me some more support
community, I'm willing to keep doing this
and to do more of it. And so
it's time to reflect on that and what we want. And Danny, before we jump off of all of this,
do you have any thoughts on LibidWadia, how it may affect elementary OS and that stuff?
Because that's sort of your neck of the woods to the whole GTK area, even though you guys
don't have GNOME Shell, you're pretty reliant on the GTK stack. And I got to figure you're
looking at this. Yeah, I mean, so far in porting to GTK4,
the changes that were made here have been extremely positive. Like it's a lot of GNOME
design language stuff is no longer in GTK4 and it's in libabuida. So this is a huge win for
desktops like us or for XFCE that have our own design language and we don't have to kind of be
pulled along behind the GNOME boat. But for the GNOME community, I think this is also a huge win for their app
developers because GNOME has never really had an official supported theming API. And with Lib
Adweighted, now it does. Yeah, I think it's an adjustment, right? It's not a downgrade. It's not
a bad thing. And I think initially maybe was seen as that. It was seen as the end of theming.
And it's just a change in how theming
is done. And I feel like it
is a new chapter in the GNOME desktop.
We're just arriving
there. It's the very beginning. It's the very
first chapter. But we're just there.
We've just turned the page. It's exciting.
42 is a great release. 43 is
going to be a great release. And I'm excited to see how
distributions like elementary OS and others that are using all those tools embrace it. 42 is a great release 43 is going to be a great release and I'm excited to see how distributions
like elementary OS
and others
that are using
all those tools
embrace it
Linode.com
slash unplug
go there partner
and get yourself
$100
in 60 day credit
on your new account
and you support
this here show
I don't know
thank you
I don't know
I don't know
but maybe
maybe that guy
could do the ad reads
from now on
right
he seems pretty excited.
I'll tell you why he's excited about Linode.
He's been using it for a while.
You know, he runs his business infrastructure on there.
And so he knows you can trust your business infrastructure on there.
But he's also got his kids game servers on there.
So like you want to set yourself up like a safe Minecraft server or maybe you want to
get yourself, you know, I don't know, a little CSGO action.
I don't know what you do.
I don't know. I little CSGO action. I don't know what you do. I don't know.
I'm not going to pass judgment.
I'm just saying go to Linode.com slash unplugged, get $100 in credit and support the show.
And they got 11 data centers around the world.
And I'll tell you, I don't always pick the right data centers.
And I'm going to own up to that in a little bit.
But you can.
You can learn from my mistake.
I'm going to use Linode the wrong way so you don't have to.
Also, Linode's constantly upgrading their
infrastructure. They got these
super fast connections. It's embarrassingly
fast coming in. 40 gigabit connections coming
into their machines. And they
are their own ISP. Then they got
new MVME PCIe storage. And I just learned
recently, Wes, a couple of our machines, we get free upgrades.
What? I thought we were going to have to pay more.
Wait, free upgrades?
You go in there, and if you qualify,
you look at your disk, and they just got a little
button. You press that button, and you
just get it. Wow. Maybe I shouldn't
be saying this. Maybe it's not true. I don't know. But it is
online, and I'm pushing that button. Let
me tell you. The button says upgrade, yeah. I'm pushing
that button, Wes. I love that button.
And I love their object storage. This
is my new, like, I don't know what I'm going to do with all this stuff, but I don't want to store it on this particular machine button. And I love their object storage. This is my new, like, I don't
know what I'm going to do with all this stuff, but I don't want to store it on this particular
machine. So I put it in object storage and that's been working so good. Everything gets its own
addressable URL. It works super easy with their upload or their CLI client that I use to upload
it because their CLI client's easy too. The whole thing's ridiculously simple. You don't even have
to be like a server expert to figure this stuff out. But if you are, you're going to be impressed with the power that they let you get to if you need to.
I'm talking like they'll even give you guides to like take one of their rigs, wipe off their image, and put NixOS on there if you're that kind of maniac.
They'll do it with you.
I've done it before.
Don't threaten me.
I'll do it again.
I'll do it.
Don't challenge me on this.
I'll tell you what.
Go try it for yourself and see what we're talking about.
Go see why Wes spins up his stuff over there.
He's spinning up servers all the time.
You'd think he's like some sort of DJ spinning the tracks.
Yeah.
I don't listen to music.
I just listen to that WebUI.
That's right.
That's right.
And that awesome, what do you have keyboard?
The coder keyboard or whatever it is?
Yeah.
Code keyboard?
Code keyboard.
Yeah, that's right.
I'm sitting there on calls with Wes.
Clack, clack, clack, clack, clack,
as he's setting up the Linodes.
It's like how I know he's doing something, actually.
But, you know, it's a great thing
is we all can use it, right?
Wes can over there, you know, he's fine.
He's using it.
But, you know, I can stumble my way through it too
and act like I'm an expert as well
because it's pretty straightforward
and they got great documentation
and just the best damn customer service in the world.
That's what I hear all the time from the audience. So go try it out with $100 and support the best damn customer service in the world. That's what I hear all the time from the audience.
So go try it out with $100 and support the show.
Linode.com slash unplugged.
That's Linode.com slash unplugged.
Just about two years ago,
we launched the Jupyter Colony Matrix server
as something of a living experiment, let's say, here on the show.
And since we spent the last week pulling that same server back from the brink,
we thought it best to share what we screwed up, what we've learned, and there's a lot,
and how we're going to try to build things back a little better.
That's right.
We did mess up.
We messed up bad.
I acknowledge it.
Sometimes we get these wild ideas.
Yeah, maybe a little over our head.
And we think, what could go wrong?
And then we realize we've stumbled into something that is old school complicated.
Not new school complicated, old school complicated.
That's all right.
That's my power zone.
I can deal with that.
I didn't realize I was signing up to manage an exchange server,
but that's what I'm doing basically.
If you're an old IT guy, you know what managing an exchange server is like.
That's what managing a Synapse server is.
So maybe some terminology,
just so that way we're all on the same page keys right now.
Matrix is actually the protocol, like HTTP is, right? So you've got Matrix, which is a
protocol. Element, the chat client, is like a front-end. It's like your web browser. There's
multiple front-ends. Element's probably the most feature-rich. And then there's Synapse. Synapse
is the back-end matrix server that we're using. Dendrite's another one. Is there other? I don't
know. But Synapse is the one that's deployed at any kind of scale. Yeah, you know, and that's kind of been the one that the team behind
Matrix have been using is like the sort of primary implementation development stuff. And, you know,
Dendrite's a new implementation, but hasn't necessarily caught up on all the features,
which kind of right now, after they go through the, you know, the RFC kind of process, then
they're kind of usually first implemented in Synapse. And I guess the sales pitch,
the short sales pitch for Matrix, if you're not really familiar with it, is it's a decentralized federated chat system that supports direct encrypted communications and public chat rooms.
And it features a lot of the same kind of nice-to-haves that your Mattermost, your Slacks, your Discords have.
You can have a server with multiple channels.
You can paste a Gif or a gif
i don't care what you call it you can paste it in there it'll display it you know image previews
kind of the rich like chat features you get from a modern chat system they're all in matrix a lot
like how discord does it but one that you can self-host and one where you could say create an
account on matrix.org and then join our matrix server where you'd find all our rooms and stuff.
It's great.
Yeah, you know, I mean, instead of a sort of proprietary implementation
or even just a particular implementation here,
you've got something that has a generic protocol and has a spec
and then can have a plethora of clients.
And yeah, you've got that old school internet feeling like with email
where you don't have to go through some gatekeeper.
You can just say, yeah, I want to have a home server. And then you can start participating
with our home server, with the main one, with any of them. Yeah. And I want to make it clear.
What we are about to tell you would not be a problem if it's you and like a handful of people
using a matrix server. You could probably run it on a Raspberry Pi and you'd be happy, right? What we're about to
tell you is a journey you go through when you start with like a hundred users and then you got
nearly 3,000 users and just like all of the things that change when that happens. That's what we're
going to talk about today. So our initial deployment was a bit anemic. It was basic. You know, we were
just really wanting to get some hands-on experience so that way we could talk somewhat intelligently about it on the show.
I thought we had so much to learn, but we initially deployed a four-core Linode box
with eight gigs of RAM and like, you know, a fair amount of, I don't know,
150 gigs of local SSD. It wasn't a lot. Yeah. Again, right? Like when we just did it kind of
simply because we'd been talking about matrix, we obviously thought it was cool
and it felt like, okay,
well, if one of the things is about that,
you know, that we highlight about this
is that it's open source
and you can run it yourself,
we better run it ourselves to just try it.
So we just went with like a super simple
Docker compose setup,
which was just two containers, right?
Like we got Postgres for the database
and we've got the synapse container,
just the, you know,
the one right from stock upstream
running the main Python application. This is the good and bad about things in containers too it's
not about the technology it's like it does enable you to quickly get up and get running but it's not
necessarily configuring the most optimal situation for a high demand production use and you just have
to kind of be aware of that uh and you know this was fine two years ago when we maybe, you know, after a year had 300 people on there.
It was getting a little bit slow,
but as long as Wes logged in from time to time
and made sure the disk wasn't filling up, we were okay.
Yeah, you know, did our updates.
I got to say that really like the upgrading process has always been nice.
They've got great, you know, release notes that kind of call out
if there are things that they're making breaking changes. And other than that, you just update the tag on the Docker image and away you go. So that part's been nice. They've got great release notes that kind of call out if there are things that they're making breaking changes,
and other than that, you just update the tag on the Docker
image and away you go. So that part's been nice.
Yeah, and solid updates.
They're almost always introducing
something that Wes and I are like, oh, that's worth an upgrade.
They've been really good about that.
And Dendrite's moving along really well, too, but
Synapse legitimately has been
making solid improvements nearly every time we want
to do an update. So it really hadn't been too bad.
But over the last 10 months or so, the Matrix colony has grown.
In fact, I would say even kind of like at the beginning of the year or so, I wasn't really sure, like, are we going to keep running this Matrix thing?
It's kind of becoming a pain in the butt.
I think I groused about it on air once.
You know.
You know, maybe, yeah.
Well, we were just, that was the first steps of us realizing
that we either need to decide to like commit to this
or rethink our involvement with it
because it is, with the size that it had become,
it did take some resources.
I do like to take a think about it
because we kind of haphazardly
just implemented Discord a few years ago and i
think we're still kind of dealing with the consequences of that so i thought you know
we'll take it slow with matrix we'll see where this goes and then when it started to get to
this pain in the ass stage i thought i really haven't made up my mind i really haven't committed
to this yet you know where i actually think things really kind of changed to tell you the truth
is when the whole free node thing went down and then all of a sudden to tell you the truth, is when the whole Freenode thing went down. And then all of a sudden,
a lot of the projects that we like to communicate
with started popping up on Matrix.
And I think
that particular instance also
made people realize how nice it
was to have something that could be decentralized
so projects can run their own instance
but still participate in
the overall network.
I'm on a Red Hat instance, I'm on a GNOME know, I'm on a Red Hat instance.
I'm on a GNOME instance.
I'm on a bunch of other projects' servers,
but I do it from the JB server.
And then it just kind of started accelerating.
And in about three months or so,
it just really kind of started growing a lot.
It picked up, and that's awesome.
And there's a really cool decentralized community growing on there.
And as a result, I've been spending a lot more time on there.
And I realized this is there's a lot of really good conversations.
People are kind because it's still in that kind of phase where it's not huge, you know, but it's also big enough that there's always some conversation going on.
And then that room aspect of it kind of breaks things up.
Yeah.
Yeah.
You know, we've got there's different communities that can all kind of breaks things up. Yeah, yeah. You know, we've got, there's different communities
that can all kind of flourish there,
but it still has that,
you have to opt-in kind of functionality
because, you know, you might already have Slack or Telegram
because of your friends or for work or something,
but if you don't do an open source things,
you probably don't yet have elements.
So you have to want to be there.
And it's, you know, not a great process to get on board,
especially if you want to onboard directly to be there. And it's, you know, not a great process to get on board, especially if you want to
on board directly to our server. You have to like change defaults in the on the login screen. It's
just not like as great. And I know it's a little confusing, right? Like if you're not familiar,
you have to figure out what a home server is and then figure out which one might be right for you
or where it makes sense to go do that. Right, right. Yeah. Do you want to host your own thing?
Do you want to just join matrix.org? Do you want to set up an account on the JB system?
Right.
Now, it's kind of like when you start with Linux,
you got to pick a distro,
and sometimes people just get stuck there.
But over the last few months,
it seemed like people were moving over.
They were trying it out.
And then within the last,
literally like the last,
since the last time we did the show,
we realized the load was getting
way too damn high and then one morning i woke up to a telegram message that said i think the
matrix server is down i'm not sure if the matrix server is up can you check on that and sure enough
it was and i realized pretty quickly that we had used up all of the available disk on that system. Sure had.
That basically kicked off a series of nights of Wes burning the midnight oil to try to rescue the matrix server and him and I doing like emergency preparations to like, like
maybe migrate a server or like spin something up entirely and all this kinds of stuff that
we had to do.
Some of it was fun.
Some of it was a bit hectic.
entirely and all these kinds of stuff that we had to do.
Some of it was fun.
Some of it was a bit hectic.
But we realized, I think, probably what, by day two,
we're probably going to have to just totally rebuild the way we've implemented this
if we really want to scale it to like maybe 10,000 plus users.
Yeah, you know, we'd been kind of interested
in utilizing the new worker system that Synapse has added
so that you can kind of deconstruct it
from just this one monolith Python application
to have that work shared across different processes.
But that takes some plumbing
because now you're taking that system apart
and it needs abilities to be able to coordinate
between the workers
and you have to add Redis to the system,
none of which is super hard,
but it just has its own particular setup
and we're not developing this and we haven't gone down that path before. So it was just a new,
you know, several new docs pages to read and get familiar with while having it on a live system
that we were hoping to, you know, not totally break after we just got it back online.
And of course this all starts happening. We really realized Tuesday, this is really screwed.
The system's really screwed up now.
The problem was, it was down for a while,
and then when we brought it back up to sync with the rest of the federation,
essentially maxed the server out.
And then the other problem we discovered was that when a user would log in
and sync all of their rooms, a single login session of a single user
on a 2,000-user system would peg the system to 100%.
We also realized pretty quickly
that while we had thrown more hardware at this
because we thought that's going to solve the problem,
we'll just throw hardware at it.
So we upped it from four cores to six cores.
We went from eight gigs to 16 gigs,
and we sliced off another set of storage.
This is like the third time we'd sliced
off more storage at this point and we figured there that'll fix it but we discovered by default
synapse is kind of well it's sort of silly single threaded it's kind of amazing how single threaded
it is and so you can throw six cores at it but that doesn't matter because it's using one core
unless you do this worker thing.
So that was really our realization is, OK.
All right, let's get this thing to a stage.
That way it survives the live event on Thursday.
Yeah, because, of course, we just started to push people towards using the matrix server, right?
That was part of the live stream.
Let's get people to sign up for matrix.
And now here the matrix server has fallen over.
So you banged out like a couple of generic workers is I think what they're called yeah you know they can kind of just just do anything and
then there's some more support you can split things out further uh we haven't yet walked
down that path probably something we want to try uh still still more reading to do really but this
was just an attempt to like okay well yeah maybe we can split off some of the stuff you can do some
of the you know responding to the federation events and syncing there and then we can have
some other things that are handling and interacting with the clients,
just try to keep those a little more
responsive. And it seems like it's worked. It's not,
it's still not great. It's not where I think any of us
want the server to be in terms of responsiveness,
but I think it's been usable, at least.
Yeah, it's still a little slow,
but it's usable, and we want to work on that.
We have, and what we realized
too is sort of our hodgepodge approach
of adding disk sort of randomly at the machine, various like local disk and network disk and object store, all these random mixes of disks.
Well, we were constrained by the lowest common denominator.
So our disk throughput was essentially completely pegged.
We were maxing out our disk IO.
We were single threaded on the core.
And because we'd been down for a while because we had used up all the disk from storage,
it was a nightmare while that thing synced back up.
It just completely slammed the box as people were trying to sign on all of
that.
So we realized we got to do this workers thing.
We got to readdress the disk situation.
We kind of started making a list of stuff.
And also I'm told Wes that version three of the sync protocol is going to be
lighter weight and faster and it's close.
So we're going to get a little assist.
That's the nice thing, too, here, is they are making improvements
as we try to make improvements.
To their credit, they really have.
Like, I remember, you know, maybe six months or a year ago,
we would often have issues with the memory use of Synapse.
Just over time, it would slowly, between it and the way it was kind of using Postgres,
like, the system would just really eat RAM.
That doesn't happen anymore.
Yeah, that's true.
That has gotten better.
So that's nice to see.
Also, I got to admit, because this was a this was a test that we were doing for the show.
We deployed this system in California.
It's quite close to us, but maybe maybe not the rest of you all.
Not for an international audience.
Not the best location for it. Maybe not the worst of y'all. Not for an international audience. Not the best location for an,
maybe not the worst, but not the best.
So I think, you know,
one of the things we would do differently now
is we would deploy to a centralized data center
and we would kind of look at some of our options here.
And I think when you consider all the improvements
we could make for a large-scale implementation,
one would be better utilize the workers,
go even further with that.
Each major, like, functionality aspect
of the backend Synapse server
can be handled by a worker.
And some of them, it's like one worker
to handle, like, presence.
That's a worker, right?
But something like Federation Sync,
that's like 10 different workers you can set up,
each with their own individual config file
and NGINX
rules you may or may not have to set up.
Yeah, right.
Because you might want to start setting up NGINX load balancing or you use Regex to direct
some traffic to the worker that's going to handle that traffic.
So there's just a little, lots of moving pieces to reconfigure.
This is about the time I realized, you know, I should have just contacted the folks at
EMS and just said, hey, would you like to sponsor a few episodes and in trade just host
our matrix community?
Because I'd like this to be faster for people.
I'd like it to be a nice experience with Matrix.
And we are not experienced Matrix admins
or Synapse admins by any stretch.
We've just kind of realized
we need to take it more seriously.
And so we've thought, well, how would we build this better
if we were going to redeploy it?
And I think one of the things we're going to do
is experiment with the NixOS build,
kind of in theme right now, as a way to kind of learn the edges of this thing, to see if we could
build something with multiple workers out of the box and something that uses a centralized data
center with NVMe storage, right? Just one pool of big NVMe storage. And maybe something that
collects metrics out of the box, because you discovered
there's actually quite a bit of information
we could be getting from the system
to better understand what it's doing. Yeah, you can
have Synapse emit Prometheus
metrics, which is great. So from that on, we just need
to start sucking it up and throw it in
some dashboards. Yeah. So that's something
we want to play around with, and
maybe inevitably we decide,
okay, NixOS, this isn't the way to go.
Let's go with this community answerable solution
because that's pretty popular.
Yeah, definitely.
And I think even regardless if you use it
or use pieces of it,
it's an awesome store of community knowledge
around how to admin a Synapse box.
And a lot of the, you know,
it's got pointers to tools of like,
there's an admin web UI you can use.
It's got notes on database maintenance,
which is another area I think we need to spend some time either deciding how much history we're going to keep for some of these rooms or, you know, spending some time compressing past state events. There's just one of the things we run into was our database was getting larger than we expected.
And Synapse really needs it to be pretty, pretty fast if you want good latency.
Yeah, that's tricky.
That is something we'll have to figure out too.
And we want everyone's input on all of this.
We're going to collect your feedback.
We do have a matrix ideas room on our matrix server,
which will be linked in the show notes.
But if you go on there,
if you go on the colony.jupyterbroadcasting.com matrix server,
you'll find it.
I'm picturing maybe a future build.
Maybe it's NixOS based.
Maybe it's not.
It's NVMe storage.
It's centralized data center.
And maybe it uses object storage for, like, the image files and video files that people upload to the Matrix chat.
Yeah, that seems pretty reasonable.
I could see something like that.
Just something that we could build that we know could run for a while.
I think six cores and 16 gigs of RAM so far has been sufficient.
I feel like once we optimize the Synapse configuration,
that's plenty of hardware to throw at this.
Yeah, that was the other part too. You know, we kind of, we got
the default working Synapse. Like, oh, hey, look,
it works. It's doing federation rate.
Cool. And then we had
not done performance tuning. So we've also turned off
presence. Maybe we should never have had it on.
I don't know. People have thoughts about that too, maybe.
Yeah. Matrix Ideas Room.
That'll be linked in the show notes.
Go over there and let us know what you think
because we're kind of putting all this together right now.
The system's up, it's running.
It just doesn't perform at the level we'd like,
but we're tweaking it
and then we're going to eventually probably migrate it
to some new ultimate matrix server.
Right?
Doesn't seem like the plan.
Yeah, I think so.
And just another reason,
maybe the other part we're kind of experimenting with here is
maybe you want your own home server
if you're into that kind of thing, rather than
signing up for ours. That's true. And you won't
run into any of these problems if it's just you and
a couple of friends or just yourself.
You really
would not have any of these issues. It's just when you have
thousands of users and they're always on there
24-7.
And then, like a jackass, you hold a live stream where you tell everyone to go sign up for your matrix server the very week that it's collapsing under its own weight.
And then, of course, make us all have to work super overtime so that way we can at least get it working and functional so that way the live stream isn't a total disaster.
Which it still was touch and go.
There were moments the server barely made it.
Yeah. It was definitely
lagging there for a while. But hey, conversations
happened. And sats were given.
And sats were given.
Speaking
of that, that live stream
is posted at
xers.show slash 84.
Brent did the yeoman's work
of trying to quickly collect up everyone's lightning payment addresses and paste them did the yeoman's work of trying to quickly
collect up everyone's lightning payment
addresses and paste them in the doc. Nice work,
sir. I think we did okay. I think we
hit everybody, but hopefully
crossing fingers.
And then Wes had to turn it into
a battle when he tried to boost
people before I could boost people.
So the sats were flying.
I love this. We gave away 60 unique lightning donations,
usually around 1,000 or 2,000 sats each,
and we gave them to 42 different individuals
on the live stream.
32 new people, at least by our count,
signed up for our Matrix chat,
although the great thing is
those numbers continue to roll over the weekend.
They've gone up.
We featured an interview
with the Fountain FM developer, Oscar.
That's in the live stream. We also covered
some JB projects that the three of us are working
on. That's all in there. Extras.show
slash 84
if you'd like to participate in that.
And if you'd like to send us one of them boosts
that we talk about all the time, go grab a
new podcast app at newpodcastapps.com
and then get yourself
set up with something like Blue Wallet or Strike.
Grab a few sats,
stuff them in your
Lightning Wallet,
boost away!
That's how you do it.
newpodcastapps.com
All right, guys.
I think we've waited
long enough.
People want to know
about this NixOS challenge.
We're going to put the
details in the show notes,
so if you miss any of this,
hopefully we'll have it
listed there. But it is time to officially kick off the next challenge
with some goals. Number one, study the next expression language. Number two, set up at least
one Nix or Nix OS system because Nix is also a package manager available for your current distro.
Install HTOP using Nix and then join our Nix Nerds Matrix channel.
Come in there, tell us how it's going,
tell us how you're doing,
share all that kind of stuff.
And then last but not least,
post a screenshot of either HTOP
or for extra nerd points,
a full working graphical desktop
over on our GitHub,
which we will have linked in the show notes.
The idea is, the challenge is
to complete all of that by the end of April.
At the end of the month, we'll do a review of how it went for everybody, the moments
that stand out to us and what our plans are as a team to keep using Nix or not and what
we're going to deploy with all of that.
I challenge thee.
So you've been officially challenged and Brent's already all in.
You've created a GitHub for people to use where they can post screenshots and stuff.
Tell us a bit about that.
Yeah, we kind of put together some ideas to post a bunch of screenshots there instead of necessarily the Matrix room because then we could just review them all.
It's actually really fun.
And there are discussions happening with each screenshot as well, which I find really great.
So we set up the GitHub's discussions, which is actually super useful for this.
But bonus, and I might not have told you guys this, I also set up a section to share some config files and discuss them as well.
Great idea.
Excellent.
And also some resources that people are finding really handy.
So there's some activity happening there.
I think, you know, if you're
familiar with GitHub, that's a really natural place
to have these discussions. If you're not familiar with GitHub
like me, great way to learn.
This is just like the perfect opportunity
if you've ever been tempted by the idea
of trying out Nix. We're doing it here with you.
The goal is pretty doable.
You just got to get HTOP going. And we've got
the Matrix community and the GitHub
as resources. And the people in the Matrix community
have been so awesome
about answering questions
and helping people figure out
and get their heads wrapped around this.
Yeah, I've been really impressed by that.
So good.
It's a real nice thing.
And if you're not sure
if you want to try out Nix,
just ask yourself,
would you like a totally
deterministic Linux box?
It's always this way.
It's a static thing.
When an update changes,
it rebuilds and it tests.
And you can roll back if things don't work.
Think about how powerful that could be in a production environment.
Like I'm picturing studio machines running NixOS that's rock solid.
If some update comes along and it screws something up before a damn show, we can just roll back and just kick it and do the show and not worry about it and fix the problem afterwards.
I like that you're still not willing not to do updates right before the show.
You're just trying to plan around that.
No, I'm doing updates.
There's a red dot in the system tray.
I got to do the update.
It's got a number on it.
It says there's a bunch of them, Wes.
I can't not do the update.
What am I, an animal?
And so from time to time, we're going to do the check-in.
It is a month-long challenge.
So really, I mean, come on.
You can spin up a VM.
You can go get a Linode going.
Just try it out.
Try it out.
And then let us know how it goes.
And we'll kind of check in from time to time.
Because we're going to be traveling next weekend.
I mean, we'll probably be still talking about Nix.
It's like all, if you look at, so we all collect links throughout the week for this show.
And literally, the only thing we collected this week was Nix links.
And some Matrix links, obviously, because we were dealing with Matrix.
But it's like we are just obviously going to be thinking about this
and talking about this, but we'll check in.
You know, and just to point out, too,
if you don't want to go the full Nix OS yet,
you don't have a spare system, you don't have a VPS or a VM around,
you can install Nix on any old Linux box you got.
Just start with that, and you can install Htop from there.
Could just be a great way to manage software
on a distribution that you already use.
Maybe a new way to solve that problem.
So the next challenge has officially begun.
Get the details in the show notes.
We do have a bit of feedback, though.
Mr. Brent, do you want to kick us off?
Sure.
I figured we could talk about NixOS feedback that we got.
I've been sort of watching the chats go by, especially in our Matrix channel,
and I thought I would grab a few that were interesting.
So Nivoyu suggested, honestly, it's everything I hoped NixOS would feel like.
This is exactly how I want my systems configured,
because it feels like one of those Arch bootstrap scripts,
but I actually know exactly what's going on here.
That was pretty funny and
very good feedback.
It feels like Arch in a lot of ways,
but then when you tell the Nix package
manager to go do its thing,
and you're like, I really haven't told it much. I just told it
I want a system. And then it just goes
and it builds a system.
Or even with NixOS, you just say like, yeah,
I need this service to be running. Here's a little config for the service. And then it figures out, oh, well, you're going to need that package. You're going to need a a system. Or even with NixOS, you just say like, yeah, right, I need this service
to be running.
Here's a little config
for the service
and then it figures out,
oh, well,
you're going to need
that package.
We're going to need
a startup system to you
and we're going to...
Dear NixOS
or dear Nix Package Manager,
I want containers
and then you tell it to go
and you end up with a system
that can run Docker
and Podman,
you know,
or you could do just a Docker
or a Podman system.
I also found with
the package caching
that it's really speedy
whenever you make any changes, even
quite large changes. So I thought that
was a nice experience too. I see Josh wrote in.
You want to take that one, Mr. Payne? Yeah, Josh
writes in, quote, I've already
been hacking on this config and learning the ways of
Nix because I've determined this
is my dream distribution.
Whoa! Strong words, Josh. I like it.
We are in the new distro energy
phase right now. Like, it is strong right now. Like, wow. Although, in the new distro energy phase right now.
Like, it is strong right now.
Like, wow.
Although, I got to say, we're like three weeks into it.
So it's like sticking around.
And, you know, part of the goal here, too,
is to figure out where the limitations are, too, right?
We get it.
Like, when you adopt a weird system that's out of the mainstream
that does things differently, that, you know,
needs some workarounds to make some things work,
that's going to be part of the experience.
And I'm interested in like where that falls and how much that is
and how hard it is to kind of figure out those workarounds.
How does it feel in practice day to day?
And now, as the French say, it is time for Le Boost.
Merci à tout le monde pour Le Boost.
This week we got user 406, who got in early.
5000 Sats says, I can't make it live on Sundays, but wanted to send you a boost so you know I was out here.
Had to get into Bitcoin for this. Here's hoping to see you next Sunday one day.
Thank you for getting into Bitcoin. You know,
what I feel like is
there's going to be
a lot of people
who are just trying this out
for the first time
to send us a boost
and I'm all about it
because I think we're leaving
people with more options
down the road.
And I still recommend
if you're in the States
or a couple of other countries,
just grab the Strike app.
Super easy to get started.
Really painless.
And you know what, user 406?
Whenever you do get that Sunday day off,
we'll be here waiting for you.
That's right.
Every Sunday.
Well, Toxel boosted in with 800 sets,
writing in about LUP 451.
I'm a fan of JB's show since mid-2012.
Wow.
I've never written any feedback, but Chris's
speech in the last episode inspired
me to try out the Fountain app and
write up a boost.
Thank you for 10 years
of the best tech podcast content out there.
And by the way, the sets
I'm sending you are from the OSM
BTC.org donation fund.
It's a nice way to earn some sats while contributing to the
OpenStreetMap project. Wow.
Wow. That is fascinating.
Thanks, Toxel. That's
so great. You know, it's one of the fun things about
these boosts, too, is I feel like we've
been hearing from folks that we've never heard from.
And we definitely hear from people we have heard from, too.
It's an interesting new channel, by the way.
Crashmaster18 sent in a thousand sats
yesterday for 451.
Thanks for the Fountain FM recommendation. I got the Bitcoin Lightning set up. I like the boost
idea. I use the Cash app to fund my Fountain wallet. I also joined the Matrix using Element
Easy Peasy. Thank you. Oh, nice. I feel like that needs a bell. All right. You know what?
One bell. Something shifted in my head
at really the beginning
of the year,
and I really started
thinking more about
pushing decentralized stuff
and stop just no longer
really trying to push
centralized services
onto the community.
And I think that's what I like
in part about the Lightning stuff,
and that's part what I like
about Matrix.
It's decentralized.
It's self-hostable.
I think it's the future.
Cardinerd wrote in with a thousand
sats, been using NixOS as
my main workstation for about six months
now, so
longer than we have.
I'll never look back.
I really look forward to seeing how y'all feel
about it. Also, Raspberry Pi
plus Nix works like a charm.
I will say, I think
ultimately, I am very, very excited about trying that for my home network setup.
Now that I'm thinking about redoing the home setup and we're getting into Nix,
I'm just thinking, how great could that be for the home setup?
Right.
You know, Alex divulged to me that he has a few pies laying around that he's not using right now,
so maybe we can do a bit of testing when we're out there next week.
No name with 500 cents boosting in yesterday on
LUP 451.
Podfather recommended your show. And yeah,
best Linux show I've heard.
There you go. We got a recommendation
from the Podfather. That's gotta
be serious. Check that off the list.
Shout out to Adam Curry. Respect
to the Podfather. And check out his
podcast with Dave Jones, who we've had on the show before.
It's Podcasting 2.0. That's the name of it. You can find it over at podcast.index.
It's a great podcast if you are interested in maybe where podcasting could be going.
They do a weekly show. The tools that they are building, the open source tooling that they're
building, and how they want to keep podcasting decentralized. It may be a little weird.
That's all in their show. It's a good show. Limiting Factor wrote in with 13,500 sats yesterday.
He says, what a pain in the ass this was to set up, lol.
But much like everything we do in our community,
it's overly complicated for its own various reasons
and yet totally worth it.
Amen.
I do appreciate it.
That's kind of our style.
It is early.
Like this is one of the things we thought about
with Matrix and The Boost and, you know,
even when we launched the shows, guys, seriously,
even when we started doing the shows 15 years ago,
it was like, we're kind of appealing
to a lot of early adopters.
We want to make shows for everybody,
but we want to make sure we're also making content
for early adopters because that's us.
And we want content that's made for us too. And we don't want just like willy nilly, like hype men, early adopters. We want people that think about this stuff deeply,
that try this stuff, that build this stuff, that work with this stuff. That's why we're running
our own node. That's why we're running our own matrix server, right? Because we want to get
hands-on experience with this stuff and truly take advantage of the fact that it can be self-hosted because if people like us won't do it, who will?
Really no one?
Yeah. There's got to be an audience to use this self-hosted stuff or else they're just
going to stop building it for folks. And that's part of my thought. And if we can encourage people
out there to also have fun with the technology, because it is a lot of fun, and also create a
bit of a demand for the self-hosted technology, for this peer-to-peer technology, for this decentralized technology, for open source
technology. Well, and the more of us that are doing it, you know, as we've seen just with,
you know, with the next challenge, with the self-hosted show, just the more of us that are
interested in it, the better we can all help each other. I have a fun pick for us before we get out
of here. What sneaky last minute pick.
This is wild, man.
So I'm going through my notes because I heard from the developer of Userland.
It is an app that's available for Android that lets you run Linux on Android.
Like Ubuntu or Debian or Kali Linux.
You don't have to root your device.
You just essentially get a fast local VNC connection to theux that's running in a self-contained app vm it's so cool and i actually i tagged this
for us to to feature this as a pick three years ago wow and then i never it's still going yeah
that's a great sign right like i went to check on it i'm like well surely it's dead now no no it's
still going and i had a chance to talk with the developer it's'm like, well, surely it's dead now. No, no, it's still going.
And I had a chance to talk with the developer.
It's open source.
It's GPL.
It's up on GitHub.
We'll have that link.
It's available in FDroid as well, which we do like seeing that because there's just not enough stuff in FDroid.
There's good stuff in there, but not enough.
And so it's really nice to see that.
You just install this like a regular app and then you can run a full Linux distro on top of an Android device.
regular app and then you can run a full Linux distro on top of an Android device.
They also, if you check, once you go to that link, if you look at their Play Store apps, this isn't really a pick, but this guy's
like a bonus pick. They kind of use the same trick for two other things. So they
have the whole user land for Linux distros, but then they have Foxbox
which is a desktop Firefox. Oh, like the desktop version of Firefox
on your phone.
Yeah, so you know like when you go to Reddit or other sites,
you're like, would you like to see the mobile version?
No, I don't.
You don't get that when you use the desktop version of Firefox, right?
Plus you get all the Firefox extensions and stuff.
Yeah, sure.
But then the other one that actually feels like
maybe this should be a million-dollar product
is called CraftBox.
And it's a VM that fires up a Linux machine
and just runs a hosted Minecraft server
on an Android machine.
So maybe you got a fancy phone,
you got a tablet you're not doing anything with.
You could actually turn that
into a local Java edition Minecraft server using this.
And all of this is GPL3.
So wanted to give a shout out to that.
My Android phone can finally be useful.
Great.
Chris, it's so interesting that you brought this up and didn't tell Wes or I about it
because literally yesterday I was playing with an application.
I guess there's another bonus pick called Termux that you can install via FDroid.
And I wanted to simply do some rsync backups
to my phone and play around with
what I could do with some old phones.
And this is so the same
line of thinking, and it makes me smile that
somehow we always seem to get to the
same place at the same time. So, thanks for that.
That's great. Of course, link over at
linuxunplugged.com slash
452. Alright, gentlemen,
I think that's everything, yeah?
I think we came to do what we wanted,
and we got our self-support shoulder waiting for us.
And Nick's expressions to write, I think.
We have some Nick's work to do.
We got a whole matrix system to rebuild,
and rebuild it better.
You know, we're going to call it the million-dollar matrix system.
Like the million-dollar, you know, you get the reference.
Of course.
Yeah, of course.
We have the technology.
If you'd like to join us live and participate in our mumble room or chat room or just listen live because it is a different show, well, you can do that on Sunday.
We do it at noon Pacific over at jblive.tv.
And we do have the calendar linked at jupiterbroadcasting.com slash calendar where you can get it in your local time.
See you next week.
Same bad time, same bad station.
Of course, don't miss Linux Action News.
We did a roundup of a lot of things coming to Linux 5.18.
Plus, we talked about the new Rolling Rhino, a 1.2 release.
Mix, remix, unofficial.
We got the details and more over at LinuxActionNews.com.
Be sure you go grab that.
Of course, our members feed.
You get the full live production or an ad-free.
It's your choice. You can support the show at unpluggedcore.com or support all the shows on the network
at jupiter.party and get all of the bonus content. That's how you do it. And links to everything we
talked about on our website, including our contact page, linuxunplugged.com. And we'll see you right
back here next Sunday. so i'm uh i'm trying to find a spot to like take my son camping for like his 13th birthday but i
i'm out of ideas.
Like we've been everywhere.
And I'm not saying that to brag.
It's like I really can't think of anywhere to go.
And I promised him like a trip.
And now I'm just coming up blank.
Has he been out to the Olympics?
That was what I thought.
I suggested the Olympics to him.
And he's like, oh, I don't know.
I kind of like to be somewhere in the woods.
That's like that's a rainforest.
Yeah.
That's like the most woods you can get.
I need to find like a great spot and suggest it to him is what I got to do.
I've been thinking about like, where would I want to go if I were his age?
It'd probably be like swimming, hot tubs, you know, hiking.
He loves all that kind of stuff.
Hey, he can have breakfast in a hot tub here if he wants.
I want breakfast in a hot tub. A journey to visit Uncle Brent.
There you go.
I've got woods.
And we show up and he's got breakfast ready for us.
I mean, while you guys are going out east,
you might want to swing down to Florida and catch a lunch.
There's quite a few these days.
I really...
Our initial plan, we were planning to swing down to Florida.
That was initially the plan.
I just couldn't make it work from like a budget standpoint.
But the first draft did have us going to Florida for a bit because I've never been other than just like staying over at an airport or a hotel for a night to then fly out somewhere else.
Like that's my extent of Florida travel.
Yeah, but in super secret, huge Disney guy over there.
Yeah, it's a magic kingdom after all.