LINUX Unplugged - 466: The Night of a Thousand Errors

Episode Date: July 11, 2022

We were fixing servers all night, but at least we have a great story. A special guest joins us to help make a big show announcement. Special Guest: Tim Canham.

Okay, boys, how are we feeling about the news that Lennart Potterine has ended up at Microsoft? Hey, I hope he got a nice payday at least. You got to figure, right? That's a pretty good get. There's really no clue as to what he's doing over there. There's been speculation he's working on WSL. Do we even know like a title or anything? Oh, I love how this got discovered, which was purely by accident, right? Through an email? Yeah, yeah. And it was sort of revealed after it happened quite a bit you know of course leonard famous for creating pulse audio

and then later systemd huge contributor to the desktop as we know it today and uh now at microsoft for reasons we don't know but we do hope you know it was a great move for him leonard you ought to move up to the pacific northwest be in the home base and uh shoot us a message we'll take you out and show you some of the best places to eat, right? Take Leonard out on the town, ask him all of the secrets about Red Hat and SystemD. Hello, friends, and welcome back to your weekly Linux Doc Show. My name is Chris. My name is Wes.

And my name is Brent. Hello, guys. Welcome on in. We got a big show, a big announcement this week. We'll have a very special guest to join us to help with our big announcement we've been waiting to tell you for weeks. Plus, we burnt another night with a big server outage we do it so you don't have to guys we'll share you what went wrong and we'll share you our attempt to fix it wait that's why we do it we're doing this intentionally i think so that's

what i tell myself then we'll uh we'll run it all out with some boosts some picks and a whole lot more so before we go any further let's's say time-appropriate greetings to our virtual lug. Hello, Mumble Room. Hello, Chris. Hello, Wes. Hello. What a fantastic showing and hung in there with us through technical challenges today. We had a power outage before the show started.

How even? It was crazy, right? And that was even after Wes put in all those UPS batteries. Well, we forgot the battery part. But next time we'll get the batteries. Well, and someone's kind of got to join the circuit by holding both ends. Right. Hey, I want to say a special good

morning to TailScale. TailScale is a VPN protected by WireGuard's noise protocol. We love it. We use it. It will change your game. And you can get TailScale up and running in minutes We love it. We use it. It will change your game and you can get Tailscale up and running in minutes. Get it on all your machines. Build your own flat mesh network in minutes. So go say good morning to the folks over at Tailscale. Go try it on up to 20 devices for free at Tailscale.com. Freaking love Tailscale. You got to go try it out. Tailscale.com.

freaking love Tailscale, you got to go try it out. Tailscale.com. Let's get right into this week's episode because Brent has put together a very special treat for us. We've got two brunches and one of them is on theme today. Yeah, previously, if you remember, we did a brunch with Quentin Stafford Frazier, which is really great. Check that out, extras.show.86. But this week, the real highlight is a brunch I did with Tim Canham. That's at brunch.show.87. You can find it there. Tim is a fascinating gentleman. We've had him on Linux Unplugged previously, I believe earlier last year. He is a senior software engineer at NASA's Jet Propulsion Laboratory and had a very deep hand on the Mars helicopter project. And so he and I got a chance to sit down earlier and talk about all sorts of

pretty amazing topics that I think all of you would really enjoy. Everything from software and hardware that's powering NASA's Ingenuity helicopter, but also, and I think this is the extra juicy stuff, what it was like to kind of craft JPL's switch from Solaris to Linux back in the day, and how they're using open source tools and projects and the philosophy over at JPL these days. So here's a little snippet. Tim, thanks for joining me on season two of Brunch with Brent. I really appreciate having you. Well, thanks.

It's great to be here. Tim, for those who don't know, you joined us early last year on Linux Unplugged. Could you give us a little bit of an introduction of who you are and what you do? Well, I am a software engineer at NASA's Jet Propulsion Laboratory in Southern California. And for those who might not know, you have NASA as a whole, but then there are centers around the country that have various responsibilities. And the Jet Propulsion Lab, their responsibility has always been robotic spacecraft. You know, we don't have a pilot. We don't do man launches.

So all these spacecraft you've read about over the years, like Voyager and Cassini and the Mars rovers, those were all built and launched and operated by the Jet Propulsion Lab. And so I started there in 1991 coming out of school, and I've been there ever since, except for one brief trip out to industry in the mid-90s, but I've been back and doing that ever since. And my background is I'm an electrical engineer, but during my studies,

I also studied computer architectures and software. So much of my career spent living in that, straddling that world between software and hardware. And so over the years, I've worked on a number of projects at JPL. I've worked on, when I first came out as the globe rotates, all these deep space spacecraft can be in view of the Earth. So there are these gigantic antennas. One of them is 70 meters across. If you can imagine an antenna that's most of the width of a football field wide. So early in my career, I worked writing software to operate those systems.

And then as my career went on, writing software to operate those systems. And then as my career went on, I switched to the flight side, worked on Cassini for a while, worked on the Curiosity rover, the one that's been on the surface for 10 years now. Much of that software was brought forward to the Perseverance rover, so I have software running on that rover as well. But obviously, in the last five years or so, I've been working on the Mars helicopter, which is a small semi-autonomous helicopter that is a companion to the Perseverance rover. And so we operate the helicopter via the rover, and it does flights around in the vicinity of the rover doing scouting, taking pictures, getting a lot of great flight data. And so that project launched with the rover last year. We landed in February, so we're almost coming up on a year of the rover being on

the surface. And then during April, we did a series of primary flights to prove the concept of flying on Mars. And then they went so well, and we got so many awesome pictures that they decided to extend our mission. So we've been doing an extended mission of flying around and doing scouting for the rover ever since. And it sounds like you were in part, let's say in part, but you can accept the whole thing if you like, in part responsible for some of the Linux adoption that went into, you know, I think this was a D project where it's kind of experimental.

You know, I think this was a D project where you it's kind of experimental. And you were telling me earlier how Linux was a natural progression from things like Solaris back in the day. How was that transition for you from one to the other? And why was Linux interesting to you back then? And why is it interesting to you again today? Well, coming from the Linux angle about 15 years ago, when JPL was very much a Solaris house, I saw kind of the democratization of the platform as a good thing. Because with Solaris, you know, it was this big corporate package and you bought their hardware with it.

And it was very much a silo. You bought the hardware with the software, with very much a silo. You bought the hardware with the software, with the services all together. And they were also extremely expensive. If you rewind the internal clock, but you know, back in the 2002, 2003 era, you know, one of the standard Sun workstations was anywhere between $15,000 and $20,000 a pop. They were very expensive. Back then, that was actually even more money than we think about these days, right? Exactly. And as I mentioned, I think it was this podcast, you know, our previous, the Linux Unplugged one was, I discovered Linux and I put together a system out of

basic parts you can buy from Fry's Electronics. May it rest in peace, you know. I was thinking you were going to say RadioShack, but I put together a PC for a couple of grand at work and ran Linux on it. And it compiled our code at the time twice as fast as the Sun machine did that cost seven to eight times more. But I'm surprised that that experiment hadn't been done before, in a sense, like that the Solaris guys weren't like, oh, we got to really keep track of what else is out there. And so what was the incentive for you to even give that a try? were in Linux, you know, build environments and things like that and compilers. And I saw it as a way to take advantage of them. And also just trying to be a good, if you will, a good steward of JPL's money, right? If we could have a project where we, because sometimes the developers would have to share machines because they were just so expensive. And, you know, the idea that you could you could have more machines that were more powerful and more people's desks just kind of was a multiplier of people's efficiency and a better

use of jpl's money and so i that's why i started looking into it for my project and we essentially turned on a dime at that point and we switched switch from solaris to linux and then when i was hired on to do curiosity the rover project they were still using or were planning on using sunboxes from the previous project and i just said well let's try this you know let's switch and use these linux machines instead because they're much more powerful and they're cheaper so we can buy more of them and for more developers and uh you know they were skeptical at first but once i made you know i showed from my trade study that we did that it worked better they pretty much switched to and now it's very pervasive on the flight side of the house of jpl to use linux for not only development but the ground systems

themselves they run on linux and so back 15 years, and I don't want to claim sole credit for this because there's other people that were advocating for it. I was just in a position where I could actually do it. But what's happened in the last five or 10 years is that there's been a big switch to containerization and cloud tools. containerization and cloud tools, right? So I was able to get us to where actually using Linux on physical boxes, which was a big change. And then in recent years, they've switched to, you know, running a big giant VMware instances. Now they're starting to transition to doing everything on the cloud. So there's like this AWS government cloud where the AWS provides services to run containers. And so JPL is switching a lot of their infrastructure to use them instead of physical machines. But it's still all living within a Linux world. And so I'm glad I played a part in that. I don't want to overstate my role, but I think it happened at the right time to kind of catch the trend. it was, it happened at the right time to kind of catch the trend. Well, in many ways, it sounds like you're really just taking the attitude that, you know, new technologies need to be looked at and considered and implemented if they really

make sense. And that I would imagine at a place like JPL is kind of, it should be the way to go. And, and okay, we rely on what we know, but we need to still stay very current because some of the new technologies allow things like, you know, but we need to still stay very current because some of the new technologies allow things like, you know, the Mars helicopter to be as small as it is and to do some of the things it's doing. So I would say kudos to you for keeping your thumb on new technologies and not being afraid to continue to dive in even, you know, even, even this far into your experience with, with all of that stuff to not stagnate is no small thing I'd say.

experience with all of that stuff to not stagnate is no small thing, I'd say. And I don't think it's overt, but it's very likely that JPL getting a lot of institutional experience with Linux to the point where they trusted it for some of their critical tools for the ground systems or for development, and even some of the enterprise tools that they use, you know, that are business related. When it came time to proffer up the helicopter as a platform running Linux, you know, they at least had some institutional comfort with Linux as an idea. Because back when I was first doing it,

it was kind of viewed as this, how do I put it? Like the crazy uncle who's got everything wired up in his basement. You know, the people that were pushing Linux were kind of fringe, you know, that's how they were viewed because they didn't have these enterprise solutions

like Sun offered at the time. And so if we had come completely out of the cold and just said, Hey, we're going to fly Linux on a helicopter, you know, they, there could have been a lot more resistance to it had we already not made the transition. Linode.com slash unplugged. Go there to get $100 in 60-day credit on a new account, and you go there to support the show.

Let them know you heard about it here. Linode's how we've built everything. A couple of days ago, we were working on an outage of our own making, and it was really nice to have Linode in there as our co-pilot, if you will. You know, I could go into the dashboard. I could get all the disk information i needed so we could sort things out and i was able to quickly resize the disk which gave us the breathing room we needed to get things back online it's nice not having to fight linode's interface or linode's

capabilities or having to like trick their customer service into helping they have just the best support too and that's always kind of in the back of our mind if we ever get really stuck. And their S3 compatible object storage has been such a useful tool for us. I think once you wrap your head around it, you're going to love it, even just for your backups. Of course, they've got cloud firewalls,

VLAN support, a powerful DNS manager, super fast networking. They've got 11 data centers around the world. They've been doing MVME PCIe upgrades for their hard drives, AMD EPYC CPUs for their big CPU rigs. And they've got pricing that's going to work for an individual or for a business. It's 30 to 50% cheaper than those hyperscalers that want to lock you into their crazy esoteric platform. And Linode has just recently rolled out Kali Linux support. I kid you not, guys, at one point when I was back in the quote unquote industry, I almost just laugh at myself just

saying it now. But I kid you not, we were seriously investigating building a solution like this ourselves and selling it to clients. Because there's real utility in installing Kali Linux in a remote, safe cloud location and then checking your network. You know, even if it's just a couple of machines or it's an entire business network, it's such an incredible tool. And they have it in their app marketplace, too, so you can get it up and going in just seconds. That's Linode. It's powerful. It's fast.

It's Linux to the core. And they support some of your favorite shows and some of your favorite community projects and events, too. So go say hello. Get $100 in 60 day credit and support the show. Go sign up and see why we love it so much. Linode.com slash unplugged. Brent, that was a fantastic talk with Tim and the entire conversation is available at extras.show. We've already published it, so we'll drop a link in the show notes for that as well. But one of the reasons we wanted to play that chat is because it ties in with an announcement that we have here on the show. So let's start this off because there's a lot to it.

We have a very special meetup that we're planning at the Jet Propulsion Laboratory on September 29th at 1230 p.m. It's a special tour from our friends at JPL. They do have some restrictions and notes, so we'll link to their public tours page. And it's going to be limited to 20 audience members. So it's going to be a little bit tricky to figure out how we do this. Now, don't worry. This is not the only meetup that we'll be doing. No. We're going all the way in

and we're doing a West Coast tour and so we'll probably do something like half a dozen meetups in that route. But we need to know where you're at and we need to be able to plan and be smart about our fuel usage. So we have spun up a West Coast crew matrix room where we're hoping

that if you are on the West Coast and you'd be interested in a JB meetup in your area, you'll let us know you're out there so we can start getting some data so we can start planning this route. We're also going to need some local experts to help us pick out locations that we could go for a meetup or maybe a spot Jupes could park. Especially, I know immediately we're going to need to know a dinner place near JPL since we can only fit 20 audience members at JPL for the tour we want to do a meetup nearby maybe like the next day or something like that where we can all sit down and do dinner so if you're in the area that could be really helpful if you get in that matrix chat room we'll have that linked in the chat room it's I also made a short url out of it to it easy. It's just bit.ly slash westhostcrew.

And that'll take you to our Matrix chat room. Tips via email and boosts are great too. But we're hoping really to build a long-term community in that Matrix chat room. So if you join us in there, we'll have a link for that in the show notes. But this is something very awesome that we are very excited about and very grateful that tim has extended this opportunity to us no kidding and uh we're huge space nerds so to be able to go down to jpl uh and see all of it down there is going to be well it's an opportunity of a lifetime for us linode is on board for another linode on the road road trip to help us get down there

so we're going to make a whole tour out of it and we're just getting it all figured out right now. We will have meetup details and locations as we figure them out at meetup.com slash jupiterbroadcasting. But phase one is really we're just trying to get some signal of if you're out there. So if you're out there and you want to do a meetup in your area on the west coast and I include basically Idaho, Oregon, Washington, California, that general area. we almost never stop in the portland area we almost always just blast right on through so if you're out there let us know uh you can send us a boost and say you're in the area or you could join that matrix chat room that'd

be great or send us an email we can start figuring it out but we know ultimately our end destination will be the jet propulsion laboratory where uh, where we will meet up, do a small tour there, and then we'll have, you know, subsequent meetups. It's just super exciting. It's a great opportunity.

And it's really nice because there's not, um, there's not a lot of events going on. We're starting to see more starting to come back, but there still hasn't been like the full return of events. And we've been, we've been missing it. We've been missing the audience,

you know, especially after what a great time we had in Raleigh recently. Yeah. Yeah, which... And we're feeling a little jealous we don't get to go to the London meetup. I know I am. We may have a Raleigh Matrix chat room soon as well because Brent's going to be in the area and maybe doing some micro meetups. It's true. While

Alex is in London, I'm going to be at his place. That West Coast Crew Matrix room, that again is bit.ly slash West Coast Crew. And we'd just like to get an idea if you're out there, where you're at, if you've got any suggestions that could hold, you know, 30, 40 people or something like that. And the near JPL too would be really helpful. Super excited about that. We'll have more details to share as well as planning that'll happen in office hours. So stay tuned. Just a quick spot of housekeeping.

I want to mention that you can become a member. You get an ad-free copy of all of the shows at jupiter.party. Of course, you can become just an Unplugged Core contributor at unpluggedcore.com. You also get access as an option to the live show feed, which is like twice the show. Usually it's a lot of Brent cussing, but other than that... Mostly. But you got to listen to find out, really. Yeah, Brent, you potty mouth.

But yeah, jupiter.party to become a network member and support the independent podcast here. I mean, the reality is guys, we do a niche, you know, we're, we're never going to, we're never going to get like a million views on YouTube or, uh,

go viral on Twitter. Talking about no matter how many ridiculous faces you make. I mean, you keep trying. I try. Yeah, I try. We really,

really thrive on direct audience support. And so your membership is a great way to do that. At jupiter.party. All right, we have a bit of a tale of woe this week. Here I was, 10 p.m., Thursday, minding my own business, making myself a snack before kind of getting ready for bed. You know, the usual evening. I see a message on my phone from Brent. Nothing too unusual you know he uh

we've been chatting a lot lately and we tend to stay up late too we sure do but there was something about this message sounded simple hey is matrix down for you but i had a sinking feeling that was gonna change the course of my evening. I even paused for a moment because I was like, surely it's me. You know, I have pretty terrible internet over here and stuff. So usually the problem is just like I got to wait a half an hour. And so I did. And things didn't change.

And I just thought, is it too late? No, I'll just try. It'll be fine. It'll be fine. And so, well, I asked and the answer kind of surprised me. And then go figure. We all were kind of doing something different in the matrix rooms at different times. So one of us would inevitably going to notice, but it's always Brent. You know, I was sitting down to contribute again. I've gotten this routine this week of sitting down late at night and

contributing to our new website, which has been really great. And so I sat down as usual, as I had the last, you know, four nights before then, and I was just going to check our little matrix chat room. We have a little website matrix chat room, everybody keeps in tabs of what's happening there. And I couldn't update people. And that's really how it started. I sat down just to get started and then I couldn't do step number one. And so I looped both of you guys in. Yeah. I mean, we can't have a sad brand. No. And of course there I am, right? By this point, I got one kid down. I'm managing two other kids. My wife Adia is out that evening. She's hanging out with her lady friends. I'm managing two other kids my wife Adia is out that evening she's hanging out

with her lady friends I'm literally just like trying to get everything settled down for the night and well we decide we all gotta jump on a call I'm just doing dishes while we take care of the matrix server oh that's nice so I'm sitting there doing that stuff

one kid falls asleep while we're on the phone. Two other kids still missing. We got to track them down because it's way past the time they're supposed to be out. But of course, now the matrix server is down. And Wes, like the pro that he is, he kind of gives us like a good solid, like, stand up meeting recap of the situation. Before we ended up down this route of, you know, dumping our entire database and restoring it somewhere else, we realized that our matrix server had gone offline, and then it turned out

that we had eaten up all the extra disk space that we'd given it just a few months ago. We were able to allocate a little extra disk space by reclaiming the stuff that ext4 auto-reserves for root, which is quite handy, and that was enough to allow us to bring the database back online,

and Postgres is pretty good about, you know, it's got its write-ahead log and stuff. So it can restore itself pretty well from that kind of failure, usually. And it seemed to restore just fine in our case. But we did have enough room to do a preliminary run of this Rust program that compresses the state grips. So we did get that to run. They hopefully provide a repo that has a Docker file.

So you can just, if you already have Docker Compose going for Synapse, you can just tell it to go build this and then run it. And then they, the first time I used it, they only had like a manual tool, which we can look at and maybe see, like that might be a way to figure out how to do more if what it auto did wasn't sufficient.

But now they have an auto tool that'll go through all the state groups for the various rooms and stuff and compress the ones it thinks it needs to compress. So we can do future rounds of that as well. Assuming it works. So once again, our matrix server was down. And we just a couple of months

ago scrambled and really upgraded this box. But what gets us is the database growth. And it's the state tables, too, which is not what you'd expect, right? You'd think it'd be people's conversations or you think it'd be images. But it's the state's table that's growing and growing and growing. And you came across a cool little utility that actually helps us do some proper maintenance on this thing. And what was that written in there, Wes? Oh, it's written in Rust. This is great. I wish this was built in, to be honest with you. Yeah, that would be nice.

Perhaps in a future version, someday. But yeah, it kind of goes in and it looks at some of the state group cache that they're using to be able to quickly answer questions about the state of various rooms. And then you can just compress some of those entries so they don't need as many individual rows and kind of shrink that giant table. Because I think right now, that's by far the biggest table in the database. And it's something like 100 gigs. More than that.

Maybe now it's 100 gigs. Yeah, it was more before. It didn't start that way, yeah. So we had to get the box back up online. I mean, you fill up your disk, services crash, things go offline, and then you're kind of sitting there with options. Like, do we want to migrate this to Linode's managed Postgres so we're just not worried about a Postgres server anymore?

We've been tempted. I mean, it's a recent offering. We've been hoping to see it, and it's great that they offered that now. Do we want to maybe change data centers because maybe this isn't in the best location because we kind of spun it up on a lark, so maybe we should do, you know, like all these questions kind of came up and we had to answer which technical debt are we going to solve to get this system back up and online? And can we play around with the storage sizes to give us a little flexibility? So once we had that conversation,

of course, as things are going on, as you know, Wes's dogs need to go out and i'm doing dishes we kind of sort all of this out and so wes gives us a recap again not some a professional i see so are you thinking of moving it to a disk that has more room and then or can we avoid doing that so i think first things first we will shut off the. Then we do a dump of the database to a file. And then from that, we can import it into our new Shiny hosted database. But we want to get that process started because that's probably going to take some time because we have a painfully large database at the moment. Yeah. And even just moving it to that hosted system would take forever at its current size.

Which was one of the issues, right? This thing was like nearly 300 gigabytes when you included all the different tables and everything in there. So like moving it around, not necessarily an easy job. Most of the rest of the database was a reasonable size, though. Templates are built in. Postgres is the internal one. And Synapse. So yeah, Synapse is the only thing we care about. And as we see here with the 223

gigs. Right, that's the one. That's the one. That's the problem child. I think the 7000KB is probably not too big of a deal. At one time it was. At one time it was, indeed. Actually, it looks kind of like a minimum because they all seem to have that. There's really a lot to this.

And most people wouldn't run into this if you didn't have thousands of users that, you know, were coming in from all these different matrix servers across the federation, right? If you just set up yourself a home server with a small community, you'd be fine. Yeah, we've gotten ourselves in a somewhat particular situation. Yeah. And as we were debating, like, how do we want to do this? Do we want to move rigs? Do we want to change things up? We kind of realized this is all our own fault.

Because I suppose the thing to do would be to put it in the same data center at the PNM. Yeah, it looks like we've got everything in Fremont at the moment. California, yeah. That doesn't seem like the best place. No, no, that's probably a consequence of us spinning this up on Alark a few years ago. And we're like, well, we should make it close to us. Wes, I feel like this very call right now is the consequence of us spinning it up on Alark a couple of years ago.

Touché, touché. One crazy night, and we're still paying the consequences for it. And speaking of crazy nights, I'm literally trying to figure out where my kids are at because I know they're at a birthday party, but they were supposed to be home already. It's crazy over here. As we're talking on another device, I'm getting a call from my daughter letting me know that there's being cake served over at a friend's house for a birthday party. And you're not there?

Wow. Don't they know we have a matrix server issue? Well, they're just trying to make you feel better. They weren't offering cake. Oh, I see. I got it all wrong. I don't know exactly what went wrong, but I guess the mom went out nine hours ago to pick up the cake and just got back now at 10 o'clock at night. And so now they're serving cake at 10 o'clock at night.

My kids are supposed to be asleep. They're supposed to be over there having a sleepover. Now they're getting... And of course, my daughter thinks this is so great. She's got to call me to let me know. Dylan wants to get out of bed and walk over there in the dark. Get a cake at 10 o'clock at night.

Jeez. Oh, man. I think it's cute that you think people sleep at sleepovers. Gosh darn serving cake at 10 o'clock at night. That's just crazy. Save it for the morning. Yeah, breakfast cake, right? So we realize, okay, I got to go take care of kids stuff.

Wes has got to walk his dogs. Brent's got to get back to watching Star Trek Discovery. That's the most important one. So really, clearly the thing to do was for Brent and I to piss off and leave Wes with all of the work. And so we left the call not knowing how things were going to go. Alright, well, I don't know that we need to

hang around. It seems going to keep going on both fronts for a little while. It is compressing this. I don't know how big it's going to be compressed, but I don't know. Well, I guess we, I guess we could leave it as a cliffhanger for Brent and I, then you could tell us.

There you go. Oh, true, yeah. You have to set up and then... Should I close my Matrix client then? Oh, that's right. Oh, no, we can't. Yeah, I guess you'll see. We'll know if it's online, but we

won't necessarily know if it's on its own server or if it's on the hosted thing or how late Wes stayed up. I know a little bit because I increased the disk size for the Linode. Bless you. We have a massive rig for our matrix server, mostly because we're still figuring out how to optimize it, and then we're planning to shrink it back down. When we upgraded, an additional amount of storage was allocated to us that we could actually resize our disk to that we never took advantage of when we upgraded the RAM and the CPU cores. And we didn't realize that at first, because in the previous setup, we were using all of the fast disk we had available to us. Yeah. And so when that part clicked, it was like, oh man, we could increase the available disk size for the database, get the database back online. And then that's where I, that's where I, that's, that's all I know. So what did, what, what happened after we resized the disk and what did you do to get the

server back online? I mean, not much really. At that point, we just had to bring the database. I mean, we just kind of had to start things up again, you know? We had thought about much more dramatic migration plans, but once we realized that we just had more fast disk right there locally, it's hosted on ext4 right now, and that's just how it

came to us, so we just sort of resized that file system on the now much bigger partition, and spun things back up. That was one thing you did have to do. After we did the resize in the Linode dashboard, you have to go in and resize the file system. But I was actually giving you an opportunity to mention that you can run that tool and we were actually able to get the database down about 150 gigs in size after

running that. I think there's more we could do. Oh, definitely. Yeah. Yeah. And we're in a better position now. We've actually been planning sort of a little matrix sprint to shore up some of the matrix stuff that's been on our bucket list. We just weren't planning it for this week. In a part, it's like every time something like this happens, we learn there's a new set of tooling that we can take advantage of. And I think our matrix instance is the size where it's not considered like a huge instance. So there are people just ahead of us that are discovering these problems and creating solutions like this Rust tool or like the Synapse admin interface or, you know, trying out workers at scale. interface or you know trying out workers at scale and then our instance being a little bit smaller we kind of we eventually get to these problems after they have and so we're taking advantage of all of the upstream work that's being done to solve these problems before we hit them and then when we hit them there's almost always a tool available now like that's been the upshot

or there's almost always a solution or legitimately like twice in a row they've just added that feature or just changed that default in the synapse server like a release or two before we needed it yeah yeah really that keeps happening the docs have also gotten a lot better for like the administration side of running a synapse server so um it's really nice to see i think that's one of the benefits of all the pain that we do put ourselves through the benefit is we get to watch that growth and how the community is developed and i think you know each time we kind of dive back in like this time we you know we know way more about sort of the internals of the the database schemas and how that's working and so i think we have more confidence in that side of things even if you

know it took a late night another thing about it is it's a bit of practicing what we preach like i really have been on this whole decentralization i think is key for free software long term matrix is part of that and there are upsides and downsides to decentralization and choosing to self-host and it would be unfair for us to come on here and only ever represent the good things and never represent the bad things and if anything i think maybe we could represent some of the better things uh probably a little more often because one of the reasons we're willing to go through this is there's a really great community on there. And there's a growing network effect with other free software projects and communities that are happening all the time. And it's getting better. And, you know, the various clients are getting better over time as well.

And so in one sense, it's quite impressive what, you know, what they've built there and all the capabilities, especially compared to other platforms. Like it's getting much closer. I have a final question for you both. We seem to kind of dive into the matrix server either by choice or not every, what was it? Every two months or so? Yeah, it was right before Raleigh.

So it's been a bit. Do you have any predictions on what will happen in the next two months and what will bring us to doing another late night session? Well, maybe migrating out of Fremont, perhaps. Maybe. I know something else we want to do is we want to get the Synapse admin backend up and running.

I don't think that'll cause an outage other than us restarting services and stuff. I know that's next. Some more database optimizations are next. We definitely have some more performance gains to get. It's a lot better, but there's clearly more work to be done. I think that's sort of going to be the target of the next sprint. And now that we have some disk space and we've reduced the size of the database, we can focus more on maybe some of the performance side of things

and less on the keeping things running side of things, which would be nice. Also, shout out, you know, we've been getting various tips both on the Matrix server itself, but then also from boosts and feedback from audience members who have way more experience and understanding of Matrix and Synapse than we do, and that is always super helpful. Yeah, we get tips in and that is super great. Thank you, everybody. Bitwarden.com slash Linux.

Get started with a free trial at bitwarden.com slash Linux. A team, an enterprise, or an individual. It's bitwarden.com slash Linux. This is the easiest way to store and share and sync sensitive data. Bitwarden is open source and trusted by millions of individuals, teams, and organizations. It's a tool you've got to have. And a good password manager, that's great. But Bitwarden is more than that. It's really a secrets manager. It'll help generate usernames. It'll store one-time authorization tokens for two-factor.

And of course, if you're in the enterprise, it's fully customizable. You can turn features on and off using enterprise policies to adapt it to your business needs. And if you're an individual user like me who likes to know there's a self-hosted option as well, well, Bitwarden contributes upstream to the Vaultwarden project to help make sure they stick around and help that community thrive. And they've got incredible support in their apps for just all kinds of great power user features and user-friendly features for people that are just learning how to use software like this. Some of my favorites are like the fast user account switching. It's so easy now to switch between maybe like a personal account or your business account, or maybe you have an account for an open source project you're working with. When you're auto-filling now, it'll just let you switch between user accounts.

And on iOS, the face ID integration is killer. It's like you're living in the future. It just scans your face, opens up the password database, gets the right password, auto-fills it. And the reason why I think that's super critical is it means maybe I signed up on a website or maybe I signed up using the app, but I can take that sensitive data across my desktop and my mobile, and I can use complex, rich passwords on mobile and stay safe using Bitwarden. And with the username generator too, now I'm using a unique username at every site and service I sign up for. I should have always been doing that. Bitwarden's always thinking of that kind of stuff. That's

their job. So go try it out. It's easy to get started. Visit bitwarden.com slash Linux. Try it for yourself or your business. Maybe you know someone, a friend, a family, a colleague that could use a little bit of help in this department. Send them over there. Nudge them in the right direction. Bitwarden.com slash Linux. We got some amazing feedback again this week. Thank you, everyone. We did get a few songs in for our pump-up song.

What would you call that? We're trying to find a great pump-up song. Our pre-show starter song. It's true. It's true. what would you call that? We're trying to find a great pump up song. Our pre-show starter song. It's true. It's true. You can hear those if you're a member and you get the extra long feed or on PeerTube. You can check it out there.

It's a lot of fun. Pre-show mantra maybe? Hey, I like that. And so thank you, Justin and Andrew and Jeff sent one in as well. Keep them coming. I think we'll find one in time. Right, guys? I think. Yeah, I think we'll find one in time. Right, guys?

I think. Yeah, I think so. That implies there's a deadline, which I'm not sure we have. Yeah, I don't want to put like a time on this. I feel like a song commitment is a big commitment. It's a long-term relationship. And I don't want you to rush things, Brent. It might be the most important relationship any of us have.

It might be. Well, Levi. Oh, yeah. But we also got other feedback, which has been really great. And I think, Wes, this one's right up your alley. Well, maybe both of us, since we've been talking about it. Casey wrote in and says, I heard on Office Hours that Wes was giving Brent some Python tips.

A specific example Brent gave was a Python REPL. I know Wes does some Clojure, which has one of the best REPL experiences. So if he's recommending a REPL for Python, that's something I want to know about. As much as I wish I could write Clojure all day, Python still pays more of the bills. Care to spill the beans on this REPL recommendation? Yeah. Okay. I think a REPL is just super handy because it recreates this feeling of being really connected immediately with the computer. It's much like using a command prompt, right? Like Bash, it's like having a conversation with the machine where you can touch the data structures, you can feel them right there.

You don't have to construct an artificial test environment or something. Not that that's bad, but it's just, I think when you're, especially when you're exploring a new API or you're trying to learn and you don't have a lot of previous understanding on how a new mechanism works or something. It's just, it's a great way to learn. Totally true, Casey, that Clojure has an amazing REPL experience. Python can't quite compete for a lot of reasons. One of that, one of them being sort of the culture isn't there as much, especially now with type hinting and stuff, there's been more focus on the static analysis side of things, which don't get me wrong, it's also great and wonderful and there's a lot of good tools around that now, but there's less interest in building APIs that are REPL friendly necessarily. That said, B Python is a nice one. It doesn't have as many fancy features,

but it's got great tab complete and a nice little pop-up menu. So it's friendly to interact with. And then IPython or Jupyter notebooks as well. It can be one thing, but if you just want it in like a terminal, I Python works great. They've also got a auto reload extension, which is one way to get a decent REPL experience. Anytime you input a new expression or statement on the command line, it'll load,

reload all the stuff that you've imported. I also have rigged up with an async file watching library to kind of do the same thing and just do like a hot reload. Anytime I rewrite any of the files on my disk. It's not perfect but it works pretty well and then if you kind of design things, stay away from

classes and objects if you can and just have sort of functions and pass in all of your state explicitly so that you don't have to like rebuild your classes. You just have some dictionaries and other things like that, data classes, etc. that make things a little bit better in the REPL environment. Super, super fascinating. Wes,

I've got to say thank you for having long, late-night conversations with me about all of this stuff. If someone was just learning Python, would you recommend just diving into this right away, or is this more of an expert's tool? I mean, it could certainly be both, but yeah, I think having a REPL

available, you can just use the built-in Python one, but if you use something like IPython, you get just a lot more of a friendly environment with things like autocomplete and tabcomplete and readline support. And yeah, it can be a way to just sort of type in stuff you see from tutorials without having to set up a complicated environment to get it all to run. I mean, you will need to figure that out too, and you'll want to figure out how to get a test runner going and all those things. But just being able to type IPython on the command line and start playing is super nice. And also, shout out here, Gigatexel in the IRC room

mentions PT Python, which is another great choice. Okay. And of course, I'm sure the next thing Gigatexel was going to mention that Coder.show for Coder Radio

where we talk about Python quite a bit. Casey mentions too, yeah, Jupyter notebooks are a fantastic way because they're quite visual and interactive and there's plenty of places to go use those for free online. So, yeah, Jupyter notebooks are a fantastic way because they're quite visual and interactive. And there's plenty of places to go use those for free online. So yeah, check out Jupyter notebooks if you haven't.

And of course, we're not going to get out of the feedback section without a little discussion about Merkle trees. wrote in, It was great to hear you talk about Merkle trees on the last Linux Unplugged. They're my favorite, most underrated data structure. They're incredibly simple to implement since it's basically just a big key value store. And they're great for syncing because you can just rsync the key value stores around

and then atomically update a few references or branches. I once also wrote a minimal implementation of Git in about 300 lines of code, which he links to. sounds fascinating awesome and now it is time for the boost well our first boost of the day is from purple hummingbird five days ago with 2,222 cents i'm a duck. D.U.K. Duck. Loaded with talent. Hey guys I just got my Steam Deck. Sending you a row of ducks for your coverage

so far. Is there a gadget or new tech each of you are looking forward to? Thanks for the show. Hmm. I bet this is a hard question for you to

answer. You're not really a you don't covet much in terms of tech. You're very practical when it comes to this stuff. I'm really curious to

see what like the next generation framework stuff. Oh yeah. The framework laptop. Yes. I think that's sort of of all it's a little nebulous but that's on my my radar i think that'd be a really good work machine for you right what do you what about you brent you got a gadget or device maybe the dev one the new dev one waiting for you i almost forgot about that yeah apparently it's being held hostage somewhere that out. We're gonna get to that soon, I'm sure. But yeah, I'm sure.

You'll enjoy that. I think almost immediately on my list is actually some hardware that Jeff sent me. Thank you very much again. He sent me some equipment to boost my local wireless network here from one building to another. And it's been a great cat pillow so far. I hesitate to

remove it from that use, but I'm looking forward to setting it up real soon and, uh, jumping into iPerf and giving it a bunch of testing and seeing how optimized I can get it. Yeah. I am looking forward to hearing how that goes too. Um, Purple, I, I just got my confirmation and I said yes to the Steam Deck. It's like, I've done the payment thing now. I got my, your i said yes to the steam deck it's like i've done the payment thing

now i got my your things coming soon email so i think i'm going to answer steam deck because i had a realization i last week even i was kind of on the fence about oh do i really want one anymore you know maybe i'd rather have a dev one but I realized in my situation in the RV, having a portable gaming device is freaking perfect because I sometimes need to be up front or sometimes I need to be in the back or sometimes I want to be outside. A console hasn't served me super well because of that. And I just realized that that's exactly what I've been needing. And I've been wanting to game at home in the evenings, but it just hasn't worked out

because maybe like that's the evening I have my Think pad that just has an intel graphics card in it or whatever and you know so I've never gotten around to it I think I will now I'm actually pretty excited about the steam deck coming in and of course I'll share my thoughts and whatnots on it when I have a chance to look at it and play around with it for a bit Gene Bean wrote in three days ago with a thousand sats. Thank you very much. Boost. They wrote, don't forget the newer Raspberry Pi imaging tool will also let you set all of your Wi-Fi info and user info on the GUI. So Wes, I think that's a little shout out to your adventures there last week. Yeah. Hey, that does sound handy. Yeah. That is actually something that I forget because I don't use it very often,

but when you're trying to get people introduced to the raspberry pi the raspberry pi imaging tool is very legitimate try it probably will do what they need don't do it the hard way like i do for some reason a quantum boosted in with 500 sats and i wanted to read this because well first of all i think it's important this gets talked about second of all i think there's a little bit of shenanigans going on here uh they write when trying to order the dev1 machine for my dev team last week we were told by hp that the dev1 is not for quote professional developers and available only in their consumer division we were also told by hp they have no plan to offer a similar experience for professionals in their commercial line.

Why do you think they would make a developer laptop and then not offer to developers where they work? First of all, that does sound really silly. And anybody who's ever worked in a medium to large company can probably already do the math in their head and go, oh, yeah. Yeah, you got the enterprise sales team and this is this has been marked as a consumer laptop. So those two shall not pass. Right. Like you already know where this is going. What's weird? And it's silly, right? Maybe HP should take care of that. So that's why I wanted to read the boost.

But what's interesting about Quantum's boost here is this exact message verbatim was sent to Coda Radio maybe three weeks ago. Only I note that they don't say three weeks ago in this boost. They say last week. This email was also sent into Destination Linux, and now it has been boosted into us all verbatim somebody has a bit of a mission out there to try to take things down a notch uh not necessarily an invalid point but just saying that i've noticed this go around and i want the listeners to be aware. Somebody is trying to manipulate an emotional reaction from people on this subject. Should they fix it? Yeah, HP should probably fix that. Is it surprising that there's divisions between enterprise sales, business sales and consumer sales? No, not at all. Welcome to Corporate America. I think what is surprising to me, in a way, is that System76, they really pride themselves on building all of their stuff for professional developers. And it's kind of odd to me that this happened. I hope it's not happening more often, but maybe they got a fix for it.

It seems like a great laptop. I haven't touched it yet. You guys would definitely know. But would you say it's ready for the kind of quote-unquote professional developers chris i mean i could see maybe hp only views enterprise sales as maybe laptops that have like the hp pro management stuff in it i mean there's probably some sort of division where they check yes this is an enterprise box or not but yeah right it's got to be more like that or the, how the support contracts work out or how they, you know, they get you parts or whatever.

It's not, it's not about the actual experience of the laptop in the hands of the developer. And also I will point out it's, it's also, it's a commentary on the company that this person supposedly works for, that they are so inconceivably inflexible that they can't come up with another way to order these laptops. Like that stuff's also very typical for companies but so annoying because you know like the finance guy could put it on the credit card or maybe they you know like there's a way they can fix this and they could order these laptops as you know anyone else can but their company is also super inflexible apparently right was this just a i called my rep he looked in his catalog, it wasn't there, sorry.

Sorry, if we can't do the PO process, we can't order the machine, right? And meanwhile, they're angry. HP's like, sorry, this isn't a consumer machine, we can't sell it to business. It's like both companies are being typical companies. And it's really silly. I just want to give you money for your laptop. Right? And it's like, this is one of the things I really appreciate about being a small

business, is it's just you go to the website and you order the laptop. It's not really a big deal. Another boost from DeanL070. Two days ago, 10,000 sets. Boost! Just wanted to say hi from Australia. And thanks for all your hard work. Your shows have given me many ideas to try and further my Linux knowledge.

Hey, that's great. Thanks for the boost. And that's exactly what we're going for. And I love the check-in from Australia. And it really is great because also Lightning is a global network. Sats are sats. Here in Washington and there in Australia, a sat is a sat. Yeah, once you like get the onboarding stuff figured out, it doesn't matter where you are, right? The internet has needed its own native money and it's just really cool. So thank you on Australian Boost. It's great.

Exception boosted in as well two days ago with 1, 2, 3, 7 sats. That must mean something. You got me with all this Nick stuff, looking for a spare laptop to play around. Kind of reminds you of the Arch Challenge from forever ago. Oh, yeah, it does. A little bit. Oh, gosh, that weird Gentoo experiment. Where we had that long tail of like, well, all right, I'll give it a go. Okay, I'll give it a go. I've been talking about it for so long now, I guess I have to. And to be fair, we

are trying to like dial it down because we know not everybody cares about Nix, but we are really, it's probably the most excited I've been about Linux in a few years, right? It's really very impressive stuff. And what's also so fun about it is it's a major rethinking of the way a distro works. So it's like, you got to be willing to go for a big shift. Right. Yeah. That's just it. It's like a whole new paradigm in some ways. But if you're willing to, and you've got nothing really on the line, so why not try? Like it has been my last thing and then I'll stop because I could just go on and on. I have been surprised how great it has been on my thinkpad i thought when we

started this thing okay this is great for servers but i'm never going to want to have a prescribed laptop like this no man like it's now made me rethink like how i want to do the studio systems because i want them rock solid right yeah so one of the great things i love about using nix as a workstation or perhaps studio equipment is the updates all have to like pass and everything has to be good before it gets applied. So you do a build and it checks everything. And then you make sure there's no like weird conflicts or any crazy config changes. And if something like,

you know, your bootloader breaks or something like that, it airs out and you don't switch to a broken system. You only switch to that new updated system after it's passed all the tests. And then you reboot into this new clean environment. And it's just, it's so solid. It's so solid for a workstation or for a production system. I loves it so much.

Yeah, because we got another boost in here. This is something of a special boost, though. It's from Sue CD two days ago. 29,287 sets sets that's our baller boost that's a baller in our parts yeah especially because this was a manual boost sent via the blockchain and then a message to you via Matrix. Yeah. Hello from Portland. I've been listening to Jupyter Broadcasting for about 10 years.

Well, you were asking for a Portland check-in and we've got one. Yeah. All right. Well, there's one. Now we need to know if there are more. My Portland signal is up, y'all. But as you check out the West Coast crew, we do have a West Coast crew Matrix room now. Jump in there

and maybe give us some hot tips on where we should take things. We have a couple of thank you boosts to get into or at least one. 500 sats from 412 Linux wishing Brent safe travels. Brent's going to be traveling

to Alex's house in the near future and then eventually at some point he'll be making the trek to the studio either by car, train or something. we don't know

yet but uh we have brent's node now online he is participating in the split for the next we're doing for the month of july so for i guess three more episodes we're gonna do a 50 50 split of the boosts with brent so that way we can build up a little bit of a fuel travel fund so when he's done at alex's he's gonna come here he's gonna he's gonna of a fuel travel fund so when he's done at Alex's, he's going to come here. He's going to do it. He's going to do it. He's threatening me. No, we'll just guilt you into it.

It'll be real easy. Can we use some of that fund to get a tracker we can put on him to verify he's on there? Oh, that's a great idea. It's built into the dev one, I think. But really, you know what? It was fun setting all that up, that node and the community opening up channels.

Yeah, you were impressed, too too by how much liquidity came in on the node in a very, very short amount of time. Can you speak a little bit more to that? Well, about 400,000 sats of it came from me. So a lot of that liquidity was me sending you some sats. But then, yeah, we definitely had some community members step up and open up channels. And so the way it works in Lightning is

you have people who put some sats on each end of the channel and then you can move Bitcoin or sats in between there really super quick. And that's why it's so dang quick. And then it only really has to settle on the blockchain when you close those channels. And so you get super fast Lightning payments. And so people open up the channels

and it's a peer-to-peer network. And you think about it, you just kind of peer with who you're going to be getting boosts from, but necessarily, or Lightning messages from. So like i peered that node with the fountain app and with the podcast index and with jupiter broadcasting's node and then if you are a peer with one of those nodes clients can find routes to you it's so much fun to work with and watch it kind of the network discover itself and build out so it's online and we'll be doing the boost splits for a bit to help

with the travel fund for brentley so we can be doing the booze splits for a bit to help with the travel fund for Brentley. So we can come to the studio and we can engage in projects and now prepare for our September road trip too. Cause I think it'll be getting close to that. Oh yeah. You know, Chris,

I have also been keeping helipad up so I can watch the booze come in as they, uh, as people send them in. That is so fun. I, you've been talking about how that like basically drives your, your mood all day and just pumps it up whenever you need it. And I got to totally agree. It's really great. It's yeah, it's my it's exactly it's my top off when I'm feeling kind of like frustrated. I'll be like, I'll open up. I'll open up helipad. And it is so great for that. You got to get on that. Let's put you in there, too. You got to get in on that.

Let's put you in there too. If you'd like to send in a boost, just go get a new podcast app. Go do it. Why not try it out for a little bit? I've been using Fountain. Podverse is great as well. Castomatic if you're on iOS.

And if you don't want to switch podcast apps, Breeze, B-R-E-E-Z dot technology or Boost CLI. You can boost from the command line. They're all listed at newpodcastapps.com. Hey, you know, it looks like the most recent antenna pod update added podcasting 2.0 chapter support

so that's come along nicely as well that's a number now that's several several podcasts I don't know how many but I think it's a handful of the podcasting

2.0 features that have been added to antenna pod now it's really great to see that because we've got about a 30% audience share

with antenna pod it's probably one of the more popular ones yeah and it's a great client and you can combo that with Breeze so you can keep using antenna pod audience share with AntennaPod. It's probably one of the more popular ones. Yeah, and it's a great client. Mm-hmm. Mm-hmm.

And you can combo that with Breeze, so you can keep using AntennaPod, and then you can use Breeze to send the boost in, and then there are some patches out there already to kind of integrate those two apps,

but it's not done yet. I also want to say we did get a postcard in the mailbag this week mentioning that, hey, we shouldn't forget about those who are members. We talk about

Boost a heck of a lot, and sometimes we forget to thank the members. So I just want to say to all the members, thank you for continuing to support us. I know you guys are getting tons of benefits over there, too. So I just want to say thank you. Thank you for being a friend.

Yeah, unpluggedcore.com. You get an ad-free version, you get the live stream. Hey, what do you say we do a little Pixies? This is fun. What'd you find, Wes? Well, you know, as we were having some disk space problems, you need some disk space tools sometimes. NC2 is an old favorite of mine that I use all the time,

but I thought, we love Rust Pix. What else was out there, right? Surely someone's made some of this stuff in Rust. And, oh yeah, DUA-CLI. Disk Usage Analyzer is a tool to conveniently learn about the usage of disk space of a given directory. And yes, it's parallel by default and will max out your SSD. It's also got that nice little support to optionally just delete right from that interface, which I like about NC2 as well. Oh, that is super handy. But you know what?

There's a few more links in there. And we have a very special bonus pick that Brent threw in. more links in there and we have a very special bonus pick that brent threw in yeah this is maybe a really nerdy pick but um from the brunch that i did with uh tim he a long time ago wrote a framework that has been open source for a long time now and it's a flight software and embedded systems framework called F prime. And it's directly straight from Nasta and, uh, is available to anybody.

It even runs on the Raspberry Pi. They have it, they have it tested on the Raspberry Pi. Their nightly build is, is piped into a Raspberry Pi so they can test it every night. It's, it's really, it's super fascinating.

So it's so great. Okay. What, what if we built a really janky drone? I mean, I'm down. Right? Like a Linux unblocked. It won't work very well. It'll probably crash immediately, but.

Yeah, I mean, I'm down. Can we just absorb for like a hot second here that this thing is a Linux copter, right? And then on top of that, the framework it's using to be semi-autonomous is also open source and we're about to go down and meet the folks that put all this together and we're going to get a tour that's all about to happen right now how great is that my head's going to explode it's so neat so neat it's so worth even if you're never going to use it go check it out we'll have a link

in the show notes right because pretty cool yeah i i We'll have a link in the show notes, right? Because it's pretty cool. Yeah, I know Tim's a fan of the show. He's probably listening. So thank you, Tim, for all that wonderful insight. And it's actually his brother that got him hooked on to listening to Linux Unplugged. So thank you to Tim's brother, too. We owe his brother a serious couple of beers, right? It's a reminder, too, of just the power of word of mouth.

That's very true. That is what makes podcasts grow. It's not, you know, it's not really like a practical for somebody to invest an hour of their life into something that nobody's recommended to them. It just doesn't happen very often. So it's a great example. And then the connections that that creates, right? Second order effects of your recommendations.

You never know where it's going to go. It helps that we just have a really bad-ass audience. That's exactly it. That's exactly it. So never know where it's going to go. It helps that we just have a really badass audience. That's exactly it. That's exactly it. So let's wrap it up. But I am so excited about this West Coast meetup. I have missed you guys. I love doing these meetups.

It really recharges us. It's a great opportunity to connect with the audience, get a sense of what people out there are thinking about that isn't from the social media bubble. It's just so intensely valuable for us in terms of just like getting signal from the audience and getting an idea of what's on people's minds and getting a really good picture of what people actually care about. A different slice and more diverse info from y'all.

So right now we have the Matrix Room. Link for that's in the show notes, bit.ly slash West Coast Crew. room link for that's in the show notes bit.ly slash west coast crew and we will then begin to build out the meetup page as we start to get signal from the audience and we start to get an idea of where we're going to go and we'll start talking about that in office hours as well so it's a lot there but it's going to be so much fun and of course we're extremely grateful to linode to helping make all of this road trip stuff possible. It's not like this is a cheap time to do a road trip on the West Coast. In fact, you could not figure out a more expensive time to do a road trip on the West Coast. We should be taking a road trip to Central America, to the plains. It's a lot cheaper to drive. But we just did an East Coast meetup. We just did a

Denver meetup. We have a London meetup coming up in just a few weeks. It just seems like it's time to revisit the West Coast. And what an opportunity to go to JPL. So we're going to do it. So we'd love to hear from you. Send us in a boost or go to linuxunplugged.com slash contact. And then don't forget, we do this show live. That's like the high bandwidth way to interact.

See you next week. Same bad time, same bad station. Every single Sunday, noon Pacific, 3 p.m. Eastern over at Jupiter.Tube. That's the PeerTube instance we stream on. There's no Python bot taking us down. Usually it's just me.

And if you want an audio-only version, jblive.fm. And of course, linuxunplugged.com slash subscribe for the fully edited, fully produced version, because we want you to get. And of course, linuxunplugged.com slash subscribe for the fully edited, fully produced version, because we want you to get it every single week, and Drew just does a fantastic job on the show. That's linuxunplugged.com slash subscribe for that, and links

to everything we talked about today, well, that's at linuxunplugged.com slash 466. We try to make sure that's just full of goodies for you. So go take a look at that. Thanks so much for joining us on this week's episode of the Unplugged Program. See you right back here next Sunday. you know chris you made me so happy just there because it's taken six months and I've been pretty quiet about this, but you're starting to say next Sunday the way you used to say next Tuesday.

And I have been waiting for that for so long. You know, I understand it had to like settle in for a bit, but I'm encouraging it. Yeah, it had to be its own thing bit, but I'm encouraging it. Yeah. It had to be its own thing. You mourn the loss. Yeah. I didn't feel right too early. I loved saying, see you next Tuesday. That was the highlight of the end of the show for me right there, right?

And, you know, we switched to Sundays, and it just didn't feel the same. I felt like I had to give it time before I moved on and started, you know, building relationships with new ways of ending the show. But I felt like that was one of our better endings, you know, mostly. Also, you notice I switched to just, I'm just going to start mentioning the JupyterTube. In fact, I think maybe at some point we should just redirect JBLive.tv to JupyterTube. I know that's crazy, but we got big things in the works. We talked to Alex Gates, the podcasting 2.0 consultant on Office Hours this last week, and chatted

with how he's modifying PeerTube to make it work with podcasting 2.0 standards, but also to make it work with more live streaming options, build a few things that are just really small things that are missing, like in the RSS XML file and whatnot, to make it totally usable for podcasters. And it's all

just coming together. It's so freaking just, it's all coming together. It's so freaking exciting. There's like work being done by dozens of different community members, adding these different features at different areas of the podcasting stack. And it's really impressive to see it all kind of come together.

And so that was, that's why I reached out to Alex. He's like, will you come on so we can just geek out about this? Cause I'm back into PeerTube again and I know you're making these changes, and I wanted to find out, is he planning to upstream them and all that stuff? Yeah. That's a good interview. Well, thank you.

Thank you. I thought Brent really made it, though. It was a good Brent episode. Oh, wow. On average. Yeah. Well, thanks. I mean, it takes two, doesn't it? Well, at least a tango. It's been neat seeing PeerTube, you know, developing too. It looks like there was a release just 12 days ago.

Oh? Oh? Well, it's time to upgrade then. Yeah, we should also there's some important upgrade notes I know here so let's read those first. Let's not even, nah, let's just skip that, Wes. What could go wrong? Is one of the upgrades like maybe the view counts? I know, I know Chris, you've had some.

Don't get your hopes up there. I try not to grouse about that, but it is slightly annoying that the view counts i know i know chris you've had some yeah yeah don't get your hopes up i try not to grouse about that but it is slightly annoying that the view counts aren't accurate because it's like underrepresenting the views and the other thing that's kind of annoying is like when you drop when you stop live streaming it loses those numbers too and then you start all over and you're like but i but i okay but know, then you're not supposed to care, right? Because there's no algorithm, right? It doesn't really matter. We don't even have comments. It's true.

We don't put those numbers anyway. The whole thing is supposed to be a practice in me not caring, right? So I try not to. But as like, as a content creator who wants a signal on what's working and not working, it would also be nice if it was accurate. That's all. Yeah, that's fair. Can I ask you, gentlemen, since it's kind of on point for this episode, how is the PeerTube

server doing? Do we know? Is it performing fine? Oh, yeah. Yeah, because we're doing the, so right now we're doing the minimum viable PeerTube where we're just using it for our live stream archive and we're not like creating channels for every show and uploading the edited versions and we're not having community members contribute videos it is a really solid little rock solid instance the only thing we got to do is make sure that we got the storage back end all

tied up but that's pretty quick change yeah we should do that yeah well i'm sure once it runs out of disk space we will obviously it'll be an unplanned sprint that's how we do it