LINUX Unplugged - 406: Mars Goes to Shell

Episode Date: May 19, 2021

Tim Canham, Mars Helicopter Operations Lead at NASA’s JPL joins us again to share technical details you've never heard about the Ingenuity Linux Copter on Mars. And the challenges they had to work around to achieve their five successful flights. Special Guest: Tim Canham.

I mean, in the meantime, I should probably get my computer here started up. Let's see, let's start up my... There we go. Yep. It's just the most reliable choice for the live stream. No, Chris. No, just no. Hello, friends, and welcome back to your weekly Linux talk show.

My name is Chris. My name is Wes. Hello, Wes. This episode is brought to weekly Linux talk show. My name is Chris. My name is Wes. Hello, Wes. This episode is brought to you by the all-new A Cloud Guru. You know, they are the leader in cloud, Linux, and other modern tech skills, hundreds of courses, and thousands of hands-on labs. So go get certified, get hired, and get learning at acloudguru.com. Well, coming up on this episode, we're going to make good on a promise to you.

Tim Kennum, the Mars Helicopter Operations Lead at NASA's Jet Propulsion Laboratory, joins us again on the show. He's been living on Mars time, and they have had successful flights. They have, well, they have made history, but not everything went perfect. So he's going to join us today, and he's going to get really technical about what broke, how they fixed it, and some of the awesome, like, action movie kind of workarounds they had to do to get that thing flying again. And then after that, we're going to come clean about a service it's time to retire here on the show. It's something we launched here on the podcast. a service, it's time to retire here on the show. It's something we launched here on the podcast. And towards the end of this episode, find out which project of ours is just not

really worked out the way we thought it would. So we're going to have an exit interview and we'll find out what didn't work. But before all of that, before we get into any of that, we got to bring in our virtual Linux users group. So time appropriate greetings, Mumble Room. Hello. Hello. Hello. our virtual Linux users group. So time appropriate greetings, Mumble Room. Hello, hello. Hello. Hello.

Hello. Thank you for making it. This is going to be our last live stream for the next few weeks just because we're traveling. There will still be shows in the feeds. We'll have more details about that in a little bit. But I'm really glad you guys could make it because I'm going to miss

you guys. This is something we do every single Tuesday. So then to not do it for a few weeks is just going to feel really weird. So each one of you has a special place in my heart. And if you'd like to join our show live, we do it typically on Tuesdays. So this is your heads up. Maybe like in a few weeks, like say, oh, I don't know, June 8th. Come on back here and hang out with us at jblive.tv on a Tuesday. We have all of the information at jupiterbroadcasting.com slash calendar.

Really? There's no excuse anymore, right? Because you've got these upcoming weeks to get your audio set up, tuned, tweaked, maybe practice with the lup lug, and then you'll be ready to join the live stream when we're back. And let's be honest, Wes, put in your PTO request so that way you can come hang out, you know, or slide the lunch hour. Good idea. I also want to let everybody know that our friends at A Cloud Guru have a Linux system maintenance course.

This is a course that's designed to give you the knowledge and utilities needed to just maintain a Linux box. So you get started with getting programs installed and set up, but then they have sections that cover backups and which directories you should consider when you're backing up the utilities to create backups, how you can use physical medias, if that's your jam. And then it kind of ends with automating some of the communication and also how you can communicate with other users on your Linux box. It's a, it's a standalone Linux system maintenance course,

but I also wanted to mention it because it's part of the path towards the LPIC 2.201 exam. So if you are getting your cert on, or you just want to maintain a box, we'll have a link to this in the show notes because it is kind of a long URL, but we'll have that in the notes and you can go check out their Linux system maintenance course, which probably everybody could benefit from because a lot of the stuff I've self-taught. So you can go check that out. So let's not delay, actually. I think, you know, this is usually the time in the show where we would do news. But realistically, what is bigger than making history with Linux on Mars? So we were really thrilled that Tim could come back and join us. As I mentioned, he is the Mars Helicopter

Operations Lead at NASA's JPL. And he was on the show in early March, right, Wes? Yeah, early March in Lepp 396. But a lot has happened since then. I mean, we were kind of looking forward. We heard about the successful landing, but there hadn't yet been any flights. Yep. He joined us earlier, and we started by talking about getting off of Mars time, which he had just been living on. Tim, welcome back to the show.

It's great to have you. It's been amazing to watch five successful flights on the surface of Mars. Yeah, it's been extremely exciting. It's performed beautifully, even beyond our expectations in many ways. So I think, right, you were on Mars time for a bit, and now that's wrapped up. How was that? It's very different. I mean, it's very stressful on your schedule because you're walking close to an hour every day right through the night. And it actually happened at the most stressful time when we were trying to debug the hardware issue that we found on the surface of

Mars. So I was actually hopping between Mars time and Earth time because I was in the lab debugging the issue that we saw. And then we would hop on, if you will, metaphorically hop onto the helicopter in the middle of the night and try some experiments to see if the fix we were trying worked. So it was a very stressful time. But at the same time, we came through it and we were able to fly. So that was very cool. Hopefully you're catching up on sleep now. Yes, for sure. That issue that you ran into, that was what delayed the first flight, right?

Can you give us some details on what happened? Yeah, absolutely. I don't know if you're familiar with what the concept of a hardware watchdog is, but it's a hardware function, if you will, that monitors a CPU or a processor and the software running on it. It's a form of a dead man's switch. Remember those movies where the bad guy would hold the switch in his hand and if he got shot by the police, it would open up and he would blow up, right? Right. So watchdog timers are a form of a dead man's switch that's used very commonly in embedded software to make sure that the processor is behaving. So if you're in a safety-critical function or, you know, very timing-critical function,

and for some reason the software or the processor goes off in the weeds, a watchdog will stop that processor from doing something bad or reset it. And the helicopter hardware has a watchdog timer. And the way the watchdog would work is once the software is booted up, it gives it one second to boot up. This is on the flight controller, which is the microcontroller, not the main Linux processor that we've spent a lot of time talking about, but the microcontroller. Ah, some of those hardened ingredients you mentioned. Right. And so it gives it about a second to boot up.

And then it looks for essentially a special code that we send it. We send the hardware. There's two codes we have to send alternating. You can call it A and B, where in computer E's, the B is the one's complement of A. So that makes it so the software has to do something deliberate. It flips all the bits of the A pattern to do the B pattern, and then flips all the bits of the B pattern to go back to the A pattern. And so the design with the hardware was that the software has to send these alternating A and B patterns every cycle on that microcontroller,

which is two milliseconds. That's the duration of the cycle. Very short cycle. So at the end of that one second window, the FPGA, that's a field programmable gate array. It's a form of programmable hardware that's very popular amongst embedded designers. If you have like hardware, you know, which is all the individual components on a board, but then you have these field programmable gate arrays, which allows a hardware designer to design in specific functions into the hardware. And as long as that hardware is still on the ground, you can plug in a cable and update the hardware design without having to fabricate new chips or anything like that.

So it's very popular among hardware designers. And so the software boots up and starts sending the A, then the B, then the A, then the B. So depending on how long it takes for the software to boot up at the end of that one second window, that hardware would see either the A pattern or the B pattern. Does that make sense? Yep. I'm following you. And according to the design, the hardware was supposed to accept either A or B. And we found out once we got on Mars that the hardware actually required that you send A first. And over time, as the hardware has settled in, if you will, and each hardware has

its own clock that can run slightly slower or slightly faster, that's what clocks the processor. That's what clocks the processor. And over time, the boot time of that microcontroller drifted such that on Earth, for the most part, all through our testing, it just happened to send the A pattern. But then when we get to Mars, go figure, it decides that it's going to boot up either slightly slower or slightly faster. And the first pattern that the hardware saw was the B pattern. This is probably not the kind of incident you expected. I would imagine you thought some sort of physical damage might happen or something might get jarred loose. I bet you weren't expecting something like this.

So how did you even start the research process on that? It totally came out of the blue. It's very interesting, but this is not atypical for a project that you sweat and you stress and you lose sleep over things that you think are going to be a big deal. And they don't end up being a big deal, but something comes completely out of the blue and smacks you in the side of the head. And this is really what happened. And we basically just opened up the FPGA code itself, they have simulations they can run on the FPGA hardware, and they're able to reproduce the case where the software boots up slightly faster or slightly slower and sends the B code

to the hardware instead of the A. Wow. Good thing that simulator is accurate. Yeah. It's a nanosecond accurate simulator. So we were burning the midnight oil. We were up many, many days, you know, 15 hours a day, just trying to dig through and find out what happened. And once we had nailed it, even though it's on Mars, there's always a maximum that once you can reproduce it and nail it, you have a pretty darn good chance of fixing it. It's the ones that are really hard are the intermittent ones that happen once in a while. So it's hard to reproduce.

But after about a day of just intense study and simulation, the hardware designers and I were actually able to figure out what was going on. Then we had to come up with a solution. We actually found, if you will, a hardware hack to get around it. There's really two solutions. One would be use the existing software and find some way to get it to fix itself, if you will. Or the second alternative is a software update where we change the software so that it, in essence, sends both the A and the B pattern every cycle instead of every other cycle. And so the hardware sees the A pattern first and then the B pattern and then it's happy. So there was a relatively easy software fix. You say relatively easy in the code change there,

but I don't know, the prospect of updating remotely, that seems a little dicey. Right. Yes, so getting it to Mars is the challenge and we did it. I mean, we did the software change. So we were trying to develop these two paths in parallel. So you can probably understand why this was a very stressful time,

because at the same time I was debugging the issue in the hardware with the hardware team, I was also making a software update and testing it on the rover that we have in what they call the Mars Yardage APL, which is a replica of the rover on Mars, because there's a whole set of procedures we had to develop in the space of 24 to 48 hours, all these procedures for updating the software on the helicopter and test them. And so I was doing all that in parallel. My wife was getting very nervous because I was so stressed about it. I wasn't eating that much. And because there's all

these eyes on it at all levels up to NASA wanting to know how we were going to fix this. And so it was definitely one of those Apollo 13 moments. And so we got the software tested. It's actually the software update is sitting on the rover if we decide to use it. But we actually found a fairly clever workaround on the vehicle itself that did not require the software update. So that's got to be the safer route then. Yes. And it actually shortened the time because we had this very limited window. We could do this experiment on Mars and time was ticking. And based on my own estimates, based on all the times it would take to transfer the software to the helicopter and then retest it, we would probably lose at least a week.

What is it using to communicate back to the rover, the helicopter? Is it a Wi-Fi network? Is it something else? Well, you're going to laugh at this. It's a Zigbee radio. No way. No kidding. Yeah, it's the same radio people use in their IoT devices in their home. Yeah. And we have one on the helicopter, and then the helicopter has an instrument payload on the rover called the base station. So they both have these radios and they talk to each other. That's how we communicate back and forth. But what we figured out was on the helicopter, I mentioned a software update. We had built in from the beginning, the ability to update the

software on this microcontroller. We actually did it many times when we were testing because we'd do some testing. We'd say, oops, we need to correct that issue in the code or try this different approach. And so we would update that microcontroller. So one thing you don't want to have happen is as you're updating, right, you're writing to the flash on the microcontroller, which takes a certain amount of time. You don't want the hardware saying, hey, you forgot to stroke my watchdog, so I'm going to shut you down, right? That would be bad. Please leave your Mars helicopter plugged in while your system is updated. Yeah, exactly. And so we had a special, we called it the patch mode in the FPGA,

where when you put it in this patch mode, it would disable this watchdog check for as long as you're in that mode. And then when you would exit that mode after you were done patching, the hardware would look again for those watchdog transactions that I mentioned, the A and the B. So we found out in our lab at JPL that the timing is, there's all kinds of charts and things that would probably bore you. But we actually did a whole bunch of analysis and we found out the timing was just right, that if you exited this patch mode, it would actually make it so that that A transaction will line up about 85% of the time and it would work. We're like, wow, how do you like that? Right? And so in essence, we have these software backdoors because that's what software people like to do.

And we have these commands that are already on the helicopter that allow us to poke into the hardware directly and change that mode to be in this patch mode, and then we can exit it. So normally when we're doing patching on the software, it's a much higher level function. You send a command that says, I want to write this file to the flash on the microcontroller,

and the software executes at a high level to do that. And obviously we didn't want to update the software, we just wanted to put it into this mode temporarily and pull it back out. Does that make sense? Just to adjust the timing to kind of get these cycles between the two systems more in sync. Yeah, to kind of force it to go in and out of this mode. And as we found out in the lab that when it exited this mode, serendipitously, 85% of the time, it would be happy with that A-B pattern. That's excellent. Very clever. So every

sequence that we run, every command set that we run now during a flight, it has those transactions in there to basically hotwire this hardware mode to force it in and out so that we can fly. And we did have one flight. We had flight four that canceled because we just caught the timing wrong. I was going to ask you about that. I was wondering. Okay. Yeah, the fourth flight on April 29th, right? Yes, that's right.

So that's a case when the timing was just off slightly and when we exited that mode, it didn't work. And so the sequence failed out safely and we just ran it the next day and it worked. Wow. That is, I love hearing clever solutions like that, that avoid the greater risk of, you know, remotely flashing the copter. There are a bunch of Star Wars nerds on the project and if you look on twitter on the jpl twitter feed on may the 4th we have the helicopter sent out a may the 4th be with you message back to earth so that was kind of fun but we have sort of a running funny meme for that workaround that we found to get the helicopter to fly

and that is the scene when han solo is trying to get away in the millennium Falcon and he walks in and the Millennium Falcon won't start. So he turned around and he smacks the panel and everything lights up and he can go. Right. That's kind of what we're doing on the helicopter every flight. Hey, if that's what it takes, it's still making history. So what then has worked better than expected? Like what, what were you maybe very concerned about that turned out to be a non-issue or something you didn't consider that's just gone fantastic? Well, we were really sweating the energy story because the helicopter is small relative to the size of something like the rover. So it's difficult to model the energy, meaning how much energy the solar panel will absorb and fill the batteries, how much heat leakage there is out of the

helicopter, and how much it will expend overnight keeping the helicopter warm because we have to keep the batteries warm overnight or else they can go into a deep freeze and then, you know, the helicopter is done. And so we were really, we were spending a lot of time trying to figure out what this model would tell us. And the model was pretty pessimistic and it said, you're kind of borderline on energy. And so we were coming up with all these contingencies about, well, how can we do it at a different time of day?

Maybe we can only do one flight because that will exhaust the battery and it won't keep us awake at night. And as it turned out, we dropped on the surface. And then the next day when we talked to it, the batteries were bouncing off 95%. Wow. We were like, this is fantastic. And basically, you know, the battery health has been great. There's been even some days when we wake it up at noon and it's sitting right at 100%.

It's, you know, fully charged by noon. And what that really allowed us to do was to extend the maximum time that we could fly. We had been advertising, and you may remember this from the first time I spoke to you, that we had been advertising that we could fly up to 90 seconds. Because past that, we didn't know if we would have enough energy left after we landed to keep the helicopter warm through the night. Well, we had so much energy, we basically took the limit up to two minutes of flying. And that's what allowed us to do this flight number four, where we went 130 meters to the south and then back. That's essentially two and a half football fields in one single flight. And that was a pretty epic trip. And all kinds of great things came out of that

trip. We were able to scout the landing location for flight number five, which is where we are right now, and get some really, really good imaging on the way of the surface. We actually used our black and white camera and took on the order of 60 pictures as we were flying towards scouting, you know, scouting towards the landing location. And we got a whole ton of pictures right as we approached the landing site and we were able to take a bunch of color pictures. And when we went back to the north again and landed, we still had really good energy. We were sitting at, I want to say around 60 to 70%. Amazing. That's got to be beyond your expectations. Yeah. We were just elated at that because it opens up so many possibilities.

And it seems like we're just energy rich, which we just didn't expect. So what about the Linux systems? Last time we kind of talked about the journey it took to get Linux involved at JPL and onto this copter. And we were talking before about some of the hardware and low-level software details. But you were talking about the images. How has the Linux stack actually done? Well, the processor and Linux have done very well. We tripped over one thing.

It's kind of funny in a way. We have a command on the helicopter that if we want to list all the files in a particular directory, it takes each file in the directory and it gives us the size and the name, but then it also computes what's called a checksum. And basically it's sort of like MD5. If you're familiar with that utility on Linux, it gives you a code that says, you know, this

is a check of the actual binary itself. Oh, sure. And we found out that on some of the larger files, it would take sufficiently long to compute that checksum that on one, actually two different days on Mars, we actually tripped the, remember I talked to you about the watchdog on that microcontroller? Well, there's actually another one that's watching the Linux processor, the Qualcomm ARM processor that has a three-second expiration. And apparently, the time it took to read those files and compute that checksum was just, it locked up the Linux kernel just long enough that it tripped that watchdog. And we ended up getting the processor reset. And so that was one of those

moments where we're looking at the data on the ground and we see about 50 files into the listing, and all of a sudden the helicopter goes off the air. And we're like, that's not good. Uh-oh. Oops. But, you know, the way we designed it, we always set the alarm clocks for the next day before we do anything else.

And so the helicopter just came back the next day just fine. But there was about a 24-hour period there where we were like, yeah, we know that's the design, but it also just kind of vanished and we hope it comes back. That's a long time to wait for the next log time. Even though you know that's how it's built, that's got to be weird. Yes. And I was kind of joking with one of my co-workers, one of the other engineers on the project. And I said, that would be a little embarrassing if the helicopter went down in history as being killed by LS. You know what I mean?

I hope this is sort of a success story for future experiments that might use Linux. But that wouldn't have been if that was the case. Yeah. And it's not Linux's fault per se. It's just doing what it likes to do which is you know when it does file accesses it right it's not a real-time operating system so it says okay you want me to go walk through this huge file i'm just going to grab the kernel for that time yeah and the watchdog did its job yeah exactly it was one of those things you discovered i i remember there was a very senior manager jPL who's been doing projects for years and years and years.

And coming into these projects, I always assumed there was this ironclad requirement that you exactly know every single thing that a spacecraft is going to do once it gets where you want it to go. And he said very plainly in a meeting, he says, well, you know, when you get to a location, you start to discover the personality of a particular spacecraft. You can't test every single scenario on Earth the way you would encounter it because you're not on Mars or you're not at Saturn or you're not orbiting Jupiter or something like that. You really start to discover over the time what the personality of a particular spacecraft is and the things you have to avoid, the things that you thought were going to go wrong and didn't and then like i said the things you didn't even have any idea would go wrong but did and so really we've been learning the personality of the helicopter as it's been sitting there on the surface of mars and we just learned don't

use that command because as i said it's not the fault of Linux. It's just a feature, if you will, that on a normal desktop would not be a big deal. But we have this watchdog watching over us. So, ironically, what we ended up doing was we do have yet another backdoor on the software that allows us to execute a command on the shell as if you were sitting there at the Linux prompt and typing something. Oh, cool. Okay. So you can run pretty much anything on the helicopter you want. And what it does is it takes the standard output of that command and puts it into a log file that then you can transfer back to Earth and see what actually happened.

So what we did is after we tripped over that particular behavior, we actually switched to doing LS, actually running LS. One shell command slowly at a time. Yeah, we just, and we found out that because LS was running and we put it on a different core on the processor, right? We directed it to a different core. We can do that with this command. We can say run on the third core as opposed to the first core and we can direct it.

We basically, since then, have been doing LSs on those directories and then running the MD5 command on those same directories because we use the sizes of the files that were generated during the flight to estimate data volume that the rover has to downlink to earth and how long it will take because it takes a certain amount of time to transfer from the helicopter back to the rover and so on and so forth. So we needed those sizes. So we basically just switched to using LS. Core utils to the rescue. I'm kind of taken aback by the fact that you are actively essentially using the shell on this copter on the planet of Mars from Earth. That's just incredible to me. Yeah, it gets even better in that we do spend a lot of time with the helicopter up transferring these large logs and images from the helicopter back to the base station that I've mentioned on the rover

over the radio and then subsequently transferring those files from the base station to the helicopter. And because those files are large, it takes a significant amount of time, you know, to transfer those on each hop from the helicopter to the base station, the base station to the rover. It consumes significant amount of time, you know, to transfer those on each hop from the helicopter to the base station, the base station to the rover, it consumes significant amount of time. And so once we started using the shell command, we just said to ourselves, you know, why don't we just try compressing these files and see what happens. And we did some experiments here on earth and the good old fashion BZIP2 utility, you may be familiar with it on Linux. We basically do BZIP-K because we don't want to delete the original files. And we take all these log files and we compress them into BZIP files. And then we transfer those.

And that actually shaved probably 30 to 45 minutes off of each hop from the helicopter to the base station and the base station to the rover. It significantly cut back our transfer time just because we just compressed them by running this utility on the command line on the helicopter. And how are they transferring between copter to rover and up? Is it some proprietary sync solution? Is it SCP? I mean, what is going on there? It's an old-fashioned serial UART. I don't know if you're familiar with that. Remember the serial ports from the nineties and the eighties, right? They essentially have one of those on steroids.

It's, you know, the signaling is very robust so that you don't get noise in the rover that can interfere, but it's, it's relatively slow by modern standards it's a 115k baud connection between the base station and the rover for those of you played around with modems in the past right you remember how you have these standard speeds of serial ports based on the 8080 pc back in what 1982 they've just kept them kept these standard speeds for 40 years. So they set up these connections between the rover and the base station. And so we initiate a transfer across that serial line, that UART line, from the base station to the rover. And so the rover then takes that data and stores it in the form of its own internal files. And then whenever the next pass, the spacecraft pass, when the spacecraft goes overhead, they relay those files up through the spacecraft and back to Earth. So when we're

actually looking at our data, we're seeing a bunch of these binary files on storage on the ground, and we just run our tools to decode them back into the data that we got from the helicopter. Wow, that is really impressive. What's next for our little helicopter here? And what's next for you? I mean, you're off Mars time now, right? Yeah, although Mars time is sort of aligned with Earth time because it circles around. But after a certain point, the rover projects, they realize that the human cost of just always going to Mars time, it just wrecks people's schedules and they get more and more tired and more prone to make mistakes. Somebody think of the humans. Right. So what they do is for the first part of the project, this is very common for Mars projects, everybody's on Mars time because they want to make sure the vehicle is working and stable

and doing what it's supposed to be doing. And then they go to these kind of a modified schedule where when Mars time is in the middle of the night Earth time when humans don't want to be up and doing things, they'll take one day on Earth and they'll schedule two Mars days. They'll do two things at once and then they can process it during the day when humans are awake. So the shift kind of slides from early in the morning Earth time to later in the afternoon and into the evening Earth time. So it's still sliding, but it slides during the Earth day. It doesn't go through the night anymore. That's a little nicer on your schedule. Yes.

So they're able to juggle and not be as hard on engineers doing it. But because Mars is cycling, you know, every three weeks or so, it's working its way through a normal Earth day. And you can get, if you will, results the same day that you do an activity. So Mars is kind of in that mode right now where it's about three or four hours behind us, if you will. And so we're able to get the data and look at it on a more normal human scale. But we did finish our 30 Sol experiment experiment and we met all of our requirements. I mean, our original promise was we would do these three canonical flights and we pretty much nailed them. They were great. And we always said that flight four and five would be more

adventuresome flights based on what we discovered in the first three. And we did exactly that. So we're able to check the box and say, yeah, we did it and we did it in spades. And we met our criteria and we got all the data that we want from those flights for future projects to use to design their own particular missions. And really validate that this is a worthwhile investment. Yeah. So we switched modes, if you will. The first 30 sols were, it's a technology demo. We're just trying to see if we can fly on Mars. It's really what we're trying to show. Was this even possible? Yeah, exactly. Nobody knew if we could do it. I mean, that's the bottom line. A lot of people were skeptical. Yeah, well, it's very nice

that you had a chamber, you know, but you had a cable. You don't know what it's going to do when it gets there and so on and so forth. And we, it just nailed it. Our chief pilot was looking at plots of altitude and position and rates and all that. And he says, if I had looked at this six months ago, I would have said it was from our simulation, not from Mars. It was so close. We were just shaking our heads. It's another one of those things that we thought might bite us, but never did. We were worried about wind tossing us around. We were worried about, you know, the performance of the flight system not being what we expected, but it was just dead on. And we also got these pretty glorious color pictures as we were aloft. You know,

we were up at five meters. That has to feel like real tangible, you know, something you could take away from all of this work. Because that copter is up there forever now. But you have these pictures. That's real tangible proof that you were successful in this. Yeah, and the pictures came out really great. And if you go to the JPL website for the Mars helicopter, there's all these great color pictures that we took. And you can even see in one of them. That's very cool.

You can see in one of them, cause it's sort of fisheye though. You can see the edges curve and we actually caught perseverance in one of our pictures. It was off in the distance watching us with its camera and you could just see it at the corner. So there's actually this really cool picture where we're looking South when we're on that flight three,

when we're flying to the North and we're taking pictures sort of as we're flying backwards. And you can see in one shot, you can see the rover off to the left. You can see the helicopter landing spot in the middle. And then to the right, you can actually see the place where the rover landed. It still has the blast marks from the rockets as they were landing. You can look in the picture and you can see these two scoured circles where the rocket blasts scoured all the surface clean just as the rover was landing. You get this great picture of everything all at once. And so I'd encourage everybody to go look at that one picture. If you look around, you can see it. But that really proved that we can actually fly and it works.

And the rover project saw these color pictures and they were just like, wow, these are really good pictures. And so they said, you know, since the helicopter is doing so well, we're going to actually propose this extended mission, if you will. And it's really an operations experiment, you can use that term, where we're switching from just seeing if we can fly and surviving to, well, let's try to fly around and act as a scout. So we'll fly longer distances, we'll fly over more challenging terrain, we'll take pictures and we'll bring them back down to Earth and we'll say, you know, how could a potential rover mission or other mission use these pictures? And the rover team doesn't need us as a scout. They never did, but they're still going to opportunistically look at these pictures and see, hey, maybe there's something over there interesting. Yeah, it's valuable data for them. Yeah, so the idea is that we're going to start looking at,

we haven't fully planned all the flights yet, but we're going to start looking at these more distant features and locations to fly over there and get some good shots with the color camera and, you know, bring them back and see how well they can be used by people to figure out, hey, there's a cool feature there. But we aren't going at the same cadence we did because during April, April, they dubbed the month of ingenuity where we pretty much were the whole show. The rover was prioritized to work with us to get flights every few days to get the data between that. And they are actually moving on to their own mission, which is to start to explore that southern area.

It actually helped because originally their plan was to do a sprint away from the landing zone to get to the what they call the delta which is where all these interesting geological features are but they've decided in the meantime that they want to poke around to the south of where they landed and look at some interesting stuff they've discovered down there up close so they since they were loitering if you will down in that southern area for a while they said well why don't you know we're not going to drive away from the helicopter for a while. So why don't you guys do this operational experiment? You won't fly as often. We're flying probably once every two to three weeks instead of every four or five days. It'll be more intermittent. And they're deprioritizing our data. Like, for instance, for Flight 5, we're just starting to get the data down from the vehicle because they're prioritizing their own science instruments, which we're totally fine with that.

This whole thing is a bonus for us. We're living within the new paradigm, which is let us do our own thing, maybe not as often, while you're doing your thing on the rover side and see what we can get out of it. on the rover side and see what we can get out of it. So we're going to probably be flying every, as I said, every few weeks and seeing how well it works potentially through the summer. Will you pick up and move to essentially where the rover is going to be going to? Because I would imagine you have to stay within communications range. Yes, we will stay relatively within communication range. We found out that that's the other thing that we were worried about

was we didn't know what the radio performance would be. We were worried that obstructions would get in the way that we'd have what they call a multi-path where the radio waves bounce off things and cause interference with itself. And what we found is the radio is performing beautifully. It's almost amazing. We actually keep a counter on the helicopter of how many packets were dropped. So the helicopter attempts to send a radio packet to the base station. And if for some reason it doesn't work, it tries again. It tries up to three times. And during, I think it was flight five, we received our first single counter that said that we had dropped a

single packet. So it's kind of mind boggling that over the entire project up to flight five, we didn't drop a single radio packet. Doing better than my Wi-Fi at home. That's the truth. It's kind of ironic that there was a couple of times when the internet connection failed at home and we were doing operations, talking to the helicopter on Mars through my iPhone hotspot. Amazing. And the radio on Mars is working great, but not evidently my internet connection. Oh, wow. Infrastructure's hard. Well, that is, that is just, it's such an incredible story and it feels like it's a technology success. It's a human success and it's just been also a historic event to witness.

And I just, I have so much gratitude and just want to say so much congratulations to you and the team. Just remarkable what's happened here. And I really appreciate you running down just some of that with us. I feel like I could talk about this with you for the entire day, but I know at some point we have to bring it to an end. So Tim, thank you for joining us and updating us on that little Linux copter. You're quite welcome. And it's been great

talking with you guys. It's always good to talk to people who are space geeks like ourselves, because, you know, we really are space geeks living the space geek dream, right? If you think about it, flying a helicopter on Mars,

how many people will get to do that? And so having me as a kid from a small town, being able to do something like this, it's still hard to process sometimes that, you know, we're going to go down in history with people like, you know, the Pathfinder, the Voyagers, all these different amazing historical spacecraft and projects. And we get to take our place in that group that it's not a pride thing for me. It's just a sense of amazement that I get to do something like this. Linode.com slash unplugged. Go there to get $100 in credit for 60 days on a new account.

And of course, you support the show. $100 in credit. Well, that lets you see why Linode is our hosting provider for everything we've built in the last couple of years. Linode's infrastructure is solid. It's flexible. And you can focus on your project, not your infrastructure. You get 11 data centers to choose from around the world. And every service level, every Linode, every data center is just backed by the best customer support in the business.

And, man, when that matters, it really makes all of the difference. And at every step of the way since 2003, Linode has asked themselves how they can use Linux to accomplish their next task. I know how that feels. They have built on Linux in brilliant ways and they've contributed back to the Linux community in a lot of brilliant ways too.

I love that their dedication to the platform is baked into their product. And us longtime Linuxers, if that's a term, you can really see it. You know, that's special when you're using a product and you can kind of notice that the company behind it truly loves Linux. I love picking up on those things.

And a few months ago, just as an example of how sometimes I'll randomly use Linode for my own personal stuff, I wanted a cloud-based component to my sync thing setup because I like to sync between the studio and I like to sync to the LadyTubes RV. And one sync server to another sync server, it works, but it's kind of slow. You know, I want like more transfer. So you can tweak sync thing and you can really open up the floodgates if you bring in another sync node. So that's what I did.

That's what I used a $5 Linode for. And now I kind of use it to do a couple other things like jump posts and stuff. But, you know, SyncThing, it's great. It's a great open source project and it just transparently is moving my data at the file system layer. I don't even notice it. It feels like I have a network-wide file system. at the file system layer. I don't even notice it. It feels like I have a network-wide file system.

And Linode's so fast and so reliable that I move a ridiculously impressive amount of data that I'm kind of embarrassed to say, and they do it just with no fuss. So head over to linode.com slash unplugged. Get that $100 in credit for yourself. Go see what it can do. Go try this stuff out. Go build something.

And, you know, maybe learn something. There's a lot of ways to host, but there's no company like Linode. Go see why we choose Linode every single time at linode.com slash unplugged. You know, Wes, we have a few things to get to this week because we're going to be traveling over the next couple of weeks and off the air. So we have some details we should probably cover. You know, we still need to find the keys to these chains. I don't know if we're going to be able to leave the studio. Yeah, you know, these podcast locks, they're pretty tough. So for the next two weeks, we won't be live. We do have some great content lined up.

And then we'll be back in studio, all chained up on June 8th. In the feeds next week, our AMA episode comes out. And it was a lot of fun to record. We did it with the LUP plug this last weekend. And I think you're really going to enjoy it. I was surprised at how great it turns out. And you still have a chance to get your questions in. Go to asklup.com or use the standard contact form at linuxunplugged.com slash contact.

The LUP plug will continue on while we are traveling, though, every Sunday, noon Pacific. That's in our mumble room. We have mumble information on our website. And we have our calendar to get the live time or the plug time at jupyterbroadcasting.com slash calendar. Oh, I should have mentioned this earlier. I made a thing. I made a thing and I want to share

it with you guys. I made like a $5 Linux can fly sticker to celebrate this incredible milestone that Linux is flying on the surface of Mars. The first rotocopter flight on another planet was made possible by Linux and free software. And look at how cute this is. Bold, brave on the surface of Mars. I made this myself, Wes. What do you think? Are you impressed? It's the rotocopter on a stylized version of Mars in a cut sticker.

So it's, what do you call that kind of? Yeah, it's like a die cut sticker. So it's got, you know, the wings of the rotocopter or the blades. It actually extends a little bit out. So it's not a perfect circle and it really gives it a really nice 3D depth effect. Yep, and it's just five bucks.

So JB makes like probably $1.80. But if you're a member, if you're a core contributor, you can use the same promo code I gave you a couple of weeks ago, and it will take 15% off even this. And it's at jupitergarage.com, and you can grab the Linux flies on Mars sticker that I made myself.

I'm very proud of it. And I totally ordered a couple myself because I want them around the studio. This is, every time I look at the sticker, I'm going to think Linux and free software flew on the fricking surface of Mars. It's pretty awesome. So I wanted a way to celebrate it

and I thought, why not make it available for everybody at jupitergarage.com? So go check that out. And I won't be mentioning it again because we're traveling, and the shows are already recorded. So you just got to remember yourself, I suppose. And last but not least, I'll mention the Telegram. We'll still be popping in there from time to time, even when the shows are prerecorded and they're not live. We'll still pop in there, jupiterbroadcasting.com slash telegram.

Okay, so it's time for an exit interview with PeerTube. It has been great. We are so excited about the future of this project, but we have some problems. And so we have decided to make the tough choice about decommissioning the Jupyter Broadcasting PeerTube instance that we set up in episode 388 of the show. So a little while ago. Not something we do often or necessarily take lightly. No. No, we generally don't.

We generally just keep it running and support it. And support it. But, you know, we also want to, this is really about refining the scope of software and services that we maintain and not overdoing it and keeping what limited time we have invested in the right areas. And part of me just wanted to keep this going and just kind of experiment with it. But I've been getting feedback about it. And there's kind of this public perception that if it's online, it should be a little bit more than it is. perception that if it's online, it should be a little bit more than it is. So I have put up a longer explanation of our decision at jupiter.tube if you'd like to read about this. Put just really kind of short though, I think PeerTube is and should be the future of decentralized video on the web. There are other platforms out there right now, but PeerTube is a pure peer-to-peer play with no silly cryptocurrency that's going to be a problem for them down the road. It's just a YouTube in a box that you can install yourself.

And I think it is good to go today for open source projects like a distribution or a community event that maybe wants to do a little live streaming and then make their talks available online. I think personally, PeerTube is there. And so this is not taking away from any of that. It is an incredible piece of software, and they only recently rolled out live stream support, and we were both very impressed with what they've done there. Oh, yeah. I mean, there are still some issues for our use case,

but just to add that at all in a project like that, including, I mean, initially web torrent support, and then now they've got a new P2P HLS-based system, that's fancy. And they've got some really clever tricks with FFm to pull in a few videos, um, like an import feature from Twitch or really anything that YouTube DL supports, which is crazy fancy. Um, really it comes down to when you're using it at scale on an ongoing basis for like, you know, four or five shows plus two or three or four live streams and a team of more than a handful of people. And you start adding all these things together and it's kind of unclear how you scale multiple shows across multiple channels. Do you set up an account for every single show and like a network account for live streams and then personal accounts for each staff member who's going to be on there and interacting

with the comments and interacting with their shows? It just becomes an incredible amount to manage, but it also becomes kind of cumbersome to automate because there's like all of these little okay we have to go create this account do this instead of just like an api that we can use to to just upload video productions as our automation system completes them the live stream support is really solid but it's still a little tricky to use especially if you want to have your live stream auto-publish after you're done. If for any reason during a live stream, if you drop, PeerTube will not allow you to reconnect to that stream.

And for some intermediate services like Restreamio, when that kind of lockout happens, they just mark the entire RTMP feed as always bad. You have to go delete the integration and reset it up, which is disastrous during a live stream. And then really, and this was what I was worried about when we started, there's no great way to mass import, right? So they have great ways to import maybe a YouTube playlist or... Yeah, you just want to pop in a link or something, easy. But if you have a whole SFTP server full of backlog

and catalog... It just doesn't even support that. What we would love to be able to do is some kind of directory import. We could point it at an FTP or SFTP URL or whatever with maybe a text file in there that has some of the metadata like name and description

and then have it import 400 videos. so that way we could do a mass migration but they don't have anything like that there's no rss feed import either which would be really nice so that while the video import options are good today they're just not as expanded as we need and so since it doesn't address all of these use cases and we've also seen kind of a fade in adoption as people have kind of went back to youtube and twitch um we're going to just take this moment to hit pause on it we're going to take it private we're going to continue to have it installed and upgrade it and

test it privately and then when we think it's gotten to a point where we can support it fully and, and, and like go all in, like just really go all in where we, we publish every single show. We have channels that you can subscribe to. If you just want this one show, all of it's tied into our automation system. We figured out the storage issue because storage does become a problem. Even in our light testing,

we've already eaten up 30 gigs and we've only been publishing two shows. So, you know, all those little things have to be worked out and then we're going to make it public again and try to go all in on it because I really do like it. I think it is very good software. I think it's one of the more exciting free software projects out there right now. So it does hurt to not be able to just totally embrace it right now. But I have to be realistic about what our abilities are as a small team. And so what I've done is I've put together a page that goes into some more detail. Specifically, I talk about how I think they can maybe defeat the network effect that YouTube has. And I tried to link to the previous coverage that I could find, which looks like four previous shows, and I gave timecode links to those segments if you'd like to hear our past kind of discovery

of PeerTube to implementation of PeerTube. And now to this phase for now, and that'll all be up at jupiter.tube. I think we just found ourselves in a sort of awkward spot of not being all in, finding some rough edges that we weren't really prepared to necessarily deal with or wait for the project to get to in time. And we liked it so much that, yeah, if we're going to use it, we want to do it right. We want to go all in. We want to be able to have it as a firm foundation for the future of Jupyter Broadcasting, which would be really nice. I mean, it would be really nice to know, like, here's a spot we can put stuff. It doesn't matter if YouTube's going

to choose to demonetize it or censor it or take it down for some other dumb reason. We've got our own accessible catalog in a way that's a lot more useful than some of the archive systems we have now. But to do that's going to take a lot of work, right? It's going to take investment, and I think we have to evaluate when the right time is to expend that energy.

But I am looking forward to continue to play with it, and already since we started using it, they've updated and made a lot of nice improvements, and I'm sure that's forward to continue to play with it. And already since we started using it, they've updated and made a lot of nice improvements, and I'm sure that's going to continue. MailRoute.net slash Linux. Go try out MailRoute today and get 10% off the lifetime of your account and start with a 30-day free trial. No credit card required.

How great is that? Support the show and try out a service that is going to make email a lot better for you. For 24 years, MailRoute has focused on its core competence, providing cutting-edge email security. That's mailroute.net slash linux. We're using MailRoute for the new Jupyter Colony mail server, and MailRoute can protect your mail server with a suite of services

designed to remove spam, viruses, and my favorite aspect, prevent downtime. And I know that sometimes it can be tricky with your ISP. Perhaps they're blocking certain ports or trying to play tricky with your email, or maybe for some reason your server ended up on a blacklist and so it's not getting email anymore. It's just gone into the black hole. Yeah, I hate that. And I've had to help clients fight that problem too. I've deployed MailRoute for my clients in the past, and now I'm using MailRoute today, and I want you to try it out at mailroute.net slash Linux. It solves that downtime problem. If you need maintenance or maybe you just have an outage,

they will queue email up for you, and then when your email server comes back online, they forward it right on. They were the first to build an email filtering service back in 1997, and that's what they've been doing since. They provide one-click migrations for both Office 365 and Google G Suite. And they have very simple onboarding realistically for any mail server. It's just you change a couple domain records, you add a couple of settings in the mail route service.

Then you're good to go. And you're getting real-time logs. You're getting their filtering. And you also get critical compliance for federal government contractors and others that have to meet CMMC compliance. So go check that out as well because that can be a huge plus. Your account also includes controls to stop spam and phishing attempts and viruses and get a whole heads up on all of that. When you need it, MailRoute is there for you, and they'll make email better. So try out MailRoute today and get 10% off up on all of that. When you need it, MailRoute is there for you and they'll make email better. So try out MailRoute today and get 10% off the lifetime of your account.

And start with a 30-day free trial by visiting mailroute.net slash Linux. Protect your server, protect your business, and just make email better with MailRoute. mailroute.net slash Linux. We have a really great pick for those that want super high quality music, but want to host it themselves and are looking for something to take the music off their server and get it on their desktop or maybe their mobile client. We have got a couple of great picks. And this wasn't even inspired by the fact that another music service announced their, quote-unquote, lossless audio streaming, which is all fine and good, but I got a library full of flags, Wes.

Yeah, exactly. They might have some things that that online service doesn't have. For sure. But how are you easily going to get access to them? Because not everything's happy with a flag file. So there's a couple of pretty well-known home media hosting software packages. We talk about them pretty frequently on this show and on Self Hosted. Then top of list is Plex and Jellyfin. So let's start with Plex here.

This is how I'm solving this problem today. And it's called Plex Amp. Plex Amp is a mobile app for iOS and Android. And they have a Linux desktop app image, which I think is just like an Electron app, but it's kind of great that you can get it on your Linux desktop. And it is a dedicated app just for streaming music from Plex. It doesn't see the other aspects of your Plex library. It looks like somebody created a dedicated music app

that just happens to use Plex. And it's beautiful, it's clean, and it's really solid, has a lot of nice features, and I can even use it to play audiobooks off of my Plex server. So that's Plex Amp, and you can get it for mobile devices or over on AppImageHub, they have an app image of Plex Amp as well. That's really nice, and it does look beautiful. Now for us Jellyfin folk, there's FinAmp,

which looks relatively new and is a little harder to find. It does support Android and iOS, but it's not on the Play Store or the App Store. You can find it in F-Droid or build it yourself for iOS. Sounds like those things are coming down the road. I like that you got to work for it a little bit. Although I think we should just make a sub pick right here, F-Droid. We've mentioned it a dozen times on the show,

but it's so important that on Android, a alternative app store remain viable. And I feel like just as a vote for alternative app stores, we should all have F-Droid installed on our Android devices. And then you do get some great apps like Finamp, F-I-N-A-M-P. So you do have to have the media serveramp, F-I-N-A-M-P. So you do have to have the media server component for both of these running on your server. This is kind of assuming you've already done that.

But if you have, this kind of just plugs right in. And if you haven't, you really should, because it's great to have your media automatically organized and available on all kinds of devices and streamable to friends and family. It just feels super cool. And it can often be the gateway drug into even more self-hosting. So go check out the links we have in the show notes at linuxunplugged.com slash 406 for that. Also, a shout out to 1Password, who has formally announced 1Password for Linux, their password manager.

And it has some cool details. And they wrote up a blog post that goes into some of it. And it is filled with all the buzzwords you could hope for, yes, including our favorite, Rust. You don't say. You'd almost think we plan it that way. Well, maybe that's why we're talking about it. No, no, what's really nice here is to see them excited about Linux.

And actually, they took the opportunity of developing Linux support to build the new cross-platform foundation for the rest of their apps. I don't think they've actually updated them yet, but it sounds like that's the plan. Yeah. The front end is like a mix of web technologies. TypeScript and React is in there. And including WebAssembly.

Yep, yep. And then as they put it, Wes, they bundle it all together using Electron that allows them to integrate deeply with the operating system. But before you go Electron, it's kind of rad because that acts as a front-end client to this like 99% Rust back-end. They diagram the whole thing in this welcome Linux to the 1Password family blog post where they even share pictures of the team with their Ubuntu boxes, like building the prototype apps and demoing it to each other. It's all kinds of adorable. And while I'm very happy these days with Bitwarden, I have fond memories of 1Password.

But more importantly, 1Password is more and more widely used in the enterprise. So this now means that an enterprise-provided solution, I've had an employer in the past who just gave everyone a 1Password account as part of their security practices. And now that means Linux users can participate in that. Yeah, you're no longer stuck to the web client. And I just thought it was cool to see them sort of knowingly talk about a lot of the features out of the box, like all the integrations with things like GNOME and KDE and the system tray and clipboards and key rings and wallets. Now, a lot of that stuff is made possible with Electron, but you can tell that they know that there are boxes here that they should check off. And that's impressive to me. Like, this is not a lazy Linux board.

They've put in a lot of time here. Yeah, they really have. Very nice to see it. Thank you to our core contributors at unpluggedcore.com. As an ongoing thank you, I should say, we give you two feeds, the limited ad feed, nice, tight, and ready to go, and the longer all mistakes, all screw ups, full live stream feed has a second option. You pick which one works best for you. And don't forget to use your special

promo code to get 15% off the new Linux flies on Mars sticker, which will be available for a little while. And I won't be giving you any more heads up or reminders because this is the last live show we're recording for a while, but you can find it over jupitergarage.com and then use your special member promo to get 15% off. And thank you to our members who make this possible. Thank you to A Cloud Guru, too. They sponsor this show, and you can find them on social media.

They're just slash acloudguru just about everywhere that is actually a social media page. If you want more Wes Payne, go check out Linux Action News. He's over there helping me break down the news that matters every single week like a champ. That's right. Hurrah!

Like a news ninja. He just sort of swings into the studio on a Sunday, lands in his chair, and starts breaking it down, you know? It's kind of impressive. And I'd say join us live, you know, but we won't be here next week. But come back on June 8th. See you next week. Same bad time, same bad station. Only we'll see you in two weeks. You know, I can't travel back

in time and get him to change it. What do you expect? That guy's probably, I mean, I hate to say it, but he's probably not alive. I know you were thinking it. I was thinking it too. So what do you want me to do? I can't do anything about it. It is a little awkward. It's a weird way to go out, but you know, that is what it is. Links to everything we talked about today at linuxunplugged.com, all kinds of links over there. Our matrix server, our mumble server, all of it. So go check that out. It'll help you get even more out of the show when we're not on the air, but we really appreciate you listening or streaming or being a member or whatever it might be. And we'll see you

right back here next Tuesday. Now, we should be clear that contractually, when Chris says, see you next Tuesday, next Tuesday means the next Tuesday we're here and not the next calendar Tuesday. Well, I just think it just means the feed. You know, like, we'll see you in the feed. Because honestly, the shows that we did while we're off the air turned out great. Yeah, don't miss those.