LINUX Unplugged - 396: How Linux Got to Mars

Episode Date: March 10, 2021

Tim Canham, the Mars Helicopter Operations Lead, shares Linux’s origins at JPL and how it ended up running on multiple boxes on Mars. Plus the challenges Linux still faces before its ready for mission-critical space exploration. Special Guest: Tim Canham.

Chris, we have Rust news. What is it, Wes? The Core Utils Rewrite in Rust project, well, it's now available in Debian. And it's good enough to boot Debian with Gnome, install the top 1,000 packages, build Firefox, LLVM, and Clang, and yes, even the Linux kernel. You know, Wes, I think show's over, you know? Show's over, man. Hello, friends, and welcome into your weekly Linux talk show.

My name is Chris. My name is Wes. Hello, Wes. Hey, this episode's brought to you by the all- New Cloud Guru. actually here to talk about Rust. We have a very special out of this world episode. Tim Kennum is the Mars Helicopter Operations Lead at JPL. And he shares with us on the show today Linux's origins at JPL. He was one of its lead advocates at getting it in the institution at the Jet Propulsion Lab. And how it then went from this risky kind of chance OS to running on multiple boxes on Mars. That's something that hasn't been super clear,

but there's lots of new and good information in this chat we're about to have with Tim. And he also tells us about some of the challenges that Linux still is facing before it's ready for the really super critical computer stuff, like some of the control computers that are in the rover on there. But we're going to get to that in a little bit. We'll have some community news.

We got a couple of really great classic picks and a bit more. So to help us massage all of that into something that's listenable, we're bringing in our crack team of experts. It's our virtual lug. Time appropriate greetings, Mumble Room. Hola. Hey there. Hi. Good to see team of experts. It's our virtual lug, time-appropriate greetings, Mumble Room. Hola! Hey there! Hi! Good to see all of you. Thanks for making it here on a Tuesday. It's always fun hanging out with you guys. We've been going for a little bit on the live stream and having a good time. And when I got on the live stream really kind of about a half

hour before I normally do today, it was really in the pursuit of serving the show. It was really because of my responsibility and duties to the show that I had to play video games. And this is pretty great, actually. Probably most of you have seen this. If not, well, you're going to hear about it now from me. The Steam Link app is now available as a standalone thin client app for Linux boxes. And it's available as a flat hub, a flat pack, or it's also linked in our show notes. You can just get a tar of it. And it is a minimal application that was brought together by the

folks at Collabra and Valve to make it possible for essentially any kind of low-end machine that has a fast enough network connection to stream games from another system on the land. It's Steam Link. We know about that. That's been around for a while, but now this is built into a little app. Hmm.

So is the news here then that you don't have to install Steam and get that all set up to use this feature? You got it. And you could have it so you could launch it on your laptop. You don't even have to have a Steam account. And if my machine upstairs was logged in and ready to go, you could stream them right from my computer. And you really don't even have to have anything more

than an Intel graphics card to make it possible. I have two minds here. One, as they note in the announcement, special thanks to Collabora for helping make this possible, and they just do so much good stuff. The other part that's funny about this, though, is what took so long? I mean, wasn't the Steam Link hardware running Linux and presumably running some of this already?

I had that thought, too. I did wonder. It must have just been there just wasn't resources involved to make it happen, and so they, in a sense, outsourced it to Collabra. And it's good. It works really well. It detected my Xbox-compatible controller. It automatically detected my workstation upstairs running Steam. It detected that there was plenty of bandwidth. And then once you connect, you're put into Steam Big Picture

mode, which Valve has put a crap ton of work into. There's a lot of information that is in the Steam client that big picture actually manages to surface to you in a pretty reasonable way, which only serves to make the remote Steam link experience even better. An example is I could launch a game and big picture mode had a nice way to tell me that the Vulcan shaders were being pre-rendered and that I was going to wait a bit. And then it detects that this is going to take even longer. Would you like to just skip it? And it presents all that information in a way that doesn't break the big picture experience.

That's impressive because it's already kind of hacky and annoying just on the regular Linux desktop so that it works at all in big picture on Linux is nice. It's smoother in big picture for sure. It really is. No doubt about it. That's the experience they've put the more recent effort into. So it will detect all of the Steam machines on your LAN,

and then it will detect if you have a controller that's compatible with Steam, and then it detects if your connection to those machines is fast enough. It gives you these three green check marks, or not, depending. And then when you pass those three tests, you just hit start playing. It brings up the machine's remote library. You see every game that's on that computer and you can fire them off. And including Proton games,

I launched Star Trek Online. I played Batman Arkham City. I played a little Left 4 Dead. And I played Hot Shot Racing. And a handful of those are actually just Windows games, but they just work beautifully. I find it fascinating that there's a FFmpeg bundled in here under their Steam link folder.

Yeah, I think it's right. It's got to be. They're just relaying the input events, and it's streaming as fast as it can, an H.264 stream, but it clearly has some allowance for bandwidth because in every game I tried, I did not spot any compression artifacts, and I have an eye for that kind of stuff,

and I didn't see any compression issues at all. Really what I saw is that I need a new computer upstairs because my machine upstairs is just a dog. It was pretty great, Wes. And you combine that with the news that Proton has enabled 7,000 Windows games now to run on Linux of various quality, to be honest with you.

That's pretty remarkable. Well, and that Proton keeps getting updates, you know, like there's a new version of their Soldier Linux runtime container, also Proton Experimental on top of regular Proton releases. It really, it's hard to remember the old days sometimes because it's so good right now, even when it's of mixed quality. But it's just, it's so fascinating to me that some of my, you know, the friends, acquaintances who I occasionally play games with who are on the Mac, and it used to be like there were a lot of games they could play, I couldn't play.

And now that has just flip-flopped entirely. Yes, great point. You don't buy a Mac to play PC games anymore at all. You can play iOS games. But, yeah, you know, I want to put it in what I feel like is a fair context. I feel like Proton is undoubtedly a win. But I feel like out of those 7,000 titles, the reality is only a small handful still are platinum. And that number isn't really changing a lot in terms of Proton support level platinum. in terms of Proton's support level, Platinum. And I have definitely had mixed results, more than I would like.

So I want to say it's, Proton, for me, has worked out to be an 80% solution. I am not complaining about that. But I'm just also trying to be realistic. It has not, even games like Destiny is a great example. Destiny 2 works for a bit, then it's broken, then it works for a bit, now it's broken again. I was about to say that, you know, games are often a great example. Destiny 2 works for a bit, then it's broken, then it works for a bit, now it's broken again. I was about to say that, you know, games are often a moving target, as

is Proton. So I think it's one of these things that if you play a little bit of the older set of games, you're going to have a good time. And if you're on the bleeding edge or AAA world, okay, it might be a little dicier. What's really nice, though, is you bring all this together. You can Steam Link too. Like, I was connecting to a Linux box upstairs. But if you had a Windows 10 machine on your LAN, you could use

that and stream that to all of your Linux computers. That's kind of nice, right? You can hide that away in the basement, pretend it doesn't exist and just present a nice, you know, Linux face on everything. I had a failure state where I had alt tabtabbed out of the Steam Link experience to check something on my local machine. And the alt-tab was grabbed by the Steam Link and my local machine. And so it alt-tabbed out of the game and over to my console. And so when I went back to the Steam Link experience, I was actually looking at another window on a separate monitor on my computer. And I know we've had people write in and they've told us that you can actually break out of the game

and you can actually just completely stream the desktop. So I began using the machine through Steam Link. Because I have the terminal open, I just launched other stuff and I started messing around with the computer and I could manipulate all the windows on that monitor that I was connected to from my computer down here through Steam Link.

You just found yourself a brand new remote desktop solution. Yes, that's what I'm saying, is there could be some ways to have some fun with that. So if you know a way that I could maybe do that on a Chromebook or something like that, let me know at linuxunplugged.com slash contact because I saw that peek behind the curtain and I realized this could be a killer way to just remotely stream any application from a machine to another machine and get GPU acceleration. That could be magic.

Linode.com slash unplugged. Go there to get a $100 60-day credit. And, of course, you support the show. Linode.com slash unplugged. Linode is where we host everything. We had a challenge on the show last week where we asked the chat room to go after a machine, and we built that on Linode because we knew we could take snapshots, we knew we could destroy it if we needed.

And one of the methods that we took with that is we built a test system first to see if it was possible on Linode, and then we cloned it. And the clone was actually what we put out into the public. And Wes and I had a control test machine that we still had up and running that we could have flipped public if we needed to. And it's really simple to do all of this with Linode. And it makes approaching upgrades and managing your servers feel a lot less risky

when you can get an automatic snapshot that is clearly labeled and easy to understand in the dashboard to tell you exactly where you're at. And you can just run one immediately if you need or even clone the box. Their cloud manager is top-notch. And they try to make it approachable to everybody, and they do a great job at that.

But what I really have appreciated is usually a click or two away, I can actually get the commands I need to run on the Linux box and get even deeper. They really don't hide that stuff from you. And if you're a longtime Linux user, you're going to appreciate that. And it's really impressive

how they've managed to strike that balance. And it's not just by accident. They didn't just figure it out overnight. Linode's been around since 2003, and they've really focused in on doing this type of stuff really good. I was telling this story on Linux Action News, and I'm going to tell you here because this is, it's one of the examples that I like to cite about Linode

when people ask me, well, you know, I use another provider right now. I'm not really sure why I should switch. I mean, I love that $100 credit, but, you know, what other reason? And the thing I hear from people who switch is it, to me, they say, it feels like the networking is faster, especially between the data center to data center networking. Like Linode's got that dialed in more than other providers do. And people are like, oh, yeah, you'll see it in our telegram.

Oh, I agree, yeah, yeah. Well, I'll tell you why. There's actually a reason for that. In 2016, Linode took their entire networking infrastructure into their own hands. They wanted to guarantee low latency and a certain level of performance between their data centers that they just couldn't guarantee unless they became their own ISP.

And in 2016, they did just that. How about that? And they set the foundation to tackle all kinds of designs for the future and to really strengthen the connections between their data centers. And something else, and this is on their blog. I'll put a link to this in the show notes. This is what they say on there. As we worked on expanding our global network,

three things were non-negotiable. Maintaining vendor diversity, balancing flexibility and control, and this is my favorite one, the third one, and incorporating Linux at the network level as much as possible. And then in parentheses, since we're Linode, we're very good at Linux. And they are. And it's that kind of stuff, And then in parentheses, And now they have 11 data centers worldwide, 40 gigabit connections coming into the hypervisor, super fast native SSDs,

and boxes from $5 a month to massive compute machines. And they have object storage available, cloud firewalls, load balancers, Terraform support, and a lot more. So go check it out. See what you can do with that $100. There's a lot. Linode.com slash unplugged. Go there,

support the show, and play with any of these things and let me know what you think about it. I love hearing how it goes for you guys. It's a great way to find out what you're using Linode for. And then every now and then it makes its way into a read on the show. So get started at Linode.com slash unplugged. And then let me know how you're using Linode at Chris Lass on Twitter or the contact form. Either way would be great. Linode.com slash unplugged. And then let me know how you're using Linode at Chris Lass on Twitter or the contact form. Either way would be great. Linode dot com slash unplugged. Before we get out of this world and start talking about Linux on Mars, just a couple of notes in the housekeeping. I want to mention again our Telegram group because as things eventually one day return to normal, any kind of activities we end up doing like meetups or events, whenever that does happen, that Telegram is always one of the first places that we're coordinating. JupiterBroadcasting.com slash Telegram.

The conversation just continues. Often on Wednesday mornings, you'll see people discussing what we talked about here in the show. And if you want to get in on that action, that's where it's at. The LUP plug remains strong and consistent. Our LUP gets together every single Sunday. It's not a fluke. It's something you can rely on and something you can participate in. It's a really nice experience. It's on our Mumble server, just there in the lobby. And we have information on how to get into Mumble at linuxunplugged.com slash mumble. And I think that's really everything we have for the housekeeping, Wes. That seems shorter than normal. I feel like we're

missing a plug here. Normally, we make a big old mess, but not this week. Although, if you aren't already checking out the Coder Happy Hour on Mondays live, you probably should be. It's really been a highlight of my weeks. I'm still usually stuck at work, but it's the start of ending my day, and I get to unwind with you and Mike. Aw, shucks. Thanks. Thanks. Yeah. We do that at 5 p.m. Pacific, 8 p.m. Eastern at jblive.tv. That is the Coder Happy Hour, and that is our housekeeping. So if it just wasn't perfect, since we got together last, NASA's Perseverance rover has driven on the Mars terrain for the first time.

It's not a lot, but it's significant. It matters. Yeah, this first little adventure was only 6.5 meters, but once it starts pursuing its main science goals, regular commutes extending 200 meters, that's like 600 some feet, or more are expected. The part that really stuck out to me though was that the mobility system is not the only thing getting a test drive right now because back at the end of February, they actually upgraded Perseverance, or Percy for short. They upgraded some of the software running, replacing the computer program that helped actually get the landing done

with the program that will control it while they investigate things on the planet. And if that isn't the most stressful firmware upgrade ever, I mean, come on. Hey, right? No kidding. Tim gets into that system a little bit, and it's remarkable in a way that shocked me. And he also gets into the multiple Linux systems and all of that. So Tim is the Mars helicopter operations lead at NASA's Jet Propulsion Laboratory, and his team specifically is responsible for the Linux-powered helicopter that is in Perseverance's belly. But we were just super excited to have the opportunity to chat with Tim

because his team has got to, you know, this is, it's a historic moment for them. And to get an opportunity to talk to one of the individuals responsible for getting Linux into this organization, well, that was special. Attention, Perseverance, stay pleased. On the defense of law. and fun. I say very safely, on the surface of my... Tim, welcome to the show, and thank you so much for joining us.

We've been geeking out all morning about chatting with you. Well, I'm a fellow geek, so I always like to talk to people like this, and my brother's a big fan of your podcast, and he's the one that said, you've got to talk to these guys. Well, Talon, we said thank you, because we completely agree. and said, you got to talk to these guys. Well, Talon, we said thank you because we completely agree. And just before we started, we were talking about your tenure at JPL, and it looks like it's somewhere around 30 years now working for JPL. I imagine that you started there very young.

Yes, so I got my undergrad in EE and computer engineering and then interviewed around, but the top of my list was JPL because I knew of all the different projects I did from reading up on them when I was a kid with National Geographic and all the Voyager visits to the outer planets. And I was just super excited when I saw a poster at my school said, sign here if you want an interview. And I did. And the rest was history. I came out of school moving from upstate New York, very much of a rural area,

and moved to the big metropolis of Los Angeles and started at JPL. So you have a background in computer technology. So what kind of job were you looking for at JPL in that time? I'd done a lot as an undergrad with computers, you know, taking a lot of software classes and then computer architecture classes. And I also had a background in electrical engineering. So I understood the circuits and the design of how they worked. And so I wanted a job coming out of school in that area. And so I, when I heard onto JPL, they were looking for somebody who had software background, but also had the engineering background as well, to be able to look at schematics and understand how the different boxes in the system worked. And so that was my entree

into JPL was that I came in early as a graduate from college and started working in the Deep Space Network. That's a big network of antennas that JPL uses to track the different spacecraft that they're talking with. And they were looking for somebody who was a software engineer, but also understood hardware, so we could program the ground controllers that would operate these transmitters and receivers in the deep space network. So that was my introduction to JPL, was in that field. And then in the late 90s, I kind of switched sides, as they say at JPL, from the ground side to the flight side and started working on the Cassini project, if you remember that one. That was the big spacecraft that orbited Saturn and took a lot of measurements and pictures over the years. Just breathtaking photos, yeah.

Amazing. And then, you know, they had this very dramatic burn into the atmosphere as a goodbye. had this very dramatic burn into the atmosphere as a goodbye there's a very a very amazing video of that you know a rendering of that on the web if you want to search around for it but i worked on the i didn't actually work on the flight software itself i worked on a a simulation environment where you would simulate down to the instruction level what the processor could execute and built models that were all the different peripherals on the spacecraft. And so you could take the actual flight software and load it on our simulator and it would run as if it was on the vehicle. So they use that as a testbed, a virtual testbed for many

years on that project. Boy, I could see how that'd be immensely useful. And at what point in this journey do you discover Linux and start working with Linux? So JPL for many years was a Solaris house. Right. You know, when I came out of school, everybody wanted a big Solaris workstation. Yep. You know, the old Spark workstations. They were pretty cool. They were cool. And so I got, you know, I kind of came up the, cool. And so I got, you know, I kind of came up the, you can call it the Unix curve, learning how to work on Solaris. And, you know, the flight project side used Solaris a lot. And prior to getting on Curiosity, which is the big rover that's still on Mars and operating, I had been on a project where we were doing a lot of software technology development,

and we could kind of see the writing on the wall that Solaris was starting to decline in the marketplace. So in our own project, we went and started looking at alternatives. And I just bought a PC made out of random parts, you know, just commodity parts at the time, and loaded Linux on it, and took the software that we had on that project and compiled all that software on Linux. And it was kind of a somewhat comical moment, because one of the senior engineers on the project had gotten his $15,000 Sun Solaris workstation that he wanted as a senior guy. And we had the software compiling on both.

And I took my little commodity-level PC and compiled the software, and it compiled twice as fast. And he just had this look of... What did I waste all my money on? Yes, he just, just this look came over his face like, I can't believe that. And he ended up pawning off the whole system to an intern and then went off the model. It was kind of funny.

What era, what year-ish was this, would you say? This would have been early 2000s, probably. Very early 2000s. And so Linux at the time was kind of a niche thing at JPL and sort of an underground movement, if you will, because, you know, Solaris was the enterprise class system that everybody supported and believed in. And, you know, they built enormously robust machines. You could drop them off of a cliff and nothing would happen. And you had support.

It had an ecosystem. It was the established operating system at that time. It was sort of a no-brainer. Right. And so when I came onto Curiosity, I was one of the first five team members for Curiosity. And I was tasked with setting up a development environment for that particular project. And I said, why don't we use Linux? And everybody around the table kind of looked at me and said, really? Isn't that the thing SCO is trying to sue out of existence?

And so there was, you know, it's kind of this, there was kind of a view at JPL that it was sort of a, you know, a disreputable corner of the operating system market in some ways. Because, you know, it hadn't had as much of the enterprise time under its belt then. And it seems like in your environment, that matters more than it would to even most enterprises. Yes, you're right, because they had a lot of tools that have been written specifically for Solaris, because it's Linux. I mean, it's not Linux, it's a flavor of Unix, so there's many similarities. But the underlying architecture was Spark and not Intel.

And so I just made a push. I said, listen, Linux is where the market is going. You know, you could already see the decline in Solaris even at that moment in time. I said, if we just keep replicating these old platforms, we'll be left behind. Sure, it seems like even though at that point in time, I said, if we just keep replicating these old platforms, we'll be left behind. Sure. It seems like even though at that point in time, that was still a bit of a stretch because

people probably assumed Solaris would be around forever. I mean, even if I recall my timing right, even kind of switching to the Intel architecture itself would have been kind of looked at as a bit risky, right? So, I mean, you were just talking about a new OS, but you're talking about a CPU architecture that was built for PCs. You know, I used to work with a data center mainframe guy who called them Mattel inside. Yeah. Yes.

I mean, there's a famous Dilbert cartoon where Dilbert and his friends are talking about, you know, programming on their PCs, and the bearded, grizzled Unix veteran came along and flipped him a nickel and said, here, why don't you buy a real computer? So you're kind of fighting two different headwinds in that case. Yes, but to JPL's credit, JPL is always balancing internally. Yeah, we want these

firmly established technologies that we can trust, but we're always open. If somebody can make a good case for us, we're always open to looking at new things. And they said, well, why don't you go off and look into it for us and make a case for it, and we'll do a trade study, and then we'll decide what we're going to do.

And so I went off and i configured up you know a machine spec and install linux and show that i could get all of our tools running on it and and the fact that it was fast and they're relatively inexpensive compared to sun work stations to replicate and so we ended up kind of pivoting and starting to use Linux for our development tools. And what was interesting was that the other parts of the project, like the ground system and the test beds, you know, started to take interest and came and asked me for my machine spec and looked at it and ended up adopting it for their own, you know, basically a PC with Linux.

And so now, for Curiosity, and now especially for Mars 2020, they've pretty much pivoted to Linux. And they've taken it a step farther, and they have a lot of, like, VMware infrastructure in place where they have a lot of different virtual machines that people can log onto and run their tools and do their analysis. So it's really been a big pivot in the last 15 years. It's kind of curious if any of the open source nature, but also sort of the you can just

install and run it without licensing, has been useful internally. It's just a lot more flexible, and certainly you all are pushing the limits of what is possible. It seems like that might go hand in hand. Well, JPL also, it's very important to them to have support contracts in place. And so for many of the official, if you will, the official workstations and environments that they install, they have support contracts in place and they try to standardize on a single image. So it's sort of a blending of the two worlds, right?

They still don't want people to go off and just ad hoc build up these Linux environments that they don't know about the level of patching and security and compliance with all these other requirements. You're not installing Arch Linux. No, there aren't. And so it tends to be flavors of Red Hat that they get support contracts for. And they, you know, they have a very robust system administration environment now. So they've kind of adopted Linux, but at the same time they've made sure that it's robust in an enterprise way

to make sure that the machines are secure and the environments are standard. Because the one thing that can bite you when you have tools that run on Linux is you have to very carefully version control the tools that you use. Right, right. And so if you rev Python from, say, 3.6 to 3.7, and everything starts breaking all of a sudden because there's been some change, that can really hurt you as much as things in other areas.

So they very carefully control versioning on these workstations so that they don't get bitten by tool changes. And so this sort of sets the stage, I would imagine, for some cultural acceptance of actually using Linux in a mission on Mars. But I don't imagine it was automatic. There had to be an established methodology for doing something versus Linux. And then not only is it Linux, Tim, but one of the things I'd like to get to

is the open source software running on top of Linux too. But before we get to that, could you give me just a little bit of an idea of what it was like to actually convince people that it's going to be okay to put Linux on Mars? It's going to work out. Let me explain it to you this way, is that NASA and JPL have these different classes of missions.

And there's different levels of, if you will, reliability and vigor and verification that are assigned to those different levels. So the big flagship missions, like the rover itself, they call them Class B. So there's a lot more scrutiny about what kinds of software you get to run, what kinds of operating systems,

what kind of testing program you do, what kind of verification, the level of reviews. So there's a whole litany of things that you have to comply with if you're going to design and fly one of these Class B missions. But JPL is also aware that they want to try new things. And so they have a classification of Class D missions, they call it. And the helicopter, the Mars helicopter is one of those classifications. They've had other ones like, I don't know if you've heard of CubeSats, they're small compact

satellites that you can fly. Many universities are doing those kinds of projects. And so they tend to be lower profile. And if they fail, it's sad, but it's okay, because it's not a critical aspect of the operation. Right. And so the helicopter was defined as one of these Class D missions, because if the helicopter crashes, for some reason, it doesn't affect the rover, right? They can continue to go and do their primary mission of going and getting rock core samples. And we can't really hurt them. And so there's a lower level of scrutiny, if you will, on that kind of mission.

And so the main work that we had to do with respect to the rover was to verify that nothing that we did could hurt them in terms of how we interface with them and so forth. And so because that's true, because the Mars helicopter is classified as a technology demonstrator where they're willing to try new things, willing to accept a higher level of risk, they didn't really push those hard requirements on them to use the traditional operating systems and computing platforms that they usually put on rovers. Because, for instance, the processor that's on the rover is really a Mac power PC from the mid-90s. Not exactly modern. Yes, exactly. It's like a 200 megahertz power PC, but it's been re-engineered from the silicon up to be radiation tolerant.

Right. So that it can, and very thermally tolerant. So it can take a beating, you know, a variant of that same processor is on the curiosity Rover and it's been operating for, you know,

going on what, 10 years here without any problems. And so they build those to be super tough, but that also means that they tend to be older technology slower. Yeah. Right. And so the build those to be super tough, but that also means that they tend to be older technology. Slower, yeah. Right. And so the helicopter itself, we could not use any of those older processors because not only are they not very powerful processors, but they're big and heavy and they consume power, a lot of power. So we couldn't use those old processors just for that reason alone.

So we had to use something modern, something that had a lot of processing power because we're running the guidance loops on the helicopter when it's flying it's measuring sensors running it through control algorithms and you know sending commands to the flight system at 500 times a second and on top of that we're taking pictures we're we're taking black and white pictures down to the, in the direction of the surface 30 times a second. So it's taking a picture 30 times a second, analyzing it for features. Is that F prime right there? Is that where F prime is doing some of its magic? That is a, if you want to call it an instance of F prime. So F prime itself is a software framework.

It doesn't have all that built into it. So we made, we took that software framework, it doesn't have all that built into it. So we took that software framework, which we developed to JPL. And is open source. Yes, and is open source. So you can get a Raspberry Pi and go to the GitHub repo for F prime. And you can build a little demonstrator with a Raspberry Pi and see how it works. And many of the same software components that are on that repo are on the helicopter right now. Sitting on Mars. it's amazing. That's right. And the helicopter code itself is not on the repo because that's proprietary intellectual property of JPL. So we use that framework, which is something that I developed back in the 2013-2014 time range

to develop the software. But because the processor for the helicopter has to be very powerful, we couldn't use that. I mean, the processor on the helicopter is 100 times more powerful than the processor on the rover. It's an ARM, like a multi-core one too, right? Yes, it's a quad-core ARM processor. It's built by Qualcomm. So it's a Galaxy S5 era processor. If you wanted to draw a comparison. So the way Linux got into the picture was that this was a development board that you could buy off the internet

for about 400 bucks back in the day. It was actually used for people to do drone development. If you wanted to prototype a drone, you could use this particular board. Sure, I mean, it sounds like cheap prototyping and that's basically what you guys were doing. Right. And so this board is very compact. It fit within the envelope that we wanted and it had the IO, you know, because we're talking to all these different subsystems on the helicopter. So it had the sufficient IO that we needed. And what was really important was it already had all the software in place for the camera systems we use because we had been doing some asking around.

And if you just take a modern camera and you try to program for it in Linux, it can take many months with an expert to get it all up and running. No doubt. And we're not experts at writing camera drivers for Linux. We just aren't. And so we had partners at Qualcomm who helped us. And this thing all came packaged, and it came packaged with Linux. Sure. It's a 3.4 kernel based on Linaro.

I don't know if you've heard of that flavor. Yeah. Linaro is sort of an embedded version of Linux. It's kind of aimed towards embedded applications. And they've done a lot of good work for ARM support generally, I think. Yes. And it came prepackaged with these camera drivers

and the IO drivers we wanted. And so it was a very natural place for us to step in and use it. And so that's how Linux got onto the Mars helicopter. And because it was this Class D mission, we didn't really get any pushback from JPL about making Linux the operating system. That's fantastic.

And so if I'm kind of getting where you're going with this, it kind of means that this Class D style classification means that a little less pressure. It gives you a chance to try something like Linux. And if it proves itself here and proves itself a few more times, I would imagine, it means it could eventually be moved up to something like a Class B style mission? Yeah, perhaps. I would say it's gaining more trust in that field. Right. Yeah. You know, as time goes on and now that there's the Linux real-time patch that gives the kernel more of a real-time aspect to it. Is real-time one of the major things that gives JPL pause when using Linux? Yes, because Linux is really not designed truly to be real-time. Even the Linux kernel itself

has modes you can run it in that makes it more real-time. But Linux was really designed to be a multi-user fairness algorithm kind of OS. In the marketplace, you have very purpose-made algorithm kind of OS. In the marketplace, you have very purpose-made real-time operating systems like Wind River's VXWorks, which is the operating system that is on the rover. And that way you get much higher guarantees about when tasks will finish. You just can't have another process taken over, right? I mean, when you're trying to fly autonomously on Mars, there's just, there's not a lot of room for failure. Right. and so when you're doing very critical things like that, you have to order things in the operating system by means of their priority.

And so when you have a fairness algorithm in Linux that just kind of hops from user to user and makes sure they all get time, it doesn't lend itself to that. Because you don't want your image compression algorithm to get in the way of your, you know, your flying or your landing algorithm, right? That it would be silly for image compression to take over for a minute and you crash a billion dollar vehicle. Yeah, it sounds ridiculous. Right. So you so when you're dealing with very critical activities like landing the rover,

you have to have an operating system that you really, really understand how it's going to operate with respect to all these deadlines in the software. And so that's the void that Linux has to cross to get accepted on some of these more class B projects is kind of be engineered in such a way that it can make these kinds of guarantees for performance. And improve it. Yes, exactly. Is the Ingenuity helicopter, is it's 3-4 kernel?

Does it have a real-time patch set against it? Is it real-time? So interestingly, it doesn't. It has the preempt patch on it, but it does not have the real-time patch. And it just happens that it's super fast so that it can keep up. And we also, so the Linux installation on the Snapdragon is not the only processor we have on the helicopter. We actually have a coprocessor.

It's a microcontroller built by Texas Instruments, and it's a microcontroller that's really designed for automotive uses, so it's very rugged and very temperature tolerant. Sure. And so the main flight control loop that we do, there's two parts of it. designed for automotive uses. So it's very rugged and very temperature tolerant. Sure. And so the main flight control loop that we do, there's two parts of it. There's one that's on the main processor, you know, the Linux processor, and there's another part of it that's on this microcontroller. And that one, we actually run bare metal, what we call bare metal, which means there's no operating system. It just gets a signal from the hardware and that runs a control cycle and then it waits

for the next signal from the hardware. There's no operating system at all. And so that particular processor is the one that's in charge of the immediate flight control, if you will, keeping the helicopter in the air. And then the Linux processor is the one that does the, you know, the more of the navigation and the imaging. And so if that Linux processor falls behind, we have built-in robustness to the software that it can tolerate some of those slips if it happens. You've got like a hybrid setup there to try to take advantage of both sides of things.

Right, we give each processor the job that it's good at. Yeah, it sounds like that secondary coprocessor is pretty foolproof. So is that also the system managing the thermals? Because my understanding was this tiny little helicopter spends a ton of its energy just keeping itself warm enough to survive. Yeah, so the third part in this triple play here is that we have an FPGA. I don't know if you guys are familiar with what that acronym means. It stands for Field Programmable Gate Array. It's a it stands for field programmable gate array.

It's basically a sure. OK, a digital logic chip. And that one itself is very much radiation hardened. So the helicopter spends most of its time asleep because we don't have the energy just to leave it up and running like a, you know, data center server. It's it's instead what happens is we manage wake sleep cycles with a helicopter. So the FPGA, we essentially program it, and we say, okay, we're setting our alarm clock for 10 a.m. tomorrow morning. Please wake us up then.

And then we go to sleep as a processor. So the whole system basically shuts down except for that FPGA. And that FPGA has digital logic built into it that will basically run a thermostat to keep us warm overnight. So that gets us into a very low-power sleep mode so that we don't expend energy just sitting there waiting for instructions. Just the minimal amount of control to keep things operational, keep that clock counting until you wake up and check in again.

Yes, exactly. So that makes our operations very challenging because we have to manage that alarm clock and make sure that we're up and waiting for it when the helicopter comes back up because we can't just come up at any time and say, hey, helicopter, go do this for us. It's not just waiting for us to talk to it. We have to coordinate all these wake up and sleep times so that we can do things when we need to do them. And then I imagine it also, there's a component of, it has to be aware of a battery level and solar charge and whatnot too. Yeah, that's, we manage that really on the ground. You know, in our, we have models that we developed in our

ground tools that track the battery state of charge throughout the Martian day. And we try to time events on the helicopter so that we maximize the amount of battery energy that's available for a flight. So the software that runs on the helicopter is paired with a number of these ground tools that we use to track its state as well. And then if something's drifting from what you expect,

you just sort of send an update to it to change whatever? Yeah, we would typically just change our plans for that day. We might move it later in the day to give it more charge, if that's what happens. So when are we looking at that thing kind of starting to do its, you know, because it's still in the belly right now, right, of Perseverance, and then it's going to come out and sit out for a little bit, and then it's going to be a little bit before it even flies. So, I mean, we still have some time here, right?

Right, we're still attached to the belly of the rover. We charge the batteries every week or so. So, interestingly, the helicopter has this Qualcomm board, but we also have a box on the rover that's an instrument payload. We basically just took the same avionics cards that are in the helicopter and constructed them in a different way and interconnected them in a different way so that the the base station box that's attached to the rover also has the qualcomm processor and also has linux and also has f prime on it so this box acts as a radio relay to the helicopter so the rover will turn on this base station as an instrument payload and communicate with it and pass it the files we want and then internally we take this base station and the base station as an instrument payload and communicate with it and pass it the files we want. And then internally we take this base station,

and the base station passes the files we want to the helicopter when it wakes up. And that's how we operate it. So what you're saying is there's actually two Linux boxes that are going to be active on Mars. Right, and there's actually three, which is kind of fun because did you guys see the EDL landing videos? Yes. Wow. With the parachutes billowing out and you could see the dust kicking up when it was looking down and the rover was touching down. It was incredible.

That was also another Linux installation. And this one was an x86 box, right? Correct. It was a compact PCI, a ruggedized PC, and it was just running pretty vanilla, from what I understand, pretty vanilla, a 4X, you know, the developer sent me the numbers. It's a 4.15 kernel. Wow. And it's running Linux, and they had a whole series of just USB cameras attached to all these places on the surface of the rover. And this was also another one of those class D

instruments because it was attached to the rover, but it wasn't used in the landing to find a place for the rover to land. It was basically a giant video recorder. So during the landing at the right moment, they just sent in a command and said, hey, start recording. Good luck. Right. So if everything had gone wrong and for some reason they hadn't recorded the video, it wouldn't have affected the rover landing itself. It just would have made it so they didn't get all that cool video. And historic video, too. I mean, the fact that it worked is fantastic, and it means that Linux took part in capturing really, truly, truly historic video. And

I heard a shout-out on stage to FFmpeg for doing some of the heavy lifting there. Yes, and I was talking with a developer recently. They even used Python scripts on that box to do the image. So after they landed, they had saved all the images, and then over the next week or so, they started transferring all this video, and so they used these Python scripts on board to do some of the processing of the images

on the actual Linux box before they shipped them over. So you have a bunch of different open source things operating on that one. And so I guess they get the credit for being the first Linux installation to run on Mars. But you'll have credit for the continuous installation, right? Because their computer, I assume, is now shut down. It's shut down, but they're going to try to use it for other things. Like they have a microphone that they used for... They attempted to get a recording on the way down through that microphone,

but it didn't work. They didn't really elaborate on why it didn't work, but they've tested it since, and it seems to work. So they may use it for recordings throughout the mission. It's sort of a bonus feature. So if I ever get a chance to play the sound of Mars on this show, it'll have been recorded on Linux and it'll be played back

and recorded on our end using Linux. That's pretty great. Yeah, if you get it from that microphone, they have another microphone up on the mast that's also, they're able to point it in particular directions and take recordings. Okay, so those ones might not be, okay.

Yes, but I guess you could say we get credit for being the first Linux installation They were able to point it in particular directions and take recordings. Okay. So those ones might not be. Okay. Yes. But I guess you could say we get credit for being the first Linux installation to operate on the way to Mars because this instrument box that I told you about that we designed has the software on board to do the charging. So we had to keep the batteries maintained. And so we would wake this box up every week or two, and it would execute commands to keep the charging going. So have you kind of, I mean, it's still kind of early in a sense, but you've hit some pretty significant milestones here. Have you gotten a sense of, in terms of Linux, what's worked, what hasn't worked, what might be adjusted for future missions? Do you have any of that idea yet? I think that as we move on in Linux, I think what we want to look at

is getting much better at doing things like configuring the kernel and understanding what's going on in the kernel and being able to do things like apply this real-time patch that I mentioned.

So the path forward, I think, is to study Linux and understand it better as a real-time environment and just increasing our expertise in how it works. Because we're new to this game. I mentioned that earlier operating system VxWorks. I was one of the VxWorks specialists for Curiosity. I was the one who configured it and understood it and wrote the software that would interface with it.

Have to get very up close and personal. Right. So I come out of that world where we have to know all these things about the operating system. And so we actually had a developer on our team that was very much Linux savvy. So I wouldn't classify myself as a Linux expert. I would classify myself as kind of maybe a power user in that domain,

you know, a flight software. But we had somebody else who was digging into the innards of our particular distribution and figuring out how to configure it and get the things working that we needed to. So getting more of that expertise in a field that typically doesn't have it for that particular application, I think that's where we want to go So getting more of that expertise in a field that typically doesn't have it for that particular application, I think that's where we want to go

moving forward at JPL is to understand, you know, how Linux works in that environment and try and study it and characterize it and test it so that we gain that trust so that we can use it for more critical applications.

Are there any kind of future like class D style missions that are in the works that might involve Linux that you know of at this point? There are some in the pipeline. I don't know if I can speak to them now because they're more in formulation and JPL doesn't want to, you know, break news on things like that. But it does seem possible. Sure, sure. And I think there is a growing place for these kind of Class D applications of it. As a matter of fact, there was a CubeSat that JPL developed and flew and operated a couple years ago.

If you want to look it up, it's called Asteria, A-S-T-E-R-I-A. And it was a Earth-orbiting spacecraft that would do, it would attempt to, you know, do alignments with distant stars to do observations. It would attempt to, you know, do alignments with distant stars to do observations. And it used F prime, the same flight software framework that we're talking about. And it used a flavor of Linux for that board as well. So, OK, all right. I love it. So we've had this partnership between F prime and Linux for a while.

F Prime and Linux for a while. So I think definitely these projects that have used it and have started to build a resume for Linux will help us to more mainstream it in this kind of project in the future. And there's definitely interest. I would imagine there's many tales of over the years of getting Linux further and further adopted, but I just appreciate getting some like little peek,

little insight into how it's all kind of unfolded. And it's kind kind of unfolded. And it's kind of one of these stories where there's more to come because we have eventually at some point, the Ingenuity helicopter is going to fly. And I'd love to chat again and just kind of get an idea of how it went and what you guys learned and all of that, because that's still a milestone that as we record this is yet to happen. Yes, it's very exciting. And it's very exciting, not just for Linux, but for spaceflight in general, because this

is something that has never happened before. It's sort of a Wright Brothers moment where we're trying something completely new. We had to develop new techniques. We had to construct the helicopter in a very special way to operate on Mars because Mars has 1% of the density of Earth. So you can't just put a, you just can't grab a commercial drone off the shelf and put it on Mars. You know, we had to build lighter and longer blades that spun faster and make sure that we were very lightweight. And that was a challenge in itself. And so doing this for the very first time is going to be super cool. If everything goes well, we'll have some nice color pictures coming out of that

because these cameras that came with it, we have basically a 13 megapixel color camera. Wow. Oh boy. And I love a good drone shot. I always love drone shots. Yes, it's a half a gram. You know, it's a tiny little square camera that came with the board. It's basically just an adapted cell phone camera.

If everything goes well, we'll get some good pictures from aloft on this helicopter that we can share with the world when the time comes. I mean, I just, I think of the weight restrictions and the atmosphere restrictions or limitations that had to go into making this thing even conceivably possible. And it's mind-bending. making this thing even conceivably possible. And it's mind-bending. But I can see long-term how this would be essentially invaluable to any manned mission, having something like this little helicopter, a little more developed, a little bit longer flight time, of course, that could go out and just do some reconnaissance ahead of them if they're out walking around,

or even be streaming back pictures as fast as it can back to ground so, you know, ground control can see what's going on with the walk. I mean, like, there's so many uses for a little, even something that just takes pictures that flies around. There's so many uses for that on any manned mission in the future, I would imagine. Absolutely, and you can look at the picture aspect, but one of the things that a helicopter can do that a rover cannot is it can go to these features that you can't drive over. So if we wanted to inspect, say, a cliff face or a deep ravine or something like that, you can't send a rover there to look at it.

So you can imagine uses where you would send a scout like this to a really difficult to access place, take a bunch of pictures. Future helicopters might be larger so that they can carry an instrument payload. So you might have like a spectrometer or something that can scan a formation and get data. So the nice thing about the helicopter is not only can it get places quickly compared to a rover because the rover just crawls across the surface to be safe, but it can only get there quickly,

but it can also go to places that the rover just can't. Wow. And it starts here. I can see why you would work at JPL for 30 years. I mean, what an exciting thing to work on. Even just getting to work on one rover would be remarkable, but two, Tim, is incredible. So congratulations to you, to the team, to everybody. I mean, this is so exciting for us as space geeks and Linux geeks

to see these two things come together. Amazing. Yeah, so you can imagine it's very exciting for me as well. Having grown up as a kid, reading all these stories about JPL and all their intrepid adventures and getting to actually work on this

has been a real boon for me. It's been a lifetime dream to be able to work on something like this. Well, thanks for coming on and sharing the details with us. You're quite welcome, and thank you for having me on your podcast. Boy, that was a treat. That was fascinating. I think there's new details in there that haven't really been talked about much publicly.

And what a story, Wes, from this thing they're going to try to now actually running on multiple computers on the surface of Mars. And you might say it's out of this world. You know, it's really fascinating to see a space where Linux is still proving itself. It's kind of already conquered Earth, but the constraints for a multi-billion dollar mission on another planet are a little bit different, and I get that. I'm just hopeful that due to the open source nature here, you know, as JPL and NASA adopts more Linux and puts it in more difficult situations, that's just going to be another shared use case that improves that kind of real-time story for other kernel users. Yeah, it was interesting, and it makes total sense to hear how important it was that the Linux kernel support real-time to make some of this actually possible. And I'm hopeful that

we'll see in the future way more devices that are this Class D style mission device. There's tons of support equipment that isn't necessarily fundamental to the core operation of the mission, but would be useful to have and add supplemental value. So that would seem like a great use case for Linux for a long time. I mean, I wouldn't be surprised if eventually it's the predominant type of equipment. But we'll only have to just wait and see. But it was really cool to hear all of that and to see that maybe one day it could go even further. So thanks to Tim for coming on the show.

We really got him at just the perfect time because after this, I think he said, did he say next week Mars time is kicking in for his team? Is that what he said? It's very soon. Yeah, pretty much. They've got to switch over to living on Mars time. How about that?

And not all the teams have to do this all the time, but his team does because they're the helicopter team and they're going to be doing flights and stuff. So it's pretty hardcore. So I'm really glad we got him when we did because once he goes into Mars mode, he kind of just goes AFK. And so it was great to chat with him. And maybe when he emerges from Mars time, we can catch up with him and see how things went. Yeah, I mean, I think we'll all be hoping that the first flight of this little helicopter goes well. Wes, before we go, why don't we do a pick or two? These are honorable mentions because they have been on the show at some point in the show's nearly 400-run history.

But I can't remember the last time we talked about them. And they both deserve a little attention. I'm going to start with Oaken Audio. This is the audio editor everybody should know about. Everybody knows about Audacity. God bless Audacity. It's a great project. Amen. But Oaken Audio, this is one of my secret weapons for production. If you know any clip you ever hear on this show that I've clipped for, like if we have a news clip or something like that, it's in Oaken Audio. Always is. I love this app because it'll open up every audio format,

but also every video format. It has a really nice, smooth UI. It's great for just grabbing a bit of audio and exporting it or cutting something. It's about time I try it because I remember, I think Drew recommended it in the past, and I sort of gave it a go, but it was at the time, it was hard to get installed,

but just looking right now at their download page, they've got Windows versions, Mac packages here, and then a whole bunch of Debs, RPMs, and even a Pac-Man package you can just download, so it seems like things have changed. A Pac-Man package, you say? Yes, yes. And then, let's give mention to Natifier. We've talked

about it before, I feel like we have. But Natifier is an application that lets you take a website and essentially convert it to an application. It's basically Electron for yourself. But the idea would be that maybe you've got to use webmail, or you get stuck using a few web apps, and you just want them out of your browser. You'd like them to have their own process.

You'd like them to have their own entry. You'd like them to have their own entry in your taskbar. That's where Nativire's kind of handy. Yeah, we're all stuck with it, but a lot of the benefit of having a native app is some of that native desktop integrations, if that app supports it, but then also just having a different icon, having a different entry

in your alt tab so that you don't have to manage every single thing in another tab. From your lips to developers' ears, Wes, that's all I want. That's all I want. Well, that and I wouldn't mind it to be a native application. This is an interesting challenge. This is one that Canonical is taking a crack at with Flutter.

And it's a problem that is, it's apparent to anybody who's been listening for a while. Developers have a whole mixed bag of incentives. Sometimes they want to develop for mobile. Sometimes they want to develop for several desktops. Sometimes Linux is just kind of a byproduct. And we've ended up with Electron, for better or for worse. But Flutter is perhaps another path. Unfortunately, not one that I'm too excited about.

Not really happy that it's hitched to Dart and Google. But it is something Canonical seems to be pretty excited about. And they're writing their future installer in it, and they've committed to writing every future Canonical desktop app in Flutter as well. So there's a lot going on here, and I think a lot of people are passing over this story. One of these just sort of glaze over and pass by.

I would encourage you not to pass over this story. Go to linuxactionnews.com slash 179 and listen to this week's episode where Wes and I do a breakdown so you can kind of get where they're coming from and kind of the scale of commitment that they're making to this thing. It's more than you might suspect, and I think it's worth your time and attention. So do go check that out, linuxactionnews.com slash 179. And with that, I think, Mr. Payne, we're going to wrap it up. So I will just say this. Join us live next week. See you next week. Same bad time, same bad station. We do this here on Tuesday at 12 p.m. Pacific, 3 p.m. Eastern. If you do the Twitter thing, you can find out about it by following us at Linux Unplugged or follow the whole

network at Jupyter Signal. There's a whole cast of great shows over at jupyterbroadcasting.com. There's also even a calendar to find out when we're here live. Hey, that's a good one. And all the links, all the links, linuxunplugged.com slash 396. Oh, my gosh. Oh, my gosh. Also, our subscribe link is over there, our Mumble server info, our Matrix server.

Basically, all the stuff we kind of just casually mention all the time on the show, we try to have links to it at linuxunplugged.com. That's the general idea. But we just appreciate you listening. Thanks for downloading and listening, too. We'll see you right back here, not tomorrow, not Monday, not Wednesday, not Saturday, but next Tuesday! All right, everybody, let's go vote. JBTitles.com.

Everybody go vote at JBTitles.com. Everybody go boat at JBtitles.com. We've got quite the selection. Yeah, there's a lot of boats going today. We've got a new Jellyfin release, Wes. 10.7.0 is out. Did you see the new shiny? No, I haven't had a chance to upgrade yet, but I guess I know what I'll be doing after the show.

Watch out, though. It sounds like there are some irreversible database changes, so you can't go back. Or at least make your backups first. You're running in a container, right? This should be easy. Yeah, do your backups. Do your backups.