Embedded - 93: Delicious Gumbo

Episode Date: March 18, 2015

The Linker post for this episode is RTOSs and Brownies. Joel Sherrill (@JoelSherrill) spoke with us about real time operating systems, free and open source software, interns, and space. RTEMS home page and wiki Google Summer of Code (the FAQ is the best part!) and ESA Summer of code (awesome tagline: In space no one can hear you code). The LEON is the ESA Sparc core with open source VHDL and extensive use by ESA. Some projects RTEMS is used on include the Magnetosphere Multiscale Mission, theExPRESS Logistics Carrier, Mars Curiosity, and the Dawn spacecraft that is visiting the Ceres asteroid.

Thank you. Look there for my advice on winning the Hackaday Prize and the after-action report Chris wrote on Lane and Osh Park. Today, we are joined by Joel Sherrill to talk about real-time operating systems and free and open-source software. Hi, Joel. Thanks for being with us today. Hi. Thanks for having me. Could you tell us a bit about yourself? I went to the University of Tennessee Chattanooga, was lucky enough after I graduated to move to Huntsville and work on Unix minicomputer device drivers on the smart that has turned into be RTEMS, which now 25 years later is still the project I lead and I'm happy to have users all over the world. So RTEMS is Real-Time Executive for Multiprocessor Systems, is that right? Right, it is indeed.

And the history is that it was developed here in Huntsville by myself and two other people who are still at online applications research and it was released as open source by the U.S. Army Missile Command. It was always intended to be open source and in fact they sponsored it as an open source project so they would have the ability to review the source, analyze the source, not have to track licenses. All the basic freedoms that come with free software actually make a lot of sense in embedded systems where you tend to have long life custom systems that you want to analyze and debug on bizarre and unique hardware. So it really made a lot of sense. And since that time, there have been a lot of high energy physics, radio telescopes, space applications, and industrial applications. So it's been an interesting ride. It's pretty early for an open source project. We're talking about the mid-late 80s, right? I started on the project in 1989, and I can remember in 1991 actually first having email with Ian Lance Taylor, who wrote the original GNU binary utils. And so we were already using the GNU tools by 1991 for cross-development.

Okay. And that is pretty old. Yeah, that's pretty. But was there even a term for open source software then? Well, the Free Software Foundation predates that by a few years. I don't remember the exact dates. Because I remember at the Unix company I came from before that, we actually did have 9 Track Tapes with GNU Emacs on it already. So that was the big thing of a full-day project to get GNU Emacs up to run.

Sorry, Nine Track Tapes. Okay, so going back to RTEMS, is that what you call it, or do you have a different way of forcing it? That's exactly how you pronounce it. It's a real-time operating system. It's like MonoVista's, I forgot what they have, and MicroCOS from Labros.

And Green Hills and ThreadX. And PumpkinSalvo. PumpkinSalvo. And Nucleus and VxWorks. The basic history is that Right, and Nucleus and VXWorks. And FreeRFCoast. The basic history is that about the time the project started, an operating system called PSOS Plus was being resold as VME exec

by Motorola Computer Group, who is also another branch of Motorola that's long gone. But Motorola wanted to claim that it was an industry standard. So they submitted the PSOS API to the VME Industry Trade Association for standardization. And that evolved through two efforts, the real-time executive interface definition, RTEID, and the open real-time kernel interface definition, RTEID, and the open real-time kernel interface definition, ORCID, which RTEMS is effectively what we now call the classic API, is based on that API and essentially looks very much like PSOS Plus and Nucleus and to a certain extent ThreadX because apparently we all have some heritage back to that first attempted standardization.

But eventually, that effort, not surprisingly, was very contentious and didn't officially get ratified. And the people went on to work on the POSIX threads effort and what became POSIX 103.1b, I think 1996. And about that time, we added our first POSIX thread support. And we did that to support GNU Ada. And basically over time, we've added more and more POSIX support. We use the same C library that Sigwin uses. So at this point, even though we're single-process, multi-threaded, we have about 90% of the full POSIX spec. We don't have fork and exec.

We don't have memmap or shared memory. But largely within those constraints, everything is there you would expect for a single-process Unix system. We've got various file systems, networking, USB. It's a pretty full software stack capable of running a lot of things right out of the box. Why does anybody pay for RTOSs if RTEMS is so great? Lots of times it's because the ones that you pay for, A, have huge marketing budgets. Yeah.

That's the first thing. The second is there's a lot of comfort. People get used to using a particular product and they don't change. So there's a lot of inertia. We are trying really hard. We've worked a lot on our tool ecosystem, but we don't have some of the flashy tools that some of the other people do. Although that is improving, we now have an effort to use the same timeline visualization engine that the Linux Trace Toolkit uses.

So hopefully in the next few months you'll be able to capture timelines. Well, we can capture event timelines on the target system now, but the visualization is the next thing. So we use all the standard GNU tools, and that gives us a strong base for portability and commonality. But needless to say, our expertise is not in the fancy GUIs or in the marketing. So, I mean, the first thing you get is there's not as much marketing. Sometimes people are very comfortable with what they have. Sometimes there's flashy product spaces, flashy features. And although we have been flight certified by the European Space Agency, we don't have the paperwork to sell for the

certifications in other domains. So if you're looking for something that's got certification for like a FAA or FDA type certification, there are companies whose products have that certification. They're perfectly happy to take your money and give you the paperwork, and we just simply don't have that. We do have the test coverage of the style that would probably meet level A-type test coverage, but we don't have the rest of the paperwork.

It's interesting you mentioned marketing because I'm going back to my past and thinking about which RTOS to choose for a project for a large company. And I found it extremely difficult to sort through the things that the various companies were telling me. Oh, yeah. And they would always focus on just very idiosyncratic things like, oh, we can context switch in X number of microseconds, or we have this or that. And they're always just the weirdest trumped up stuff, and it made it very difficult. Do you have any advice for people who are trying to decide

what to actually look for and to kind of cut through that? I've done some help consulting for other people. What I tried to do, to be fair, since I didn't want to come across as biased in giving them advice, is I tried to give them a pattern to follow. The first thing is, if you've got an idea what target architecture you're going to run on, it's pretty easy to start filtering them out based on what they support. If you're interested in running on some weird MIPS variant, I mean, that probably eliminates a lot of the RTOSes.

But the next thing is to start looking at do you really want features like mutexes with priority ceiling or priority inheritance? Do you want symmetric multiprocessing? You know, do they charge you extra for a TCP IP stack or USB stack? And you just kind of have to start looking at what features do you really need and what is that at least technically, without worrying about the license or the cost, that at least starts to bound it in. Try to look at it as what computer science features you need

and try to get away from the marketing. Because, I mean, there are some RTOSs. you mentioned UCOS, which I think, if I remember correctly, UCOS 2 only allowed one thread at each priority, if I remember that correctly. Is that correct? I don't remember. And you're calling it mucus, right? No, I said UCOS. You said mucus.

That's pretty funny. Oh, well, I tried so hard to call it micro-COS because I've heard they get pretty mad when you call it mucus, especially at the Embedded Systems Conference when you walk up and say, hi, how are you? Okay, UCOS. We'll go with that. Yeah. And maybe we'll edit that out or not. We'll see how that goes. But if I remember correctly, it only has one thread task per priority level. That's pretty good. They may or may not fit with your design. So you should just consider what you want as available as design features in the RTOS

because they will impact your application. So and that's kind of my advice to people is just make sure that they have the features that you want because there's no point working doing a lot of work in your application to make up for a feature that the OS didn't provide. But that goes back to the marketing budget. Sometimes they come up with a super spiffy, like you said, with mutexes, with priorities and all of the other things. Somebody has that and they use that as a differentiator. And I don't know if I need it. I mean, I might. Maybe I should just pay the big bucks to get an operating system that has

all the features. At least that's what it always felt like when I was saying to my manager that we should go with this one because it has all the features, which I never really believed, but it just seemed like a safe choice. We might need it someday. To some extent it's true because whatever you purchase and you start the project with, you're probably stuck with it until the project is long, dead, and cold because it's painful sometimes to switch. But one of the nice things about RTEMS is that although we had an initial set of requirements driven by that RTED Orchid spec and we've added POSIX, a lot of the features have been very strictly driven by users. So the things you're talking about, priority ceiling, priority inheritance, we've long had those.

We also have something, so I can do this right, MSRP, which is an extension of priority ceiling to symmetric multiprocessing systems. We've pretty much all of the more advanced computer science real-time features we've tried to implement as users have requested them and needed them. So at some level, RTEMS reflects what users have really wanted and needed. So there's a surprising number of features in it. And the other side of that is, does it become too bloated? Well, we've tried really hard to focus on not putting unnecessary dependencies between things.

And you should be able to get a completely linked static executable on, say, an ARM thumb for about 16K and still completely bring up and shut down the system and have a task with a reentrant C library. So we're balancing the needs of the hobbyists with the high integrity space environment. So that's, but it's all driven by user features. I mean, we even have support for rate monotonic scheduling, earliest deadline, first scheduling. So one of the features that got added in the effort to add symmetric multiprocessing was the concept of a pluggable thread scheduler. So you as a user application can actually specify which thread scheduler you want to use, whether you want to use a deterministic priority scheduler, which everybody's used to, or an earliest deadline, or a constant bandwidth, or one of the symmetric multi-processing schedulers

if you're on that type of system. So we've tried very hard to stay pure to the computer science. So the algorithms have solid theoretical foundations while also trying to listen to users and users submit their own features. So there's a lot of diagnostic features that have been added by users who simply found they needed it to hunt down some odd bug. So one of the good things about an open source project is that when it's done right, it does become the commons for a group of people to put their best knowledge and best experience in. And that's really kind of neat when you get people from the automotive or the high energy physics or the space community or the hobbyist trading their techniques and they all kind of go into a pot and come out as some delicious gumbo. I like that analogy quite a lot. As I'm sort of remembering what symmetric multiprocessing is and nodding along blankly to priority ceiling and inheritance, I think we should actually go back a little bit and work on the definition of an RTOS. Some of our listeners are electrical engineers.

Hi, Chris Gamble. So the classic textbook definition of an operating system is just a set of software that provides an abstract interface to hardware and services. So that's still true today, no that the operating system has some features which enable you to write software which can meet deadlines or meet some performance characteristics. It often, it should mean that the operating system has algorithms which execute in predictable amounts of time. So one of the key things is not to be fast, but to be predictable. So you get the same execution time each time you make a service call. So there's not a lot of variance. And that's deterministic, is that word?

Right. Yeah, that's the word I should have used. No. That is the word. That's the word I should have used. No. So that is the word. That's the right word. And so we've worked very hard to make sure that algorithmically as much of our temps as possible is either executing constant time or is algorithmically bounded based on a design constraint like maximum of 256 priorities. So that allows you to guarantee the time.

And the other side of that is by focusing on the algorithm rather than tricky code, I think right now there are 18 processor architectures supported. So the implementation is as much in C as possible, and that means that you get the same execution characteristics on each of the processor architectures because they're algorithmic correctness and algorithmic optimizations. But real-time covers a lot of territory in what people think, what types of requirements, and what type of design patterns people use.

So that's why there's so many real-time operating system solutions out there, because they generally all have some domain or design pattern that they're useful for. So the basic idea is that compared to something like Linux, with this, you could write down on a piece of paper, okay, I'm going to get a hardware event at X time, and the interrupt handler is going to run X time later, and it's going to take this amount of time, and you could actually budget and be pretty confident

that I'm going to be able to respond to this hardware event within X number of milliseconds. Whereas with Linux, yeah, it's going to pipe through the kernel and something else is going to get scheduled in, and there's no way of saying, yeah, I'm going to be able to respond in a certain amount of seconds. Whereas with Linux, yeah, it's going to pipe through the kernel and something else is going to get scheduled in. There's no way of saying, yeah, I'm going to be able to respond in a certain amount of time. Right. We've made great efforts to make sure that's exactly the case. With so many different users, I mean, hobbyists and mission critical things don't seem like a huge overlap. How do you make sure that you retain low latency

until you do have a fast response time or at least a deterministic response time? Well, I mean, the hobbyists mostly focus on running on lower end, I shouldn't say lower end, more inexpensive hardware than, let's say, a satellite. So we where... I hope so. Right.

There are some cheap satellites out there, but not like that. Well, a lot of hobbyists are interested in running on, say, a Raspberry Pi or a BeagleBot. Yeah. And, I mean, honestly, a lot of the space community is, too, for prototyping. But for the purposes, so they use different hardware platforms, but they don't, a lot, I mean, most submitters submit hardware-specific code. Not nearly as many submit code back into the operating system proper. And those just get highly analyzed so you don't seal a lot of churn on the operating system algorithmically unless someone really has found a new way of doing

something and then it'll get seriously discussed so for example there's a few um well actually the We have board support packages. The European Space Agency popularized a Spark processor for use in space, and we have board support packages for every variant of that. We know anytime somebody submits code that would touch one of those board support packages that it has to be very rigorously reviewed because everybody who's using that code is essentially a space application user. But if somebody submits something for the Raspberry Pi, as long as we can duplicate the results they had, then there's a pretty high confidence that it's okay. So there's a, it's a strange notion to apply to an open source project, but there's parts of the code that are highly critical and you have to be very protective. And there are other parts, like we have a board

support package for a Nintendo Game Boy Advance. I mean, if somebody tells me it's broken, I mean, and wants to fix it, that's okay. I mean, that's obviously not going to fly. So there's just a difference in the view of what I would consider real production board support packages versus ones that are pure hobbyists. That makes a lot of sense. I mean, it's a matter of mission critical, who has it and who doesn't? Right, and that's why it's important for users to speak up. So you do know that anybody who uses one of those Spark board support packages is almost certainly headed toward a space mission

because they're available and space-hardened, and no one else really uses embedded Spark anymore. But there's a lot of critical ARM applications too, but they don't tend to feel BeagleBone. They tend to feel maybe a custom board based on a similar part, or maybe even the same system on chip, but they don't tend to feel the BeagleBone. Well, you have Atmel AVR as one of the supported architectures

given the Wikipedia page, also the 68K. So those are pretty low processing power architectures. How do you run a whole operating system? I mean, I guess I did run PSOS on 68K many years ago. 68K was a Mac processor back then. Right. Yeah, but it's old. It is old.

And the AVR is new, but it's 8-bit. Right. The AVR is almost a push to see how low we can go. The 68020 was actually the first processor Artemis was written for. And the main limitation there is not as much the processing power as it is the amount of memory that you used to see on those systems. Some of the larger features just tend to take more memory than you tend to have on those low-end systems. And that's something we have to battle in the test suite because, for example, some of the file system tests create a RAM disk of at least a megabyte.

Well, if your target board doesn't have at least a megabyte and a half or two megabytes, you can't give a megabyte to a RAM disk. So a lot of those tests just will not fit in the um on the particular target hardware so believe it or not a lot of the processors we fly on um are from 14 to about 25 megahertz that's a fairly normal space uh qualified processor the-end ones tend to be in the 130 to 200 megahertz for space applications. Now, on the ground, in normal embedded applications, without those kind of critical requirements,

the processors obviously are a lot faster. Most of mine aren't, but okay. I believe you. When you talk about BeagleBones and Raspberry Pis, those are a lot faster. Most of mine aren't, but okay. I believe you. When you talk about BeagleBones and RaspberryPi, those are a lot faster than the ones I usually use. Those are what I consider large, fast RTEMS platforms. Those are large, fast computers that are pretending to be embedded systems. Sorry, I'm a little opinionated on that one.

I'm such an embedded system snob. It's fine, because one of our users actually has used the Raspberry Pi as an RTEMS development system just to prove it could be done. Exactly. And it does appear that it is about the performance of a mid-90s Spark

Sun workstation. Based on what we all remember of build times around that period. I remember being so excited to get one of those on my desk. It was so powerful. Now it's a $20 computer. It's pretty amazing when you think of it in those terms. Yeah, that $10,000 workstation is now a Raspberry Pi.

I don't have to use the build server anymore. I have my own fast computer. It just shows how things have changed. But the reality is, normally it's more important to be predictable. And you pick the processor for whatever the job is. Embedded systems, in our class of real-time operating system,

you tend to build special purpose devices, which are effectively running some single application, and the whole set of everything from the processor, its speed, the amount of memory, storage, and peripherals is custom picked just to meet that applications needs so we just tend not to see huge boards because of the people the cost of those is very expensive in volume hobbyists and critical applications people the both are interested in dead code but for different reasons this is how there's some bizarre overlaps between the user, what appear to be completely different users. The hobbyists generally up until the Raspberry Pi type generation

was trying to squeeze into very small boards. So any code they didn't need literally would not fit. So there was a whole effort from hobbyists and students to help us trim down any code that didn't need to be linked in to a minimum image. And that benefits the critical application people because they don't want dead code because that tends to not do well in safety analysis. Exactly.

Every line, I mean, if you're looking at how can this line break my whole system, you want to have as few lines as possible. Right. And it's funny, they both want as few lines as possible, but they want them for completely different reasons. And the hobbyist, it's a make or break. It won't even fit on my hardware if I don't do this. So is RTtems lightweight embedded processors are so crammed giving up flash and ram to an operating system can be painful it's very small i mean it obviously depends on the

code density of the processor and what features you use. But on the low-end arms, you can do a complete base for an application in 16 to 20K. Even on the embedded sparks, you can start a realistic application in the 32 to 40K range. Is that flash or RAM? That would be Codespace. Codespace. We'll call that Flash or ROM. It's ROMable, yes. But a lot of that on the other side of that, the sky's the limit because if you

start pulling in TCP IP and file systems and lots of features, that tends to mean you have lots of user code too. And every little bit takes a little more code. And do you have the RAM numbers for an ARM too, Andy? Other than the RAM numbers, I'm trying to remember. We have some examples of tuning down the system and

recalling a four-task example can be under 7 or 8K of OS memory. I'll ask you for the tuning document to put in the show notes. Okay, so that actually would work.

But a lot of people decide all of this is just too much. You know, operating system, why bother? I can write my own little itty-bitty scheduler. And then you make a message passing system and then you've accidentally got an RTOS. But you start with this idea, I can solve it better for less, I want to say cheaper,

but really it's less processing, less RAM, less code space used. Do you have advice for people to decide when do you really need an RTOS versus when you really can just go bare metal and go on your own? I would say in general, if you find yourself already talking at an abstract level about your application as having naturally concurrent activities, like I'm reading from one device

while I'm supposed to be writing another, if it becomes quickly too complicated to do like in a single control loop, you probably are looking at multiple threads so I mean at that point it's pretty obvious that it's probably better to look at an operating system the other thing I tell people who why what would be a good reason to use our Tim's even if you thought you could do it bare metal well the first thing is it doesn't cost you anything to spin there There's no money cost. There's no license to track. So if nothing else, out of the box, you have a working C compiler, C library, and hopefully a board support package for what you need. So you can at least start out with a hello world that's

working and you didn't have to do any work. Because if you do start to roll it your own, you do end up becoming responsible for things like where's my C library coming from, what's my compiler, if I start to need a file system, what am I going to do? So sometimes it's just better to share your maintenance with other people. So sometimes even if you don't need another thread, just having a good base environment that you can ask questions about and report problems on is worth having it to start from. I totally agree.

And I think if you find yourself desiring to roll your own, so to speak, I think you need to maybe question why you're doing that. Because there is some merit to doing it yourself as a learning exercise. But there's so many options like Artem's out there that, okay, you can drop it in and it'll work. If it's too big, then you start thinking again. But it just seems like an ego trip almost to say, well, I can do better than several operating systems that have been developed for the last 30 years. And there's a lot of pitfalls for race conditions and timing windows that you may or may not think of when you're doing it as a one-off.

Even though no one is perfect, anything that you're not developing on your own has some history and provenance and prior usage, so it should be less buggy. New software is buggy software. So anything you can reuse should always be a better choice. That's an interesting attitude. A lot of people have the opposite, which is if I didn't write it, it's no good. The other person who coded

writes crappy code. And you're saying, no, other people have good code and there's a good reason why they did that. And maybe it isn't perfect, but we should at least respect that they're not all idiots. That's what I tried to do. And I've seen over the years, there are obvious examples where there is some really bad code out there in the world. But what I've noticed a lot on Artem's is on every project, you have a fixed amount of time to get from not working to something running. So the first person who, say, submits a device driver for our teams, it will work well enough for them on their target platform. The second person who comes in and uses it has logically about the same amount of time to get their project up and running,

and they either will port to another architecture of that device driver, or they will tune it. And so over the course of two or three, first two or three times that a particular piece of software is reused, on our TIMS at least, the first person tends to get it working. The second person tends to either deal with performance issues or portability issues to other architectures and by the time you get to the third or the fourth user most of those cross-platform and performance issues have been completely resolved and then the code goes very stable so it's the other thing that's i would like to speak highly of the Artemis user base.

A lot of our users are coming out of the high energy physics, radio astronomy, and space community, the ones who speak the most. doing generally doing very safety critical high integrity applications like building controllers for fire systems or gas detection systems there been some automotive controllers so they each of our users is generally pretty smart in their own domain. So you're kind of in some ways getting this, again, the community's knowledge comes together to give you the best of breed. And that's what you hope for. I mean, the first time you write something, you make all the mistakes the first time. So you hope the second time somebody comes in, you get those many eyes. So that's what you want. You want eyes on code. And the first time you write something,

there are just not many eyes on it. So you were starting to talk about applications, and I want to distract you onto that. So it's been to space. What's it been to space on? What are the space missions? Space, space, space space space okay so we've been since we were lucky enough that uh the nasa mms multi magnetospheric somebody ought to look up mms i never get it right so the both nasa and the european space agency around the year 2000 started looking at RTEMS seriously for mission software. The proof of technology mission that preceded MMS, that is a solar mission that just launched,

they started work on that in 2000, 2001. But in both the European Space Agency and NASA, you need a risk reduction proof of technology type mission before anyone will fly you. No one wants to be the first to use any technology. There's always a risk reduction mission. So since then we have been extremely fortunate. I think there's 20 plus European missions we're on, which includes Herschel, Planck, the Galileo's GPS satellite constellation. Let's see, Probo V, which is a vegetation monitoring satellite. There's a number of other satellites. And on the NASA side,

we've been fortunate enough that JPL has used us in a number of radios along with a number of other NASA agencies. And so a lot of a lot, if not all, of the radios in the Deep Space Network I've been told are running rtams i have no proof of that other than what someone's told me there's we are on some very cool things like the dawn mission which is in the asteroid belt now and has been sending back pictures from the asteroids we're on um we're actually on mars curiosity in a rate at least a radio we're on the mars reconnaissance orbiter we're on the uh what's the coolest and hardest to make look impressive

application it's called the express logistics carrier elc on the space station it is essentially a box that experiments plug into on the outside of the space station. It is essentially a box that experiments plug into on the outside of the space station and the communications backplane runs RTEMS. It is really cool. It looks like an aluminum box and it's hard to say, but it is an extremely cool application. So, Solar Dynamic Observatory,

if you've ever seen the videos of the huge solar flares, those tend to come from Solar Dynamic Observatory. MMS, which just recently launched, was another solar mission from NASA Goddard. So, there's a lot of the space applications. High energy physics side, Stanford Linear Accelerator Center, a Canadian light source in Saskatchewan, has a cyclotron, Argonne National Labs, Brookhaven. Most of the Department of Energy National Labs use RTEM somewhere in a high energy physics. There are other uses outside of Europe and the U.S., but those are the ones that usually come to mind. So nobody's going to be able to say, I can't use this because it hasn't been proven in mission-critical applications.

No. It's got a very long history with NASA and the European Space Agency. Last week's guest, Lane, said that he chose Purple as his boards because then when he looked in other projects, he could say, oh, that's one of my boards. Don't you wish you could kind of do that with the software and say, oh, yeah, Artemis is on that. I can see because it…

Oh, I wish we could, but that's one of the hard things. And back to the marketing budget since no one signs a license unless someone tells us that artem's is inside something then we really have no idea um so we were fortunate a few years ago to find out that uh bmw motorsports actually used artem's for a control system in one of their, uh, race in their, in their racing motorcycle.

But apparently they had been using it for nearly a decade before they told us about it. So, I mean, that's kind of typical. it's, it's the sad thing because unless we lock up on a, on a paper, someone tells us something, or luckily, a few years ago, EADS, which has now been merged, well, I guess they merged that back and that's just now part of Airbus.

Someone published a conference presentation on their history with RTEMS, and it listed over 20 missions, and a lot of those I had never heard of. If someone listening is using RTEMS, how would they tell you? Usually, sometimes they post to one of the mailing lists. Sometimes they just privately email. Occasionally, I think my favorite story, how we found out we were on Mars Curiosity. I was at a conference. Somebody was giving a talk. Somebody reached over my head, handed me a coffee mug and said, I think you want this. And that's how I learned we were on Mars Curiosity. And I couldn't get up in the middle of conference to find out what this coffee mug meant, but I was pretty sure I liked it.

That's cool. Well, I have some sort of link for later in the show notes, so if you are using our times, we will get a way for you to let Joel know, because that is pretty cool. But I want to switch gears for a minute. And you are doing Google Summer of Code and ESA's Summer of Code in Space. Right. These are amazing university student programs. I think this is the 11th year for Google Summer of Code.

This is probably the fourth or fifth for the European Space Agency summer of code in space. And what happens in both cases is open source organizations apply. If their application is accepted, then they have to have wish list open project ideas and then they apply through either Google or the European Space Agency's websites to take on one of these open project ideas. The students work with people from various open source projects to get the project idea put together. Our proposal template looks a lot like just a normal contract

proposal this is what i'm going to do there's a midterm evaluation and a final evaluation and we help the students set their goals for what they're going to do and if the students are successful, just look, the ESAS is €4,000 for the summer. Google's is €5,500, which based on the current exchange rate means that the Google one is worth quite a bit more money right now. The interns or the people who do this, so they get paid, and you don't have to pay them because Google or ESA pays them. Right. And are they remote? They are normally remote, and normally we don't meet them. I have met a few students over the years by making sure I knew where they were from. So if we got in the area, we'd take them to dinner or something. But most officially during the program, we're probably not going to meet them.

Because I mean, like last year, I had a student from Egypt and I obviously didn't just bump into him randomly one day. But interns are tough or they have been for me. It feels like I put as much work helping and teaching them as if I'd done the work myself, though there are other rewards with watching people learn stuff. How do you manage to keep them on track and not distracted by going to the beach every day or anything else that would interest a university student? Luckily, we've gotten really good students. So we've only had two out of about 70 students that we've had to fail at midterm. But we also tend to get graduate students, which is a little better.

But we try to make sure that we have a group chat meeting once a week where everybody gives status. We try to monitor. We assign at least two people to every student. So even if they're not technical mentors, they're tracking them to make sure that we've heard from them at least two people to every student. So even if they're not technical mentors, they're tracking them to make sure that we've heard from them at least and check in on them if something goes wrong. I mean, sometimes students have had family issues during the summer. But, you know, everybody expects that they'll go on vacation or will go on vacation during the summer. And you're

just supposed to be up front and mature about it to say, I'm going to be gone for a week. So don't expect anything. Because if you disappear in the middle without saying anything, that's really bad. It's just like real life at work. You don't show up for a few days and people start to wonder. But if you tell them you're sick or you're going to visit your mother, then they don't worry about you near as much. But you're right. We put a lot of effort into making sure the projects are achievable. We learned that making sure the students can do like a hello world type project before we will even consider their application ensures that they can actually do something with our teams. So if they can't get

to the point they can do that, then they're probably not going to be successful in their project. But at least if they meet that minimum hurdle, they at least have a good running start. And since I know we have some college students listening, how do the students get chosen? How if they meet that minimum hurdle, they at least have a good running start. And since I know we have some college students listening, how do the students get chosen? How do they apply? Do they apply to you or to Google? They should talk to, well, I mean, let me be fair first. There are 137 free software organizations involved in Google Summer Code this year. There are approximately 20 in the European Space Agency's program.

So the first thing to do is they need to go to the websites for the two programs and see which projects are involved and get in contact with those free software organizations so they can see what projects they have. Because both of them tend to have a lot of projects. Every open source project you have ever heard of has at one time or another participated in Google Summer of Code. Everything from games to KDE and GNOME and the Free Software Foundation, Apache,

all of those plus smaller projects that are very good. And everybody who does it really enjoys doing it. But after they've figured out what project they're going to do by discussing it with the mentors from a free software organization, then eventually they will file an application via the Google or ESA program websites. And once we figure out, once all the applications are in, from an organizational standpoint, we look through them and figure out how many of them we think have a good chance of success. And ESA normally only gives one, possibly two slots per organization. So we would have to pick the student, best student there.

On the Google, we balance it based on how many mentors we have, how many students look like they can succeed. And I think we've done anywhere from five to nine or ten students in a year through Google. So a lot depends on how many mentors we can have and how good the students look like. But the way it's done is, I mean, Google has an amazing number of slots. I think last year they had almost 1,300 students that participated.

So that's a very large program program and it's a very successful one uh hugely successful and last year was the 10th anniversary and so that i don't remember how many students were in the first few years but most years we've been in eight years and i think every year has been that we've been in has been over 800 students. So somewhere between 800 and 1,300 every year. So there are probably close to 10,000 students at this point who participated in it. It's funny that we're so focused on coding. And it was just the thought crossed my mind is we kind of need a summer of solder for electronics projects for companies, but it's not as easy a thing to do.

You can't set up an electronics project remotely as easily. No, that's harder because there are other programs that do similar things on smaller scales. I think some of the larger free software projects have the non-profits have held similar programs, but I don't know of a hardware focused one. I don't either, but I'm sure someone will tell us about it. They're usually like contests or when I see things like that. Well, I guess that is the point in which I mentioned the Hackaday Prize is happening and it is a contest. And maybe if you do hardware all summer and work very hard, you can win the nearly $200,000, which is very cool. But you'll also learn a lot whether you win or not.

Okay, well, I think while I have a whole bunch more questions, I am almost out of time. I did want to ask about the RTEMS GPL. You don't use GPL exactly, so it's not like if I use RTEMS, my code becomes sticky and GPLed, and now I have to set it free. No. One of the things we're very strong on

is that none of the code in RTEMS that's shipped as part of the main body of RTEMS has any license that places any obligation on any end user. So the RTEMS license is actually almost identical wording to what used to be used in the GCC language runtimes that could be linked with any program you compiled with GCC. So our intention is that we know you normally statically link our code with your application code, but we have no intention of placing any obligations on you.

We don't want to have your code have to be open sourced. We don't want you to have to redistribute anything. But on the other hand, you are completely free to redistribute the source, which is really handy in the scientific community because a lot of times the code does get handed off between different research institutions, universities, various research projects. And so there's really no licensing to track. They own their application, and they can just hand off RTEMS and whatever else came with it if they want to.

So it's also a good way to support open science. So if you want the application to move along, there's using RTEMS in no way discourages that. In fact, it makes it easier. Which isn't what you can say about ThreadX or any of the others. I shouldn't pick on them. It's just something that comes with the general open source free software model is that if you have all the source and you have the rights to redistribute it, then that's kind of a benefit when you're doing a community or you need to hand off things to a customer

or you simply have to support something for 20 or 30 years. And that's a lot of our users have very long life cycle projects that really is where open source and embedded space makes a lot of sense so Christopher do you have any more questions? I have one question, you live in Huntsville right? yes so in the bustle of all of the excitement about technology

we usually hear about Silicon Valley and Seattle and cities like that. But Huntsville is a real hub for doing aerospace and space science and software for aeronautics, right? Right. There's quite a bit here. I think this is one of the largest research parks in the U.S. It's the home of NASA Marshall. It's where the Von Braun team came after World War II. So this is where the Saturn V and all of NASA rocketry has been done. It's also the home of Army Aviation and Missiles.

So the helicopter community and the missile community from the Army is based at Huntsville. There's a surprising amount of telecommunications companies, and there's even some biomedical research here now. I was surprised a few years ago to learn that the actual tissue samples from the Human Genome Project are located about a mile and a half from where I'm sitting now. That's very cool. Well, and I didn't want to forget the M in Artemis didn't always stand for multiprocessor, did it? No, and Google is happy to remind you about this too.

And Wikipedia. Actually, Wikipedia does too. So that one's still there. The original M was because it was for missile because it was developed for the missile command. And then it went through a period where we realized it could have been used. It wasn't restricted to missiles. So the M was broadened but not improved when it was changed to military. And then we started realizing that no open source project would ever get any users if the M was military.

But it's been multiprocessor for years and years and years. Probably by 91 or 92, it was already multiprocessor. I would have to actually go find things to actually know when it changed. It's that long ago. So you guys do marketing? Occasionally. Mostly by marketing, we tend to go to, there's a few workshops and conferences that we like to show up at.

But generally every other year we'll show up at Embedded World in Nuremberg. All right. Well, since we are about out of time, Jewel, do you have any last thoughts you'd like to leave us with? No, I think I've talked a lot. All right. Well, we'll see if anybody has used RTIMS and wants to let you know about a neat project they've worked on. And anybody who wants to apply for Google Summer of Code or ESA Summer of Code, if you can't just type them into Google, there will be links in the show notes. has been Joel Sherrill, Our Time's Project Lead. Special thanks to Andre from the Great White North. No, not Boston.

Thank you for connecting me with Joel. Thank you also to Christopher White for co-hosting and producing. And you know, I say that every week, but it isn't like, hey, sweetie, thanks for taking out the trash.

I get a lot of credit for this podcast and many compliments on my voice. But if it was up to me alone, I'd probably sound like I recorded in a bathtub with the water running. Luckily, Chris cares a lot about making us sound more than tolerable. And as demonstrated by a recent fledgling podcast that never posted its first episode, producing a show is hard work.

Audio editing is not for the impatient or faint of heart. I couldn't do this without christopher so when i say thank you for producing i really mean it wow and enough ranting so thank you for listening contact link on embedded.fm blah blah blah also stickers if you send your address do let us know if you're using artis because that's just neat. Mars curiosity. And a final thought to leave you with this week. We're losing too many really great people. So this one's going to be from Terry Pratchett.

The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it.