Embedded - 329: At Least 32-Bits, Thank You

Episode Date: April 30, 2020

Kate Stewart (@_kate_stewart) of the Linux Foundation spoke with us about the Zephyr operating system (@ZephyrIoT). Some Zephyr docs: Getting Started Guide Supported boards Two projects using Zephyr: Open Artificial Pancreas System (openaps.org) Zephyr on a Hearing Aid talk at Embedded Linux Conference 2019

Welcome to Embedded. I am Elisa White. I am alongside Christopher White. And our guest this week is Kate Stewart of the Linux Foundation. We're going to talk about the Zephyr RTOS. Hi, Kate. Welcome. Thank you very much for inviting me. Looking forward to talking with you on this one.

Could you tell us about yourself as though we met at, I don't know, an online embedded systems conference? You probably will have seen me at some of those, yes. I've been working in software for about 30 years and open source for over 15 and have been involved with the Embedded Linux Conferences for several of those years, as well as the Linux Plumbers Conference. So I tend to be focusing on the embedded side of the technology space and helping to see what's emerging and making it a little bit better. I started pretty much for the compilers way back when and have liked to be between the software and the hardware interfaces

and working with the hardware. So that's kind of my happy spot. And Zephyr is one of the projects that I'm working with at the Linux Foundation. I'm a strategic director there. And we're basically looking at how can we apply the best practices into an open source project that can be used when Linux is just too big, because Linux doesn't get much smaller than two meg these days. And Zephyr is how do we make sure we have a good place for people to collaborate? And that was sort of the motions behind it. So that's one of the projects

I work with. Okay. And I have lots of questions about Zephyr. Me too. But before we do that, we want to do a lightning round where we ask you short questions and we want short answers. And if we're behaving ourselves, we won't say why and how. Are you sure? Are you ready? Okay.

Okay. Go for it. Should we bring back the dinosaurs? No. Compiler question. What's your favorite token in any language? Does anyone have a favorite token? Semicolon, actually.

All right. Well, actually, no. One of the projects I work with is something called a Cregit. And what it does is it replays a Git history for a log, and instead of storing things line by line change, it stores it token by token changes. And that semicolon tends to be very useful for figuring out where statements and logical syntax elements end. Yes, tokens are cool.

Do you like to complete one project or start a dozen? Start a dozen. Do you like to complete one project or start a dozen? Start a dozen. Do you have a favorite processor? That one would get me in trouble. Okay. I need to be neutral. What planet or moon would you visit if you could? Earth's moon.

I'd love to see the view in the other direction. It's a short trip, too. After all these years. Yeah, there's moon. I'd love to see the view in the other direction. It's a short trip, too. After all these years. Yeah, there's that. And it may even happen, you know, within our lifetimes. Do you have a tip everyone should know? Tip everyone should know. Always assume that people are trying the best

and don't assume people are being malicious when they're irritating you. That's a good tip for getting along in the world. Well, in open source projects, we know there's issues with trolls. And usually it's just a matter of sitting down and really understanding what they're doing and trying to make sure the empathy happens. So community building is something that's an art in some ways and i think we're figuring out some things we still have a long way to go in other places and since we're all remote these days with covid we're doing more of the type of community stuff that we usually do online and more of the community more of the uh wider population is going to start to experience

what we tend to do online so zephyhyr, are these best practices related to community building or to code or both? All of the above. So with Zephyr, we looked at what was sort of working with Linux in terms of processes, things like making it easy for people to contribute. Zephyr works with the DCO, which is a developer certificate of origin, and that's the same that Linux uses, where people sign off on their changes. So they retain the copyrights to their changes,

and that means that things don't get unpredictable for them based on who the copyright holder is or some CLAs that have been signed and things like that. So the Linux kernel has been very effective at using that for 20 years, 20-plus years. And so that seemed like a good model for us to pick up and work from when we started the project off. We wanted to make sure it was neutral governance

so that no one company controlled the entire project. And so when we wrote up the governance docs, we made it clear that we're keeping the decisions about finance and where we spend the money separate from the technical decisions, which is another lesson we learned from Linux. And we also have the concept of doing Ltss ltss are what um they're long-term stable releases okay and so the kernel has various long-term stable releases and generally that's what the distros pick up to put products out on and it also is where other people take and make products um from and the idea with that is that there's one place that we can focus on making sure that the fixes, their important go. And if people want to sort of do that, they're not constantly chasing a moving target and shifting interfaces. So this gives them enough of a stability that they can build their products on top of it.

It's the same concept you see with Ubuntu, right? 16.04 LTS and 18. And 2004 that just came out, for that matter. Yes. Actually, in my background, I was Ubuntu's release manager for a couple years, about 10 years ago now. And so I learned a lot of lessons from that experience. It was very formative in me understanding what communities are and when they're working well and when they're not and how to, you know, how to sort of navigate that side of

it. So that was, yeah, that's been very formative for me. And how do we make good communities happen and how do we keep people motivated and make it a good place for them to work in? There's a lot of things that I think that the Ubuntu community is right. So pulling some of those lessons in subtly has been something that we've been working towards. There's a few things that, you know, the Linux community has that we're trying different paths. For instance, we don't have like one dictator at the top of the pyramid. We have maintainers and we have a TSC chair who sort of fills that role. And that chair is elected every couple of years. So we have continuity and, you know, maintaining the TSC, but it's, you know, different voices and it has, you know, the community has a chance to change over.

And we've gone through, you know, a change in chairs already on the project. And the same way in our governing boards, we also change and everything's under election. So it's the peers working with their constituencies to have confidence in them. What's a TSE? I'm sorry. It's a technical steering committee. Okay. Okay.

So it's basically where the maintainers, like with Linux community, where there's maintainers over pieces of code, and that's where they're sort of meeting. And so when we're adding something that's new or potentially controversial, that's where they sort of can meet to thrash out the differences. What's controversial? I mean, what kind of things can be controversial with an operating system? Oh, well, two different people have different proposals on how to do a certain

piece of function and the network and things like that. And so it's discussing the pros and cons and figuring out what's going to be the best future so that the right technical decisions can happen. Or if there's a couple of different approaches for doing things on security, that's interesting. One of the areas we're working on right now is what coding guidelines does the Zephyr community want to follow? And that's an area we're sort of trying to all come to agreement on, specifically because we're going to be going

after a safety certification as a project. And so we'll have to comply with certain, certain of the safety standards require this type of thing. And having MISRA compliance is something they're looking for. So figuring out which rules we're going to adhere to, which ones we're going to make a formal exception, and then making sure we have the tooling in place. But when you start talking about how code is formed, that gets people very passionate. And we all want to make sure we can get Zipfler to that stage. So we're looking at, okay, how do we get the guidelines that everyone can

live with in place? So that's, I think, queuing up for the TSC discussions right now. It's been in the safety committee up till now. This is, this sounds like a lot of discussions, but this is an actual operating system. It's not just about the organization. Is it an operating system? Is it a kernel? Is it really a real-time operating system? What is Zephyr? Zephyr is a real-time operating system, and it has a reasonably complete stack around it now with a lot of peripherals, the same way Linux has a lot of drivers and infrastructure.

Zephyr for organization purposes. So it is a true operating system. It has been evolving, and there's lots and lots of code going in. I just checked before I got on this, and there's now over 40,000 commits in the Zephyr repo. And given that the project just opened up in, I guess, about four years ago, for an embedded project, that's pretty spectacular with that rate of change. So we've been working that up, and there's 672 contributors right now. And when I go and look at insights on GitHub itself,

as of today, right now when I'm looking at it, over 129 authors have pushed 1,300 commits to master and 1,400 commits, or 1,409 commits to all branches. And this is in the last month, so in 30 days. So to put it in perspective, I think we're sort of averaging now over one commit per hour. And the Linux kernel community, the Linux kernel itself, the upstream is about 9 something, 9.4, I think, commits per hour.

So it's active. So it's active. So we're actually, like you say, we've got a nice velocity and a lot of the content is, there's a lot of really good features going in and a lot of polishing and people are finding a good place to do some, you know, to from, and then they contribute back. And so that kind of thing makes it really quite exciting to see what's emerging. One of the surprises for me with this project was when we were aiming for the MCUs and MCORs for the ARM processor, as well as other architectures in that embedded side. And since we actually started, we've now got a very interesting range of architectures in here,

including the R cores, which are sort of a logical one on the ARM side. But then A cores has just started getting added. We started off with RISC-V 32-bit ones, and then put a 64-bit RISC-V port in. Intel has been there since the start, and it's been x86 and then x86-64. And we're seeing things like open power coming in, which I would not have expected that to come into,

a 64-bit power PC to come into an embedded OS like this. What's the reason to scale way up to the desktop class chips? Is it... I guess I haven't looked into Zephyr's architecture that deeply, but is it a full process model that can take advantage of an MMU

or is it more like your traditional ThreadX and stuff like that? No, it's a full processor. You can take advantage of the MMUs. And I think they're putting some of the more desktop type of cores in because they want to use sort of in mixed systems. Right.

Where they're using Zephyr to do a limited function, but they're running on the same cores as other and using other interfaces. Linux may be on other cores, and certain functions get partitioned between different cores. Okay. So we're talking about a multi-core system and running Zephyr on some of them. Yeah.

Okay. So one of the chips that's listed is the Cortex-M0, which doesn't have an MMU. How does Zephyr... It seems like that's a dividing line. Either you run apps and you have a protected memory space for the apps, or you don't really run apps. You run a kernel and a scheduler, and you have tasks and processes, but they're not independent. How do you handle that? So what Zephyr has is it's using the same K config infrastructure that's in the kernel.

And it uses device tree. And so you're configuring the parts you want to have working, and that's what lets you actually statically build it for a certain footprint. Zephyr gets as small as about 8K. But once you start putting some of the communication stacks on, you're obviously going towards 512. But we can do a lot of things in very small footprint. And we do that by statically compiling what we need. But you'll see that a lot of the work and a lot of the architectures and everything else is selectable. And that's what

lets you have that flexibility. The flexibility is great. And you talked about the velocity and people committing. As an embedded software engineer, I hate it when things change. Well, that's why we have an LTS for you. Okay. So you can basically say, okay, none of my APIs are going to change. It's going to be stable for the next two years. I can innovate on top of that, and then I can decide if I want to move to the next LTS. And when I see Zephyr is supported on a chip on the Zephyr page, what means that the code is in the repo and that there is a certain amount of testing that's been done, that it's building and so forth. Some of them are different stages of completeness.

And I think it's a matter of figuring out which one you want to use and how close it is to your target. There are certain ones that are very well tested and very well exercised, and there are others that people are adding. So I think it's a matter of looking and sort of seeing what you want to be using and what you have access to. How do I look? I mean, what am I looking at? Am I just looking at Git commits in this area, or is there something that will tell me,

oh, this chip is fully supported with all of its interfaces and this chip well you can run zephyr on it but don't expect a spy driver and we'd love to be that way um the documentation is really pretty good um and love there's a lot of um information we pull out of the documentation out of the code for the documentation and then if you look at the docs.zephyr, it goes into more detail about the level of support on certain of the chips. I'd love to have a master big table of, you know,

which chips, which drivers have well-tested and everything else, but it's an open-source project, so no one's started generating that yet. Is that, I mean, documentation on a normal project is always one of the hardest bits. How do you convince people who aren't necessarily doing this as part of a professional role that the code alone doesn't actually stand alone? Well, this is, I think, where we're sort of lucky in that the companies that are working on Zephyr have had a strong belief in important documentation.

In the early part of Zephyr, we've had David Kinder from Intel, who's done an absolutely excellent job of keeping the documentation up to date and has made Zephyr's documentation look really good. Unfortunately, he's moved on to another job inside Intel, but he's put us on a really good sound foundation. And so I'd encourage you actually to look at our documentation. And if people like to work in documentation, obviously help is always welcome. If you see an error, just open a bug. Or even better yet, a pull request. Because all our documentation is up on GitHub too.

It is, and it's pretty easy to read, and it's not just Doxygen outputs. Right, yeah. No, like I say, kudos for that or to David Kinder. Like I say, he did a beautiful job. Your list of supported boards is really long. You already mentioned Cortex-M0s, the Cortex-A53s, RISC-V. I didn't see DEC-al alpha on there, though. No, no tech alpha. No one's given us a simulated version of it yet. However, there's, like I say, there's a pull request waiting, looking for getting Spark in there now. Oh, God.

Yeah. Like I say, we're seeing a lot of people, especially with the FPGAs and working with the open architectures. There's a lot of revitalization happening on some of these coding families, like some of the architecture families. So 8051 will live forever. Oh, no. I haven't seen that one yet. We do need to have at least 32 bits.

Thank you, please. For 32 and 64. Oh, gosh. Yeah. That was 16, wasn't it? If I remember is correct. No, no, it wasn't 32. Okay, fine.

No, it wasn't. Oh, 8051. Yeah. It's eight. It's an eight. Okay. Thanks. What do the boards have in common i mean do they all have usb j tag flash tools or

i think they all have basically some sort of serial connection that's coming out most of them have some degree of some sort of interface coming out at least one way or the other. And yes, having the J tags in the connections, but I don't know that all of them do. I'm not sure if there is really common denominators amongst them all, other than they're all using the same kernel. I couldn't make a statement about the peripherals without a lot of work. Okay. How do I decide which operating system to use? I mean, there's FreeRTOS that's out there, and there's Zephyr, and there are many, many,

many others. I mean, you work for the Linux Foundation, you work on Zephyr, so you're going to tell me Zephyr, but... Oh, I also work on the Linux Foundation. You work on Zephyr, so you're going to tell me Zephyr. Oh, I also work on Linux, too. I also work with Linux, and I also work with other things. But yes, of course, I've got my biases. How do you think people should go about trying to choose? The way they should go about trying to choose is look what their end goals are.

What are you trying to accomplish? Are you trying to sell a product? Are you trying to do something as a hobbyist? That would be one aspect. What level of support is there? How seriously, you know, what does it cost you to get security updates, for instance? What does it cost you for moving, you know, getting things that you care about upstream? You know, can you retain your rights?

These are the things that I think are important, you know, that are worth looking at. I also think, you know, is how clear is the licensing? That's important to look at. Could someone come after you for patents after you've put a product out and made some money? One of the choices Zephyr made was to go after the Apache 2 license because it had an explicit patent grant in it. So people who are contributing to it are granting any patents associated with the code that they're contributing. And, you know, these are the sorts of things. So I think it's a question of, you know, what are you trying to do? And what's important to you from a, is this something you're just doing for yourself or is this something you're trying to

scale? Okay. So let's say it's something I want to scale that I want to use this professionally. Okay. So what you want to make sure is that you've got clean licensing because you do not want someone to be coming after you, after you've started shipping. So that's the first thing to look at. Next thing to look at is, okay, the patents. Then it's going to be, how easy is it for me to work with the code base? And can I find developers to work with the code base, who have worked with the code base?

Then once you start scaling it, are you trying to, you know, and it's out in people's hands, do you have a way of letting them know if there might be a vulnerability in it? One of the things we're doing right now with Zephyr is we actually have a security team that has now been working since the start of the project. So if you look up on the MITRE site, you'll see that Zephyr is actually one of the few open source projects there that is a CNA. And so the project has a PCERT team or Project Security Incident Response Team

that is responsible for, you know, reviewing any vulnerabilities that come in and then figuring out with the rest of the team how to address them and then reporting back to MITRE. So we are working on, you know, we have several that we're working on and we are being very transparent about them and letting people know when they're fixed. And then those who have products that have registered with us, we're letting them know within the embargo period too.

So we're taking security very, very seriously. And we want to make sure that Zephyr can be used in product safely. And in terms of safety, there's also the whole element of, you know, is your product going to be used in a safety-critical context? Zephyr is actively working towards being able to go after safety certifications. Which ones? We're hoping for 61508 initially. And there's some of our members that will also take that to 26262. We have potential interest from some others towards the medical device space.

But 61508 gives us a good grounding for all of the others. That's the ISO? Yep, that's the ISO 61508. And what we're sort of doing is, you know, if we can get to the stage where we can get that certification as a project, I think, you know, and we're making the, keeping the sources available to everyone. So, you know, you'll have a pretty good base. So the next LTS, LTS number two, we're sort of working towards that for next year. And we're hoping to have everything in the stage where we can go after those certifications with that next LTS2. So, like I say, we're working towards it pretty tightly. We've got a safety team that's working

on it, looking at things like the coding standards, as well as, okay, what evidence do we need to do? One of the things that we'll probably be talking about in an upcoming technical steering committee meeting is how we're going to be mapping code back to requirements, which is what a lot of these safety standards call for. Traceability, yeah. Bingo. Yep, exactly. And so we've got some ideas on how we can do a lot of that within the code base so that we can generate a lot of the stuff.

And as well as, you know, the testing and so forth. So we're sort of raising a little bit of a trail here on this stuff by doing it out in the open. And it's kind of exciting. That's actually part of the part. It's one of the parts of the project I'm super excited about, is that we're making progress here now. And security and being able to be certified, I mean, those are the two reasons to buy an RTOS. Mm-hmm. Because that makes that huge thing somebody else's problem.

But one of the things with Linux and I love Linux. I use it a lot. Um, but it is sometimes free as in a puppy. Um, sometimes it is, it is a lot of work to get it working and to get it configured the way I want. And it's super flexible, but it's so flexible that I spend all of my time worrying about my operating system and not enough time worrying about the application, which is what I really wanted to spend all my time on.

Does Zephyr suffer from that? I don't know how to phrase that question. In Linux, there are operating system distros, right? That basically do a lot of the configuration testing, keeping it up to date, pushing fixes at you, you know, Ubuntu, Red Hat, so forth, right? In Zephyr, we are starting to see our ecosystem emerging with solutions there. There's one of our members is Foundries,

and they will basically keep Zephyr up to date for you and work with you to make sure it's configured for what you need so that you can focus on your application. So there's options emerging. And because it is such a smaller code base than Linux, there's more, you know, you will need to keep on top of it, I guess, is one thing you do need to do, unless you have someone that's there to help you.

And there are consultants out there who will do work for you on Linux, as well as they also work in Zephyr. So some of the embedded Linux consultants that are fairly well-known in the industry right now, like ConsultCo is one, BayLibra is another. I know that they've worked and created Zephyr products for people already, and they also do a lot of work in Embedded Linux. And to a large extent with Zephyr, if you're familiar

with Embedded Linux and the integrals of Embedded Linux, Zephyr's pretty similar. You know, using device trees, using the k-configs, this type of information, and the structure of the directories is very clean. So things are pretty much where you roughly expect them to be. this type of information. And the structure of the directories is very clean. So things are pretty much where you roughly expect them to be.

And if you've got suggestions for how things can get better, obviously start pull requests and start the discussion. But there's a lot of knobs to turn and things to adjust. So I guess it depends on the product, right? It's true of most operating systems. And from the embedded side, it's a little... Linux has the downside of... Well, not the downside,

but it's a different world in that people use it for their desktops or as servers and things, which is kind of a different universe than an embedded device, where once you get your configuration on your embedded device, you're kind of locked down.

Never change. Never change it again. Well. I was going to ask about architectural. You said it's similar to Linux. Was it a clean sheet development, or was it let's take Linux and pare it down? Because I remember using Linux in the early 90s, amazingly. And, you know, it ran on a 486 with, I think I had eight megabytes of RAM and, you know, comparable to a nicely configured embedded board these days.

It's like, okay, why don't we just take that Linux from the 90s and put it... So was it that kind of thing or was it a clean sheet with Linux principles? Actually, it was, it ended up being a sort of clean sheet with Linux principles? Actually, it was it ended up being a sort of clean sheet with Linux principles. What had happened is that Wind River had made an acquisition of a company at one point and had their RTOS

sitting on the shelf. They have, obviously, VXWorks. And when Intel acquired Wind River, they took a look at options. And they decided to start with this RTOS that was sitting inside Wind River called Viper. And the Linux team in Intel, the open source team in Intel, basically took that and then structured it into something that was useful from their perspectives. And since it was a Linux team that was doing the initial structuring, you saw a lot of the feel and things that were common in Linux come in. Initially, it was, you know, there was a different make system

that was more common to Linux, and they actually switched that out a couple years ago for CMake because that's what the embedded developers wanted. We initially started on a Garrett system, and we moved over to GitHub because, again because that's what the embedded developers wanted. We initially started on a Garrett system and we moved over to GitHub because again, that's what the developers wanted. So the code base is, you know, very much shaped by the developers for the developers and you know, the discussions and the resolution of frictions happens at the technical steering committee level.

So hopefully the best idea works and they've, you. And they've gone in, done something, and then come back a couple of years later and said, no, we should do it differently. And that happens too. So it is an active code base and it's not like it's written once and it's that way forever. People are looking for ways of improving things. Although some parts are now very, very stable and haven't been touched at all, just like in Linux. So it's probably more familiar to people from a Linux background than somebody who's used to traditional, very old RTOS and the embedded space.

We're getting some of our members coming in from the very deep embedded space, and they're looking at trying to keep up with all the new technologies, like the Bluetooth stack and so forth, and wanting to use that to take advantage. And working with their traditional code bases, it's very, very painful. And so they start looking at Zephyr and see that it's an easy transition to just add the bits that they want, because a lot of the other stuff is there now at a level of quality they're comfortable with working on. I'm glad you brought up BLE. the other stuff is there now at a level of quality they're comfortable worth working on. I'm glad you brought up BLE because I have a BLE project that's starting and it's on a TI board and I'm relatively familiar with the TI operating system with BLE, but not super familiar. Should I ask my clients to pay me to evaluate Zephyr for use on the TI board?

I mean, I see that it's supported, but... Well, Texas Instruments was one of our members for the project. And so, you know, they're demonstrating their commitment to be there with Zephyr by, you know, helping us with membership. And so I'd say that, yeah, I think the question is, what's the long-term cost going to be to their client for the product they're going to use? Well, it's free either way. So the long-term is more about the features they get and the confidence they get from good software. I mean, the operating system is free either way. The operating system.

Sorry, client. No, the software isn't free. I'm definitely charging you. So the question here is, what's the ongoing support cost going to be? Are they likely to deal with vulnerabilities down the road? Is it like completely closed or are their interfaces open? Given it's BLE, there's potential for malicious actors to interact with it in some ways.

So if they're both free, that's fine. Do you have the support? I guess would be the question. Well, I could say, as you said, free as in puppy, what are the long-term costs? And if the long-term costs, you know, if you see support paths and everything else in both options, then they're good there. But if you don't see a support path, then maybe it's worth taking a look at Zephyr and seeing, you know, whether it's close enough.

That's actually the problem, is that with TI, using the TI operating system, that's a pretty clear support path. I mean, they're probably not going to abandon it. Their forums are not necessarily the most active in the way that I want, but they exist. And as long as I'm not coloring outside the lines, what they have will work for me. I know that because that's what they say the chip does. But I'm curious about Zephyr and I want to try it, but I'm having trouble. You know, I have all these questions like, is it certified the way it needs to be certified for TI? Can I get to the lowest power? Do I need the lowest power? Can I do all?

But that means I have to spend days and days reading about Zephyr's BLE implementation. And I just want you to talk me into it. There's no free lunch. Yeah, I'm afraid. Actually, you know, I'd probably reach out to some of the TI folk that are working on the Zephyr project. And I'd ask them. That's fair. But your BLE stack is certified and your Wi-Fi stack is FCC. All of the fund certifications have happened for all these things right um so the ble is the one we've been tracking um the wi-fi one um i think our cost basically our members have been taking it through but we don't we haven't done it as a project with the bluetooth we've done it as a project because we're pretty cutting edge there cool yeah um what's kind of

exciting for us in the sense that the Bluetooth SIG tends to use Zephyr as a proving ground before they release specs. And so we're, you know, we keep pretty current to the tip of it all and all the new technologies coming out, which has been really exciting for the project. Yeah, that is. I mean, I know that the TI operating system doesn't support the BLE 5 mesh networking. And so maybe that's what gets me over to check out Zephyr. That's good. Yeah, on the Slack channel, for instance, we've got a really active Slack channel. It's a free one. And so there's no history. So don't think you can bookmark something. Scrape it, please, if you need it to keep it around.

But we've got a lot of pretty friendly community. And you'll find people from TI and other places on those channels. And you can ask them questions directly if there's issues on specifics. Like I say, if you sit and monitor the Slack for a bit, in the general channel, you'll see that there's a lot of good discussion that's going on. And the discussion that I saw was both about developing Zephyr and using it. Mm-hmm. Yeah. In the general channel, you'll see both. And there's other channels that are more specific to areas. But you will see use and development happening.

There's a user list and a develop mail list, and it's just a matter of perspective where people seem to ask questions. But I think the most active is probably the Slack rather than the mail list. How real-time can a Linux operating system be? How real-time is Zephyr? You've hit a spot that I don't have a good answer for you. I want it quantified still. There's some work I want to do to get it qualified,

to get it specifically precise in a way that I trust. I've seen some estimates done, but I don't believe them. One of the other projects I work on is real-time Linux, so I'm pretty sensitive to this stuff. Okay? Yeah. And as a result, I've seen people obviously claiming that Zephyr is going to be able to have certain cycle times and so forth.

But when I look at the methodology, I'm not quite believing them. So I can't give you a good answer right now because I don't have the data and I have not had, I haven't gotten the data from someone I trust yet. Let's put it that way. All right. I like that answer because when evaluating these things in the past, it's always been very difficult because every operating system vendor has their own slightly different methodology. And so the numbers look comparable. Oh, we can do, you know, this context switch plus blah blah in 50 microseconds or whatever but it turns out it's a slightly different from the other

persons who claim 75 microseconds and maybe that matters maybe it doesn't and it's just this and then i also i'll ask about okay what's the jitter aspects like yes in the face of noise and then how long is it likely to be um sustained how long has it been running for without issues? And I mean, I didn't specify a board or even a family of ships. There was no answer to that question. Well, like I say, I haven't, like, you know, I've seen a couple people starting to tackle that and it's an area that I would, I'm probably, there's some people I want to be talking to this summer and I'll see if we can start to get some more details emerging on that in the fall.

But right now, like there's a lab in Germany, OSADL, or OSADL sometimes it's referred to, and they have a really good real-time lab. And I've been talking to Karsten there about them doing some runs with Zephyr and with Linux and, you know, doing a characterization. And he seemed interested. And so it's just a matter of, you know, finding the right boards to send them and things like that. So I'm hoping, like I say, by next fall, I should have some better data for you. So real-time Linux is a thing,

and embedded Linux is a thing, and Zephyr is a thing. How do I decide which one I want to look at? They all seem overlapping. Well, real-time Linux is a set of patches right now that you apply upon embedded Linux. And most of the work for real-time, the preempt RT patches are going upstream right now. We're hoping that by 5.8, all of them will be upstream.

And that we will have the ability to have real-time just be a config option in the kernel. And it being used embedded is just a comment on which options are being chosen. For Zephyr, when you're looking at embedded Linux, it really doesn't get smaller than 2 meg. I worked with Nicolas Petre when I was at a prior organization before I went to Linux Foundation. And he was busy trying to get it shrunk down as small as possible. And he had to play some pretty intricate tricks to get it down to about 2 meg. And so a usable functional system with Linux, you really don't want to be lower than 2 meg.

Zephyr, like I say, we can do 8K for the bare bones as well. And certainly we can go up to there but 512k becomes reasonable so it's a question of what's your memory footprint and then um with zephyr you're going to be using less of your computing resources so you're going to have a much longer battery life potential and so how long does your battery life need to last um one of zephyr's members is odicon and um they are doing their next hearing aid line on zephyr's members is Odecon, and they are doing their next hearing aid line on Zephyr because of the Bluetooth and because, I think, the battery life. They actually gave a really good talk last August at Open Source Summit North America, and they were explaining how they were using it, Zephyr and their products. And so, you know, that people are going to be using it in hearing aids.

Wow, okay. You want your battery life there, right? You don't want to be having to plug this stuff in. And so they find Zephyr to be a good solution there. So, you know, these wearables, you know, things that have to last a long time. One of the other applications I know that's using Zephyr right now is ear tags on reindeers. Yeah. So they're running Nordics chips, and they've basically created these ear tags for the reindeers up in Scandinavia so that the herders can track the herd.

And the reason it's effective for them to do it is because of the way that the system is set up, where is if a natural predator takes down a reindeer, the herder is reimbursed by the government for it. And so by tracking the deers and the cost of these tags, effectively, they can, you know, recruit more of the insurance money that the government would pay them if predators, you know, prey on it, and they can show the evidence. So that helps a much more natural situation evolve. And from my perspective, I look at it and I'm really impressed because you see these endangered animals with these huge collars on and things like that, and I'd much rather them be wearing ear tags so people could keep an eye on them.

Wait a minute. So hearing aids, ear tags, I'm sensing a theme here. Sorry, I was off writing a reindeer fraud screenplay. Now, there's some really cool things emerging with Zephyr. Like the creativity that's kicking into this community is awesome. Just last night, I was reading up about one of the other products I know about that's using Zephyr is Intilinium has these safety shoes. And with these safety shoes, what they're doing is letting people tap Morse and get haptic feedback in noisy environments. And so they've been wearing warts. And recently, they just came out and announced this.

Wait, wait, wait. I'm sorry. Safety shoes and Morse code. I'm not kidding. So if you're in a very noisy environment, okay, you can't hear an alarm because of you're wearing headphones. And there's a lot of heavy machinery around you. So your shoe is going to tap on you, not you tapping on the shoe. The shoe is going to give you a buzz.

Okay. So there'll be some sort of haptic feedback. And if you needed to communicate back, you can tap, i.e. an acknowledgement or something like that. Got it. Okay. Thank you. Sorry, I just couldn't picture.

And so on Wednesday, sorry, on April 22nd, the interlineum folk basically announced they're creating a smart tag that so that it helps with the social distancing and helps support them with social distancing so that it's detecting are you close enough to another worker or not. So it's using that type of feature to basically tell you if you're getting too close to someone from the social distancing perspective. That makes sense. Yeah. So you're seeing some really creative ways that people are using the technology to help keep other people safe. And one of our other members right now is Adafruit. And they're pivoted in a slightly different direction, but they're busy making face shields in New York City and trying to help things out there. So they're doing awesome work as well.

So we're super proud of that member and what they're trying to do to help. So, you know, with and without the Zephyr running, there's a lot of creativity in the community. And that's kind of exciting to see where it goes. As with any embedded project, you know, you don't know if people are using it or not. And so finding these little breadcrumbs of where people are using it sort of helps motivate, keep you going sometimes. That is so true.

Yeah. Your title is Senior Director of Strategic Programs at the Linux Foundation. That's a cool title. What does it mean? Well, it means there are certain projects and programs that we're working on here at the Linux Foundation that we consider strategic to the overall industry and how we can make things better. And so I have a portfolio of projects that I help. And so it's different roles in different projects from the director of the project to, you know, someone to help about facility communications. It's, you know, there's a lot of different things I work with. But, you know, I work with real time Linux, and I work with Zephyr. And then the other project I'm working with right now that's kind of exciting

is something called ELISA, which is Enabling Linux in Safety-Critical Applications. And again, the safety theme is kicking in here. But because of standards like 61508, they were written with a very specific mindset and a specific model of development in mind. And what we're trying to do is figure out what is it going to take to make people feel confident that Linux is safe and

you know what sorts of you know what should they be looking at in their analysis before they deploy a product that could have a safety critical application things you know our Tesla's out there right now are running Linux I don't believe they've gone through 61508 certifications. How can we make sure that these, as we get towards the autonomous driving, they need the processing power of something like Linux. How can we make sure these sorts of things are safe? How can we figure out what's important there and in other applications? And what level of safety? So it's a big open question. And we've gained people like from all different spaces coming together to try to help figure this

out. We've got a medical devices work group right now, where we're checking and looking at where Linux is being used in medical devices. And we're trying to figure out how can we analyze the interfaces around it. And we were looking at, there's an open source project called OpenAPS, which is Open Artificial Pancreas System. That's a project that's being done by type 1 diabetes for themselves. It's a hobbyist project, but what they've done is improve the quality of their life by creating a closed loop between their insulin pump and glucose monitors and stopping the alarm fatigue.

And so we're sort of looking at how Linux is being used there. It's not a key component. It's not the algorithm, but the algorithm is running on Linux. And so, you know, what's really important and trying to do some analysis there. Recently, though, we decided to apply our analysis work onto the open source ventilator options that are out there. And so we've been looking at the various open source ventilators there and starting to look at trying to do some safety analysis with some of them. And, you know, there's one we're sort of, I think, going to standardize on. trying to do some safety analysis with some of them.

And there's one we're sort of, I think, going to standardize on. Like I say, we're finishing our surveys off right now. That has a good part of it there, and it looks like it's in reasonable shape, but there's going to be communication happening to a screen somewhere for displaying the results. And that's where it'll end up fitting in, we think. And so we're actively will end up fitting in, we think. And so we're actively monitoring what's going on. And that group has got people from UL, Underwriters Laboratory, as well as some of the tubes over in Europe, you know, participating in trying to figure out how can we figure out whether

Linux is safe or not in these sorts of situations? And how can we mitigate the risks? Which is mostly what safety analysis is, is, you, is understanding your system and then mitigating risk. So we're sort of going after it from that perspective in that group. There's another group, obviously, for automotive, and we're looking at trying to form one working in the robotics and industrial too. So there's these different use case domains working,

and then we're looking at, okay, how do you break down the process like Linux and relate it to the standards processes? You know, what's the evidence that's there, what's not there, as well as, you know, how do you start looking at key parts of Linux where they're important? And, you know, there's one researcher who's doing formal verification on parts of the scheduling part of the kernel, for instance. And, you know, are there other places we should be doing formal verification? Things like that to make Linux safe.

So, you know, on one hand, you have something like Zephyr, which is very small, and you can go through the traditional processes. And then on the other hand, you have Linux, which has a whole slew of interesting domains to interact from and some hard problems to solve. You know, on the other hand, it's, you know, everyone seems to agree that it's got pretty good quality at this point. And so there's a lot of things in place to keep it with good quality. And so it's just making those visible in a way that the safety people understand.

That's a tough thing to do, both from the safety, getting people to understand, but also it's a lot of work to get anything safety certified. I mean, really amazing amount of work. Yeah. But, you know, if people's lives are going to be depending on it and there as, you know, trying to do as much of the work for everyone as we can in this effort. So there's going to be more and more of these types of safety-critical implications happening. You know, if I ever needed an implant, I'd want to be able to know I could, you know, there's an open source view of the code somewhere, and I can understand what's going on sometimes. These are, and you know, batteries to last a long time too. So these are all sort of places that we want to see, you know, safety

brought into and be accessible to the applications, because there's a lot of cool ideas out there, but we want to make sure that they can be used in a safe way. And the neat thing about Zephyr is not only are you building the thing and certifying the thing, the Zephyr part of it, you are showing everybody else how to do it, which is definitely important too. We're showing people how we're doing it for Zephyr, which has a certain set of parameters around it. And we're sort of working our way from a very small subset of the functionality in Zephyr up. On the Linux side, we're basically looking at different techniques to accomplish the same types of goals. And it's a really interesting spectrum between them. The other project I look at is that Zan, I sit in on their safety team there.

And I'm trying to make sure we cross-pollinate between all three projects. What are the best practices to use? And so in this way, these things are all strategic. And that's why I'm getting involved in them. Another area that's strategic is, you know, how do we make sure that we can have transparency in the supply chain? So do you know what's running on your device? how do we make sure that we can have transparency in the supply chain? So do you know what's running on your device?

We saw things several years ago about Harplead and that's sort of on the service base, but when you start looking in the embedded side, how do you know what's on your device? Well, unless someone gives you an accurate software bill of materials, you don't. And if you're running a BLE stack, someone could potentially hack it. If you don't know if your version has been updated, what stacks on it, you may or may not be susceptible. So we need to really improve the transparency of what we're sharing for software information so that we can use these things safely. Embedded systems are going to have to grow up soon. There'll always be a place for people to be creative.

Come on. Oh, yes. Oh, definitely. It's just thinking about all of the additional stuff that's important. Along those lines, if I got excited and wanted to try Zephyr out, what's the quickest route? Is there a specific board that's great? Or is there a tutorial that says, hey, go through these steps and you'll be up and running in hours, days?

There's a bunch of quick starter guides for various boards out there to get you to Blinky effectively. One of the boards that we really like and is well supported is Phytec actually created a board for the project. Oh, great. So you know that that one. And they've actually had extension boards on it and things like that, so adding different peripherals. But that one's very well-supported. And it's got a little LED on it, and it us um basically the bluetooth and so you can sort of

signal to each other in a set of them things like that so depending what functionalities you want to play with um the fitech board is a really cool one and there's other ones that are out there from some of our other members like nxp and so and nordic and so forth that are like the nordic thingies are all running zephyr quite well, as is some of the NXP ones. Okay. So there's some good options that are available pretty cheap, too. And if you look at the list, there's a good chance that most of the people listening to the show might have one of these boards.

Yep, I would say so. I mean, the DISCOs are all there, and the EXPLANES, and the NUCLEOSs. Yeah, this list is very long. ESP32s, that might be fun. There you go. No, like I say, a lot of people have contributed a lot of boards, and this is how we've got so many commits, right? As people are putting in what they're needing and using on a day-to-day so that it's kept up to date with everything else with our testing infrastructure.

I mean, if you're going to have to write a spy driver, you might as well write a spy driver other people can use. Exactly. That's the whole benefit of open source. Kate, thank you so much for speaking with us. Do you have any thoughts you'd like to leave us with? Sure. Just, you know, in this crazy world we're in right now, stay safe and be creative with the extra time you've got at home now. Our guest has been Kate Stewart, Senior Director of Strategic Programs at the Linux Foundation.

Thank you, Kate. Thank you. Thank you to Christopher for producing and co-hosting, and thank you for listening. You can always contact us at show at embedded.fm or hit the contact link on embedded.fm. And now some thoughts to leave you with. Zephyr, as you may know, is the name of the west wind, the bringer of spring and summer breezes. But did you know that Boreas is the north wind, bringer of cold? Notis is the south wind, the hot wind, and Eurus is the east, bringer of rain. Happy spring, Northern Hemisphere. Embedded is an independently produced radio show

that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.