Embedded - 175: How Hard Could It Be?

Episode Date: November 9, 2016

Jean Labrosse of Micrium (@Micrium) spoke with us about writing a real time operating system (uC/OS), building a business, and caring about code quality. Take a look at the uC/OS operating systems��(available for free to makers) and Jean's excellent and free RTOS books (it was the Kinetis one that talks about the medical process). Also, check out the uCProbe which integrates with your debugger to replace some logic analyzer and oscilloscope features.  Jean's blog about detecting stack overflows: part 1 and part 2. Brother to Brother by Gino Vanelli

Welcome to Embedded. I'm Alicia White, alongside Christopher White. This week, we will be speaking with Jean Labrosse, author of the Micro-COS Real-Time Operating System. Hi, Jean. Thanks for joining us today. Hello. How are you doing? Could you tell us about yourself? Well, I have a master's degree in electrical engineering and I fell in love with microprocessors just about the time they came out, circa Intel 8080 and 68000 from Motorola. I've been doing embedded systems for almost 40 years and I founded Micrium in 1999. And Micrium was actually

acquired recently by a company called Silicon Labs in Austin, Texas. I have some questions about that, but I have a lot more questions about the real-time operating system and learning how to deal with it and Micrium and micro-COS. But before we do that, we have strange lightning round questions where we ask you questions and want short answers. Sort of a getting to know you thing. You bet. Christopher, you're first. Okay, 8 or 32 bet.

32. What language should be taught first in university courses? English. Sorry, what programming language? Oh, well, I mean, you need English to program any programming language, but I would probably say C. Little or big Indian? For communications, I would say big Indian, but little Indian is the norm. That's the correct answer.

Least favorite electrical component? Electrical component, least favorable. These are, I mean, they're short, but they're tough. Ind're tough inductor oh i agree with that uh favorite fictional robot c3po oh see this is great this is gonna be fine uh favorite processor of all time um actually i like the 68hc 11 processor because it was simpler uh but nowadays probably the cortex m is my favorite not because of of the architecture of the instruction set mostly because of its built-in debugging capabilities and the fact that it's everywhere yeah fortunately or unfortunately it's all everywhere It's all we work on now, Cortex-M.

What science fiction technology or concept do you think will become real in our lifetimes? Well, unfortunately, self-driving cars is going to happen very quickly. So, that was my favorite technology concept that i always thought would be really cool to see we saw that in some movies in the past and i was i was always quite impressed with that so self-driving cars and i agree we're definitely going to see that one and soon i think yeah so from andre he asks what is your favorite Gino Vannelli song? Okay. Gino Vannelli.

I don't know how many people know Gino Vannelli, but he's a very awesome musician. He's created over 200 songs. And if I had to select one, I'd say a brother to brother. All right. I think that will do unless you have any more, Christopher. Oh, here's one. Is a cell phone an embedded system? Years ago, I'd say it was an embedded system, but nowadays I would think it's more of a general-purpose computer.

Fair enough. What is a work day like for you? Oh, that's kind of complicated. It's split between two things, writing and answering emails, getting into meetings, and then I work at night, and that's when I actually do the real work. What is the real work? Well, I mean, you know, do things that we talked about during the meetings and answering the emails of the questions people ask and things that they want to have done. So if I have some to-do things to do, and most often I do them at night or on the weekends.

We hear this a lot where engineers will say, I spend most of my days in meetings and talking to people and usually it's not people whose title is engineer it's people whose title used to be engineer and now their title is CEO or chief architect and now they spend their days in meetings and they finally get to work after work yep that'd be the case why isn't the hard part of dealing with people the work? I mean, the coding part's the fun part. So as you progress in engineering, as you progress and you get promotions and you, in my case, start a business, you get to do less of that and more of the management, the project planning and none of the fun stuff anymore. Others do that. It's so sad when you have to give away your toys and other people get to write your software for you. Yeah, yeah. It's a difficult thing to get to give away your toys and other people get to write your software for you. Yeah, yeah. It's a difficult thing to let go of, that's for sure.

So you started writing this operating system in the 90s, is that right? That's correct. And did you start writing it as part of an article for Embedded Systems Program, or did you have it and then write the article? Well, actually, it all started where I wrote the operating system because I was using an operating system and I found a bug in it. At least my team found a bug in it. And we contacted the manufacturer. Long story short, we were out of our maintenance period, and they didn't want to help us fix the bug that was in their operating system. So I decided, well, how hard could it be to write an operating system?

And I decided, let's go ahead and do it. And that was a time where lots of people decided, hey, let's just write our own operating system. This is not working for us. It used to be super common, but now it's pretty rare. How did the industry finally figure out that that was not a good use of time? Because, I mean, they pretty much all do the same thing, and it's not a good use of your time to do that i mean as as embedded systems are getting more and more complex i think you want to spend more time dealing with the value

added to your to your product instead of spending your time with infrastructure software which is you know real-time kernel and protocol stacks and so on. This is an argument I have constantly with clients. I bet. Your product is this. Why are you making infrastructure? Well, I mean, case in point, how many times people have rewritten a UART driver? I mean, something as simple as that. Thank you.

Yes. Right? So, it's exactly the same thing i mean although a real-time marketing system a little bit trickier i mean when i when i first started doing it i thought well how hard could it be you know it's context switching of registers no big deal i could do that i'm a good programmer and then after you know once i got to version 1.11 or 1.12 i figured out that was not that trivial because there's some uh race conditions you got to really really be careful uh with so uh it's not that it's not that easy so and the the other thing that's kind of interesting is um it really takes

between seven and ten years to get a product mature, such as a real-time kernel. Well, mature and accepted in the industry. So anything that comes out new nowadays, it's got some time before it gets to be matured and accepted. The baking time when all the bugs are found, or when 50% of the bugs are found, the easy 50%. Well, I'd like to think that most of the bugs have been found by now. And it's been 10 plus years. And those articles in Embedded System Programming, you wrote them after you wrote the operating system,

but they were only a few pages. I mean, it was three parts and five pages each? Yeah, it was something like that. What happened is I actually wrote the kernel, and then I contacted the Embedded Systems Programming magazine to have it published. And at the time when I wrote the article, it was actually 70 pages.

So Embedded Systems Programming says, well, we can't publish a book like that. We can't publish that. So they actually broke it down into three parts of five or six pages each, May, June 1992 me and said, hey, by the way, you know, those 70 pages that you've sent us a while back to get published in a magazine, would you be interested in writing a book? And I said, sure, why not? And I said, well, if I'm going to write a book, I might as well make it a real book, not a booklet. 70 pages to me was kind of tiny. So I decided to go ahead and write the book.

And I think it was early March, April, May or something like that, 1993, that the book actually got published. And it was longer than 70 pages, but you went over all of the how an RTOS should work and you used yours as an example. Pretty much, yeah. I explained the concepts, you know, what's a semaphore, what's a mailbox and how this context switch works. And, you know, you had, at least at the time, there were some books out there explaining about the operating systems in general terms, the Tannenbaum book, for example. And there was really no actual code that you could get that you could actually try it on the processor.

And the first processor I tried it on was a 68HC11 processor. So very easy to use on an 8-bit processor like that. And in fact, it's kind of a misconception that people think that real-time kernels are only used on 32-bit processors. I mean, they kind of started on 8-bits, actually. Yeah, these kids and their 32-bit processors. Exactly, exactly. So, you were working probably a full-time job. You were making your own operating system, and you were writing a book. Yep. After you finished the book, how many years did it take before you were willing

to even consider writing another? I went from 1992 to 1998. So, the Microsoft S2, it actually took a while before the book got sold. So, it took something like, you know, four or five years before we saw any revenue because they were giving me royalties. I mean, I say us, my wife and I, because she gets the royalties. That was meant as a joke, by the way. Sorry. So anyway, so it took a while for the royalties to start kicking in. In fact, what they did is they gave me an advance and then the advance took a long time

before it got depleted. So anyway, I really never did it for the money. I really did it for the righteousness of things. I wanted to teach others, and especially when I found the bug in the operating system that that we were using commercially it was kind of like upsetting to to see that so i decided well uh i'm gonna write uh an operating system and i'm gonna provide all the source code to it instead

of you know a bunch of uh binary uh files so that was the that was the difference it really really was not for the money it was really more for the helping the industry. And then I started doing lectures at conferences in 1993. Same purpose there. It was not, I was not looking for fame or glory or anything like that. It was just like, what can I do to help so that others don't get into that situation. And how did you go from that to making it so that there was Micrium and you had a way to support and sell the operating system in addition to teaching about it?

Well, that was also kind of an accident. In the 1995-1996 timeframe timeframe a gentleman by the name of Mike Kaskowitz who was working at a company called VLSI technologies contacted me and he basically says hey look I'm looking to use your real-time kernel micro-cos for a processor that we're actually saying I'm sorry for a processor we're actually selling. And then the processor is an ARM6. And I said, what the heck is an ARM6?

Oh, yeah, very, very popular. It's going to be popular in the years to come. And I'm going like, I've never heard about this processor. I heard about the 68000 that was popular. You know, X18 or x x86 processors were were quite popular but arm 6 who the heck would ever want to use those you know nobody heard about him so anyway so he he actually used the the kernel cc so he basically said oh yeah what i'd like to do is license your software so that could actually provide that to my customers as an example of multitasking

this 32-bit processor. Okay, fine. So I said, well, you know, what's licensing? I had no idea what that was. So I say, yeah, we write a contract and you give me the rights to actually provide that to my customers. And I write you a check anywhere between $5,000 and $10,000. And I said, well, $10,000 sounds great. So, let's go ahead and do it. So, a few weeks later,

I got the contract and I got the check in the mail and it was downhill from there. I mean, once I had that, I opened up a little BBS at home and I put my contact information in the book. Actually, it was in the book all along in 1992, 1993. So people started contacting me. That's how he got a hold of me. And oddly enough, I mean, other companies contacted me well afterwards. HP, for example, contacted me. I don't know, Kodak, a whole bunch of companies afterwards. And I said, hey, I know about this licensing concept. And if you want to use it, this is what we'll do. Before that, it hadn't been licensed.

I mean, it had been out there and free for people to use, although maybe you would have gotten annoyed if people were using and selling it. Right. And that was the concept. I mean, the concept was all along was always if you're going to use it in a commercial application, you know, I think you should do the right thing and, you know, share the wealth. But if you're going to use it for non-commercial use for a hobbyist, then absolutely go ahead and use it. I'd be more than happy that you learn about real-time kernels as well as use it for your own projects at home. But initially, I mean, I was kind of this naive engineer, did the right thing, you know, write software, very clean code. And that was one of the things I wanted to promote also. I've always been a nag about clean code, you know, very well organized, lining up equal signs, lining up comments, comment signs, you know, slash star and star slash or vice

versa. And, you know, I've always been very, very picky about putting proper comments. It's kind of selfish in a way that I wanted to have that way of writing code out there as well as here's an operating system you could play with and you could learn about. So when you were developing this at first and exploring how to distribute it and how you wanted people to use it, this was about the same time that Linux was just beginning to enter people's consciousness. And the same thing, GNU as a license. Was that kind of on your radar? Were you thinking about those things?

Or was this just kind of everything was so new that it wasn't really clear? It was new, and anything you learned about, I mean, the internet, I mean, the websites and stuff, that's not that old. I mean, it's not something that was in existence in the 95, 96 timeframe and stuff. So all the information you read about was pretty much magazines. So, you know, and I wasn't really keen on Linux at the time, or I really didn't get involved in any of that. So to me, it was like completely different things.

So, I mean, in a way, you could say that I kind of started this whole open source on the embedded side of things, not Linux was a little bit higher end systems. I mean, you look today, you have free RTOS that's out there. You have lightweight IP for protocol stack. So there's a whole bunch of stuff that came out afterwards. And oh, by the way, when I came out with the article, there's a lot of real-time kernel or real-time operating system, however you want to call them, companies that were quite upset about the article. time kernel as an article in the magazine how how dare you expose this secret of ours that we've

been you know been been charging a fairly a fortune for so uh they were it was kind of interesting in fact i mean uh oddly enough uh the reason you know like i said earlier the reason i actually came out with the real-time kernels was because um we found bugs in in the operating system. And it all started when I started with one operating system that was kind of cheap, but we could never get it to work. So we got this real-time kernel from a company called MicroDigital in California. And we were contacted then by phone. At the time, there was no email or internet. We were contacting them by phone and saying, look, we can't get this to work.

Could you please tell us what we need to do and stuff? And anyway, after spending like a month or so, we couldn't get it to work. So then we switched to Vertex, and Vertex was like $5,000 at the time, so quite expensive for a company way back in the 89 time frame 88 time frame so we decided let's go ahead and and spend the money spend the five thousand dollars and you know 90 days later your maintenance expired and that's about the time we actually found found the bug and they told us well sorry we can't we can't help you you your maintenance expired and unless you pay us back again, we won't fix the bug.

That's extortion. Yeah, so part of that whole story is more of the righteousness of things, and this was not fair. I didn't think it was fair. Look, and specifically the second operating system was Ready Systems Vertex, which at the time was very popular and was known to be bug-free. In fact, for years, they were giving a Volkswagen bug if somebody found a bug in their software. So I didn't claim the bug. I didn't really like that car anyway.

But, you know, so I didn't go any further with that. But anyway, so we found the bug, we pinpointed the bug, and it took him a whole year by the time, from the time we found it to the time they fixed it. So that's when I decided to actually write that. So... A reminder to everyone about how difficult things were before the internet, before communication was easy, before people, you know, if you've had a security bug now and talk to Microsoft and they said, pay me, the outrage would be through the roof. Exactly. Exactly. Well, it's funny because I think that the complexity is just as complex today as it was back then. Oddly enough.

Today, you know, you have high-end processors. You get better tools. You get debuggers. You get all kinds of profiling tools. You get MISRA. You get static analysis. So you get a lot of these tools that you didn't have back then. And, you know, all these tools that you have, they're great, but the complexity of the software, the expectation from customers

has gone up tremendously. And back then, we used to have, you know, eight kilobytes of code space, 256, 512 bytes of RAM, no debuggers, you can't, there's no flash, you got to erase EPROMs. Just the tools were just archaic as could be. But the applications were super simple. So applications were simple, it was difficult to debug. You actually had to stare at your code a lot more than people do nowadays people just go ahead and write code and and oh we'll figure that out you know when when we actually test it or in the debug stage back then you know you just you can't wait

30 minutes for the e-prom to erase you have to have a whole stack of EEPROMs in a tray being erased while you're actually looking at your code. So there was more staring down as more design back then, I thought, than there is today. Well, yes, because you had so much time between when you could try out a solution. And so now we've talked about this on the show, sometimes you end up sort of banging on your keyboard like a monkey trying all the options because you can.

But if all of the options took 30 minutes each, you would have to think it through. And that often makes for better software. It's just to stop just trying all the options and to think, what am I doing and how do I get from here to there? Yeah, and there's that old adage, I guess it's a word, it's an adage, don't just sit there, do something. I'd rather it be just don't do something, sit there. Yeah.

You know, you got to think about it a little bit longer and before you you you do things and and and it's acceptable nowadays that people actually write code throw it away and then write write the right code afterwards you know i've heard some people say you know oh it's it's it's okay to to write throw away code uh in the beginning because you know you'll do that regardless i I don't believe that. Okay, we've thrown around a lot of RTOS terms without defining any of them. Maybe we should come to the point where we actually define some stuff. Sure. Starting with RTOS.

All right, what is it, right? Yes. An RTOS is also what we call a real-time kernel. An RTOS means real-time operating system, and a real-time kernel is really not an operating system per se. It's the multitasking side of an operating system. So typically, an RTOS is software that manages the time of a CPU, and each of the application that each, so the application is divided in sub components called task, and each task is responsible for a portion of the application that you're doing. Each task is given a priority based on the importance of the task. And then each task typically is an infinite loop that waits for an event to occur. When the event occurs, the RTOS or the kernel will schedule the task to execute and it will

actually run that task on the single CPU that you have. Let's say it's assuming a single CPU. I have sometimes used the analogy of a nursery school teacher, that if you are running on a CPU and you're going bare metal, it's sort of like you're a kid who is doing one thing at a time, and that's okay. But as soon as you have six kids, you start needing a teacher to organize who gets to go to the bathroom when and who gets to play with the red crown and who gets to pet the gerbil. And so you have to have some organization and the RTOS

is that nursery school teacher that provides the organization and says who gets to do what, who has access to what tools and what features. Is that a terrible analogy or is that just funny? It's both. Rating your own jokes? No, no, no. No, it's not a terrible analogy. It's a good one.

I mean, the whole idea is that you have a single CPU. You have, it's a, an RTOS is a way to orchestrate things to be done in a sequence of based on priorities. You know, like not everything is important in your day-to-day work. And it's the same thing as a manager. You basically have to figure out what's more important. What do you have to do first? What do you have to do first? What do you have to do next? And the RTOS just makes that easy. Unfortunately, when we work, when we take phone calls,

when we are in meetings and stuff, we don't have this piece of software that kind of brings us back to say, oh, by the way, is this really important? Do we need to do this right now? And that's what the RTOS does. It says, okay, an event occurs, something happens. The scheduler says, the real-time operating system, is this the most important thing that we need to do?

If the answer is yes, then we do it. If not, then we keep on continuing what we were doing before. And it so happens that if there's nothing else left to do, then the RTOS runs what's called the idle task. And that's a perfect place where we could actually put the processor to sleep because when the CPU has nothing else to do task-wise, we could actually go in low power mode on most of the modern CPUs and thus reduce power consumption and save on batteries.

So what distinguishes an RTOS from just a simple scheduler that you base on a timer? Well, the basic scheduler that you would base on timer, would typically be implementing a state machine where somehow events would set flags and then you would execute those events by looking at the flags, but it'd typically be implemented as a polling loop. So you'd actually check to see if any of these flags were set and if the flag was set, you actually execute the code for that. So it's not as orchestrated as if you had an RTOS because it's all built in. There's one more thing that you need to have for an RTOS and that is each task has to have its own stack and thus each task thinks it has the CPU all to itself because whenever a task execute and something else of higher importance occurs,

then the CPU registers of that tasks are saved on the task stack. And then the new CPU registers of the new task, the more important task, gets loaded into the CPU and then that task executes. So all that orchestration is done all behind the scenes

and it's done as efficiently as we possibly can. People designing real-time operating systems spend their time figuring out how to save just about every clock cycle to do what needs to be done and as quickly as possible. So that most of the time is not spent in the operating system. It's spent doing the work that the operating system is supposed to do. Okay, so that's how the scheduler works, and it switches between tasks and then goes to idle. Most operating systems today operate that way. What's the real-time part of it?

Latency. Yeah, latency, determinism, preemptive scheduling, a preemptive scheduler is, and that's the case for most real-time kernels or real-time operating systems. Preemptive means that whenever there's a more important thing that has to happen, then it will actually preempt or stop what you were doing before and switch over to the more important tasks. So preemptive scheduling is very, very critical for an RTOS. In fact, if you remember the days of Windows 3.1, Windows 3.1 was not a real-time scheduler. So, for example, you had an interrupt. The interrupt service routine made it a more important task ready to run. But what happened was the interrupt returned to the interrupted task.

The current task was ready to relinquish the CPU, was the operating system invoked and gave the control of the CPU to the high-priority task that was just made ready to run by the ISR. So there could have been a long delay. In fact, I remember that Windows 3.1 used to hang a lot if the task that you were running was sent on a tailspin. So not the most efficient thing. I don't know of any real-time operating system nowadays

that don't have preemptive scheduling. I mean, it makes a lot of sense that priority inversion is the worst part about non-preemptive systems. Is that right? Well, the priority inversion occurs with preemptive scheduling just the same. And what happens there is that you are a low priority task. You obtain a resource that is shared by a high priority task. So you want the resource.

You typically use a mechanism called the semaphore. So you grab the semaphore, which is kind of like a flag that you just grab and you say, I'm going to own this resource until I release the flag. So you grab the resource, you access, you grab the flag, you access the resource. And during that time, let's say a more important task occurs, the event that a more important task was waiting for just occurred.

And it so happens that the RTOS will switch to that task, of course, because it's the most important one. The RTOS will actually, the more important task that task, of course, because it's the most important one. The RTOS will actually, the more important task that says, oh, I would like to have access to that resource, but when you can't have access to that resource, it's already owned by the low priority task. So that high priority task gets to be suspended, waiting on that resource to be relinquished by the lower priority task. So what happens there is that you can have medium priority tasks that occurs in between

because medium priorities are more important than low priority ones, of course. It's a preemptive scheduler. It does its job. So you're furthering, delaying the access to the resource by the high priority task, the more medium priority tasks that you have. So anyway, so long story short, that's the priority inversion. A low priority task is acting as if it was more important than a more important task. So there are mechanisms in most RTOSs nowadays.

It's called mutual exclusion semaphores. And those mutual exclusion semaphores, what they do is they actually raise the priority of that low priority task to be the same as the priority of the high priority task so that it could hurry up and relinquish the resource as quickly as possible. And then once it relinquishes the resource, the priority of that task gets lowered back to where it was. And now the high priority task could have access to that resource and if you're out there like trying to follow along without a whiteboard and maybe driving at the same time imagine that your debug task has a hold of your uart and some other tasks some communication important task comes along and needs the uart like really needs it well it has to wait for the stupid debugging to finish.

And then the debugging gets interrupted by the USB that comes in. Now your really high-priority command task has to wait for the stupid USB thing and the debug thing unless you have this mutex which says the debug has to finish so that the high-priority command handling can happen. Sounds like you've written a lot of software It has to finish so that the high-priority command handling can happen. Sounds like you've written a lot of software because you actually put explicatives to tasks and stuff. Like, yeah, anybody that's written a lot of code will call things stupid this and stupid that.

So, yeah, you've been around the bush. So, when isn't it a good time to use an RTOS? I know that you write about RTOSs and you care about them and you sell them. You're in the pocket of big RTOSs. You are totally in the pocket of big RTOSs. When should I not consider that? When should I say, no, you know, I have a thing. I don't need to worry about it quite yet.

Well, if you have a very, very simple application and you could do it with a super loop and you don't have complex communication protocols, you don't have TCP IP, you don't have USB stacks, you don't have file systems, you don't have a lot of these things, your application is very, very simple. It fits within a few kilobytes of code space. And then the responsiveness of a super loop or a while one loop would be sufficient to handle your product.

Then I think you don't need one. It starts to get complicated as soon as you start adding protocol stacks, TCP, IP, like I said, files. And oh, my God, a GUI. If you want to put a GUI on there and you're single threaded, just imagine you're painting a screen with hundreds of pixels and then your control loop will have to suffer because by the time you paint, for example, 320, what's the resolution, Quarter VGA 320 by? 240, I think.

240. That's like 76 kilobytes worth of pixels that you would have to paint. Imagine updating a display with that many pixels. If you're also doing a control loop in your super loop, there's a good likelihood you'll miss your deadlines. Yeah. So GUIs and all of the communication protocols. BLE was one I kept thinking, oh no, it's simple enough, but no, you really do need an

operating system for that, sadly. Yeah, I would say anytime you start adding complex protocols, especially those protocols that you purchase off the shelf or you get from somewhere, because you don't have any control over how they actually operate. So a real-time kernel will actually say, it doesn't matter if I don't know how it operates, I'm going to give it a lower priority task than my control loop.

And when my control loop needs to run, it will always be guaranteed to run irrespective of what my protocol stacks are doing. So clearly in that case, that would make a lot of sense. So MicroCOS runs on many different processors. Name it, it runs on it. And having ported software from one processor to another a few times yep that's not always the most easy or fun thing to do

when did you decide to commit to yes i'm going to run on everything well actually the porting process is the easy thing the hard thing is understanding the new cp architecture understanding the tools getting the eval board getting to know the debug environment and all that so i mean that's that's the hard part once you understand all that porting it is really trivial because all you need to do is you need to set up a timer interrupt for what's called time delays and timeouts. So the tick rate, for example, that's what it's called. And then the other thing is you need to figure out how do you actually save all the CPU registers on one stack and then restores them from another stack. And that's pretty much the extent of a port. So, you know, the writing

of the port is the easy thing. So, you know, how do you decide whether you need to do it or not? In my case, I mean, if I had a customer that was willing to send me a check, like I had mentioned before, then I would certainly be motivated to do the port and provide that work. But over the years, I mean, different customers, and then since we started Micrium, over the years, we have all kinds of customers asking us to port on power PCs and ARM Cortex-M, Cortex-A, Cortex-R, Renaissance RX. I mean, name a processor. So, of course, we're motivated by doing it from a commercial point of view. What about the compilers?

I mean, you know, we should have had that as a lightning round question. What's your least favorite compiler? You don't have to answer. Well, anything that's non-reentrant. And I don't think anybody has a non-reentrant compiler anymore. But there's not that many choices nowadays, right? I mean, there's GNU, there's IR, there's Kyle. And what else?

What is non-reentrant for compilers? I know what it would be for a task. Yeah, non-reentrant means that if you're starting, well, and that's the thing, it's typically used for multitasking. That's really an issue with multitasking. So non-reentrant really means that if I'm starting, well, for example, strtok, the standard library function

where you're actually parsing an ASCII string for delimiters. So what happens every time you invoke that function, it stores inside of a global variable that it keeps the current location of where it's actually found a token, the last token. That replaces that location with a null pointer, and then it moves to the next location. So next time you re-invoke string TOK, then it actually parses the remaining of that string. So just imagine if you were to call that function by two separate tasks that could actually execute at any given point, the task you know is expecting to have that pointing to one token and then the other task is pointing to a different location a different string so that's the non-re-entrancy of things so that makes sense that was sort of what i was expecting

and so when you say a compiler is non-re-ent, you mean the library that it's built? Yes. The C library. Okay. That makes more sense. Yeah. There were some compilers years ago. I can't recall exactly which ones they were, but there were some compilers that were actually designed to be non-reentrant, not just the libraries. They had to make some assumptions about things.

Again, I just don't recall who they were. No, I can see how you could optimize things so much that having reentrant... Yes, all right, I get it. Especially for really small devices or something. Yeah, exactly. So do you support all the compilers? GNU, IAR, Kyle? Yep.

Clang. Clang? What's that? you support all the compilers? GNU, IAR, Kyle? Yep. Yep. Clang. Clang? What's that? That's the new hot open source compiler. It's sort of like GCC. It's kind of moving as a replacement for GCC in a lot of cases. Wow, never heard of it.

You will someday. Nobody's ever asked. Yeah, probably. Probably. It's ever asked. Yeah, probably, yeah. Probably. It's very nice. Yeah, it is actually pretty nice. And that's for ARM platforms?

Yep. And x86 and a bunch of other stuff. Cool. So one of the ways that micro-COS has changed over the years is you got it certifiable for FAA and FDA use. Right. What was that like? Actually, it was quite easy. I didn't have to do anything.

We actually, some gentleman by the name of Scott Nowell at Validated Software at one point was having or had a customer that needed to have certification for an operating system. And he magically found my book and he says, well, I'm going to actually use this because I get all the source code. It's very clean, very well written, well organized. I could actually get this through certification. So they actually did kind of like a reversed engineering, which is allowed by the FAA to do. You take a piece of code, then you go backwards and create requirements,

docs and stuff. And he was able to create the artifacts for an avionics application where you could actually prove that the software is tested 100%. So they did test suites. So they tested 100% of the code. Everything was verified, all the different paths of the code.

And actually, they only had to change some 20 lines or so for it to be certified for this avionics application. Yep. I would be extremely gratified to have somebody else do that and say, yeah, yeah, you did a good job. No, no, it's fantastic because, I mean, it gives you a lot of credibility. You know, look, I mean, if it's certified for avionics use and it was certified for do178b level a which is the most stringent certification you could get so yeah it was extremely gratifying and like i said they only have to change 20 or so lines of code which was very very cool and so now is it uh mizra compatible so the miss so miss what then that's another controversy that I have. I mean, MISRA is meant to be for the purpose of writing clean code, you know, to make sure that you don't do any stupid things with stupid tasks, if we could go back to that.

But the way I look at MISRA is it helps developers to write clean code. That's absolutely true. But if you have your code already certified for avionics or medical applications, you know, the MISRA, even if you violate some MISRA-C rules, your code has been validated in very stringent environments. So, you know, if you don't follow all the MISRA-C rules, then, oh, well, you know, I got a better way to demonstrate that the code is robust. And that was the whole purpose of MISRA is to produce robust code. So, to answer your question, yes, we have some MISRA-C documents that explains which of the rules that we follow and which ones we don't because we don't follow all the Mr. C rules because some of them just didn't make sense for us. For example, in a real-time kernel, for example, you call an API and then the API, you pass a pointer and then

you pass some arguments. So one of the things I do in the kernel is actually check to see if each of the arguments are proper. So are you passing an alt pointer when you're not supposed to pass an alt pointer? Well, if you do pass an alt pointer, you're not supposed to, the code returns immediately. Well, in Mr. C, you're not supposed to have an immediate return call like that. You're supposed to have a single return point in your code. Well, to me, it made more sense to say, hey, as soon as I detect the error, I'm going to kick it out and tell the user you did something you're not supposed to do. So there are things like that in the code that just in the misreceive rules that didn't make sense that I decided that we would not be following.

And most people don't follow all of the misreceive rules. You choose the ones you want that make sense for your system and you accept it. Sometimes it just isn't right for you. Yeah, actually, I met one of the gentlemen that was part of the 10 people that's on the committee for the misreceive committee. And he told me if ever anybody follows 100% of the rules, because some of them are kind of contradictory, something's wrong with you. That's true. Good to know ahead of time. Well, there are a lot of rules, and they cover different kinds of things.

And that's why most of the compilers that have static tests for MISR, they have lots of checkboxes that say, well, I don't care about these rules, but these ones I do. And it's not going to yell at you for some of them. Although passing pointers and functions reminds me that we did get a question from Twitter about hooks, RTOS hooks. What are they and how do they work? Sounds painful. Nah, it doesn't hurt at all. So an RTOS hook is a callback function that is actually invoked, for example, in the idle task. Whenever the kernel doesn't have anything else to do, it's an infinite loop. So like I said before, you could actually put the processor to sleep when there's nothing else to do because all the events that all the tasks are waiting for have not occurred.

So the processor spins around doing nothing. So in that case, in the micro-COS kernel, for example, there's a function call that I call that's called OS idle task hook or something. And that's where you would actually put the code to put the processor to sleep in whatever sleep mode you want to use for that case. For example, whenever there's a context switch, whenever we're saving registers for one task and restoring the registers for another task, there's another hook function or a callback. So it used to be hook, it called hook, now it's called a callback.

So the context switch hook could be done or could be used to save things. For example, I want to save floating point registers or I want to save MPU registers on a context switch. Things that you'd want to do for, or for example, you want to turn on an output port as soon as you're starting the context switch and you want to turn it off when you're done or something to show how long the context switch code takes to execute or something. So there's a lot of stuff you could do with these hook functions. There's one for the timer.

There's one for the tick hook. So there's all these different things that are available in the kernel. So you can extend and modify the behavior of the kernel? Without having to change the base kernel itself, correct? You don't want, say, I don't want customers to go in and change the kernel, even though we provide it all in source form. We don't want them to go in and change the kernel. That's why we give them callbacks that they could actually change

and they don't have to go in the actual source code of the kernel and say, I'll change this, I'll change that. Well, the context switching one is really useful because I have done what you've said, although not to measure context switching, but for each, when I enter a task, I turn on this GPIO. When I leave it, I turn it off.

And when I enter a different task, I turn on that GPIO. And when I leave, I turn it off. And then the scope can show me exactly where I'm spending all of my times and tasks. And when I'm going to sleep, everybody needs to turn themselves off the right way, going to the lowest

power state. And so when you get to the idle mode, that's what you do. Absolutely. And in fact, there are some tools that you could actually get. There's a tool called System View from Seger that recently just came out, actually. And then it actually does exactly that, but on a really, really cool Windows-based environment. In fact, it works with Macs and Linux and all that.

But it's really cool because it gives you a timeline of all your different tasks on a horizontal timeline and you could see how long each task executed the isrs you know when the task was ready to run so all of that is is pretty neat is all displayed graphically and it requires just a little bit of instrumentation not any different pretty much that what you've done with your io port so So there's some very, very slick tools that you could use nowadays to do exactly what you were doing from that. For all that we started out with no breakpoints and flashing EPROMs to getting two breakpoints and being grateful for them,

our tools are getting better. Absolutely, absolutely. Well, the other thing, the problem is, okay, so break points are great uh you know it's really nice to have that capability capability but it's not good for everything it's good for testing algorithms but it's for real-time applications there's a lot of cases where you just cannot stop the the application to monitor what's going on. So, for example, you're mixing chemicals. You know, you can't say, oh, when I get to this point, stop.

The chemicals will still keep mixing, but the results you'll get from the mixing will have been far gone away once you start executing again. When you're running engine control, for example, you're firing spark plugs on an engine. You just can't say, oh, let's hit a breakpoint. You know, whatever you're connected to will potentially get damaged. You know, you're flying an airplane. You can't say, well, let's put a breakpoint when we're, you know, banking left or banking right just to see what the result will be. You know, your control system will go dead.

And most likely the passengers will also complicate that. So there's a lot of applications that, unfortunately, everybody does the same thing. They blink LEDs. They have a little GUI interface. They actually display numbers. They do printf statements, the infamous printf statements. So, you know, oddly enough, at Micron, we have this really cool tool that's called Micro-C Probe that allows you to see at runtime the value of variables. And the same thing, these, and in fact, we're able to display variables such as kernel awareness variable.

So we show you how many tasks you have, the amount of stack space each task is consuming, the CPU usage on a per task basis. So there's a lot of information we're able to display at runtime without interfering with the CPU. And that's, in a lot of cases, a very important tool that important behavior that you, a very important tool that you have, because how would you do it otherwise? If you're adding printf statements, you're adding artifacts to your code and thus, you know, changing the behavior. So, what you're seeing is not really necessarily what's going to be when you remove those printf statements. And that's why your code doesn't work when you don't debug. Exactly.

Well, and another thing, the number one issue that I get all the time, that our support guys get all the time, is stack overflows. I actually wrote a blog on our website. You could check out our website. I wrote a two-part blog on stack checking and stack overflows.

And that is, there is no easy way really to see how much stack space your task is using unless you actually monitor that at runtime. So what a lot of people do

is they oversize their stacks because they say, well, you know, this idle task, for example, you don't want to put four kilobytes of stack space for an idle task, for example, you don't want to put four kilobytes of stack space for an idle task because you just don't need that much. You're going to be wasting RAM that other tasks could be using. So by knowing how much stack space you're actually using

is extremely beneficial. But a lot of times what happens is you undersize your stack, your stack overflows. And by overflowing, I mean, you're writing over some variables, values somewhere in memory that you're not supposed to because you've pushed your elements on the stack a little bit too far. And you get some weird behaviors. So, you know, anytime somebody calls us up and says, oh, by the way, I've been running the application. It's been running great.

I love your operating system, blah, blah, blah. But after a few hours, I got weird things happening. Well, how big is your stack? Do you have enough stack space? Well, I don't know. How big should they be? Or how do I

know if I have enough stack space? Well, you need a tool to do that. All right. So I want to go on from there and ask about open source RTOSs. Okay. So there are some open source ones that are getting better. RTEMs and FreeRTOS are both the ones that I keep looking at and thinking, that looks pretty good. Have you dealt with the competition?

Do you look at it? Do you just not worry about it? Well, oddly enough, the free is always our biggest competition. I mean, we, you know, BigQuery themselves, real-time operating system, real-time kernels, you know, they're not cheap, I would say. But at the same time, you know, your time is not cheap or the developer's time is not cheap.

So I look at, and so our number one competitor, bar none, is free, you know, free operating systems. So we always compete against that. But, you know, it's the same thing as if I were to give you a free a free puppy you know like on the surface you think wow this is great you know i i get a free puppy um you know a puppy or puppies are expensive 500 bucks 600 bucks a thousand five thousand but but your cost doesn't stop there your cost will increase from that point the maintenance cost the shots that you'll have to give that puppy so yeah there are a lot of you know real-time kernels that are available for free but

just think about it as a free puppy and for those of you wondering perhaps you've heard me use the free puppy analogy uh i believe i i stole it from jean many many years ago okay i know i'd heard you say you've definitely you've definitely heard me say free as in puppy yes having a big moment of deja vu there yeah so you're so you're convinced then that's good well and there's there's always uh something to be said for when you're when you're making a real product in a real company um there's always something to be said for having an external party to yell at well you know you're paying for support and so you can go and say help me this doesn't work you know yeah but it's more it's mostly that it's it's support it's it's peace of mind it's somebody

that you could always fall back to because invariably as the applications are getting more more and more complicated you will need help. You're not an expert in RTOSs and protocol stack. You're an expert in your own application, whatever that is. So you don't want to spend your time debugging. Or if you have to debug, you want to have somebody to help you with that debug stage that understands the intricacies of multitasking, protocol stacks,

file systems, and so on. So you want to concentrate on your application. That's not where you're paying that money for. If you're on your own, and I see that a lot, it means, hey,

I'm going to save a lot of money to my boss. I'm going to get some free operating system somewhere. I'm going to save a lot of money. And it turns out that, I'm going to get some free operating system somewhere. I'm going to save a lot of money. And it turns out that, you know, I guess your weekends are not free or not. You got to spend weekends potentially to debug things that you could just call somebody on the support line to help you out with.

And, you know, it's like anything else. We've seen a lot of these issues that people, you know, bottom line, we see pretty much the same issues over and over again. So for you, it's a new issue. For us, it's something that we've seen many times and we would most likely answer those questions fairly easily, fairly quickly. Whereas you will, you're novice to any use of the software. So it's a lot easier to ask somebody sometimes than it is to dig and spend the whole weekend at it. Yeah, because you're going to have to, if you're taking a free project,

you either have to depend on forums and talking to the community, which will answer you at whatever rate they feel like, or you have to take on forums and talking to the community and at which will answer you at whatever rate they they feel like or you have to take ownership of it and okay you can have an internal team that's devoted to fixing finding bugs and you know making the changes you need to make so it's a trade-off and but you have to you know like you said it's it's ownership of a living thing kind of exactly and software is a living entity. Software changes. You know, we try to...

There are things that don't change. For example, the kernel has been extremely stable. We rarely make drastic changes to it. You know, if it's not broken, it's working fine. It's got certifications to it. So we don't want to change the kernel itself.

TCPIP, I i mean of course there's the new protocols we're adding uh so you know file system there's new devices we need to support so there's always so software is really a living entity new protocols and new things that you want to add new drivers new devices so um so yeah you want you want to benefit from that you know we're in the business of doing that on your behalf so that you could concentrate on your application how does the purchase of micrium by silicon labs change things for you or from you for micro cos uh it actually COS? It actually doesn't change anything. The marching order that we have from Silicon Lab is that we will be maintaining our support for all current customers, irrespective of the platform that the customer will be using. So we'll be supporting just about everything. I mean,

we made it very clear to Silicon Labs when we were acquired that, you know, what we care about is our customers. We want to support them. We don't want our customers to hang and us. You know, we need to support all these different platforms. And, you know, oddly enough, I mean, Silicon Lab has Cortex-M processors, but they don't support all the different Cortex family.

We don't have anything with Cortex-As. We don't have anything with Cortex-Rs. We don't have, of course, PowerPCs and RX and all these other platforms. So all the customers that have used our software in the past, you know, last thing we want to do is, you know, abandon them. Good. That's a good is abandon them. Good. That's a good thing to hear. Yeah. One worry I heard was that they were going to maybe not let you put your book

for free on your website anymore. That's not true, actually. The book is still going to be for free, and we will continue doing that. In fact, we have started this program called the MicroCOS for Maker. We'll maybe rebrand that a little bit differently, probably called the Micrium for Makers. But the idea is that you could use

any of our software in source form if you're a maker, a hobbyist, a student, a professor. So you could use all of our software, OS, TCPIP, Files SIM, USB host, USB device, Modbus, anything that Micrium and or in this case, Silicon Lab owns the IP to from the Micrium acquisition, then we will still offer that for free use. And it's not the crippled version. It's the exact same version that commercial entities. So the concept there is just like when I came up with the MicroCOS 1, 2, and 3 books is we provide the source code of the software for people to use. And as you could imagine, it's an honor system.

So we can't really control because we're opening that up in source form what we want to do is we want to get people to use um you know clean commercial grade software um to play with and then the uh the hope is that uh you know like professional engineers that work for companies they play our play with our software at night or on weekends come back to to work and say hey boss i tried the micrium software it's awesome and i would like to use that in our next product that we're doing commercially and of course that on weekends, come back to work and say, hey, boss, I tried the Micrium software. It's awesome. And I would like to use that in our next product that we're doing commercially. And of course, at that point, we're expecting them to contact us to license our software. And if somebody has been doing bare metal development, like Arduino 8-bit sort of development,

and they want to try this, do you have a board or project or path that you would recommend yeah it's very very good question there's quite a lot of content on our website in fact i would say if anything you could go to our website there's a download center you could download not only the the software in source form but you could also get a whole bunch of projects. We probably have 500 or 600 projects on different eval boards and stuff. So it's very easy to try the software. How are you going to continue that in the future? Well, if I could find some hours in a day, it would certainly be nice. But I mean, certainly one of the, one of the things that Silicon labs wants to do is to have me be more,

be, be doing more writing, more presentations, lectures at conferences, write books, potentially do more videos. So that's something that, that will continue.

If I could just kind of like start figuring out how do I delegate all these new assignments that I now have. But eventually that's the idea. So I really enjoy doing that. I mean, oddly enough, I mean, writing is difficult for me. My vocabulary is very limited. I'm originally from Canada and my native language is French, but I've been in the U.S. for 30 years, so I don't really have an excuse not to be good at writing and stuff. But it is still difficult, and I guess it's an advantage for me to have limited vocabulary because I don't try to use complicated words to explain concepts. I really try to get to the fact, and I really believe

in graphics. I really believe in pictures. I like to explain things with illustrations, graphics, or that type of information. So, if you were to look at my book, I think you've glanced over it. You'll see there's a lot of illustrations, and I narrate the illustration in the text so that you could actually follow along the whole process of stuff. So, I really would like to continue that whole thing. And I may have a couple books that will eventually, if I can find the time, write about. I do want to point out to listeners that not only are the books free, I think with registration on the site, there is a book about the kinetis system and medical that I particularly looked at. And not only was like the first chapter, the really good explanation

of real-time operating systems and how they work. The fifth chapter was an excellent tutorial on how to get started into medical. It listed all of the acronyms and all of the papers and all of the government documents you need to look at. It was really quite useful. So, everybody who's thinking about writing to us and saying, how do I get into medical? My new plan is to point you to that chapter. Absolutely. And I don't know which one that you actually got. Did you get the Micro-COS 2 book with Kinetis or the Micro-COS 3 book for Kinetis? Because there's two versions of it.

Okay, so there's a Micro-COS 3 book. It's even larger, so it will guarantee to put you to sleep at night. So, what it is, is the first part of the book is about 600 pages that explains the internals of microservice 3 in great details. And each chapter is organized. Like what is the, for example, what is a semaphore, how to use semaphores, and how does semaphore work. And the next chapter is what is message queue and how to use MessageQs and how do they work.

So all that is all explained in there. But in that second portion, so that's out of 925 pages, I think 600 is dedicated to real-time operating system and specifically the MicrosuS3. And then the remaining 300 plus pages really goes into details about the medical aspects of the certification and things like that but also there's four chapters one is blood glucose meters pulse oximetry blood pressure monitoring and i can't remember what the fourth one was but very very cool so we actually explain the physiology of things and then we get into an example of how do you implement that using a kernel and using a kinetis part and that is we're

using a k53 tower from from free scale which now was bought by nxp which is now bought by qualcomm i did i liked the i looked mostly at the oximeter one, and I liked all the pictures of like how hearts work. And it got down into the pretty good, interesting biology, mechanical biology and how light goes through your, it was very cool. I was happy. Thank you. Yep.

So you're clearly still excited about what you do. And I talked to a lot of people who are getting burnt out or already burnt out. How do you stay excited about this? Well, there's so many things happening. So many technology is growing at frantic rates. I mean, the IoT, electric vehicles, the connected home. I mean, there's so many things

happening. And I read trade magazines and I see, wow, this is new technology. I mean, for example, I was reading the IEEE magazine the other day, and this guy is coming up with a camera that's about the size of a cell phone that's going to be as good as a DSLR.

And in fact, he's trying to get up to like a 600 millimeter lens equivalent, but it's the size of a cell phone. It's flat like a cell phone. It's got like something in the order of 15 or so different lenses of different focal lengths and stuff. So like, wow, there's still some new stuff coming out. There's a lot of new stuff coming out. We live in an exciting time. Absolutely. I mean, like I said, the cell phone is a computer.

It's phenomenal. When I travel, I spend most of my time looking at my phone. I only open up the laptop when I need to do something very complicated, very focused that I can't do on the phone. But the phone, I answer my email. I look at websites. I book my flights. I input my expenses. I mean, there's so many things you could do with these things. It's just unbelievable.

Well, I think that's a pretty good note to start winding it down on. Chris, do you have any final questions? Yeah, sort of following up on new technology. Operating systems have been very similar for a long time in terms of fundamental structure. Is there anything new happening? Are there new concepts? Or are we kind of, this is how an operating system is going to work for the next couple of decades?

Well, I think the things that all going to be built in. They have mechanisms to protect execution of code that's not going to be visible to an application. So you could write an application, so your own application and purchase IP that you want to be protected so that either you don't want to be able to tamper with or you want to protect the IP because you put some value on it. So that ARMv8 architecture, so there'll need to be some adjustments made to the real-time kernels real-time operating system but i think it's going in that direction security is going to be a huge

thing yeah and i hear a lot about hypervisors too right um higher end systems sharing the cpu amongst different virtual environments um that would be for very higher-end CPUs. I'm not sure we're going to see that at the Cortex-M type class level. And I still think we'll see 8-bit processors and we'll see 16- and 32-bit processors for a long time without all this memory management units and and hypervisors so um i mean case in point uh case in point uh silicon lab has uh 8051 processors so and they're quite popular there's a lot of applications that an 8051 is way plenty

and in the meantime we'll keep calling them watchdogs. Well, not quite. I mean, they do some fairly complicated applications. Oh, I meant the hypervisors being watchdogs. Oh, yeah. Okay. Well, we have kept you, I think, for long enough today, although I have so many other questions.

Do you have any last thoughts you'd like to leave us with? Well, the one thing I'd like to say is if you're interested in RTOS, I would urge you to download copies of my book, of course, from our website. They're all free in PDF format. You could go to www.micrium.com. We have plenty of sample code, example and source form. You can actually play with it. We support a large number of evaluation boards. So anything that you fancy,

we pretty much support. Also, considering the cost of your time, it should be used to add value to your product instead of writing infrastructure code like RTOS and protocol stack. And the other thing is I'd like to, I had mentioned that earlier also, I'd like to urge

you to look at or download a copy of our Micro C Probe product, which is really cool because it works with just about any processor architecture. And it allows you to see at runtime inside your embedded system and you're able to look at variables any variable that you have in your application and in fact you could look at io ports and display those using gauges meters numeric indicators all kinds of neat stuff so you just check out our micro c pro product on the website you'll find it quite interesting. You won't need to do printf statements or blink lights or turn on output ports and stuff.

And if anything, there's one feature that we've added in there. It's an oscilloscope. So you're able to tap in and look at any variable in your application and throw that on an oscilloscope tray. So very useful for monitoring or looking at real-time applications. So that's it. Thank you. Thank you for your time. Those links of Kourfours will be in the show notes.

And thank you, Jean, for being with us. Our guest has been Jean Labrosse, author of Micro-COS, The Real-Time Kernel 1, 2, and 3, founder of Micrium and chief architect at Silicon Labs. His books are on the Micrium.com site. In case you're wondering how to spell that, that's M-I-C-R-I-U-M.com. Link in the show notes, of course. Thank you also to Christopher for producing and for co-hosting. And of course, thank you for listening. For those of you waiting for stickers, you're just going to have to wait a little bit longer because, wow, you all wanted stickers this time and I'm happy but slow. If you want stickers or if you just want to say hello,

hit the contact link Unembedded.fm. You can also sign up for the newsletter, subscribe to the YouTube channel on Embedded.fm, or you can read the blog, which is pretty darn cool. Okay, and now a final thought for you from Jean de La Fontaine from Fables. A person often meets his destiny on the road he took to avoid it. Embedded.fm is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive any revenue from them. At this time,

our sole sponsor remains Logical Elegance.