Embedded - 401: Oil and Water

Episode Date: February 4, 2022

Miro Samek joins us to discuss designing systems, state machines, and teaching courses. Miro’s company is Quantum Leaps (state-machine.com) which provides commercial licensing for QP Real-Time Embedded Frameworks.  It is an open source project, the code can be found on github: github.com/QuantumLeaps/qpc  One of the key concepts is an Active Object which aids in real-time system development, especially in the areas of state machines and concurrency.  Miro’s (amazing) Modern Embedded System Programming series can be found on his YouTube channel.  You can also find Miro on Twitter: @mirosamek

Welcome to Embedded. I am Elysia White, here with Christopher White. We are happy to welcome Miro Zemeck back to the show. Hi, Miro. Hi, thank you for inviting me back. Could you tell us about yourself as if we met at an embedded systems conference? Yeah, thank you for inviting me again to your show. I've been developing embedded real-time software for almost three decades now.

And I started with programming nuclear physics experiments. And later on, I worked for GE Medical Systems, where I worked on diagnostics x-rays machines. Then I moved on to Silicon Valley, where I worked for a couple of companies working in GPS technologies. But around 2005, I founded my own company, which represents my main interest, which is modern embedded programming.

I also wrote a couple of books about it, about practical aspects of using UML state machines, UML state charts, and event-driven active objects in the context of deeply embedded systems such as microcontrollers. So are you ready for a lightning round where we ask you short questions and we want short answers, and if we're behaving ourselves, we won't ask how and why and all of that? Sure, please go ahead. Do you like to complete one project or start a dozen? Complete one project. True or false?

Most embedded systems are either primarily state machines or continuous monitoring devices, or some combination of the two. But usually more one than the other. That's a long question. It really was a lot of hedging there, wasn't it? Yeah, yes, I would agree, by and large. What else is there? What's a favorite course that you took in your academic career?

Well, you know, I studied physics. So they grilled us on math a lot, analysis and geometry. I think maybe quantum mechanics was interesting. Yeah, I took that too. Pretty cool. Do you have a preferred way to learn new technical things? For example, reading the manual, watching videos, or trying it out without reading the docs? Oh, that's difficult.

I certainly read first, but then I experiment on my own. I'm a hands-on person, so I need to experiment. We might have asked you this one last time, in which case I'm interested to know if the answer changed, but I have to go check.

Favorite fictional robot? I think that I watched recently Terminator 2 again. And so T-100, Arnold, I think it's the top of my list now. Do you have a tip everyone should know? Yeah, well, I would say that in embedded systems, nothing works until everything works. Oh, I thought that was going to be a much shorter sentence. You're going to stop with nothing works? After the last couple of weeks, I would stop there, yes.

Actually, could you expand on that? Nothing works until everything works. Well, we need to always start with a working system and only make small incremental steps to keep it working at all times. Very often, for instance, users of my software take the software and immediately want to go

to their own board and their own example that interests them. Instead of starting with a set of tested examples that are provided as well. So I would recommend just buy a very inexpensive board,

even though it's not exactly your target, and try those examples first, as is, without changing anything. You already would be testing many things, the software, the compiler, everything around it, the connectivity, and then you everything around it, the connectivity, and then you can only make very small incremental steps from there. I didn't do quite exactly that thing, but that's what I did kind of starting with a new client this month

is they had a particular target, and I got some dev boards that were similar and got familiar with the IDE, which I'd never used. And I just got the examples from the vendor and tried a bunch of them and made sure everything worked and got familiar with the IDE, which I'd never used. And I just got the examples from the vendor and tried a bunch of them and make sure everything worked and got familiar with things. And it made it a lot easier when I finally did import the client software and start building it.

Yeah, absolutely. It builds the confidence that the stuff works, first of all. And then people take two large steps. And in order to move fast, you have to take baby steps, really small steps. If you take one bigger step, you will waste a lot of time trying to find out why things don't work anymore. And taking smaller steps actually ends up that you will be moving faster. I feel like this relates to when I read recipes online, where you'll read a cookie recipe and they'll say, I replaced the sugar for honey and it didn't work at all. And it's just, yes, I see that a lot with embedded systems as well, that you have to run what they said to run first and then make the changes. Right.

There's a million ways it can go wrong and only a few ways for this to go right. So probability points to the way of doing this. You have a YouTube series teaching embedded systems programming, teaching modern embedded systems programming. What's the modern there for? I call, well, first of all, what I define as modern, everything invented past mid-1980s, I call modern. Because what I observe is that most of the, and the other stuff I call traditional, that the traditional approach is still the dominating, absolutely dominating approach in our field. So we probably didn't really take, learned many lessons past mid-1980s, I would say. Which is a sad state of affairs here. But that's why I call it modern.

I'm trying to point out interesting and important developments that happened in the 90s and maybe early 2000s. But even if you apply more modern techniques, we're still stuck with a lot of old stuff, like I squared C and SPI aren't going anywhere anywhere and they're from pre-Pac-Man era. Right. Pre-Pac-Man era. Thanks. That's a nice line in the sand.

Okay. So, I mean, we've gotten past assembly. Yeah. Yes. So the whole premise of this course is I see it as a missing curriculum because our discipline falls, as you know, somewhere between EE, electrical engineering, and CS, computer science. And yet our field is distinct and different from both of those. And not many universities actually teach this middle ground.

So there is, of course, a need, but also an opportunity to fill the gap and to teach this part. Do you think that we will be seeing more embedded systems curriculum going forward? Or do you think it will always be you learned half of your your education in embedded systems in school and the other half you just have to pick up on the job i i don't really know what's in on the mind of of universities and and schools that teach this um some schools do some schools do and and are probably specializing in producing embedded

engineers but not too many right now I mean it's certainly better than when we went to school because we basically

got C++ and or Java a little bit after us and then if you wanted any embedded there was a microprocessors course in the engineering

department right yes yes of course. And now there are far more robotics teams and other hands-on interdisciplinary clubs. Right. But they often start with Arduino and maker platforms like this. And for instance, in my course,

I deliberately didn't choose Arduino for the reasons that we probably will discuss later. But it is geared towards newcomers to the field and makers and hobbies, but not so much to professional embedded developers that will then later go into the industry and develop production level code. It's not for professional? I would say that Arduino isn't meant for that. Sorry, I thought you were talking about your course and I was like, I think maybe? Yeah, so that's why we have this, I'm talking to what Christopher just said, that there is no easier. Yes, and we have maker platforms like Arduino, which didn't exist a few years back. But they are not necessarily geared towards professionals.

Fair enough. I mean, I wrote a blog post about not using Arduino. I think it was called, don't use Arduino for professional work. So you're right on there. Yeah, so I think we're in strong agreement on that. And now, and I'm teaching a course too, which is totally different from yours, and it's been a little

hard to convince people that you can't take a project that is based in the Arduino IDE into an interview and talk about it and expect, even if you put a lot of work into it, you can't expect them to take it as seriously as if you just used a compiler, GCC, IAR, Kyle, anything other than Arduino IDE.

Right. You end up using, you end up showing a lot of code in your videos. Right. And you're compiling for the Teva platform. Is that right? Actually, maybe I should let you tell. What platform are you compiling with and what tools are you using? Yeah.

So please remember that the video course started back in 2013, right? So it's a while back. And so back then I chose Tiva C launchpad board for quite the, and the choice was quite deliberate. I had the whole list of features that the board should support. First of all, I wanted it to be based on the modern ARM Cortex-N processor, as opposed, for instance, AVR or something older. Second, I wanted to have a debugger, a real debugger interface, built into the board.

I wanted to have a few LEDs and a few switches on the board. I want to have accessible pins because in later lessons I used logic analyzers, so I wanted to easily connect to the board. I wanted to have a virtual COM port.

Right now I don't make much use of it, but in the future I will talk about concepts like software tracing and testing on the target. So this will be very useful. And I wanted this to be available for a long time. And so far, Tiva is still available, can be bought almost everywhere from DigiKey and other vendors. And last but not least, it had to be quite inexpensive. So it retailed for some $12.99 or something like this.

And this is important because many people who participate in the course come not necessarily from the U.S. or the European Union, but from countries like the Middle East, South America, India, China. And for them, it would be a problem if the border would be too expensive. So that's why I chose Tiva. If I had the choice right now, I would probably go with STM32 Nucleo, although Nucleos don't have enough LEDs for me. There are some discovery boards that have rings of LEDs now.

Right. These are nice. But some discoveries didn't have virtual comport. I was quite, you know, this was an important feature for me. What did you choose? Yeah, I ended up, Cortex-M was all I limited it to. Right. So you don't specify beyond that.

I recommend strongly that they use an STM32 and a discovery board. And I gave them a list of things that they could use. And that has worked out well for me because I have cohorts and they go through together and then they talk about their boards and they end up with more information than I could possibly give them. STM is a good choice right now.

So your class has been going on since 2013, you said? And you have, was it 60, 80? It was a lot. You have a lot of courses, a lot of videos. Over 40 now, 43 or something. Have you planned out what you're going to teach when? Yeah, well, I drew a chart, a dependency chart, if you want. So which subject depends on which other subject.

And so I wanted to start from the beginning to kind of establish the terminology and common understanding of things. But then, you know, it took me eight years to get to my favorite subject, which is state machine. So that's why I had to talk about things like RTOS much

earlier than that because, so you see that I see RTOS a little bit lower level than hierarchical state machines and active objects. So yeah, so the dependency, what depends on what, and what I have to talk about

first and can then build upon it. This determines my order of presentation. How did you choose your theory of order? I mean, some things you do have to explain earlier, but did you base it on your book or just how you wish you had been presented the material? I wanted to end up with something like Active Object Framework. And so I worked backwards, so to speak, working which concepts need to be presented to get to that point. And so, for instance, at some point, I had to talk about object-oriented programming because it's a framework based on classes

and instances and inheritance and polymorphism. So I needed to cover those concepts. I needed to talk about real-time and scheduling and real-time issues and something like rate monotonic scheduling analysis had to be, mutual exclusion blocking had to be covered before that. Of course, the interrupts and race conditions have to be covered even earlier than that. So that's why, you know, I worked backwards.

So only now I'm slowly getting where I wanted to end up. It looks like you're finally to state machines, which is one of the things that I think you really enjoy talking about. Yes, I find this subject a little bit hard to explain because I see a lot of misconceptions. And the subject is complex because there are so many kinds of state machines even. And there is no single terminology to even name them. For instance, I talked at some point about input-driven state machines and contrast them with event-driven state machines. Now, input-driven state machines is something that I just found in the literature, but some people call it periodic state machines or even some other names.

But the point is that they are not exactly the same, and they cannot be translated into UOML state charts later on. They are not quite compatible. So that's why surgical state machines take eight lessons, and probably it's not the end of the story yet. It was odd for me to see that you had eight lessons on state machines and four lessons on RTOSs. And to me, those are opposite levels of complexity.

Why, I guess. Yeah, well, exactly because RTOS is maybe more crisply understood concept. You know, of course, you have to start with the definition, what you consider RTOS. Some people consider RTOS the whole software. It's the kernel, it's the file system, it's the communication stacks, and so on and so forth. So I like and I prefer to use as narrow as possible meaning of the word. So for me, RTOSas is the Arthas kernel only. This allows me to be more precise when I talk about things,

not to put too much meaning into one word. So for instance, when I narrow it down that way, Arthas could be explained in about six or seven 30-minute lessons. Of course, I didn't cover in detail the communication mechanisms within the RTOS, but it was precise enough for my purposes. Interestingly, things that many people will consider tricky, like context switch, could be explained within maybe first two lessons. Actually, only second lesson showed how to automate context switch in assembly.

But things like semaphores, mutexes, and other communication mechanisms could not be explained that quickly at all. This is where the complexity of the RTOS is, and many people don't appreciate it. In order to use those mechanisms correctly, you have to be quite versed, and these mechanisms are also very tricky to use correctly. Because of concurrency or other reasons? Yeah, well, I think that the biggest problem with Inartos is the issue of blocking. And this is the root of many problems in Inartos. So, of course, you can have race conditions and there are mechanisms to prevent them, such as mutual exclusion mechanisms, such as mutex, for instance. But then you introduce additional blocking, potentially,

and this, of course, interferes with your real-time response. For instance, I saw on your show, it was an interesting episode, 3.3.5 I have here, patching on the surface of Mars with Joel Sherrill, I guess. He talked about Artem's open source, Arthas. And he talked, among others, he talked about HICAP in the real-time response, where a highest priority task that normally took one millisecond to execute, all of a sudden started to execute in 17 milliseconds or something like this, or of magnitude worse.

And in the same episode, he explained the rate-monotonic scheduling and this method of handling real-time responses. And then the problem is immediately apparent for me that RTOS, even though has been designed specifically to enable real-time monitoring scheduling, for instance, at the same time, because of those interactions and mutual exclusion mechanisms, defeats the purpose right there. Because sometimes you have a completely unresponsive system

that apparently doesn't meet any real-time deadlines. Yeah, I think that's an interesting point, because I think of mutexes and semaphores and all those things as fairly simple concepts, but most of the places I've used them haven't been in places where preserving a real-time response was super critical. They were using mutexes to, you know, keep things from clobbering other things, make sure things were happening in the right order. But adding the additional requirements

of not having those things get in the way of your real-time response just makes things a lot more complex for figuring out how to architect your system. And they're not great tools for that, like you said, because they're blocking. Right. Rate-monitoring scheduling, for instance,

does not take into account mutual exclusion. And if you want to, really, then you would have to take the worst possible case and count this towards the CPU utilization, a central concept

in rate-monotonic scheduling. And then what you will end up doing is that most systems wouldn't be schedulable, meaning that they will have no chance of meeting the deadlines always. This would work for soft real-time systems, but for hard real-time systems, it is unacceptable. So should we ditch all the real-time operating systems and just use state machines? No, no, no. That is exactly the point that I introduced a real-time operating system and had to do this because it's a stepping stone in understanding active objects and hierarchical state machines, the event-driven state machines that work with active objects. So what active objects are, there's actually a design pattern that prescribes how to use an RTOS, if you will, safely.

So there are just simply restrictions on the naked threads on top of the R2 threads, that when those best practices are implemented, you'll have a chance to build a correct system much better than without them. So first of all, you should avoid any sharing of resources among threads. Any sharing of resources among threads, but you can communicate between threads. I could pass something. Right. Now, the second thing is, in order to communicate, you have to designate special objects for that. Gotcha.

And these are called events or messages. And they are specifically designed for communication only. And they are exchanged between those threads in a thread-safe manner. And it is kind of a component that is on top of the RTOS. Some RTOSes, many RTOSes, for instance, provide message queues. So message queues could be used to exchange those events safely between those tasks. And so this means that you should prefer asynchronous communications

by means of those events. By asynchronous, I mean that you just post an event with a queue, but you don't wait for the handling of this event for the processing. And then the third thing is that you should structure all your threads around event loop, meaning that there will be just one single point of blocking within an endless Y1 loop in each thread or each task at the top of the loop. And then when an event arrives,

this blocking call unblocks, and then event is processed in the application-level code. So some application-level code is called to handle this event. The event has to be handled without any further blocking, and then you quickly return back to the event loop. It's funny that you're using the TI board because this is a pattern I see a lot in the TI code, especially their BLE subsystems, that every thread really is its own little kingdom, and it spends most of its time waiting, and then it doesn't do any waiting until it gets back to waiting for whatever happens. So there's no, oh, I received some data, I'm going to print it out the serial port and wait for that to be done. No, no, oh, I received some data, I'm going to print it out the serial

port and wait for that to be done. No, no, no, no. If you receive the data, you can print it out the serial port, but you can't wait. You have to go back to your single point of waiting. Right. It's funny because my first, when I started my career, I started in networking protocols. And this was how all code was written, because networking protocols work this way. You're passing messages between separate devices, and you're receiving messages, and then you see what kind of message it is, and that's an event. And then you have this loop that handles it, and then you handle it, and you get out, and that's how everything was architected. And so when I did other things, that seemed like a natural way to do it, but I don't see a lot of people using this pattern that often, day-to-day anymore. Yeah, so there are, of course, many dangers now with programming.

I mean, many pitfalls that you potentially can fall into when you start doing this. So first of all, Ardus provides a whole assortment of mechanisms, all of them based on blocking. And so it is very tempting, for instance, in your event handler, in the code that is supposed to handle an event very quickly in return, to call something like a small delay or blocking delay.

Or you have a situation where you send some information, for instance, a packet through I2C port, but then you expect some reply. So what you do is you wait in line for the reply. So this obviously holds the whole event loop processing and you are becoming unresponsive to any new events that start accumulating in your event queue and can actually overflow it.

So this would actually derail the whole mechanism. The point is, though, that RTOS provides so many of those mechanisms and when you buy an RTOS, you pay mostly for those mechanisms. And each one of them is actually something that should be avoided. So you're paying for the things you shouldn't use. You shouldn't use. Are you familiar with any of the free RTOSes? Free RTOS or Zephyr or I think there was a MicroCOS one that was out.

Yeah. For instance, the QP frameworks are ported to all of those that you mentioned. To FreeRTOS, to MicroCOS, to ThreadX, to Emboss from Seger. And, of course, to POSIX that QP can run on top of Linux and other POSIX compliant RTOCIF. So yes, I am familiar. The QP framework, that is what your company actually makes and that's what you sell. Right. That's right. And how is that related to active objects?

Well, this is an active object framework. So it provides active objects as first class items in the framework. So there is an active object class from which you can derive your own class and specialize. There is strong support for hierarchical state machines. So the behavior of your active object is specified as a state machine. So there is, of course, support for events and thread-safe event passing. You can pass the events directly from one active object to other active object or from ISR to active object. But there is also the publish-subscribe event delivery

where you can publish an event and all subscribers will receive, the event will be multicast to all those subscribers and delivered in a thread-safe way again. There is support for dynamic events with parameters that can be mutable events that are then allocated from thread-safe and deterministic event pools. And then the whole event recycling is handled automatically in the framework.

And there are many other things that the framework supports also. But to some extent, it's about keeping things, I don't want to say clean, because that's not really the right word, keeping the objects separated as an architectural thing, not as a last-minute, just naked RTOS, just for instance, free RTOS, then you have all those mechanisms, event flags and message queue and so on and so forth, time delay. But they are not very useful, as I said. Maybe they are even harmful. On the other hand, you don't have events. You don't have publish, subscribe, and other thread-safe mechanisms for delivering those events, because the events don't exist in the first place. You don't have any support for state machines, or let alone hierarchical state machines.

So what you end up doing is you have to build all those mechanisms on top of the RTOS. And instead of everyone doing this on their own, I'm just thinking all of those things are completely reusable. And so you can think of this as just the extension of an RTOS. The most important thing that differs between framework and there's a difference between framework and RTOS, is that a framework works by inversion of control. So when you use a framework, you write the code that the framework will call. So you will write, for instance, a state machine code

that it will be called by the framework when the event is available. This is in contrast to writing threads in the real-time operating system where you write the whole body of, so every single time you start with Y1 loop and you choose which blocking mechanism you will use in this particular thread or task. So the inversion of control is just a defining characteristic of the framework versus a library like an RTOS. But you did say that it works on top of an RTOS. Right. Okay.

Because those state machines, they need to be scheduled. course you could you could use very simple mechanisms that would for instance execute everyone every such state machine to completion and then look what is the next active object that has some events in their event queue and maybe choose the highest priority one and then execute this one in such a cooperative non non-preemptive way. This is one way of doing this. But it is also possible to combine the whole approach with a preemptive kernel. And in that case, the active object, for instance, one active object can be running very long run-to-completion step.

For instance, in a GPS receiver, a position fix is a floating point calculation iterative that takes a few hundred milliseconds to complete on a small ARM processor. In the meantime, you have to track the GPS signals every few hundred microseconds, really, that fast. So you need to, in this case, you could use a preemptive kernel to do the signal processing at much higher priority

every single time preempting a very long run-to-completion step. And this run-to-completion step then will eventually complete. And everything is fine because there will be no processing of for instance next event still in the middle of processing the first one. So this will be all fine from the perspective of a state machine but you will be able to meet hard real-time deadline in a high priority thread.

And this system is, by the way, much more compatible with rate-monotony scheduling because exactly you don't have situations where you share something through mutexes or something like that. And you don't extend artificially your CPU utilization. So the whole method works much easier and cleaner. It's much easier to analyze such a system. You don't spend a lot of time talking about your framework on the videos I saw from your YouTube channel on modern embedded systems programming.

Well, that's right, but it is coming more and more, and it's already kind of being used. I smuggle it piece by piece, so to speak. For instance, in the lesson of RTOS, on RTOS, I've used not free RTOS. At the very beginning, maybe I used micro-COS or something like that. But at some point, I switched to the kernel built into the framework because the framework comes with a selection of a few real-time kernels. And one of them is called QXK, has been used in the last few lessons where I exactly demonstrate hard real-time behavior of an RTOS and using illustrations with logic analyzer traces.

I show, for instance, priority inversions and how to remedy them with priority ceiling protocol and so on. So, yes, the framework is being increasingly used. In the lessons for state machines, I used it, for instance, to demonstrate the semantics of hierarchical state machines, how events are, what actions exactly are triggered by events in a hierarchical state machine, in which order the exit actions and entry actions are handled, and so on. And this was already using the framework. Okay. Clearly, I skipped around. But you have more than 40 lessons at like 30 to 40 minutes each that start with interrupts and getting started.

But I didn't get any commercials on YouTube. And I wasn't even using Christopher's account. This is a lot of work. I mean, this is a lot, a lot of work. Yes. This is a lot of work. I mean, this is a lot, a lot of work. Yes, and because you do something similar, I know that you have an idea how much. Well, that is because the revenue of the company is not coming through this. The company survives on the revenue from selling commercial licenses to QP frameworks. So QP frameworks are available under open source license, but this is GPL, so it has the strings attached,

meaning that if you use GPL framework, you are obligated to publish your application level code as well. And it would be very easy for companies to publish the code, create a free GitHub account and put the code out there. But actually, most companies don't want to do this. They prefer to pay money and buy a commercial license which frees them from the obligation of GPL. And so this is how Quantum Leaps makes money.

And the YouTube course is just one of many channels of promoting this to the developer community. I feel it's as good a channel of promoting QP as the Embedded Podcast is a channel for promoting logical elegance, which is to say, you don't do a lot of promotion, do you? That's right. Well, you are doing the exact similar thing, I suppose. You just educate and you teach, in a sense. And that is what small companies can do. We can out-teach the competition, so to speak. For some reason, big companies like Wind River will not teach for free, but we do. It's nice. I mean, I totally agree. And clearly, because we're doing similar things.

But it is weird that it's the smaller companies who tend to want to reach out more. And the larger companies are going to instead make it all about their company or make you pay for lessons. It's odd. Talk about an inversion of how things should be. Right. I never understood, for instance, the policy of microchip, that they were still selling professional or higher optimized versions of compilers for their PIC processors. They make money by selling tons and tons of chipsets,

chips out there. And they should be giving the best possible compiler in the hands of the developers and not charge, you know... For them, these are really pennies. I never understood their policy. But I think that in bigger companies, everything is about return on investment. And they have to have some demonstrated value immediately available.

If they don't see it, they don't do it. I don't really, as a small company, I don't really know what my return on investment will be. I have only sense that it is worth doing. And because I think with the small companies, the return on investment may not be cash. We don't necessarily do everything by revenue. We can have our return on investment be satisfaction and people telling us thank you and that sort of thing. Right. Yeah. And people, for instance, when I mentioned that there is a need to fill the missing curriculum, if you will, for embedded developers, that is also an opportunity to exactly provide this for them.

And if you do, you will earn their trust and that you know something about that, that you helped them, and then they are more likely to use your product. Yes, you too were influenced by, if you build it, they will come. I understand that. Okay, so back to the videos. How much of what you do is pre-written versus how much is done on the fly? Everything is pre-written for me. I actually write down my script. First of all, I start with a code. If I don't have a good idea what kind of code I will present, I don't have a lesson yet.

So I think a lot about the code. I experiment. You'll be surprised how difficult it is, for instance, to show failures. For instance, stack overflow or race condition or some other failure. So I experiment what is possible. Then I write down the script, and then I edit it down. Well, I remember one thing, a story somebody told me that Blaise Pascal,

that was this French mathematician from the day, he started one of his letters like this. I apologize that the letter is so long, but I didn't have the time to make it shorter. So I do take the time to shorten my original script. And so in the end, just so people maybe understand, I figure that is about an hour of my time for every minute of the video. So a 30-minute video will cost me about 30 minutes of time. 30 hours.

30 hours, sorry. Sorry, sorry. Yes. You know, this includes writing the code, preparing the diagrams that I show, the editing, and so on. Does that make it hard for you to go to conferences that won't pay you and be okay with putting together a conference talk, knowing how long it takes you to present things? Well, again, I see it as if it was easy, then anybody could do it. So if it is hard, and I chose to do this the harder way,

people appreciate triple notice. I had some comments that some people wrote to me that this is the only... They watch my videos at 1x speed, and this is probably one of the only few that actually they don't speed up the videos. That is a real compliment. I take it as a compliment, right?

You show a lot of code in your lessons. How much time do you think people spend writing code versus reading other people's code? Well, I think that code is read much more than it is written. So it has to be legible. It has to be nice and elegant. But speaking of writing the code, I don't think... Well, I still, for instance, try to write as much code as I only can. And I take pride in this.

This is an important part of my day. As opposed to many companies now outsource it to the least bidder or somewhere. And the code is not the highest possible quality. So I would just say, pay attention how you write your code and then this will pay off in every possible way. This is the most valuable part that you can contribute to your company. I agree with you. And I must have gotten my question wrong because I wanted to know for people who are learning, they often jump into wanting to write code, which is fun. Writing code is fun, but they often don't want to read other people's code.

And so you put a lot of code on the screen, and I appreciate that because it encourages them to read it. But do you think outside that screen time, people are becoming accustomed to reading code instead of just starting to type? Yeah, well, just like any literature, you have to read a lot before you start writing. So that's why I would say people new to the field should read a lot of good code and choose wisely which one they consider good, elegant code, and then emulate that. Any recommendations for good and elegant code? I somehow think it's not going to be the Linux kernel. Probably not.

You're trying to get emails? Yeah, that's why, for instance, books like Jean Labrosse's MicroCOS, they were a very good source of elegant code. I would, and I might be biased, I would recommend QP, which is also open source and developed on GitHub, to read that. So there is good code, well-explained code out there. I agree.

I usually suggest Adafruit's code as a very nicely put together system where you understand you're implementing a lot of different features. It's still code that maybe isn't quite as optimized as it could be but if you are thinking about a system where you need to implement 10 different sensors

and you want one interface, that's the code to look at. Because they've thought about that. Yeah. Well, someone said, I think that Donald Knuth was, that premature optimization is the source of all evil. So yes, I would agree that overly optimized code is obscure and not as clear as it could be. And we should probably put

clarity of the code above the absolute last byte of memory and absolute last CPU cycle. Especially with our processors now. They're big enough, they're fast enough. We don't need to optimize. And when you do need to optimize, start with what the compiler wants to do.

Unless you're on a pick, in which case you don't want to pay for that. I have some listener questions. One is from Simon, and he asks, What mistakes slash problems can I avoid building into my system by properly using state machines? What are the easiest ways to use state machines improperly, and how can I avoid falling into those traps? So if you start using state machines, and by that I mean event-driven state machines, in which you generate events externally to the state machine, and then you send those events to the state machine for processing.

So if you use, if you mean those state machines, then if you start using them in the context of active objects, and by active object, we talked about it, it is the set of those three best practices that they encapsulate. So if you start using those, you can avoid a lot of problems. You will not have to deal with race conditions, because if you don't share things that can lead to race conditions, you are safe with them. You will be sharing only events that are guaranteed to be delivered thread-safe from the source to the destination. So no race conditions, you are safe with them. You will be sharing only events that are guaranteed to be delivered thread-safe from the source to the destination.

So no race conditions, no mutual exclusions mechanism. You won't need to even know what the mutex is. You will also be able to meet your hard real-time deadlines much easier and

apply RMS, rate monitoring scheduling method, for instance, to mathematically prove that you will meet your hard real-time deadlines always. It will be also easier to reason about your system because every event will be processed to completion as opposed to have some convoluted path through your code. And also, if you start using state machines, you will avoid a lot of spaghetti code. We talked about spaghetti in my last time at this podcast. So now, what are the easiest ways to use state machines improperly?

The improper use of state machine would be if you would start mixing event-driven paradigm with sequential paradigm, by which I mean if you start putting blocking calls inside your state machines. So for instance, if there's an action of your state machine, you will call it delay, time delay function, or you would call a blocking semaphore or something like that.

Don't do that. So you will be better off not to use state machine at all than to put a blocking call inside your state machine. So my point here is that please don't always realize what kind of paradigm you are using in which thread. You can mix those. You should never mix them

within one thread. You can make a hybrid system so one thread could be completely event-driven and the other thread could be sequential. That would be all right. But don't mix them in one thread could be completely event-driven, and the other thread could be sequential. That would be all right. But don't mix them in one thread of execution.

And so always realize which one paradigm you are using and stick to it. Don't mix them. They don't mix. It's like oil and water. This is why we need names for them. And we need agreed, we need shared names for them so that when you look at somebody else's code, you're like, oh, you're mixing the interrupt or event-driven state machines

with the sequential state machines, and you shouldn't do that. So let's separate those things out. Not only this, when you are using state machines, you need to realize that, oh, I am using an event-driven state machine here, or that is a case that I'm using an input-driven state machine. Input-driven state machine would be quite easy. They are quite easy to recognize because each state starts with the if statement, in which you test some global variables.

In UML nomenclature, this would be called a guard condition. It's a fancy name for if and else. So if every case in your state machine starts with an if, this is an input-driven state machine. It is not an event-driven state machine. So it is all fine, but I'm just suggesting that you recognize which type of state machine you are dealing with and then stick with that. Don't mix those either. Okay. Keep separate event driven, interrupt, and sequential. Okay. Doug G. asks about using QP on multi-core, multi-processor systems.

Have you done it? How did it go? And any interesting notes? Well, as I mentioned, QP has been ported to RTOSs and, among others, to POSIX. So if, for instance, you use it on top of embedded Linux and this embedded Linux runs on a multicore, then QP is running on multicore. But that is mostly transparent to the programmer, the symmetric multiprocessing typically, and you don't really do anything special.

I have not used QP yet on a small multi-core. Now we have chips that have for instance a small M0, Cortex-M0 and then M7 or something like this. I have not used them in such systems yet. I have however used QP on distributed systems, meaning that there was two microcontrollers talking to each other over a serial port. And it turns out that the Active Object model is very suitable for this because it is kind of like a thin wire communication through events only and nothing is being shared, which is suitable for just serializing those events and sending them over the network

or over the serial connection in this case, and then deserializing them on the other end. And actually, if you use so-called communication proxy design pattern, then you can have a proxy active object that actually does not do the processing, but just only sends events back and forth between those two processes. However, this active object looks to other active objects

in the same address space as the real thing they don't know, that they are posting events to an active object that actually lives in some other address space in the processor. So what I'm trying to say is that the active object model is very suitable for distributed systems and will be probably also quite suitable for multi-core. But I have not made that experience quite yet. You've mentioned hierarchical state machines. Can you give an example of what that is? It happens often in the state machine design that you have several states that need to handle this given event in the same way.

However, in a normal classical flat state machines without any hierarchy, because they react to some other events in a different way, you will have to put them in a separate state and then replicate the same responses, the same transitions in all those states. So this, of course, inflicts repetition, and that's why normal finite state machines, the traditional ones, tend to explode. In the 1980s, David Harrell came up with a more advanced version, which he called state charts. Later on, they were adopted into UML and sometimes also called hierarchical state machines. And this is the mechanism that you can now put states within states.

And those states that are nested within a bigger state, they inherit, so to speak, the behavior from the super state. So now you can have this common transition or common reaction to events coded only at the level of the higher level state. And all states that nest within, if they don't prescribe how to handle this event, this event will not be forgotten or dropped on the floor. It will be propagated to the high-level state and handled there. And so in that case, you will have a mechanism of overriding this event, if you like, so the substate can prescribe how to handle this event on its own, and it will be the case of overriding a virtual function.

Or if it ignores it, it will be handled in a common way by the super state. And that way you can reuse the behavior. And so you avoid the repetition. And that's why those hierarchical state machines, they don't no longer explode as your problem grows in complexity. I tend to like really hands-on examples. So I'm going to try to give one, and if I'm wrong, you can tell me. So I have a toy, and it talks, blah, blah, blah, play with me. And if I hit the volume button, it has a click, you know,

high click or low click, depending on which way I put the volume. But if I hit a different button, the A button, it goes into a different thing. It starts talking about the letter A. And in the letter A, if I hit the volume button, it still gives me the same clicks. But if I was in an old-timey, traditional finite state machine, the A state would have to point to a volume that then would have to point back to an A state. So it'd be a volume that was in the A state and only belonged to that.

But with hierarchical, I can say, look, if anybody presses the volume, you're going to play the click sound. And if you don't want that to happen, you have to replace it with something else. Yeah, that is a good model here. Yes, that would be a good application of state hierarchy, right? I tend to use toys for all of my state hierarchy because children's toys have so much state in them. And it's easier for me to think about, okay, well, how does that apply here?

Let's see. Oh, I saw a new book from the Raspberry Pi Foundation about teaching computer science called Hello World. Have you seen that? No, no, I haven't. Okay, well, then I won't ask you questions about it. You can. But someday someone is going to say yes, and I'll ask them questions, although it may be somebody I invite from the author list.

Because teaching embedded systems and computer science, it's different than teaching just about everything else. Do you agree with that? Well, I have not taught anything else, so it's difficult to make comparisons. I think that in my course, I focus on fundamental concepts, such as object-oriented programming and event-driven programming and RTOS and real-time and state machines and active objects. So that's why I believe that fundamental concepts, it is always good to understand those deeply.

And if you do go and show people how they work at the lower level, they tend to understand it better and deeper and then use this more efficiently. So this is my approach that I take. In my education as a physicist, we had actually a course for teaching physics, and I taught physics at the level of middle school, and I always tried to show them experiments, and I believe that showing

and demonstrating things to people is very effective to teach them I totally agree I have one more listener question from I don't remember your first name how would you pronounce that, Christopher? Which one?

M-C-N... M-C-E-N... Some kind of a handle. I'm going to go with Kevin, because I don't remember. I should, but I don't. who made a really wonderful joke about, are city machines less complex than state machines? It is Kevin. It is Kevin, okay.

How far can we, or should we, push concurrency on bare metal through state machines before it really makes sense to move to an RTOS? Well, Kevin, probably at this point meant the input-driven state machines that combine event discovery with event processing. So this is, again, the situation that you start every case in your state machine for every state with an if statement, which is supposed to test your inputs

and find out if an interesting change in those inputs has occurred that you need to handle. So you discover your events, and then you try to handle them. The point is that because it's a non-blocking process, so if there is nothing interesting to do, you don't block, you very quickly return or you just pass through that code. You can string such state machines

and put them one after the other in the forever super loop, right, in the foreground-background system. So in this sense, people think of FSMs, of finite state machines, as a mechanism that handles some kind of concurrency before they switch to an RTOS. With event-driven state machines, the situation is different, and they are actually at the higher level of abstraction than RTOS itself. So they build on top of the RTOS. So how far should you go, push concurrency before it makes sense to use Muftun and RTOS? I would say don't use a naked RTOS at all. Use RTOS only with those three best practices of concurrency.

That means use the RTOS only as an active object, as a vehicle to partition your problems into encapsulated threads called active objects. And then you can use event-driven state machines inside each of those threads. So this is kind of like an inversion of the question, because Kevin apparently puts an artist at the higher level than state machines, and I'm saying this is only because he probably thinks of input-driven state machines. Otherwise, I recommend to use active object design patterns with event-driven state machines on top of the RTOS. So go beyond the RTOS with that, not somewhere below the RTOS.

I would add that we've already talked about mixing the types of state machines is bad. And so an RTOS gives you a way to mix the types of state machines without it being bad because it limits their scope between them. So if you have a sequential state machine and an input-driven state machine, you need an RTOS so that they don't fight? Well, actually, RTOS limits your choice because you're no longer typically in an RTOS unless you do a periodic, you know, you sleep for a tick and then you start checking for your inputs. Then you could possibly use an input-driven state machine. Otherwise, RTOS prefers probably event-driven state machine

because the tasks can block on the empty event queues. And RTOS needs blocking because during the blocking time in one thread, progress can be made in other threads. So RTOS necessarily needs blocking to manage the concurrency that way. So I would say that RTOS strongly prefers event-driven state machines. But actually, RTOS can be bad for state machines for some other reason. And that is mixing the sequential blocking programming paradigm with state machines for some other reason, and that is mixing the sequential blocking programming paradigm with state machines.

As I said, it is very bad to put any blocking calls inside the state machine, in the action of the state machine. So that's why... So this is the danger of using RTOS and state machines together, because there are so many temptations, so many mechanisms within an RTOS that would block all of them based on blocking. Okay, I think I understand, Christopher. Nods. One more question for you, I think. think uh your book is practical uml state charts in c and c plus plus event driven programming

for events embedded systems and you have another one uh quantum programming for embedded systems which is practical state charts in c plus plus you really like uml state charts i've never gotten into them. Convince me, please. Well, the state chart title was suggested at some point by my editor, and we stuck with that. Maybe, you know, I should have called this book maybe somewhat differently, and maybe the subtitle would be better, Event-Dri for embedded systems. Yeah, but yes, I was trying to,

in the first part of those, these were two editions of my book, try to explain the state machines and their full potential to actually have them working at their full potential, it turns out that you need a framework around them that works on the principle of inversion of control and provides the run-to-completion execution context

for each state machine, because the state machines need to run to completion and be undisturbed during that time so they are vulnerable if somebody would call for instance you know a state machine from within another state machine and so on so so so people don't really i i came to this realization after reading of course many books that that um event of state machines actually do need a framework. And you see this in all UML tools capable of code generation. For instance, the Rhapsody tool from IBM is based around the framework called OXF,

Object Execution Framework. There are some other frameworks also supported by Rhapsody, for instance, IDF, interrupt-driven framework, and many others. But all of them are frameworks. And there are other tools also based around frameworks. So it turns out that UML state machines do require a framework. And the contribution of QP is actually to provide something very, very lightweight, capable of working with small microcontrollers.

Miro, thank you so much for speaking with us today. Do you have any thoughts you'd like to leave us with? Well, not really big thoughts. I just maybe want to reiterate that you can use different paradigms, and it is all right, but it is always good to realize which one you are currently using.

And then you need to know which one can and cannot be or like to be mixed together. So it's all right to use different paradigms, but sometimes they just don't work well together. Our guest has been Mira Samek, founder and CEO of Quantum Leaps. You can easily find his YouTube series by searching for his name,

but there'll be links in the show notes to that and many other things. Thanks, Mira. It's good stuff. Thank you for having me. Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listener Slack group for questions. And thank you for listening. You can always contact us at show at embedded.fm or hit the contact link on Embedded FM.

And now a quote to leave you with from Immanuel Kant. Space and time are the framework within which the mind is constrained to construct its experience of reality.