Embedded - 224: Interrupts to Interrupt Interrupts

Episode Date: November 22, 2017

Andrei Chichak joins Elecia and Christopher to do a deep dive into the world of interrupts. Andrei writes on our blog: Embedded Wednesdays. He has written specifically about interrupts in multiple ways: general introduction, buttons and debouncing, peripheral data transfer via DMA, and so on). The knock-knock joke comes from Chris Svec’s Embedded.fm blog post on interrupts. Jack Ganssle on debouncing buttons

Welcome to Embedded. I'm Iliasia White, here with Christopher White. I'm quite pleased that Andrei Chichak from the Great White North has joined us today to do a deep dive into a particular topic. Interrupts. Hello, Andrei. Good morning. Now, you've been a guest with us a number of times, but still there may be somebody

out there who hasn't already heard of you. Could you give us an introduction? Okay. My name is Andre Csicak, as you've said. Let's see. I was born at the age of three, lived most of my life at 28,000 feet, and I come from a land of ice and snow from the midnight sun where the hot springs flow. So you and Thor, huh? I guess we can just go on.

The next section, I mean, we're skipping lightning round this week in favor of a joke. What? Yeah. What? I want're skipping lightning round this week uh in favor of a joke what yeah what i want to do lightning round oh all right go ahead what would you like us to ask you uh let's see mountain or beach all right mountain or beach beach uh favorite wave standing cats or ferrets cats did you know that there was an armadillo that was once the size of a car no did you know that the word muskrat has nothing to do with musk or rats. Pass. Are bats a reservoir of rabies or not? No. No. Bats are awesome. Bats are so cool.

Do you know that there is continuous frequency bats and then there are frequency modulated bats? They use echolocation chirps in order... Never mind. Show about interrupts. Oh, but we have a joke first. As though it hasn't already, already all been a joke. Okay, I'm sorry, Andrew.

Go ahead. Please start again. Knock, knock. Who's that? Interrupting cow. Interrupting cow. Boo! Yeah, that's exactly how that just...

That's what the show is about today. Interruptions. And cows. And cows. And bats. And return from interrupt. Our goal is to talk about interrupts, specifically to talk about how to teach new embedded software engineers good habits with respect to interrupts. And the goal here is just to chat about how we talk about interrupts to people and how we make it clear without making

it super boring. So let's try that. Well, the first thing is we have to define interrupts, but also... Can I interrupt you there? No, you may not. I think that we should start at Arduinos. Well, that's what I was going to, that's where i was headed is most people who are starting out with arduinos and such probably don't encounter them or need to and so developing why why they're something interesting would be a good way to start is that where you were headed or yeah pretty much yeah go for it well with an ar, of course, you've got the two sections, init and loop. And the rest of us who don't use Arduinos end up using this structure all the time anyways. We just call it the big loop model or super loop or whatever.

So when your program first starts, you initialize all your hardware, initialize all your variables, and then go into a huge loop where you just repeatedly do stuff until the power gets pulled. And on an Arduino, the background structure that nobody sees, when your program first starts, they call init once and then they were just repeatedly called loop and when you're using that sort of a structure like it works really well i mean we use it for small projects all the time but uh what you end up having to do is if if you need to see if somebody is pushing a button you have to go and ask the digital I.O. port, have you been pushed? If you need to get something from a serial port, you have to go to the serial port and say,

is there anything there for me to deal with? And this is known as? Polling. Polling, good. And if you have enough of those things, that becomes most of your program's time is spent asking questions yeah and and you can basically make no progress because you're mostly just sitting around asking your various subunits do you have anything for me to do do

you have anything for me to do this isn't i mean this is the basic form of Arduino. Arduino completely allows interrupts. So we're not saying you can't use interrupts there. And to some extent, when we talk about this main loop or super loop, or sometimes it's called the event loop, especially when it's called the event loop, usually there is this idea that something is happening. Some event is happening and you may be polling for what that event is at the beginning. Do you have a serial character for me? Do you have this button pressed sort of events? But what if those events came from outside the loop? What if they were asynchronous? What if they didn't go in line with this? And so these events are generated by interrupts, really. That's the secret. Asynchronously. In Arduino, you can hide this asynchronous and event words by calling them events. And, of course, you're on a system that, you know, you've got other things to do as well if if we were just sitting here

uh waiting for characters to come in it wouldn't be a very interesting system like you have to have your characters coming in from your zero port and do something with them whether it's move a motor or uh turn power on and off i mean there's other things for your system to do. And you're not actually taking care of those things while you're polling for events that aren't happening yet. So if your events are very sparse, there's probably a better way to do it. Okay.

So let's say we are implementing just a button press. We don't care which processor it is because most of these terms go from processor to processor. So it could be an Arduino. It could be Cortex M4F, whatever. And we start out and we set up the IO line and we say we want it to be an interrupt. I'm sorry. We want it to be an input and then we hook it to be an interrupt, or I'm sorry, we want it to be an input, and then we hook it to a button.

And so we push the button and the line goes up or down depending on how we've configured everything. And then every time we pass through the loop, we say, so how's the button? How's the button? How's the button? How's the button? And we take action on the button. If we were going to do it in an interrupt, how do we do it?

You don't. You let the hardware take care of it for you. That seems harder. A lot of the processors, eh, not really. A lot of the processors have it all built in. And so it's just a matter of us getting over the hurdle of, oh my God, this is really hard to understand and tricky and horrible? Well, I think it is hard to understand because most programmers don't start with the concept of concurrency, right? You write a program and it goes through a series of steps

and you can see those steps and they're all in line interrupts is the first place where people often encounter things jumping up in the middle of what you're doing and doing something else which leads to all kinds of complexity right yeah okay before we go there i just want to get through the basic interrupt Okay, so now I have read the data sheet, and I understand that I can configure a hardware interrupt. And I have some parameters I don't understand on that, but we're just going to assume that I copied and pasted from some hardware abstraction layer. And now I'm getting an interrupt, and now I'm not. I mean, okay, let's say when I push the button,

I want the light to go on, because that's an easy thing to try. So now I push the button, and interrupt happens, I think. What's that mean? Yeah, but my light never goes on, so this did not improve my system. My system worked before. What needs to happen after I set up the hardware to interrupt? Do something. Yeah, how do I make the interrupt do something?

Well, how about if we hold off for just a second? We'll put that on the stack for a second and explain what an interrupt actually is. Yeah, that's definitely part of it. A good way to see an interrupt is a asynchronous subroutine call. So basically, you know, we've got our big loop. We call subroutines. When you call a subroutine, it takes a bunch of stuff, sticks it on the stack, executes some code. And then when you return from the subroutine, it pops all that stuff off the stack and you continue. With an interrupt, it's basically exactly the same thing, but you don't actually do the subroutine call. Some external event causes the processor to stop, put all your stuff on the stack, and then call a prearranged subroutine.

And then finally, when you're done, you do a return and you go back to where you were in the big loop. So instead of me calling a function, maybe my function is my button pressed. Instead, the hardware is going to call this function for me. Yes. And it's all so simple after that. It's all so simple. Okay, so this thing that it calls, you said prearranged function. This is some sort of contract between my code and the processor itself.

Yes. And it has a name. A vector table name. Depends on the processor. That's true. So what are the common names for this? Well, something like if you're using a microchip PIC, you've got... You should just stop there.

No, no, no. Yeah, I know, but it's... My biases are well known in that area. Your bias is showing. With a PIC, you've got high priority interrupts, low priority interrupt, which we'll get to later. But they all go to exactly the same place you've basically got as soon as that interrupt comes through you start executing code at a known location and then you have to trot through and figure out what caused it and do the appropriate thing on something like an arm they have something called vectored interrupts, where you have a table of pointers to subroutines.

And when your interrupt happens, the processor automagically goes and figures out exactly where it should go. So you don't have to figure out who generated the interrupt. The processor figures that all out for you and executes the subroutine for you. Okay, so just to make sure that we're saying this right, on a PIC and a number of smaller, older chips, you get an interrupt or maybe two different interrupts. And from there, you have to figure out, was it your serial port or a button that caused this? On most of the processors I like, I'm sorry, on most of the ARM processors, we instead have this idea of,

oh, the button interrupt calls this function, the serial interrupt calls that function. And there may still be some unraveling. It may just be one of these eight lines did something, and that causes an interrupt. And you still have to go into the interrupt and figure out which it is. So this idea of what exactly made this interrupt happen is important but you have to know your processor well enough to know whether you're untangling did i get a serial character in did i get a serial character out did i get an error and that's why i got my serial interrupt or

or i have to figure it out from whole cloth. And that thing is usually, if you're looking for this thing, that thing is usually called the Interim Cause Register. Yeah. And this goes back quite a ways. I mean, the way that the ARM does it is exactly the same way as the 68000 does it back in the original Macintosh days. Whereas the IBM PC, the original PC only had eight interrupts in the whole system. And then when the AT came around, they increased that to 15.

And I'm not sure how they do it these days, but you still have to basically deal with those original 15 interrupts and go and figure out who interrupted. So it does work very well, but it's a little bit more baroque. Yeah. And now, I mean, I set up a processor that had, I think, 45 interrupts not too long ago, and it wasn't a big or complicated processor. It just had a lot of peripherals and features. That was 45 of them being used. There were some no-ops, but yeah,

there were 45 in my table. Because something like the ARM Cortex, they have 240 possible interrupts available. Oh, wow. But they do everything from divide by zero to Ethernet package just came in. Yeah. So that's that's taking that cause register and just turning it into you can insert a function here instead of

going and figuring out yourself what what happened right okay so we we we have gotten our button press the hardware so bad at this the hardware has said hardware has interrupted whatever it was I was doing before and has just yanked me over to go into the subroutine. The subroutine that we have previously told that the code knows is a special subroutine. We should talk about how the code knows that later. But right now, we're just going to say the hardware magically put me in this function. What should be in this function? What

should be in that function is as little as you can get away with. One of the tenets of interrupt processing is you spend as little time in there as possible so depending yeah we'll get there but that's important and why is a good question and it is important but okay

we'll come back to it okay um one of the one of the things that you have to do in the interrupt is whatever your application needs to do and another thing that you have to do is clear the interrupt. Because if you don't clear the interrupt and you return, you'll immediately go back into the interrupt. So depending on the processor and your compiler, you might get code to clear the interrupt generated for you. The interrupt might clear all by itself. Or you might have to go and read a register to clear the interrupt.

Which is a weird thing. And then you need the appropriate return sequence. That's usually handled by the compiler. Yes. But it used to be the way that you returned from a function was different from the way you returned from an interrupt. And that's still true if you're writing assembly. But you don't see it very much anymore if you are writing for the Cortex family. Right. Because it's all hidden.

Well, it's a little bit different. The Cortex family, the structure of an interrupt is exactly the same as a subroutine. Yes. The difference between a subroutine and interrupt routine is completely gone. Whereas something like on a PIC or any other number of processors, instead of doing a return from subroutine, you have to do a return from interrupt. And those are assembly instructions. RTS, return from subroutine. RTI, return from interrupt.

And they tell the processor what it has to do to get back to where it thinks it should be to get back to the program counter. And when you're returning from some routine, it just has to unload the stack the way it loaded it, and it knows how it did that. But when you're returning from interrupt, it did special things, because you were in the middle of something before,

it has to put you back in the middle of where you were. It can't just jump to the beginning of main. You might have been doing something important. So right now, I don't think we should focus too much on that because we don't, as programmers, have to deal with it very often anymore. But it's kind of nice to know that there is a difference underneath all of that. But I remember having those bugs, but I haven't had them in the last 10 years.

Thank God. Oh, because when you, when you write a interrupt routine, you have to tell the compiler, by the way, this is an interrupt routine, do the right thing. Whereas you probably just call the subroutine instead of calling an interrupt routine and when it returned it did the wrong thing that was a bug yes is that you can it or you can also have you know an interesting function and so you make that the interrupt it does everything you needed but maybe you're in a special mode where you're polling for a little while so you call the interrupt yourself and then it all goes very badly afterwards if you have that situation but we should skip this because we don't have that situation very often anymore we we enter the interrupt and we

just treat it like a normal function because most of our compilers will let us do that now we don't want to be there for very long for reasons we haven't talked about but let's say we're going with this button press thing. You mentioned we need to clear the interrupt, and depending on your architecture and software and all of that, you may have to set it up so that the interrupt can happen again. That's the housekeeping part of it. But what are we going to actually put in this interrupt? How are we going to tell our main loop that the button got pressed?

Well, you could call a subroutine, but no, that would be a bad idea. You could, but it still doesn't. Okay, well, why would that be a bad idea? Why would that be a bad idea? It's sort of, this gets into why you spend as little time as possible in your interrupt routine as well but um you've got a little bit of a concurrency problem where you could actually be in that subroutine so you're you're in subroutine. You get your interrupt from the push button, and in your interrupt routine, you call subroutine A. So now you've got basically two instances of subroutine A happening, and you're getting into a little bit of God mode here where you have to keep everything nice and straight.

This can be done, but you have to keep it nice and straight to make that all work properly. And then it will return to your interrupt routine and then exit properly. So let's talk about what subroutine A might be. And I'm going to take a case that you should not do, but is a reasonable case that happens. Let's say in my main loop, I was printing out, waiting for a button to happen. And in my interrupt, I decided to print out, button happened. Now, you don't usually want to do prints and interrupts, and we will talk about that, I promise. But this is a good way to talk about this thing that Andrea was mentioning about maybe not calling a lot of subroutines, especially ones. Well, okay, so I'm halfway through waiting for a button to happen.

And I get this interrupt, and I send to my debug, print, button happened. But I was only partway through. And now it was sending those bytes to the output buffer and it was collecting them into its output buffer global so it could send them out later so maybe it said wait so far right and then i want to put in more bytes and it's going to be wait button happened in button happened yes it's just going to come out all garbled because we were halfway through something and now we're going to do it again assuming we started from zero it's it doesn't it doesn't work there's but the reason that happens is because you are using a static or

global variable in order to do that printing. You really have to worry about these calling subroutines that you may already be in if there's some sort of information that is static between calls. And there's a word for when you're thread safe like that. What is the word? Reentrancy. Reentrancy. Yes. You know, the word I was trying to get you all to say was volatile. So can we go back to what would you have in the interrupt? We would not have a call to printf. We shouldn't have many calls to subroutines,

especially ones that have global or static variables. And we should get out of there as soon as possible. But what should we have there? Volatile. Say volatile. Come on. Some way to indicate to your main loop that something happened yay oh wait wait we still haven't said the word um that's not necessarily volatile it will be i don't know what you will be but now you've got a variable that's shared between

the main and your interrupt. Is that safe? Well, it's going to have to be global because they're two different functions. Right. And so as long as my interrupt is writing the variable and my main is only reading it, that's totally safe. Well, not really, because... Depends on how big the variable is. No, no, no.

I'm going to take the point of view of the compiler. I've got this subroutine that all that it does is write this variable. Well, that's not very useful, so I'm going to get rid of that code. And I've got this other subroutine that only reads that variable, and the variable doesn't change. So I'm going to get rid of that. It has the other subroutine that only reads that variable and the variable doesn't change. So I'm going to get rid of that. It has the whole subroutine. It knows the variable never changes.

Yeah, so it can just get rid of that. It never changes. So why would you read it? Why don't I just set it once? So somehow we have to tell the compiler, don't optimize out this variable. It's useful. There's something weird happening in the background that you don't know about. So just trust me. I know what I'm doing. Don't get rid of this variable. How would you do that? And that's where the keyword volatile comes

in. Volatile tells the compiler, look, magic's going to happen to this variable and you can't tell what's in it at any time. So don't ever optimize it out. It may change. It's just magic. Just believe me. There used to be a keyword called register that was very similar, but they got rid of it. So now we only have volatile, which is better.

This is also really important on something like a PIC or some sort of an 8-bit processor where if you're working with 16 or 32-bit values, you could be... So the base data size on the processor is 8 bits, but you're dealing with a 32-bit value. That's where I was headed. You could be halfway through doing an add. So you've updated two bytes and you get an interrupt. Something

changes, like a timer went off or something like that. And when you come back, you do the rest of the calculation but on new values. So now you've got a half-corrupted value.

The volatile won't save you. Will it? Yeah, it will. Oh, really? Will it? I don't think so. I don't think so. Or you might have to put that into...

You have to put protection around that. You have to put protection around that, which we'll get to. Yeah. We have to. Yeah, but we're in deep complicated stuff from our button push example now.

So now we have hooked up the hardware and set up. To have an interrupt, we have used the function that is somehow connected to the vector table, jump table, however we get the subroutine that is the interrupt,

interrupt service routine. And in this interrupt service routine, we know we have to clear the interrupt and set it up so it can go again. And then we have this variable. We set this variable that says button has been pushed or maybe button was pushed at this time. Now we go back to the main loop and instead of asking the GPIO system, has a button been pressed? Has a button been pressed? Now I have a loop that says, did this variable change? Did this variable change? How is this better? Now I'm pulling a

variable instead of hardware. It seems like I'm still sitting around asking useless questions. That's why I get to ask the questions so that other people can answer the hard ones. I hated getting this one because it was so like, gosh, you're right. That is both useless. What do you think, Andrew? I'm masking your interrupt. I have answers. Go on. Go ahead, Chris.

Well, for one thing, looking at one variable or indicator is a lot faster than asking a bunch of separate questions. The structure of your code can be a lot better. And you can go to sleep because you know if something happens, it will awaken the processor. And so you don't need to keep asking the question because somebody will just shout at you and wake you up and then you can look at what happened. And if my goal for this whole thing

was to build something that button press turned on the light, turned off a light when it was released, I can do that in the interrupt. We don't want to do a lot in the interrupt. We don't want to do debug printfs for sure. But I can turn on a GPIO. It's not like you can't do anything in interrupts,

but there are times when your whole code is much simpler. Well, you should do things in interrupts that are extremely time-sensitive. Like if you knew that signaling to your main loop was going to cause you a 20-millisecond variable delay and you needed something to happen in one millisecond, those are the kinds of questions you need to ask should i do something in an interrupt okay christopher's talking about latency we should put that on the list of things we should talk about uh but i was just talking i was trying to get to okay you push the button the interrupt uh turns on the light you release

the button the interrupt happens again you have to configure it specially for this, folks. But you release the button and the light goes off. Now what's in your main loop? You know, chilling. Main loop does nothing. You have your setup. You have your interrupt service routine callback thing. Because your entire program is now the interrupt handler. Your entire program is now the interrupt handler. And this is really hard for a lot of people to see because it is pretty easy to go through the main loop. And even if you aren't polling, you're just using variables to transmit events, it's still a fairly linear thing that's happening.

X happens, and then Y happens. Or maybe they're asynchronous, but when X happens, you do this. When Y happens, you do this. When you push it all into the interrupt service routines or some form of callback system, you really are inverting it so that you don't have this if X, then do this. It just happens. And some software engineers,

this is more how they think because there are languages that this is how it's put together. But for a lot of people, this going from serial one step after another to parallel, oh my God, everything's happening asynchronously, is a pretty big mental shift. But it is pretty useful, because having main be totally empty means the system is doing whatever it needs to do, whenever it needs to do it. Okay, do you want to add to why is polling for a variable any better than polling for the hardware change, Andre? Well, what I tend to do when I write systems like that is my main, instead of being completely empty, will be sitting there monitoring the system.

So, read the ADCs, make sure that the voltages are within reason, make sure the temperatures are within reason. And a lot of the times you can have a variable that is pretty much bit encoded. So, when you get an interrupt and the mainline has to do something, you can just set a bit and uh in your normal processing making sure that the system is running properly you can look at that big word and say is it all zero and if it is you just continue on doing your doing your uh scan of the system making sure that everything is fine if somebody needs me to do something one of those bits will be will get set and then i can say okay i must do something greater than is that on the is that where you're going i wasn't really where i was headed but it's an interesting

uh interesting point you can use you can still use these global variables to communicate with your main loop in order to indicate the status of the system and to indicate additional things that need to happen. If I go back to wanting to print out waiting for a button and button happened, then in my main loop I could put waiting for a button and then signal through this variable that a button happened. Now, my interrupt may have already taken care of turning on and off a light for me, but it's still nice for my logs to have both. And because it's looking at this variable and it's not printing asynchronously, it will format correctly. It will finish the one waiting for button to happen and then button happened. And even if the button happened in the middle of where it was printing that waiting message, it will wait until the very end to do the button happened.

So your debug messages can use these bit masks or global variables to indicate what's happening, even if they aren't handling the actual event they aren't handling the event they aren't taking action on them um although it's often helpful to still have their global variables this whole turning the light thing on is obviously just a toy problem but it's a useful toy problem well a very common thing to do with interrupts is output to, like, as you say, you're going to print out, you know, I got a button press. But you can use interrupts to have to, so basically you fill up the buffer and you say go, and the interrupt routine takes care of shipping all that stuff out the serial port.

Yes. takes care of shipping all that stuff out the serial port yes there are a lot more interrupts than just button presses and the serial interrupt is is really useful very useful but if you so you're in the middle of uh sending something out the serial port and you get an interrupt and you immediately go into your interrupt routine and you say, print out this stuff. So now you're using, you've got an interrupt and you're going to be using interrupts to generate your output. And suddenly you've gotten yourself into a whole bunch of hurt unless you're

a God. Okay. We're going back to don't do printfs and interrupts don't do printfs don't do output logs don't do any of that unless you're willing to accept that it is likely to fail and bite you because in that case you're dealing with nested interrupts as well ah yes okay which that's not good i wasn't ready for that a lot of people aren't um okay i want to let me make sure that we have all the things for the very basics of getting an interrupt we talked about that it's asynchronous it's not in the line of of what you think is

happening we talked about vector tables. We kind of touched on context. Maybe we should define what context is. Go Chris. No. Context is all of the things around where you are in the program. It's your state. It's your state. It's, it's your program state. It's what, what is the program counter? What instruction was it about to run? What was in all of its registers? What was it adding together? What was it multiplying? And what was its next instruction in the pipeline? There are all these things around what the processor is doing right now that when an interrupt comes, it has to save that context. It has to remember everything it can about where you are. Because remember, it is just yanking you out of what you were doing

and sending you over to the interrupt to do something entirely different with no context. You go over the interrupt and the program counter now, because the hardware has set it, forced to set it, it is where this function is. But your register values, they can't be what they were before. You have nothing there. And if you do anything in your interrupt it's going to change those register values those are the the cache things those are the memory you use first and it has to then take those registers and put them back the way they were when you go back to your runtime context so there's the thread or runtime context which is what you're normally running

this is your main loop this is what you've called from your main loop. Interrupt comes, has interrupt context. It only has a small view into the system and that small view into the system needs to be completely erased so you can go back to your main context. And it's important to realize there's these two little worlds and the interrupt has a tiny little world and your main has a big world. And that's not a bad thing. An interrupt can be small and not view everything because we do want it to go fast,

which was the other thing on my list. Why are we keeping interrupts small again? I remember I got to a company and they were running their whole system out of interrupt. Every time the accelerometer interrupt came in they did the whole Kalman filter. Worked fine. It's fine if you don't do anything else.

Because when the interrupt is running, your program isn't. Right. So if your program is doing something mundane, that's its normal course of events, let's say you're doing button presses and interrupts, but that's controlling a video game. If you spend a whole lot of time in your interrupt, logging the button press and thinking about something else, and then you come back to your video game, every time somebody presses a

button, your screen's going to stop for a short period of time while you do all that interrupt stuff. And anybody out there who isn't thinking, wait a minute, I've seen that happen. Yeah, we've seen that happen. It happens because people have crummy interrupts, which is what we're trying to avoid. So if you have a program, your main program, that's doing a great deal of work for your application and your interrupts aren't well written,

you can introduce latency into your main program. You can cause jitter and pauses. You can break it entirely if it's doing something time sensitive. So that's one of the reasons to be very careful. But wait. Assume for the moment that you've pushed your button and you are now doing a Kalman filter calculation and a character comes in on your serial point.

Did you get it? In this system, there was only the interrupt, which was the only way to make that system work. No, but in general, if you're in an interrupt routine. And another interrupt happens. And another interrupt happens. And another interrupt happens. It could get, like in interrupts you have, we'll get there, but you have priorities. So everything of a lower priority waits.

So your character that's just come in off of your terminal waits. The flag has been set. The interrupt will happen, but you have to wait. If another character comes in in the meantime and you still haven't serviced it, you've just lost some data. So your first character just got overwritten.

Now you do a return from interrupt. You say, oh, I got an interrupt from my serial port and you go and get it and you've got the second character rather than the first one. The first one is gone. And you probably have an error because you have overrun your buffer or whatever. Which generates another interrupt. So the main reason to have small interrupts is that you don't know what you're interrupting.

You could be interrupting something very important. File system access. And so you do the smallest you can, you get in, you get out, and that's it. And you try not to do delays or printfs or anything that might take a long time because you might be interrupting something that is very important to finish quickly. And you can build a system where that doesn't have to be true. This whole massive system that was done in one interrupt, it is completely possible, but it's harder in the end because that system we could not add logging to until I fixed it. We could not add additional interrupts. We couldn't monitor the ADCs in a good way because we were so tied to this single interrupt. If you want to have multiple interrupts, if you want to have a good system that can

handle all kinds of facility with debugging, enabling logging or allowing users to do multiple things, to not be stuck waiting for just one button. That's where you need to make it small, because you need to be able to handle lots of things and to do it quickly. Chris said the word latency. We haven't defined that. Somebody want to go? Latency is basically the time between when you get the interrupt and when the button gets pushed and when you're ready to get back to doing work again.

So basically, initiation of the time between you push the button and the light comes on. Okay, so Chris is using the American standard definition of latency and Andrew is using the embedded systems version of latency. Okay, there are two definitions here. Whenever you're talking to somebody who cares about latency, the latter definition is the right definition. I think Chris is right. It's basically the entry preamble to your

interrupt routine. The reason there are two, though, is if you think about a motor system where you have to do something every five milliseconds. Right. And if you don't do it within that time, the motor jams or stops or whatever. Flies off.

Flies off and kills people. Whatever. And if you have an interrupt, you know, you have a debug interrupt that takes seven seconds that's really bad but even if you have an interrupt that takes two milliseconds it is asynchronous to that other one and so where does it happen does it happen at t equals zero okay that's fine we're done two milliseconds we need to service the next one in five milliseconds. But what if it happens at t equals four? Now we have a delay. We've started our debug interrupt handler. It's going to take two milliseconds. We started at t equals four milliseconds. It's going to take till t equals six milliseconds. Don't even get to me about

fence post problem. We're not going to talk about that. But now we have this problem that our good stuff didn't happen because of our long, random, useless, really important debugging, but not useful. Yeah, it didn't happen. The good stuff didn't happen when it needed to. And that's why you keep interrupts short. Now that time between when the interrupt happened, the debug interrupt, and when it came back, that two milliseconds, that can be called latency. We also talk about system latency, which is how long does it take for the processor to get from I'm running in a routine to I am running in my interrupt. That's important when you have a

real-time operating system, an RTOS, because they have a lot of overhead. And so if you need to be able to get an interrupt and be able to service it within some small amount of time, let's say you have a satellite, you have to be able to put your shields up when you detect a micro meteorite coming for you. I just totally made that up. I'm sorry. But you only have a little bit of time in order to take an action. You only have a little bit of time before you can swerve in a car or before you can put the locks on the motor because you noticed somebody's finger was about to get crushed. You only have a little bit of time. If your real-time operating system takes too long, then you may not have

enough. And that real-time operating system latency is a number that has been incredibly important in the past. As we get faster and faster processors, it's not as big a deal if it has to take 100 cycles, because now 100 cycles takes less time. So latency is really an important term, but it's hard to get your head around because it means a lot of things in different situations. In some situations, it's just how long until I handle this. In other situations, it's how long until I can get back to what I was doing before you interrupted me. So be aware that that term is very loaded and means different things in different situations. I got a good example of that. I used to work for a very large chemical company,

and one of the guys in the research and development department decided he wanted to do a nice little file transfer from his brand-new PC to our, at that point, it was a VAX computer. And he basically had the PC power typing at this reasonably large minicomputer. And he was only going at 9600 baud. This is characters being transferred between one computer and another over serial port. Not a problem. Except 9600 baud ends up being 960 interrupts per second.

And that was enough to bring quite a large mini computer absolutely to its knees. He was doing okay because he was running in interrupt time. But all the rest of the people on the computer, which would be basically we are the main loop, we all halted for 15 minutes while he did his file transfer. So this is a big deal that actually happens. Okay, that brings in how many interrupts do you do? And how expensive are they to the processor? I mean, 960 interrupts per second doesn't sound that much to me now. Now.

Now. But there are systems that would take all of the time in the world just to service those interrupts. Now we get to RTOSs, and RTOSs are actually just a little bit of smoke and mirrors. You've got a timer that generates an interrupt so many times per second. And every time that timer ticks, everything gets put on hold because you've just got an interrupt. It stacks all the way your registers and all your context that you were just talking about.

And then the task switcher kicks in, which goes through a series of tables trying to figure out who's got the highest priority, who's ready to run, and then loads up their context and does a return from interrupt. That's all an RTOS is. I'm going to let you finish in a second. Okay.

Yeah, okay. This is not the RTOS show. There will be an RTOS show someday, maybe. I feel compelled to say that that's one kind of scheduler. That's a preemptive scheduler. Many other kinds of RTOSs do not interrupt you to switch context in a round-robin fashion. It waits for your task to say, I'm done. So there's other ways to do that.

This is called preemptive multitasking. So, okay. Basically, there's been a lot of research in the last few years to do the task switching as quickly as possible. Because this is the latency that we're talking about. The time between that timer tick and the next task running. So if you spend too much time doing your task switching, then you could end up gobbling a huge amount of system resources doing absolutely nothing but task switching. So let's see, 10 years ago when I was doing RTOSs on a 16 megahertz system, I was able to get about 100 task switches per second and my system would run reasonably well. 1,000 task switches per second, forget it.

It just wouldn't make any progress. Now with the ARM chips that are running at 160, 216 megahertz, getting 1,000 task switches per second is easy, and you've got lots of cycles left over to get your work done. Okay. But we will talk about RTOSs sometime in the future. Let's go back to just interrupts. Okay, so you can get a lot of interrupts.

We want our interrupts to be short, so we don't want to do anything unnecessary. We do maybe want to signal the main loop through the use of global flags or semaphores if you're in our test land. What? I think you're bringing our tosses back in. I know, I'm sorry. I didn't do it. It was totally me. But let's go back to another situation

where you may generate a lot of spurious or silly interrupts. I like the button again. Button bounce. Andrew, I know that you care a lot about debouncing buttons uh so i'm totally throwing you under the button bus here that's okay i wrote an article about this uh for the blog not that long ago where i actually went and looked at some button bounces on uh an oscilloscope so basically what happens when you hit a button is mechanical things, and mechanical things aren't actually all that predictable. So you've got a spring that is

unloading. So when you push the button, it pushes against a spring and some mechanical leafs sort of open and close. Unfortunately, when you push a button, it doesn't just close. It hits the contacts and the contacts bounce open again and close and open and close. And our computers are fast enough that they can see every single one of those open and close events and they're happening on like the uh less than millisecond range and they can continue like jack gansell had a nice little article where he went through a bunch of buttons and he found buttons that would continue to bounce for like 16 milliseconds. So you're going to see every single one of those bounces, and it's going to continue to call your interrupt routine over and over and over again.

So if you were to have an interrupt where you push the button and it turns something on, push the button and it turns something off, and you figure, oh, this is great. I don't have to have a latching button. I can just have an intermittent button. Well, the bounce is going to make it go on, off, on, off, on, off, on, off, on, off, on, off, rather than just on, off. And you're going to see every single one of those interrupts happen. So I usually throw it over the wall. I usually throw it over the wall and tell my electrical engineer buddy to throw some capacitors on it to get rid of the bounce.

And that is a totally reasonable way to do it, especially as a software engineer. There are ways in software of helping to realize when you're doing bounces. One way is to delay enabling this IO interrupt right away so you can't get a whole bunch of them. There's also level versus edge triggered. Does that matter here? Nope. I don't remember why it's important. You need hysteresis, right? Yeah, you really need hysteresis. On some of the Atmel ARM

processors, they've got hardware debounce. Go through your manual and see if that's built in. But this is just a great example because it's physical and it's easy to understand. There's lots of other kinds of things that might interrupt in a spurious way or get into situations where it's triggering many. Like, if you had an error state from some peripheral that triggered an interrupt, and maybe it just went nuts and starts going hundreds of times per second or thousands of times per second.

This is also a situation that you might not anticipate, right? You might bring your whole system down because some peripheral is saying, help, I'm sick, and then I'm fine, and then I'm sick, and then I'm fine. So interrupts have great power to take over your system. Well, that's a good architecture point. Not necessarily in the basics part of interrupting, but when you're looking at doing a system and figuring out how things go together, be aware that there are ways that things can interrupt way too often, usually in an error state. So how do you, maybe you have an interrupt that you receive that it is in an error state, but you don't do anything about it. You don't tell it, that's fine, quit telling me, because you didn't ever expect to be in an error state.

So there's that side of you do have to handle these things so that you don't get a bunch of spurious interrupts then there's the other side where it is always possible to miss an interrupt oh no and it is uh it is bad to to start any design with the words if we're never going to miss an interrupt and if we do our system will just end up rebooting because your system will end up rebooting a lot. It's just the way of the world. When I talk about making a stoplight system as a good way to talk about state machines, one of the things I always say is to be sure not only to depend on the sensors I give you in the road, but to occasionally use a timer to go around the stoplight because you want to make sure that you're not dependent on a broken sensor. And that's kind of my rant about architecture,

is be tolerant of missing interrupts and extra interrupts because that's totally going to happen. I ripped that off from Elizabeth Brenner. Ah, well. I think it was Elizabeth. Anyway, let's see. Next on my list of things we should talk about includes resource protection do's and don'ts.

Go. You should protect your resources. You should hoard them like dragons. I think you shouldn't. Mostly because I'm not sure where this is going. Well, I think she's talking about the issue. Explain yourself. You're talking about if the interrupt handler modifies something that your main program may also modify.

This is the thing we were talking about. Atomic. It's usually trying to update something in an atomic fashion so it's not half updated by your program and half updated by your interrupt handler, right? And allowing re-entrancy with global variables. This is hard without some mechanism to say, I'm messing with this thing right now. You're not allowed to. And when we say resources, I'm saying global variables, but there's also the idea that maybe I push a button and it sends out a small spy command. Right.

That's a resource too. Right. Or I push a button and it... Writes to a file. Writes to a file. That's, well, that might be long. Or it starts to write to a file. It kicks off the writing to a file, that might be long. Or it starts to write to a file. It kicks off the writing to a file. You have to protect these

resources from other things happening at the same time. So how do we protect these resources? Well, you could say, don't interrupt me right now. Yes, we can turn off other interrupts. We can turn off other related interrupts, which is kind of dicey. Or we can turn off other interrupts. We can turn off other related interrupts, which is kind of dicey. Or we can turn off all interrupts. Anybody want to cover the pros and cons of enabling and disabling interrupts? Well, if your interrupts are disabled, obviously any outstanding interrupts won't come through.

So you've got a little bit of a time delay that you've introduced. Latency. We're going to go back to latency. Yeah, you've put overhead on your latency. And also, if something like back to the serial port, if you have a character come in, it generates an interrupt, but they're disabled,

so nothing happens. And then another character comes in, and it would have generated an interrupt, but they're disabled, so nothing happens. And then another character comes in, and it would have generated an interrupt too, so you've just lost a character. So you've got data loss while your interrupts are disabled. Data loss while interrupts are disabled. Failing to do things on timer ticks. You disable your interrupts for long enough and your RTOS goes kaplooey. And if you're doing something critical.

Oh, this happens in the Nordic chips. Once you start the BLE going, you can't ever stop. It makes it hard to debug them because you can't walk through your code once you have BLE running. But any interrupt that takes too long will trash BLE

because BLE has very strict timing requirements. So you can't just print out something and have interrupts disabled around it. So you'd have approximately the same situation with the, what is it, the NeoPixels, where the timing is very critical so if you have any wobble in your in your timing uh your patterns will just be wrong yeah you end up with different colors you end up with some that don't get uh turned on at all or turned on only to a different color. And it's because their timing requirements are so precise

that any latency in the system that isn't them just causes problems. Using the NeoPixels and the Nordic chip is kind of hard to do together. But there's lots of these things that have timing requirements that are very specific. That's why we do embedded systems. If we could just toss it onto a Linux computer, all right, maybe we should do that. But when you have to deal with strict timing requirements and resource constraints, that's where you start dealing with interrupts that are very close in personal level. Yes.

Okay, but how do we protect the resources? We turn off the interrupts. That's one option. But that's not an option for when we want to share a resource between our main context and our interrupt context. Do I have an RTOS? You can call it a semaphore even if you don't have an RTOS. A mutex. A mutex.

A mutex. Fine. Yes. A mutex. Mutex. It's such a funny word. Which means, it means what? Mutual exclusion.

Not mutant explosion. Or something that comes out of your nose when you're sick. Mutex. Mutual exclusion is the correct ding, ding, ding. Is a thing. Is a thing. That you can attach with meaning to a particular object or resource. So a global variable or a spy resource, whatever resource you want.

It's connected to it. It's connected to it in your mind. Okay. Or in your, well, in your code, but not directly. And when you are about to use that resource or modify it, you say, you grab the mutex. I own this. And I have this thing. And I have the conch.

And you always do this for any shared resource, whether it's an SPI bus or the conversation at a party. Right. And so when you have this thing then you go to modify or do your thing that needs to be protected and then you give it back you release it so what happens is if i have this thing and i get an interrupt and the interrupt comes in and says oh i want to modify that thing it tries to grab the mutex. And however the mutex is implemented, whether it's through an RTOS or some other mechanism, it will try to get it and it will either fail and get an error back and say, you can't do that. And it will have to figure out how to wait. Or it will block until the other thing is done. Now this gets into trouble.

It does. Priority. Now you have your main system, which is sort of slow and not responsive trouble. It does. Priority inversions. Now you have your main system, which is sort of slow and not responsive necessarily. It's not an interrupt, right? But it's holding onto the mutex for something the interrupt handler wants to modify. But now the interrupt handler has to wait. Which we don't like to do in interrupts.

We've just taken something that we want to run in a short amount of time and unbounded how long it can run for because it's sitting there waiting for your main program, which, hey, you could grab that mutex and then go out for coffee. And remember, the only way to get back to the main is to do a return from interrupt. That is the other problem. Yeah. So you can't wait in your interrupt routine because that does nothing other than sit there and spin on something that will never happen so this is not actually one of the best ways to go about this

so basically well this happens assume for the moment that you have to use the spi in your main line and your interrupt routine the only way to do this is to turn off interrupt so that your interrupt routine can't fail or, sorry, your interrupt routine can't fire. Or you can mask just that one interrupt that is going to be sharing the thing so that particular one doesn't happen until you're ready to hand off the SPI.

What does mask mean? I'm not going to answer that. Let's continue. No, it means. No, that's what it means. That's what it means. I'm not going to deal with that right now. Gotcha. It's like going into a kindergarten and ignoring Johnny with his constant.

But only Johnny. Just Johnny. Well, there is a concept of a non-maskable interrupt. Yeah. Some computers used to have a button that said NMI right in the front. Yeah. Okay. So each each interrupt has

an enable bit and a flag that says the interrupt is is happening right now and typically a mask bit that says

look it's it's it's it's enabled but don't don't worry about it right now. So if the interrupt happens, the hardware says, I'm going to ignore you because the mask bit is set or however your processor implements this.

But basically, the interrupt is still active, but it's masked. And when the mask gets removed, the interrupt will fire. And we have non-maskable interrupts, which are things like divide by zero or... Processors on fire. Processors on fire is a good one. Typically, you'll also have an IO bit that is non-maskable. And that... So that's kind of the halt and catch fire button.

Yeah. So have we resolved this problem of sharing a resource between an interrupt handler and your main program? I think we've established that it's much too hard and we should all just go to the beach. Rub it in, rub it in. I would say that there's other better ways to handle it. If you have other things available to you, like an RTOS.

But I don't like turning off interrupts. But if you have an RTOS, there's ways to architect this so that instead of the interrupt handler dealing with this, it signals a high-priority threat, which can service it at high priority, but be within the context of the operating system, so that you can work these sorts of things out in a more reasonable manner. So basically what you're saying is I'm working with my SPI bus, I get an interrupt, the interrupt routine says I got to use the SPI bus, but I can't because somebody's got it. How about if I just leave a little breadcrumb behind and when I do a return from interrupt, the main line will say, is this breadcrumb set?

And then it'll do the work. Yes. Or someone else if you have someone else. But, yeah. Yeah. I mean, the RTOS makes nice bread. It can leave breadcrumbs for you. But you can also do this manually.

You don't need RTOS. This goes back to the thing we were talking about with do as little as possible in the interrupt handler and reduce latency. You probably shouldn't be doing the SPI transfer in the interrupt routine anyways. You should question whether there's something that's happening in your interrupt handler that could cause it to wait for something else and find ways around that. And it is okay to signal the main context, the thread context, with what you want to do. Where I switched out my button pressing because it didn't make sense to pull on a variable instead of pull on a hardware. Now we're starting to see where, oh, well, you can't do everything in the interrupt. You do want to just set a variable and deal with it later because you are in a different context and you may be having multiple things happen.

So it goes back to that. It's not as simple as, oh, well, why am I just pulling on a variable? You're pulling on a variable because now you're at the right priority and you have the resources you need. You can check with your mutexes. You aren't waiting in an interrupt. So there's a lot of good stuff about signaling the main line or the main loop to do what you want. All right. Have we confused everybody?

I don't know. The shortest possible explanation of nested interrupts and then one more thing and then... I ain't know. Let's, let's, let's, the shortest possible explanation of nested interrupts. And then one more thing. And then. I ain't doing it. All right. Well, that is pretty short.

Okay. How about if I just jump in here for a second and then say, this has been great. Thanks. What? Oh, we were supposed to interrupt him as he was talking. And then that would have been nested. No, I interrupted you as you were talking. Oh, I see. You're supposed to interrupt him as he was talking. No, I just interrupted you as you were talking.

Oh, I see. You were supposed to. Okay, so let's say I'm telling a joke. And Christopher interrupts and starts telling a different joke. And Andrea interrupts. Now, if we don't let him interrupt, then there's no nested interrupts. He has to wait until Chris is done with his joke. Even if Andrea's joke is more important.

You can have interrupts that interrupt your interrupt handler. Wait a second now. Isn't this going to cause all sorts of problems with stack smashes and stuff? Well, that's up to the processor to be able to sort itself out and to deal with the contexts appropriately. But it goes back to you can get into some pretty bad resource problems. Because when you're in an interrupt, you often assume you're the only one there. Between when you start and when you end, you are running the whole time. But if you have nested interrupts, that's not always true. This sounds hard.

It's kind of important because you don't want your debug interrupt to interrupt your motor control thing because... or your motor control interrupt. You want to be able to prioritize these. If you're going to have nested interrupts, if you're going to allow interrupts to interrupt interrupts... Sorry. If you're going to allow multiple interrupts to happen at the same time interrupting

each other then they're going to have to be prioritized and that that goes back to what needs to run what is its priority and i you know there are times when i have used nested interrupts but for the most part i try to avoid them because they do mentally make my brain squirm a lot of the times it's it's if you write your interrupts properly it's usually not an issue um some processors don't allow nested interrupts um like the some of the picks that only have one interrupt, you have to go through all sorts of backflips to get another interrupt. Some of the later picks that have two interrupts, they've got priorities so that if you're in a high priority interrupt, the low priority interrupt will wait. If you're in a low priority interrupt and you get a high priority interrupt come through, it will interrupt you. But the idea is stay away from global variables because you're in the middle of altering a global variable and you get an interrupt

and it plays around with the global variables. If everything is done on the stack, so this is the reentrancy that you were talking about. If you do reentrancy properly and use local variables, you don't really have to worry too much about nested interrupts. But if you've got a little bit more sloppy style and like the global variables, then yeah, you're going to cause yourself issues.

All right. Did that make any sense? It made sense to me, but I knew what you were talking about. So maybe we should ask other people if it made sense. Why are you looking at me? No, I wasn't really, because you were in that direction, I was waiting for the audience to answer. Going on, because that's insane. Let's see.

Oh, I wanted to mention about phone screens. And I know that this is, it was kind of a big topic shift, except that my main question that I used to ask people if I was doing a phone call with them to do first interview phone call was, what can you tell me about an ISR? And that was the whole question. There was no... We haven't defined anywhere in this whole show. We've never said that. Well, we're going to have to now, aren't we? But the whole question was, what can you tell me about an ISR? And it was good for hardware engineers, good for software engineers. And I was usually interviewing for embedded software. And then you just let the person talk. And if you want to do this formally or you want to help somebody else do it, you make a little bingo card.

And if they can say things like, ISR is an interrupt service routine, well, good, they've already got one bingo square. And if they say nested interrupts and NMI or masking or... They say early advanced concepts? I mean, there's some basic stuff they should be saying before they talk about any other things. A vector table was definitely a square. Context was a square. ISR, IRQ, interrupt request. So that's the little thing that the hardware has to deal with.

Yeah. So that was one of my favorite interview questions. I guess I didn't really have a point to that story. So, you're going to return from that interrupt? I guess so. I didn't know it was an interrupt when I started. I thought we'd discuss it, but then I realized there wasn't much to discuss. But I'll put the picture of one of them in the show notes because I don't know, it was entertaining. And if you have to work with a recruiter, things like that, where it's sort of vocabulary bingo, is kind of nice. It means that the recruiter doesn't

necessarily have to understand all the concepts. it helps if they understand some of them but it gives you a way to do pre-screens that are reasonably fair no that's good because it also gets back to last week's question about um what you should be doing to get yourself a job in embedded systems and i i think a lot of it comes down to you know treat it like a treat it like a career and you're preparing for a career and sort of dissociate that from treating it as a hobby so rather than just uh writing little things in arduino maybe you should get into advanced things like interrupts in Arduinos and having to deal with what the hell is, why is my button

bouncing and how to get around it. Do it seriously. And in the learning about all this stuff, you'll be able to fill in your bingo card. Yes. Maybe we should well, maybe someday in the blog we'll make bigger bingo card. Yes. Maybe we should, well, maybe someday in the blog

we'll make bigger bingo cards for embedded systems careers. That would be kind of amusing. And that was part of the goal of the show is I had promised last summer that we would have some shows about more moving from hobby to the larger technical concepts.

And I was going to call them basics or something. And they do have the basics tag, and this will get the basics tag. But it's a lot more fun to just talk about the stuff instead of trying to do a formal lesson curriculum. Well, maybe someday, but not this year. All right, well, I think that covers most of interrupts. There's a lot more to it, of course, but those are the basic terms we should be looking for and maybe some ways to discuss them. Chris, do you have anything else you want to?

Oh, God, no. We've lost Christopher and Joe's phone. Andre, any thoughts you'd like to leave us with? Yeah, interrupts, they might seem kind of daunting, but if you go through, I think even in my blog posts, I get into interrupts a little bit. And the after Blinky comes reading the button and then interrupts

go through it it's it's it might seem daunting but it's a very useful tool and it's it's a good thing to have in your quiver of tools. I totally agree. Our guest has been Andrej Cicak of the Great White North, author of the Embedded Wednesdays blog on the Embedded.fm site, and an embedded systems developer at CBF Systems. Thank you for joining us, Andrej. Thanks for having me. Thank you to Christopher for producing and co-hosting.

Thank you to Elizabeth Brenner, Andre Cicek, and Chris Veck for essentially writing my outline this week. And, of course, thank you for listening. And if you have any questions, please give them to us, and we'll put the answers in the doobly-doo down at the bottom. That's true. Did I interrupt? Sorry. You did. But there will be show notes, and we do appreciate questions. Except about RTASs.

Nobody appreciates those. I'll do it. Okay, so I have kind of a long quote to leave you with. This one is from The Velveteen Rabbit. It's one of my favorites. That's why it's so long. You become. It takes a long rabbit. It's one of my favorites. That's why it's so long. You become. It takes a long time. That's why it doesn't happen often to people who break easily or have sharp edges or who have to be carefully kept. Generally, by the time you are real, most of your hair has been loved off

and your eyes drop out and you get loose in the joints and very shabby. But these things don't matter at all, because once you are real, you can't be ugly, except people who don't understand. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.