Embedded - 170: Electron Gnomes

Episode Date: October 4, 2016

Elecia tries to get Chris to do her homework in preparation for her "Embedded Software: The Tricky Parts" presentation at IEEE-Computer Society meeting in San Jose, CA on Oct 11, 2016. If you register, you can attend, in person or online! And for free!  We have some tickets for ARM's mbed Connect conference is Oct 24, 2016 in Santa Clara. Will you be in the area? Want to go? Contact us if you want one of our free tickets! (There are still some tickets remaining.) Also: their unit test framework is GreenTea (not whatever Elecia said).

Hello, everyone. Welcome to Embedded. It is just us this week. And by that, I mean myself, Alicia White, and my co-host, Christopher White. Hello. There were lots of announcements last week. There were? I guess there were. There were. There were several announcements. Let's start with the contest winners, since I can announce those.

We're all winners. Yes. Except for the people who didn't win. Well, I mean, they're winning books, which is really nice. But anybody out there who needs a book to read, there's this thing called the library. I highly recommend them. Who won and why? Cyril got within one number of my minus eight.

Will was the closest to Bob's 17. And he will be winning Every Anxious Wave by Bob's ex-wife, which did actually sound like a pretty good book. Cyril will be winning Atomic Accidents from Bob again. Roy guessed your perfect number of 496, like right after you said the words perfect. So he'll be getting Safewear. And Adam Grant will be getting Earthquake Storms because John Lehman did not choose 42,

despite the whole don't panic thing. He chose the coefficient of friction for everybody who is not in his lab, which is 0.6. And Nick, Nick the Exploding Lemur, will be getting Soul of a New Machine, because while his number of 32768 is not the same as Christopher's 128, it is clearly the closest in spirit.

All right. There were too many things near that, but nothing really close okay so those are the books those are the announcements thank you Bob, thank you John for giving away books that's awesome thank you for everyone who sent us funny numbers

yes I learned new numbers it's weird you know you can learn new numbers just all day long. 1003, 1004. Those aren't new. Just keep adding one for a while and you'll get there. Eventually, yes. But no, there were interesting numbers. And somebody did guess E plus I. And I will have you know, Warren, that I did take the magnitude of that in order to see if it was closest to any of the numbers. That's very kind.

You shouldn't have done that. But next time we should specify, you know. No imaginary numbers. Only reals. Only reals, yeah. Rational. Or only non-reals. We'll see.

So, done with that contest. moving on to another one what it's just contests forever i know it's kind of odd but kind of cool i get to give stuff away what are we giving away now well this is the embed has their conference in santa clara it's the connect conference it's a day long. Embed is awesome. Here's how you use what we do and here are our boards. And just welcome to Embed. Sort of conference.

And Embed, for those who are joining us late. Embed is a branch of ARM, the processor IP vendors. And they... You mean SoftBank? Man. branch of ARM, the processor IP vendors. You mean SoftBank? Right, SoftBank bought ARM, but I'm just going to ignore that. And Embed is a compiler that's online, and they qualify different boards to work with their compiler.

C++ compiler. It is a C++ compiler. They have a huge library and community of various people doing drivers, so it's a good place to start if you want to have an accelerometer and some buttons and some lights, and you go to embed and you get your core processor that's a STM or an NXP processor, and then you... N core processor that's a stm or an nxp processor and then you nxp about to be bought by qualcomm i can't keep up um and then you you put it all

together on embed and there's lots of existing stuff out there so how so we kind of recommend that for kind of the next baby step up from Arduino, usually, right? Yeah, it's not quite a baby step. It's a bit of a hike, but it's a definite step. A toddler step. Yeah. The preteen step. And they're going to unveil some of their test-driven development.

No. Unit tests. Unit tests. Which could be used for test-driven development. At their conference. And their conference is in Santa Clara on the 24th of October. And we do plan on going, Christopher and myself. And it's $25 if you want to go.

But if you email me, I don't know. Christopher wanted a screenshot of your iTunes review. That was a joke. Oh, Christopher wanted a screenshot of your iTunes review. That was a joke. Okay. But just the first, let's say the first five people to email me, they gave me a couple more than that. So if you're late, you can still get in with the sob story. But let's just go with the first five. I want you to say, I'm going to be in Santa Clara on the 24th, and I'll happily give you the code.

Okay. Works? Yeah. Okay with you? Yeah. You don't want to make them too weird? No. Okay. I also mentioned last week I'd be speaking at IEEE Computer Science about the tricky parts of Embedded. That is October 11th, which is shockingly close to now given how little I have actually done on this talk. You're not supposed to reveal these things. Well, all I have for the rest of our notes is I haven't really started my talk. Christopher, will you give me some advice?

Will you do my homework for me? I gave you a list of the tricky parts. Yes, Christopher's list of the tricky parts starts with frequent electrocution. Yep. It's a problem. How often have you actually been electrocuted as an embedded systems engineer? I might have forgotten. It often scrambles your short-term memory.

You have, in fact, been— Could be daily. —poked by needles of uncertain origin which was an adventure that was terrifying for a little while that wasn't really the embedded system's fault well if you'd been a web software person

was on the cart with the needles that was a bad company that was a bad company bad company rolled newspaper up and hid it and it's no but that that would fall under a different category uh electron gnomes was the other one yes electron gnomes which um i definitely am going to talk about in my presentation although maybe not as that maybe i will talk more about the difficulty of debugging systems. That would be gremlins. They're totally different.

Oh, really? What are electron gnomes? They're gnomes that are made of electrons. Shmebulak, could you be more specific? No, I just made that up before. You expect me to have a backstory? Yes. That's not how this works.

You've had days to think of this backstory. Well, I am going to... They're gnomes that steal your electrons. And then your power sags? And then question mark, and then they profit. Yeah. Well, I am going to talk about the difficulty of debugging embedded systems and the wonky cosmic ray sorts of bugs.

You're getting into real details now. We can't intermix this. This is the entirety of the notes? Well, yeah, I want you to write my talk. I thought you understood. This is the shortest set of search notes we've ever had. Because I have to give an hour-long talk. Well, let's move on to the rest of the list, because we obviously need to fill time.

Well, no, electronomes. Yeah. I'm going to pretend they're gremlins. I'm sorry that you have a differentiation. I just don't. No, that's fine. Reverse time anomalies, which I believe Andre covered in his blog post last week. Sure.

Did you have anything else to talk about? No. So reverse time anomalies, I think. I'm actually going to use Christopher's list as my outline. So if you come to the talk, you may see me making fun of him or him throwing pizza at me. You never can be sure. Or both. Or both.

Next up was glue, which you have to admit sure. Or both. Or both. Next up was glue, which you have to admit is a common problem. And I was going to talk about glue logic because I have this whole framing idea. Can I believe you're building real things out of this list? Yeah, I have the whole framing idea. And I'm going to take, I think I've chosen a Nordic processor, a BLE processor, and WS2812s, the little smart LEDs.

And I'm going to pretend to make an Internet of Things thing, and it's going to be a BLE LED. And then I'll discover that you, discover, quote, that you really can't do this because the Nordics operating system doesn't give you the granularity in timing that the LEDs need. So you stick on another processor, a little 18 tiny that's, you know, tiny and cheap. And now everything becomes so much more complicated because you have another processor to bootload. You have all this glue stuff that before you had pretty simple, turn the LED to blue, turn the LED to red, get it from the BLE and blah, blah, blah. But it's all that glue in between them. And then being able to do firmware downloading and gluing that into the controls. I don't know.

No, it's good. If you do be a leader, you're going to have to talk about the controller side. The phone, the app. Yes. Yes, actually, that was number seven on your list. Oh, all right. Number five was the electrical engineers, which you have to admit is a problem everywhere.

I didn't really I know I mean initially when you wrote that I thought it was a misspelling for electrical engineers no

so what did you mean by collectical engineers I meant it sounds like electrical engineers

so it's funny okay and so to make this as part of my talk I'm going to have to talk about how an embedded system engineer has to be

interdisciplinary you have to be a software engineer you have to be an electrical engineer you have to be able to talk to the mechanical engineers because could you please move that bolt from where my serial port plugs in and so I guess a collectical engineer alright

we'll keep that. Memorization of the liturgy. That one's easy. I mean, that was actually in my outline before you suggested it. Because. In those words. Well, no, I had acronymophobia. but we go through so many of these stupid k60 fn1 xx12 and why in the world do i need to remember all of those numbers and letters together can't we just call them like

stupid usb processor wait no that's good. I don't think that's going to work. No, inside companies, they probably have memorable code names, but once they get productized, they're, you know. I mean, I know that that one is named the way it is because it's in a family.

And I have learned that the K60 of this particular variant is closer to the K70 than to the other K60s. And so I have this one that's sort of an anomaly. And I understand how that happens when you set up the parametric chart and you go here and here and here, and it all works out so that you get these sets of letters.

It's the best description. It's a thorough description. But, man, I hate remembering them. But it's only a thorough description to somebody who's familiar with that part. And you have to remember everybody's scheme. So that's silly. Well, and you mentioned that SoftBank bought ARM, and that hasn't changed anything for anybody yet.

No. And it won't. I really needed to know that NXP had bought Freescale because half of my documentation was, or my documentation was all in NXP now but half of the forum information was still Freescale so I had to, you do have to memorize the liturgy, you have to remember what's going on

in your processor and your processor's family and all of your sensors and their families. I mean, they say that engineers are isolated, but we have lots of families we're talking to all the time. No? I don't know if that was working. Number seven? 4% chance of decapitation. Okay, this one's going to start with a story I think is that okay with you you can do whatever you want

so I heard somewhere that in a survey that 4% of respondents said they had been decapitated and I found this an amusing example of how surveys are stupid and how they just don't give as precise information as we give them credit for giving and so i told chris this

hey honey did you did you know survey respondents four percent decapitation i'd already seen the survey so i was prepared for for this conversation oh see. You were yanking my chain with forethought. Of course. I did not know that. And Chris says, you know, decapitation is a medical term. It doesn't mean that your head is rolling on the floor. And, you know. You took a front to this. We had some discussions about what percentage of the population had been medically decapitated and were still able to respond to surveys.

There's an upper bound. It's no higher than 4%. And eventually I said this was the lamest well actually I had heard in a long time. Which it was not, I pointed out. Because he had used neither the words nor actually. And well actually, a well actually is defined having used the words well and actually. I managed to meta well actually you. Yeah, yeah. I don't see why you don't appreciate this. I managed to meta well actually you. Yeah.

Yeah. I don't see why you don't appreciate this. I must appreciate it. I told the story. I think you're just trying to humiliate me. And that was all I had on the very quickly thrown together list. When you asked me this question to do your homework for you, and I took it seriously, of course. i'm not sure you took it seriously so i don't have an actual well actual uh place to put the four percent chance of decapitation uh unless i i don't know

how to fit that into my tricky parts of embedded talk. You'll come up with something. You've got a week. Oh my god, I've only got a week. That's terrible. Thanks for that. I am supposed to talk about processor profiling, map files, bootloaders, and debugging remotely. I know that because I wrote an abstract long before I started the slides. So, you know, it's a really large set of large topics

to go through in an hour. Well, I was going to talk fast. Yeah. I noticed one thing you don't have in here, despite number three on the list, is anything about timing. Time is hard. You mean calendar dates?

Both. Yeah. So if you're doing a control system in real-time stuff, that's difficult. We talked about bit banging. That's a timing thing. That can be difficult. But then calendar time is always a nightmare if you have anything you need to have a real-time clock for. And if it's an actual product that needs to be used anywhere in the world, that gets even more difficult.

Well, the precision timers can go in when I talk about how you have to add a processor for the LED control. Because you have an RTOSOS and while R and T stand for real time in the operating system, that just means you have a consistent response, not that you have a fine-grained response. And if you wanted to talk about the

calendar time stuff, you could bring up the old holiday configuration scheme for the light or something like that um oh right i could do that see oh what you should describe what it was oh uh that was your your scheme years ago to make christmas light well not christmas lights but holiday lights that you just put up once and then they know they have an internal calendar and they had color changing LEDs

and they knew what holiday it was so they would automatically change to red and green or blinking for Christmas bright green for Arbor Day brown for Thanksgiving you came home and you knew it was Valentine's Day was coming because your house was starting to pink.

Right. Yeah. Yeah. Except LEDs were slightly too expensive to do that with. Oh, isn't that? It was the control system. Yeah. It was before the WS2812s.

This was like 2006 I was working on this. And I was looking at PWM-ing LEDs in bundles of eight. Oh, right. And so you had 24 wires going through the cable. What a nightmare. And not counting ground. And it was just not tenable to build and organize and keep running. And then somebody came out with it a few years later. Yeah.

But it wasn't hooked into the calendar. It was just you could choose some subset of things. I think you actually hacked that up, right? Didn't you give a talk where you took that and hacked it so it could do what you originally wanted? I did. I hooked it to a wi-fi board right right and i didn't do the original hacking on it uh deep dark did i think that was who it was um but those were i mean he he used a logic analyzer and just looked at what the protocol was and wrote it up and did a beautiful job. And sometimes when I'm looking at Hackaday stuff and thinking about what I want to write, I think about that. I want to write something like that because it was useful to so many people.

It wasn't just me and my silly holiday lighting idea. All right. So yes, calendar is horrible precision timing is horrible what else did i miss well i mean you have the obvious things listed right i'm trying to think of the non-obvious things or the things that somebody who's an experienced engineer who maybe comes across an embedded project for the first time doesn't realize. I mean, one of the things I think of is the speed of iteration can be very slow because either you have to download to Flash or you have tools that are very slow

or your processor's not very fast or the deployment of new code takes a long time where a long time is anywhere from five to ten minutes. That's a lot different than standard kind of engineering practice where you have an application and you can compile it and it maybe takes 30 seconds to compile and then you restart it and you find a bug and you can iterate really, really quickly that way. Well, then when you do task-driven development, you have

to decide whether you're doing it on the target processor knowing it will be slow to deal with, or if you're doing it offline, which will be faster to compile, but may have problems when you don't have the same int size. Or

not even the same tool set. I mean, a lot of times you might use GNU tools for your unit tests to compile and you might use an embedded tool set,

an embedded compiler for your device. So you can get into other issues there too. But that, I mean, that's a little

corollary to the original problem, which is things are slower. And so it kind of gets back to that, you know, development habits of being much more careful. Because if you can iterate really quickly, you kind of fall into the trap of,

well, I don't have to really think about this because I can just try, you know, if I have a particular bug I'm trying to solve that might have, you know, it might be some combination or order of code. If you can iterate really quickly, you're always tempted to just try every option until you find the one that works without maybe understanding the actual problem and solving it through thought.

Yeah, I've been working on a command parser and I have to admit to do the checksum. I read the description and I thought, hmm, I'm not going to get that right on the first try. It's not hard, but there are like four ways I could implement that. I should try them all. Yeah, yeah. And sometimes that's fine,

but, you know, I think it's not necessarily a bad thing to have something external forcing you to slow down a little bit. So I guess that's the other side of the slow deployment and test cycle time is that maybe you should think a little harder. I don't know if you want to talk about that, but something that came to mind.

I'm trying to think of what else is difficult that maybe isn't obvious. You know, dealing with actual hardware instead of an amorphous computer or a server somewhere. Well, I do want to talk about actual, I mean the protons in the electronics, that while you may start with an off-the-shelf dev kit and you may connect it together and it all comes together fairly easily, when you have a custom board,

it's hard to debug because you don't have all of the signals going from one place to another. You can't look at them anymore. They're all on the board. And if you've miniaturized the board, they don't come out necessarily. You can't do that in a computer either.

You can have variables you can watch. Oh, I see what you're saying. So your visibility into the software isn't as good as... yeah. Right, and a lot of JTAG

debuggers and things, you're limited to a handful of breakpoints, maybe one breakpoint, depending on the processor and the tools, right? And maybe even less in terms of watches. So that's another limitation. But that's part of the debug thing, right? Yeah, I am going to talk about the difficulty of debugging. But what I was talking about with hardware

is just, you actually have a physical thing that you can ruin. Oh, the cost of actually being offline for maybe a long time because you only built ten prototypes and now you've trashed nine of them?

Well, I mean, it depends on who you is, but just as an embedded software developer, I mean, if you have who you is, but just as an embedded software developer, I mean, if you have a bare board on your desk, it's not the same as a computer. There are ways you can, I mean, depending on how it's designed, there are probably ways you can,

through software, ruin it. Those darn Flash. Right, if you have a Flash system and bootloader that's maybe not quite robust um you can break a system which requires you know an electrical engineer with a soldering iron or somebody with different access to fix but it's different from developing applications or anything else because you can't really break a computer with your software and the consequences of being wrong, dumping your coffee on your board,

which is bare on your desk, is higher, especially if it's something custom, right, where you've made 10. You know, those few initial prototypes are so expensive. Right. And so people don't order a thousand of them unless they're a big company or this is a minor iteration. And those board costs have come down a lot, but it's still...

It still costs you time. It's still a lot of time and it's still a lot of people... I don't know, when I mess up writing software on my computer, I seldom have to go admit it to somebody else. The flip side of that is when somebody designs a board, if it's a non-form factor board designed for development, there are ways to make it not ergonomic so you're more likely to break it. I'm not looking at anything on my desk right now.

That's a thought process that the electrical engineers sometimes need to put into it. It's like you can't just throw something together and say, okay, all the parts are there and it's going to work. Here it is to some software engineers who, we aren't necessarily the most careful folk with physical objects. So if you've got boards sticking off at right angles and daughter boards and things.

Or when they put all the connectors on one corner of the board and they don't key any of them. That's okay. Having all the connectors on one side is great. Well, it's the whole keying thing because I will absolutely plug it in wrong. But if you've got three USB connectors and they're on different sides of the board and the cable has to go over something you have to interact with, I mean, it's just stuff where you're going to be bending something or touching something and you really don't want, you know, these are not designed to be robust and banged around.

So this is maybe a request for future electrical engineers but have a little empathy because you're going to be the one fixing it yeah there are lots of times where i get boards that just aren't designed to be robust and yet i'm supposed to do extensive testing i'm still marveling at the hydraulic press thing in your office. I don't really understand that. I don't think we can talk about that. Didn't say what it was. Yeah, it is some sort of manufacturing device that has to do with pogo pins and it is ginormous. It's the sort of thing usually done with thumb screws. Well, that project has got plenty of thumbcrews, I can tell you that.

All right. Well, what else? What else are you going to talk about? You want me to continue writing your talk for you? Yes, please. You didn't tell me this was going to be a lot of work today. I'm sorry? I'm sorry?

Next week we have a guest. We'll just make him talk the whole time. In fact, if you want to go kayaking before or during, we can do that. Don't think that's going to work. No. Last I heard, you wanted to go listen to a blues show while we were recording. Not possible. What else is hard?

It's all hard. Well, downloading firmware. Oh, God. is just one of those things that when you make your proof of concept and then you want to get from your proof of concept to a shipping product, the downloading firmware box ends up being such a stumbling block

for so many people. Well, it's difficult. And it depends on your flash architecture and on your security laid out the bootloader and the bootloader if you've just done a proof of concept you may not have thought about any of those things no because you know it worked it had all of the functionality we said we wanted it it lit up when i did bleep ble stuff so therefore we're ready to ship, except you're not. You have this huge bootloader thing that can brick things,

and if you have multiple processors, how do you download for the secondary processor? It's a mess. So I was actually going to take some from my book for diagrams of different bootloaders. Has anything changed since then? Well, the Nordic system, when I last used it, they had lots of features to make downloading bootloader,

or they had a built-in bootloader that made downloading firmware a little easier. And they used a ping-pong method, so it was pretty tough to brick your system. And that's good, but then it did take a lot of space. And so many people are using the free Kyle compiler that is space limited. And so if you do that, then you have to make sure you have all of the space in the right parts. And you have to understand how to put Flash together so that you can have multiple programs in the same Flash

that don't count against your compiler because compilers are huge and they're dumb and they're expensive. They're not all expensive. No, no, but the reason I recommend using the Kyle with the Nordic is because that's where all the sample code is. Right. using the kyle with the nordic is because that's where all the sample code is and it is more expensive to port everything to your personal favorite compiler than it is to get used to that one i know somebody's gonna tell point me to embed actually has a nice uh nordic system and yet i

still use kyle if i'm doing it professionally And I know there's a GCC version, but I feel like Nordic and Kyle work together. And as part of the liturgy memorization, if I know that, then it makes my life simpler. Yeah. I'm just having bad memories of Kyle. I don't get attached to them. I don't get attached to them. You don't get attached to them?

Well, I mean, I don't. I haven't gotten attached to any of them, that's true. Yeah. Other than that whole sample thing, I want to inject, pretend, I don't know, talk about, let's say you get a bug in the downloader. Now you have to decide. And you say, oh, well, Nordic has an update and it does what I want. It's better.

But then you go and they've updated all the files, all the interfaces. And do you update everything? Do you update your code to the latest which may have new bugs or do you stay where you are and live without this feature you wanted this sdk and sample changes and they don't necessarily have versions and this isn't just nordic i don't mean to slag nordic they do a fine job as good as anybody does uh this happens with ti it happens god it happens with everybody the free scale thing uh or the nxp k60 whatever they changed the usb library and they didn't change all of their examples they only changed some of their examples and it was awful. And yet

that is something that I don't know if other software engineers have to deal with, but it is really common in embedded systems. Usually, no. I mean, the types of things you're piecing together for applications, say, are either system libraries and things. And those are carefully versioned, and they'll have the care to deprecate functions but still leave them operable for two major revisions or something. And of course, all the examples are usually updated much more comprehensively than that sounds like is being done for those things. I think it's a much less common problem.

I mean, it does happen. Well, I mean, there's Python 2. The 3.0 thing. But that, I mean, that's kind of the example. Everybody points to it and says, don't do that. Yeah. But, you know, I'm thinking about like Qt, which I'm not going to call cute, sorry.

You know, they made a major revision-breaking thing a couple years back. But they do a good job of documenting, you know, transition. Like, this is how you transition your code to the new system and the new APIs. They continue to support the older one. And it's not like a situation where you can't... I mean, they do have features that you want to grab from the new one.

So if you're compelled to move to a new feature because of that. But it's not like they've come out with new hardware, and if you want to upgrade your system to that hardware, now you have to rewrite all your software. So that situation doesn't happen as much. I think it's a problem, but I think it's much less a problem because I think the embedded hardware companies, their goal is to sell the chips.

We should be happy they gave us sample code. I mean, that's actually a pretty new thing. Yeah, it's a better situation than it was, I think. Seriously, if you're going to give me sample code of a UART driver, can't you include and interrupt? Well, that's the other thing. And tangentially, I was going to bring this up. You know, somebody coming into embedded engineering

might be surprised at how much reinventing the wheel we have to do with every project. And it's kind of painful, and I wish it would go away. Oh, yeah. But if you're coming from, you know, if you learned modern development recently, you tend to piece things together from other people's frameworks, and there's a lot of stuff available

depending on what language you're writing in and what target. There's a lot of stuff where you can just say, oh, I need this library to do this kind of thing, or there's this, you know, container or object that do this kind of thing, or there's this container or object that does this kind of database for me. So it's really more of an architectural problem than a tactical coding problem a lot of times. And with embedded, it's still very much a tactical coding problem

where often you start a new project and it's, well, here's my blank slate, and all right, I need a UART driver. I mean, certainly if you're developing on our embed or Arduino or, you know, a platform that's got a community surrounding it, that tends to be better. But I think if you're coming at something from a custom build point of view, which still is a lot of professional development, there's, you know, pieces missing that you've got to code yourself.

And you may find yourself getting something that's maybe not perfectly suited to your needs, like a real-time operating system. You might say, okay, this is good, but this messaging primitive doesn't work quite the way I want, or this interlock mechanism, or whatever. And you end up having to build wrappers around your own. I mean, that happens in all kinds of development,

but I feel like it happens more still with embedded where, oh, I have to do a linked list. Well, I'm going to write yet another container, you know, linked list storage class or whatever, whatever you want to call it and see. I think we're still doing that a lot, and that's painful and costs a lot of money. But, you know, UART, circular buffers, serial drivers, spy drivers,

I mean, that stuff's all getting written from scratch still in a lot of projects. In a lot of projects. There are a lot more samples than there used to be. Yeah. But they never quite do what you want. And so if the code is not well written, then it is easier to rewrite

than it is to read. And the samples are usually the baseline functionality. It's not going to be necessarily low power. It's not going to be necessarily high performance. They didn't even use the FIFOs, man. They have built-in FIFOs. They didn't even bother. That's what I was about to say.

Your chip might have special peripheral features, special DMA modes, other ways to do memory transfer. And a lot of times, the sample code doesn't take advantage of that. So you've got to take their sample code and spruce it up. And that, I think, doesn't happen as much in other

development worlds where it's like, okay, we're trying to make the software as the product in normal application development. So somebody making you a library, that's their job. They're making a library and they want it to be as comprehensive as possible. Whereas a chip vendor is like, well, here's how you get started.

We assume that you're a professional engineer and have written many drivers. So this is where you start. And if you want it to actually do what you want, you might have to add something. Well, and so this is definitely, I'm going to talk about this because this is very much on the way of,

I built this thing out of off the shelf parts. You built the city on rock and roll? And it works. But the difference between it working and that proof of concept and actually shipping a product, there are all of these gnarly little tricky things there. And that's what I want to talk about.

There's always minor stuff, right? You might have something out of the box that works for some configurations. But let's say you've got software and you've got a display driver and turns out you want the display to be 90 degrees or upside down. Well, now you've got to dig in

and actually fix the display driver or rewrite it or what have you. So that's where your physical things in the embedded world can impact your software in ways that maybe doesn't happen in other sorts of development. It's not like anybody comes up and says, well, we want for this application for the monitor to be rotated slightly. Well, and if they did, it would be part of the computer.

It'd be a driver. It would be trivial. It wouldn't be, oh, let me recalculate how I'm displaying things and change all of my bitmaps to be turned or, yeah. So part of my talk is going to be about map files, but I actually think that's one of the ways that embedded systems are less tricky than people think they are. So when you run out of RAM or you run out of flash space, there's a file that tells

you what everything uses. And if you know about that file, it suddenly becomes much easier to find out where all of your RAM went. And since you're on an embedded system, RAM is a fixed and precious resource, depending on your system. And so being able to read a map file or just not being afraid of going in there and trying to puzzle it out is so worth it. And yet I still meet people who, they get lost at that point. They get frozen, almost in fear. And no, no, no, this part isn't the tricky part. I can tell you where your RAM went. And I love map files.

It's just, but it's not something that they necessarily tell you about when you're learning software development. Not anymore. I know, not anymore. There were different kinds of intermediate outputs and things I remember before. Yeah, well, you kind of buried the lead there too because somebody coming from another development discipline is like, what do you mean I can run out of RAM? What do you mean I have unlimited out of RAM? Well, yes. What do you mean I have unlimited RAM?

What are you talking about? What's this K behind the number of things I have RAM of? Yes. And then I do want to talk about profiling, but I'm a little hesitant about that. I've never, well, I mean, I would be interested to hear what you have to say because i don't haven't had a lot of success with profiling tools so i always fall back to doing stuff by hand i think i actually in in my rough draft of my slide here i have something about

um you end up uh looking at how much the profiling tools cost and you say, oh, well, never mind, I'll do it myself. And then six months later, you look at how much money it took you to build them yourself. Sometimes. Depends on what you need. Most of the time, all you need is to time certain sections of code. Yes, and that isn't very hard. I mean, making a little timer system and then you run it over this command and then you print out or store or log or however you're getting your information and you say, okay, this part took one millisecond, this part

took 40 milliseconds and then you optimize the 40 millisecond part. If you're smart. Well, if you have any cycles to take out of there. And then the other profiler way the way that usually is if you are buying profiling, is to have a random timer tick. And for every timer tick, take a sample, the program counter before you're interrupt.

And then you get a view. You get a statistical sampling of where you spend most time. That's interesting. That depends on having enough samples, right? Yeah, you need to have a pretty big buffer

and then you probably need a program to take your map file and this buffer of addresses and tell you which need a program to take your map file and this buffer of addresses and tell you which functions you're in because that's no fun at all right and you have to have enough storage to log meaningful amounts and but no i like that idea um i think somebody did that recently i mean there's lots of ways there's lots of people who have done that. No, I know. I was just remembering recently that I came across that, and it was cool. Yeah, and those are things you can do yourself. Yeah.

I mean, the statistical sampling one, you do kind of have to commit to writing the interrupt, trying to figure out what random means for your system. And then you're still not sampling anything that has interrupts turned off. And so you have to be aware of that. And you do have to have that post-processing output where you merge with the map file. Which, I mean, I would write it in Python. There are map file readers in Python now. And it's not that hard.

But you're not talking about an hour's worth of work. Yeah. In other situations, you may find yourself needing to flip a GPIO and time stuff on a scope. I've done that a bunch of times, too. Yes, and you know what? I forgot to put that in these notes. And that tends to be, you know,

if you need super accurate timing, because you're doing... Oftentimes when you're profiling stuff, you think, oh, well, I'm just going to find out that this operation took a long... Like you said, find the thing that takes the longest time and optimize that. Maybe the thing that takes the longest time is composed of 50 different small-time things, and you need to see how often you're in one of those. And so it's not that you're going to get one 100 millisecond long operation,

but you're going to get something composed of hundreds of microsecond or 100 microsecond long things. And those you got to dig into a little more. So sometimes you have to use a scope for that. Yes. And using a scope or logic analyzer is important. And that's a tricky part that I don't know how to explain. Because as a newer engineer, it took me a while to understand that sometimes I had to acknowledge I wasn't going to figure this out by typing at it.

And I really needed to spend the two hours or whatever to hook up the scope and get the capture. And then it would be a five-minute solution. But it's hard to agree to that upfront cost. Yeah. Well, and it's realizing that you're blind, right? It's realizing there's some state of the system that's either too fast or that you have no visibility in, or it's controlled by hardware

that you need some extra external tool to reveal to you. And that doesn't happen much in application development because you've got visibility into everything the OS provides you. Hooks for tracing and more the OS provides you, hooks for tracing. More modern OSs even have really, really complicated tracing capabilities that give you all sorts of timing information about what function you're in, what line's happening. But if you've got a spy bus that's not working, or if you've got something else that's interacting with signals,

you can't see that from software or you don't realize that it's your printf function that is using a stupid uart that is polling for when it can send bytes and it's making your whole system incredibly slow because it's it's doing everything the dumb way and when you start out with your proof of concept, you don't care that it's going slow. You're not trying to save power. You're not trying to make things go fast. So, okay, we can get off profoundly, I think, but power. You know, one of the things that I want to talk about with power

is you shouldn't try to measure, you shouldn't try to optimize things that you can't measure. Oh. And so you need to understand how to measure power, and it's not always easy. Low power things are the hardest things to measure. I think that's the trickiest thing, is dealing with trying to optimize power.

Of all of the things, that is like master class advanced. So do you mean the difference between sleep and hibernate and super low power? Getting sleep states, right. Measuring to prove that you're actually doing what you think you're doing. You know, finding where you're finding where you're burning the most power in your code. And figuring out if that's where you should be.

I mean, if this is the purpose of your device, then if I have my LEDs and the LEDs are on, I don't necessarily need to optimize my processor to take no power because my LEDs are burning everything. It's one of those instances where you should only optimize when you need to. Yeah, but that tends to be a product-breaking optimization, right?

If you're battery-powered, you have to work on this. Because either you're going to be constrained by form factor, what size battery you can put in, or there's some requirement that this must last for this long. But don't optimize the use case. Don't optimize the LED on case. Optimize the sleep case. You have to optimize something for the LED on case.

You have to optimize battery. I mean, optimize doesn't mean making as small as possible. It means getting things right. Okay. So you may have to say, oh, my LED needs to be on, and that's dragging all the power, and there's no way I can change that. So that means I need to go to a battery that's X amount bigger than I anticipated. Oh, okay, I see where you're going.

I was going more with your battery's fixed size when you are looking at which states to optimize. Trying to optimize your short amount of actual runtime to be even shorter is not as important as making sure your GPIOs are in a low power state when you're in a deep sleep. Because it's the deep sleep you spend 95% of your time in. And so you want to figure out where it is important to

optimize. And again, that means you have to be able to measure it. Measuring things like nanoamps is a pain in the neck. Well, and there's the counterintuitive parts of optimization where you sometimes have knobs on your processor that could

slow it down, for example. Yeah. Oh, I can save a lot of power if I run this at 1 megahertz instead of 12 or 48. And so you turn things down as often as possible, but then it doesn't save you power. And the reason might be because it's taking a long time to do something that normally it would only be awake for a short time for. So you're at one megahertz. So you spend like, let's say we have a difference between low power and high power, one megahertz or 12 megahertz. So at one megahertz, it takes 12 times as long to execute some code. Well, you're out of sleep for 12 times as long.

And so it's actually better to stay in the high-powered state when you're awake to get your stuff done and get back to sleep as quickly as possible and so these are all features with ramifications and you have to think about all of the ramifications you don't necessarily want to go to sleep if you're going to wake up a cycle later because there's an expense to going to sleep and an expense to waking up and it's not it's not instantaneous and so it takes power weirdly you have to balance whether or not you should even bother to go to sleep well or or you know if you have a badly optimized a badly designed system from a software point of view, you might have just random events happening unpredictably. And you might wake up for one and then go to sleep

and then immediately wake up for another, like you were saying, and just have them distributed through time kind of evenly. And so you're actually awake quite a lot of the time. And spending that fixed cost you're mentioning of powering up and powering down, you're doing very, very often. So you've wasted a lot of power just waking up and going back to sleep. But going back to the time thing, if you consider how to bundle all those operations and say,

okay, I'm going to wake up every 10 milliseconds and do everything that was waiting, and then go back to sleep's that's been you know that was waiting uh and then go back to sleep that's a lot better because now you've you've reduced that state change to just a few times rather than constantly doing it anytime something happens and so you end up not turning the interrupt on you turn the interrupt off and instead you wake up on a timer and you pull the system yeah which is so counterintuitive. It looks terrible. From an architectural point of view, it looks very inelegant and sort of stupid.

But it's actually the way you optimize power. And yet, I mean, yeah, the tricky part of embedded systems is that they are resource constrained, which means you have to optimize for particular resources. Whether that's battery and power or cycles and power or cycles because you just don't have enough or RAM or flash. I think that that and remote debugging are the two really hard things. People don't understand about different debugging methods, and it's... You also haven't lived until you've done remote GDB. Have you done remote GDB?

Yeah, but... Yeah. Your Telnet server to your device? Well, let's see. Other things I have in my list are the non-embedded software parts or the non-embedded parts of your project. There's the mobile app, which if you did your proof of concept, your mobile app might have been one of those BLE interface things like light blue,

which just toggles characteristics and isn't pretty. But if you're shipping a product, working with that mobile app to get it right, to let it do firmware updates, to let it have reasonable security, that's all stuff you may not be able to control. And that is really tricky. And the website, if there's a big cloud website and security and making sure the security goes all the way through the system to your part. The hardware is, of course, tricky because you shouldn't dump coffee on it. And then manufacturing tests. People don't seem to realize that that's part of the embedded software job.

I don't think they don't realize it. I think they don't want it to be. Well, there is some of that. But if you can make your manufacturing tests and your board bring-up tests, if you think about them at the same time, it may be easier. And you do have to do the board bring-up tests.

You don't really get a choice on that unless you want your boards to be untested when the hardware engineer gives them to you. Which, dear hardware engineer, if you haven't tested the power system on your board i don't want it oh sorry that was uh um sorry i should just put that in email okay anyway manufacturing tests uh it's one of those things that when you go from a proof of concept to a shipping product you can't forget that and there's not anything truly tricky about it. Oh, yeah?

Other than the existence. Takes a bunch of space that you may not have. Well, or you have to do a firmware update in the factory. Yeah. I was going to talk a little bit about factory firmware update because that's odd, but you've given me so many other ideas

that I may did to that part. I'm glad I didn't put very much in my abstract factory for more update because that's odd but you've given me so many other ideas that i made to that part i'm glad i didn't put very much in my abstract because that means i can change it all i haven't written any of this down you're just gonna listen back to the podcast i totally i'm just gonna listen back to the podcast i hate to listen to my own voice but still i can manage that's all right i'll change it oh. Oh, can I be Darth Vader? Sure. Oh, excellent. So, yeah, that's my talk.

We missed anything? We haven't missed anything on my list or on yours. Environmental? I used to have more problems with environmental, but lately it seems like the processor vendors are a little more aware of that. That's boring. Yeah, it is sort of boring. The TIBLE chip I used recently, this goes back to the stupid memorization uh cc

2640 sure had this whole press to test mode or something ptm and it was it did all the fcc testing for you what like my software all it had to do was recognize that that button was pressed and call the function and it went off and did some thing in and it worked i mean that was how we did it was so weird that no now you have to explain because that doesn't make any sense to do fcc testing you have to do it in a chamber with oh no oh it just triggered but i didn't have to write any special software because it wasn't it squawked as loud as it could. It wasn't magically qualifying itself. No.

That's what I thought you were saying. It was just that my embedded software didn't have to do much. Got it. Got it. Yeah. Okay. That's fun if you've never done that, though. Go and do FCC testing.

It's kind of cool. That's interesting. Yeah. Yeah. Makes you want to put everything that's in your pocket in there just to see what's squawking. Well, I remember you'd go with lots of tinfoil and tape and try to put it over things, figure out where, you know, particular things. Emission testing was the one that was the fun part.

The other ones where they try to destroy it with, what is it? It's emission. And what's the other kind where they try to zap you? Radiative? Radiative. But I thought that was a mission. Yeah.

Anyway. Yeah. When, when they get out the kilovolt, they start touching your boards and you're like, Oh, no. Yep.

Doesn't work anymore. Yeah. It's good stuff. All right. Well, that's my talk. Uh, you all are of course invited.

It is October 11th if you come do ask for a sticker just so I know that you're there I'll be giving stickers out ahead of time probably before too and it's free maybe I should

qualify that oh they're providing pizza so it's not free it's like 7 bucks and there's a event break thing that they're providing pizza, so it's not free. It's like seven bucks. And there's an Eventbrite thing. Wait, it is free. It is free. So you can have pizza too and hear me talk.

Wow. It's like two for the price of free. But if you can't be there, there will be a webcast. Really, there'll be a webcast really there will be a webcast? you're just learning this now as you read this? this is fantastic also did you know that you will be on live television and simulcast into theaters across the country

yeah I hope not and I do plan on putting the talk on YouTube so we will figure that out. It may go on the IEEE channel, or it may go on mine, or it may go on both. We'll see. It'll be exciting. And if you aren't into videos,

this will, of course, eventually become a blog series, because if I'm going to do all this work of writing it and figuring it out and making interesting pictures to go along with it, remember we have to go get milkshakes because i need that for the firmware download picture um yeah so it will be on the blog which reminds me uh andre is like really getting into the stm32 uh discovery board and breaking it down and chris vac has gotten a little busy, so now he's posting every other week, but he's still on the 420, MSP420 from TI

and talking about the underlying microcontrollers. So you get two different perspectives on our blog right now. And I'm, I don't know, I'm just in random put-up-whatever-makes-sense-this-week mode. MSP 420. 430. Okay, good. 430. Because that's a non-contact ultrasonic level transmitter.

Yeah. MSP 420. How do you say that word? The Polysonics MSP 420 non-contact ultrasonic level transmitter with built-in four-digit LED, 26-foot operating range. This is great. Liturgical. How do you say. This is great. Liturgical. How do you say that?

Liturgical. Liturgical? Liturgy? Liturgy? Liturgical? That doesn't sound right. That's like a turtle that's been turned over. Liturgical.

There's a liturgical calendar. Excellent. Well, yeah. It's a whole memorization of all of these numbers and letters. I know sometimes I spew them out, but it is just a pain. Yes, he's been doing the MSP430 blog series. And you said something about Andre? Andre's doing the STM32 F4 discovery series.

He's coming at it more from the software top downside and Chris Beck is doing a little more on the bottom ups. At some point they're going to crash into GPIO drivers or, or maybe your drivers or DMA. I'm looking forward to it. It'll be hilarious. Let's see. And I posted my toast master talk last week which was

dorky let's just go with dorky but hey it was something and i've been pretty busy at work i posted nothing uh oh look uh over in slack christopher has said he wants to be darth vader no i was just i was just typing that because that's where we collect the titles. Oh. Yeah. All right. Yeah. All right.

Yeah, I think we're done. Oh, no, I have a couple more. Okay. Just a couple more. Just felt like it was, you know, careening to a... I know, I know, I was careening. We do request that you give us iTunes reviews, or Google Play reviews, or whatever reviews on whatever podcast system you use. And if that isn't convenient, I totally get it. I have no idea what my iTunes password is.

It's my iTunes password. Oh, I know what yours is. Well, let me share this with you. Oh. All right, I know what yours is. Let me share this with you. Oh. Alright, I shouldn't say it. Anyway, if you if reviewing is not convenient, we do have this neat

logo that now can be made into a poster, and if you have a coffee shop, or maybe a bulletin board at work, or a bulletin board at your school in the CS department, if you would print it. Maybe a billboard along the highway. That would be cool, too. If you print it out and post it, I would very much appreciate it.

If you just sit down and talk to someone and say, Have you heard this show? It's kind of cool. If you hate the show, you don't have to listen. I absolve you. You can not talk about it to anybody. Why are you still listening? And yes, if you could just talk about the show, it is helpful to us to reach more people. Reaching more people means we have more giveaways.

And more guests. And more guests. And it just all works well. And we're still not making money from this unless we have Amazon links in the show notes. And then if you buy stuff, we get like 2%. It's not. Some of you are doing that, for which we thank you, and it does pay for our hosting costs. Sometimes. Most months.

Every month for the last three? Yeah. Hey, it's cool. I like it. It's going to make my taxes harder. But that is not a complaint. All right. I did not write the whole outro thing, so I'm going to have to wing it. Wing what?

Do you have any last questions, Christopher? I know that's part of the outro. I do not have any last questions. Do you have any last thoughts you'd like to leave us with? You also didn't bring down the Winnie the Pooh thing, so we have to do that from memory. Crash. Winnie the Pooh fell down, down, down.

If only he hadn't wanted the honey so much. And then Tigger bounced like Tiggers do. All right, I really can't do this from memory, so I'm just going to say thank you. Thank you for listening. We do enjoy hearing from you. I did like the numbers competition. Feel free to let us know if you'll be in Santa Clara

and want to go to the embed conference. Competition. Conference. And thank you. Yeah, thank you. Thank you to the embed competition conference, conference, uh, and thank you. Yeah. Thank you. Thank you. Thank you.

Thank you. Thank you. You don't remember the song. I do, but I'm not going to sing it. Are you sure? It's pretty funny. All right.

Thank you. Goodbye. Embedded FM is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive any revenue from them. At this time, our sole sponsor remains Logical Elegance.