Embedded - 173: Everything's Amazing

Episode Date: October 26, 2016

George Stocker (@gortok) spoke with us about software, Jewelbots (@Jewelbots) and learning embedded systems to ship the product.  Elecia's book is Making Embedded Systems. George also recommended Getting Started with BLE and Programming Pearls. The processor we talked about was the Nordic nRF51, a BLE system on a chip.

Hello, this is Embedded. I'm Elysia White. My co-host is Christopher White. And we're going to talk about getting into embedded software with George Stalker. Before we get to that conversation, I do want to remind you that we have stickers available. Send me where you'd like me to send them, and I will send them off. Hit the contact link on embedded.fm or email show at embedded.fm to get your stickers. Hi, George. Good to talk to you today.

It's good to be here. So can you give us some background? Yeah. So for the last 18 months, I've been an embedded developer at Jewelbots. And I've been developing software for about 10 years now with a variety of people, the army, financial services, and so forth. And so I've been going backwards in my career, going from large corporate to finally a startup, now maybe maybe going back again and you you've gone from being a inexperienced software engineer to a somewhat new embedded software engineer yeah that's uh that's it's strange to go from uh developing software day in and day out and knowing where all the problem areas are and then starting an entirely new stack embedded development and not knowing where anything is and not knowing how to solve problems.

And it's a humbling experience. And that's what the show is going to be about today. Humbling, George? That seems unfair. Well, no, just reminding us that it can be humbling let's do lightning round uh we're going to ask you questions and hope for short answers and then if we are behaving we won't ask you for follow-up but that yeah yeah the rules are the rules are suggestions there's the points don't matter christopher i almost misread this one his

favorite fictional compiler. I won't ask that because I don't know how that could be answered. But favorite fictional computer? Oh, I would have to say L. Carson Star Trek. All right. What language should be taught in the first university course that teaches programming? C. And what is your favorite language to program in? C sharp. Would you rather use the

Kyle compiler or GCC? Kyle for debugging, GCC for ease of use and price. Nice. Describe your preferred brace style. I'm not asking that. Braces everywhere. Good answer. Fine. I asked it. Tabs or spaces? Depends on the team, but usually spaces. How many spaces should be in a tab? Depends on the language. C sharp 4, JavaScript 2.

And C? 2. Introvert or extrovert? Definitely introverted. What science fiction technology do you think will be real in our lifetimes? I'm really looking forward to a food replicator.

Yeah, 3D printer for food. You put in potatoes, it comes out as mashed potatoes. Wait a minute, that's not that hard, is it? I don't think I can do 3D printer for food you put in potatoes it comes out as mashed potato wait a minute that's not that hard is it i don't think i can do 3d printer for food i think it has to be the star trek thing where it just appears yeah because i watching it extrude something that yeah why have you don't really want to see how this has been made pasta maker is different it's the same it's not extruding a baked potato. Should we bring back the dodo bird? I don't

know because I don't know what the dodo bird did. So I guess that's a no. It could be responsible for all sorts of badness. I recall it being fairly large and dangerous. Least favorite planet? Mercury.

That's two for Mercury. I know. Man, it's getting a bad rap. Let's see. Let's go on to the real show. We could just continue that all day long. So you have been working for Jewelbots. You mentioned working at a startup, and that is

the startup. And what do they make? We make friendship bracelets that teach girls how to code. So they are in the Bluetooth arena of the Internet of Things space. They are small bracelets that have four lights and a motor and a Bluetooth radio. And when girls pick them up, they can pair them with each other and choose what color groups they want their friends to be in. And then when their friends come into proximity in Bluetooth range, they'll light up with the colors that they share. Above and beyond that, they can send messages to their friends by tapping on the button and then selecting the color they want to send a message to.

And they can actually send Buzz style messages to their friends who are in Bluetooth range. And all the friends in that color group will get them. The teaching them how to code part is improving on and extending the Arduino experience and making it work for JewelBot so that they can actually code them to do things that are interesting to them. Part of that is trying to come up with a social media API that they can use that can actually tie into their phone for notifications and use their Jewelbot as an extension of their phone. But the Jewelbot doesn't have to work with the phone. You're not sending BLE messages from the Jewelbot to the phone through the phone network to the other phone through the other Jewelbot. You're doing directs, right?

That's right. And maybe one day we'll include the phone in that part. But for now, it's all over Bluetooth, all local. And your LinkedIn profile says that for Jewelbots, your title has been VP of Software Development. What did you expect you were going to be doing there? Having never worked at a startup before, I figured it would be like the movie version of startups

where you get money and you hire people. And so your first six to 12 months is spent bringing people on board, building what you need, but mostly bringing people up. And the reality of software startups is different and the reality of hardware startups is even more different.

So yeah, I expected to be building and managing a team and building the product. I did not expect to actually be building everything. And that was a unique and challenging experience. What happened? The hardware startup space is difficult for investors to see, especially with a new and unproven market like a friendship bracelet that teaches girls to code. And so where software startups can put out a prototype in six to eight weeks, the hardware development timeline is much longer, months to years. And so getting investors to buy into that is a challenge. And as such, you typically need to bootstrap, use what you have and build with the team that you have under the understanding you may or may not get funding because they just,

investors, that's not their jam usually. Okay, so that makes sense. Investors don't really understand hardware. We've heard that song and dance quite a few times because you do have to physically create something and then there's production, which software startups don't have hardly at all. It's easy to have a billion copies of your software now. Yes. startups don't have hardly at all. It's easy to have a billion copies of your software now. But hiring didn't go so well?

Hiring, it's difficult to hire for something when you don't know what you're doing with it. So as an application developer of 10 years experience coming into Jewelbots, I was, you know, I said, okay, we had a company do our initial firmware prototype. And from there, we decided for a variety of reasons that we needed to bring this in-house. And so I started looking for a firmware developer. And I found out really quickly that I did not know what to look for. I didn't even know the problem space at the time.

I couldn't tell you what the difference between SPI and I2C was and even what they did. And that made hiring a firmware developer difficult. We ended up doing it through AngelList, but it was hard to know what questions to ask them when I didn't know the space myself. And that's a pretty tough problem because you are going off of what they say on their resume and how well they talk around problems without being able to incisively say, well, what's wrong with QMX? And have somebody answer that and then be shocked that, oh, they must have used it then. Right. And in the hardware world or in the firmware world, someone will have half a dozen chips to their name of, you know, firmware they've done for work. And not knowing what half of those are or how they're relevant, you start to look for, hey, who has used, you know, the Nordic NRF51822? And you start narrowing your search that way with people who've used that specific chip. And in hindsight, that was probably a bad idea

because it was a newer chip and because it's harder to find one person that's done one specific chip than it is someone who has the general experience. So you're looking for things that you didn't think you'd be looking for. And the intuitive, hey, who's worked on this chip before, turns out to be a bad way to figure out who's going to come work for you.

Yeah, okay. I mean, I certainly have my list of chips and I don't put every chip that I use on my resume because I don't ever want to use that again. But things do change quickly, too. Yeah. Especially lately. And I don't know if you worked on the Nordic NRF 8000 line, that it is at all related to the NRF 50X lines.

So, yeah, okay, I get that. So what happened? You tried to hire someone, it maybe didn't work, and then you went to boot camp? Well, yeah, we hired somebody, and we were running into issues with getting the SPI communication to work, and we didn't know why. And the person we hired didn't know why and the person we hired didn't know why. And so we finally came to the point where, you know, when we spent so much time on that little problem, we had to say, okay, this isn't working, let's try something else. And at the time, we had just started Techstars. So we were all sitting in the Techstars New York office, which is an accelerator normally for startups, but they accepted us and are normally for software startups. And they accepted us and, or normally for software startups and they accepted us. And

we were sitting around and we're like, well, this isn't working. And so I spent the next two days looking at firms that did this sort of work. Um, but they weren't really getting to us in the timeframe that we needed. We weren't, uh, we weren't higher on their priority list. And so the question came up, they said, well, why can't you do it? You're a software developer. And I said, but I've never programmed hardware in my life. I took a semester of C in college, and it wasn't even C, it was C++. I don't know enough, but I'm willing to learn. And they said, okay, learn. And it turns out there is a company called The Bar Group that does embedded development boot camps did you think it was going to be like did you think it was going to be like okay this is just some incremental stuff on top of what i already know or were you trepidatious that okay this is a whole new field and i have to learn really quickly a lot of things i had access to our source for our firmware so i was looking through it and i was trying to figure out what everything did. And the first thing I did was look at our hardware schematic. And our hardware schematic had pin numbers on it. And it also had

an alpha designation, which turned out to be the physical location of the hardware pins on the chip where it sat in place. So it'd be like A1 through A8, and then that's just a grid. And I did not know what that did. So I spent a lot of time just trying to figure out, you know, what was important information on the data sheet and what wasn't important information. So going from a, I'm looking at the data sheet, this is all I really know. And I'm looking at the code I have, I have no idea how these things fit together. And it was very alien to me. I tried to use our data sheets for our LED driver, and it was amazing how much information was packed into that and not knowing what the useful information was and what it wasn't.

So going into the embedded boot camp, I was scared because this was way different than anything I had done, and I didn't have a frame of reference for it. And what did you get out of the boot camp? up spending four and a half days building on lessons going from the very basic, this is what a register is, to actually by the end of it, programming the whole board to do a scuba diving management system. And intermixed with that were really deep lectures and, well, deep lectures and overviews of the different parts of embedded development. So by the time you came out of it, you understood at least where to look up all the words and

basically how to write software at a very basic level for a specific piece of hardware. But it wasn't based on the Nordic chip? No, it was based on some other chip. And I have the board somewhere in my office, but yeah, it was nothing that I was used to. We used the free version of IAR and it was, trying to remember back to it, but yeah, it was completely different than what I later used. And at the end of the bootcamp, they took you and they showed you what an RTOS did and got you to start using an RTOS as well. But they went through a lot of different parts of embedded development. Were the other people there proficient software engineers?

For the most part, it was hardware people looking to get into firmware. There were very few actual software developers day to day. They were either MEs or EEs that were looking to get deeper into embedded firmware. I would say you'd be in a better position than a hardware engineer going into that. It depends on how it's tailored, though. At least you speak the language. Yeah. Yeah, well, that's true. And so you came out and you were ready to be an embedded software engineer. How'd that go?

I spent the first few weeks trying to get everything running on GCC with the latest version of the Nordic chip. So we are the latest version of the Nordic SDK. Nordic at that time, we had been based on version six of their SDK. They had subsequently released up to version nine in between the intervening time. And so my first task was to get the S130 soft device running on the chip and getting it ported from SDK 6 to SDK 9. And that took me about three weeks the first time to get it all running. I spent a little bit of time on the phone with Nordics engineers. They actually sat with me and took me through some of the things like the linker settings that I had missed and things like that. But I did end up getting everything upgraded over three weeks. And that was a good way to start because I was able to learn their API and how they decided what things were called and where all the different parts of the system were. But it was challenging because I didn't quite know what I was doing. So I was guessing for some of it.

Okay. And then you got that working. And you were like, I'm a software engineer. I know enough how to do this. I have been doing databases in full stack. I know how APIs work. I know how registers work now. And it was all simple, right?

I wish. So the first thing, well, not the first thing. I guess that would be the second thing now that we've already talked about the first thing. The second thing was figuring out, I've really got a blank slate here. We have nothing but demo firmware on here that doesn't do anything that we really can do for in production.

Let me start sketching out the system and sketching out what this thing is actually supposed to do. And to do that, I started to look around for different boards or different open source embedded systems that I could say, okay, this is how they work. This is how they do that. And I found a few, didn't find enough that I was hoping for. And I didn't find enough of code that I was hoping for to show me how to speak the language or how to do things the embedded way. So it was a part of. Okay, so what are the good things about the embedded software community? The good things are is that you're right next to the board.

You're right next to the hardware when you're developing. And so there aren't too many magic things that can be happening. And the community is usually very direct about things that you're missing. And those are really, really good things when you're starting out. So yeah, that. Okay, what are the not so good parts? It's a little more insular, a lot more insular than other software communities I've been a part of in as far as open source, in as far as instead of just one Stack Overflow that everyone uses. Every chip manufacturer has their own forum or their own Q&A site, and they have varying levels of use, and they don't follow the same rules as stack overflow so when

you're trying to get the answer to a question they'll literally link you to the docs and say have a great day and that's when you're starting out that's really tough to do because if you knew what the docs meant you wouldn't be there the answer to your question is in this thousand page document why didn't you look there yeah so. So we were, I tried to, when we ported from six to nine in version nine of the Nordic SDK and stop me if this is a little too inside baseball, but they had something called the peer manager, which is a way of managing Bluetooth peers that you've bonded with. And I spent, must have been a week and a half just trying to figure out how that magic worked, because I knew how Bluetooth connected. But I, you know, didn't see anything in

the code that would say, hey, start the bonding process. And so I ended up having to put that aside and coming back to that later. But there are just certain things that they expect you, rightfully so to know, you know, all the things that happen in you, rightfully so, to know all the things that happen in Bluetooth. And there's really no primers that take you through it from, hey, this is what you should do if you're doing Bluetooth development with this product. So there was a lot of reading, a lot of learning, and a lot of realizing that those one-line comments in the forum posts or in the Q&A site actually might have the answer to your question and not skipping over them because

it's just a one-line comment. And you were doing something a little odd, which we alluded to earlier, where most BLE devices speak to a central, a phone, usually a smart device, you were not only being the peripheral, you were having this pure model that's more of a mesh networking idea and less of a parent-child relation that most BLE systems have. You were doing something odd. Did that make it worse? Much, because I knew it was odd. I knew enough about Bluetooth to know that was odd.

And I had spent some time skimming over the Bluetooth specification, which I never recommend. So I knew it was odd, but I did not know. And what was really tough to find were to find the answers I was looking for because no one really does that. When you pair your Bluetooth speaker to your phone, your Bluetooth speaker is never going to say, hey, I need to talk to another Bluetooth speaker in the same way I can talk to this phone. So yeah, that was different. And it was challenging to find those answers and to say, hey, no, I need to be a central and a peripheral. And luckily, the Nordic chip, the S130 soft device provides support for that.

But because it was so new at the time, there wasn't a lot of people using it. And there was not a lot of example code to go off of besides Nordic's very basic examples. And Nordic has amazing documentation and they have amazing examples you can download and use with their SDK. But you have to know what they're supposed to be doing in order to get the most out of them. I think that's something that we run into all the time is that you have a piece of hardware and you're making a product. And usually when you're making a product you might be doing something new because that's the whole point of doing a product.

Exactly. So you talk to the vendor and they say, oh yeah, it's in our feature checkbox. It's right there. It does this. And you get a few weeks into development and you learn you're the first person using this feature of this chip.

Nobody's ever done it before. And there might be an example, but if you're the first person, maybe nobody's ever run the example more than internally at some companies. So that's a big challenge is, oh, okay, I'm actually breaking new ground with this hardware.

Well, and sometimes you have multiple things you want to do. And this example might work on its own, but you also need that example and you have to mash them together. Oh, it doesn't do that. And you find out there are conflicting parts of the system where there are examples, as you were saying, they're doing one thing by itself. And then you put two things together and you realize,

oh, the radio is interrupting what I'm trying to do with flash storage, and now I have to redesign the entire way that I store and save friends and load friends because it's conflicting with the radio, and that's causing problems for me. So yeah, those are things that their examples don't typically cover. There's an application note for them, and you learn by doing it how they go around it. But when you're using so many different parts of the system together that aren't typically used, you don't know those rough edges and their examples don't have those rough edges. So it's a learning experience. And I believe you used the Kyle

compiler for most of this because that's where Nordic keeps their best version of their examples. Is that right? That's right. And their SDK later on, their SDK became better and it was easy to download. And their SDK was finally kind of segregated away from being so dependent upon Kyle to get the good examples. The Kyle compiler was great for in the beginning when there was no soft device. And I just wanted to debug hardware stuff that we were doing the hardware board bring up. Later on, when I started to use the soft device,

it would, whenever I would stop it or stop it on a break point, it would freeze the whole system and crash the whole system when I restarted it. So I used it less and less for debugging and more for the beginning of getting all the hardware brought up. I tried unsuccessfully to get GDB up and running. And so my GCC, you know, we still have the GCC compiler stuff in. But that's solely for, you know, compiling it for when you want to just compile it through the command line and you don't want to use Kyle.

We don't actually do any development on that just because it was hard to get GDB up and running. So I have a survey question for you. I'm going to ask this of all software developers who move into embedded systems. So what do you think of the tools that we've got here? The way you ask that, it sounds like there's a longstanding in-joke.

Well, you did just describe how you couldn't actually use a breakpoint because the operating system, the RTOS on the Nordic kills you. You can't use GCC because you can't get GDB to work, even if you could use the breakpoint. And you have two compilers. You've mentioned IAR, but that was for the class. But you have two compilers you're supporting. Yeah, what do you think of our tools? Aren't they great? I now understand why embedded developers I've met are sometimes cranky. That explains why you're so angry all the time.

It is different because, at least in the development I've been doing, both in C Sharp, JavaScript, and other platforms, a lot of time is spent making the tools, I don't want to say easy to use, but easy enough to use that you're not fighting with them all the time. With embedded development, it feels very Old West-ish, where you have this tool that's been the same way it's been since the beginning, and no, we are not making it better. There's no need to. We learned the tool.

And so that's strange to come into, because I fully expected there to be an easy way to get GDB up and running. And maybe there is. And part of the problem with being a new person in embedded development is that I don't know if I'm just doing it wrong or if it really is this tough. So there's that. It's like, I'm sure this has to be easy. People have been doing it for decades, I think. This has to be easy, and what am I missing? Absolutely nothing. It's very humbling to literally be fighting with the tools and the compiler, and the compiler winning every time.

Well, and you have the problem that if you get GDB working and for some reason something breaks, you don't know where in that stream was. Was it the compiler? Was it the GDB connection? Was it the programmer? Was it your software? And so you end up going back to the safest possible path. Isn't that thing you don't like to say? It starts with an open and an OCD.

Open OCD. No, he didn't have to deal with that. Yeah, well, so we just started this last week. The first thing we started doing was creating the Arduino board support library for our board. And I was looking into open OCD because that's how Red Bear Labs, that's how they load their Nordic code onto their device. But it looks like they use that as a, I guess what's called an ISP, like a systems programmer for their board. But since our pins that we do the initial flashing on aren't exposed to the end user, we can't use that. So we're still trying to figure out the unknown question is can we just use the existing

you know serial utils to put user code onto the jewel bots and so that's what we spent the week figuring out was getting our Arduino our library up and running and figuring out those parts of it. Okay so what were the difficult parts that afterwards you've learned that you've learned them and then afterwards you were like, if only I had known some small thing, it all would have been simpler. I did not realize how something like a simple for loop can make the entire system unresponsive. In the beginning to do the light patterns, I was simply doing a for loop and iterating through each light, incrementing the value in the register. And that worked. It did exactly what I needed it to do.

But then I tried to press a button and the button wasn't working while that was going on. And I was, what's wrong? It's, why isn't this working? And I pretty quickly realized, wait a minute, this is a single core outside of the interrupt context. It's not going to listen for events from other things if I'm in this loop. So I had to restructure how I wrote code because I was used to writing code in systems that are used to having a lot of consumers. And in this case, ours didn't. So that was difficult to get around and to redesign the parts where we needed light interaction and someone to press a button at the same time. You mentioned that to me about how systems don't block on for loops. And I'm like, well, I mean, it's a for loop happens over here, but it doesn't stop anything

else. And in embedded, when you don't have an operating system or you're only running one task, yeah, the for loop is all you do. You only do what's in front of you. Yeah. That's really a mental shift. I ended up using data structures and design patterns that I never even knew existed. And part of it I got from books I'd read on the topic, not to plug your book too much, but your book was amazing on this. Thank you. Programming Embedded Systems.

And so data structures, things like a circular buffer, which your readers are probably like, yeah, that's of course a circular buffer. Everyone uses those. Those are part of the system. I had no idea. And one of our constraints was, you know, we need to, friends would come into proximity and they would come into proximity as the Bluetooth radio found them, which of course was happening several times a second. And the lights needed to respond to them, but on a separate system. So the lights needed to respond as they come in, but they couldn't just kick people out because your lights would be flashing all the time

as people from different color groups came in. And so the solution ended up being two circular buffers, one that handled Bluetooth people coming in and another that handled telling the lights what colors to display. And those were on separate timers. And I pushed from one and pulled to the other at separate schedules so that they didn't interfere with each other and you could still use the system. But that's something I'd never had to think about before and was not in my initial system design because I didn't realize that would be a thing. So I ended up also building my first complete state machine for this embedded firmware.

After 10 years of software development, I had never actually needed a state machine before this point in production code, and I ended up building one for this. So that's an interesting thing, realization, is coming into embedded and seeing that there's nothing waiting for you to put it together. You have to kind of build things from scratch. And it's one of my complaints about Embedded because I've done some application work. And having come from Embedded to application work,

I have to resist the urge to re-implement things from scratch and say, oh, I need a linked list. Well, I'll go make a linked list class over here. Or I need a map, some sort of map database. Oh, I'll go implement that from scratch. Now, just grab a map class or a list class and you're done. It's just standard library stuff.

So I can imagine coming from that and going to Embedded and it's like, where's all my furniture? Right. And not realizing. There's a hammer and a pile of wood over in the corner. Get to it. I'd taken C++ for a semester

in college, so I knew about the C++ standard library, but I didn't realize until I started doing embedded work that each chip is a bit different and, well, each architecture is a bit different, and there aren't really standard libraries you'd think of them in the application world where you do have the things like the lists.

You do have your data structures. You do have those patterns sitting around you. And it actually felt really good building it from scratch because with web applications I've developed or anything else, there's always been, you know, most of the system really was there, a scaffolding. And it was just, you know, building the features you needed and deploying it. But here, I literally built it from scratch. And that was a very rewarding feeling. And that was one of the things I like about doing embedded development is that feeling of creating something from nothing. I'm glad there was something you enjoyed.

I loved it. It was a very humbling and a very trying experience, but I loved it at the end. During it, I didn't think I would get out of it. But now that I'm sitting at the end of it, I really appreciate having done it. And I really do think that other application developers should at least give it a try. Wait, wait until you see one of the bracelets on Target's shelves. That's where it goes from being a, oh, thank God it's over to, I can make this, I can touch it, I can tell people walking by they should buy it and it's weirdly magical yes and uh but also a little little humbling still because uh you know since it's yours you know where all the weak spots are you know where the bugs are and uh we do a lot we did a lot of alpha testing uh and continual

testing as we release out new features on the bracelets but there's always that worry that someone's going to use it and say, oh, this is terrible because it doesn't see my friend as soon as they come in. And I can't be there to tell them, yes, that's because battery life is a thing and power management is really difficult, and we're trying to balance the two. That explanation doesn't work for anyone. So you just have to work through it and hope that you know no one notices the words like you do exactly and you do get accustomed to these things that you used to think were bugs and now you realize exist for a reason i mean

what well like having a delay for for power optimization reasons you do that because it it means your battery lasts longer and but some things are just bugs it's but that's technically a feature he's trying to make it better and a feature it's physics well damn physics. Let's see, what other things did you have trouble getting accustomed to and embedded? I would present on the Nordic dev zone on their Q&A site if I was just being dumb and, you know, this was really easy or if it was actually a problem. So I spent a lot of time reproducing programming problems on a bare board, bringing it up and, you know, seeing the whole state of the system as it came in. And even with programming applications and web, that's really easy to do, but it's a little more difficult to do on the hardware side. But I got really good at it because I needed to find out, hey, is this actually my bug?

Is this two pieces of the system interacting in a bad way? Is this something that is actually an issue? So it got really good at building up systems from scratch to test out issues. But that's something you generally don't have to do as much of on software because it's just the software stack. You pretty much know it's just your code that's the problem. With hardware, especially with a board that, you know,

you've never really tested out all the features for you don't know if it's you if it's the board if it's the sdk because not a lot of people have used it uh you don't know which one of those it is so every time you go through a problem you have to you know one by one figure out which one it isn't and then go to the next part of the process i want to go back to something you said at the beginning of that excellent answer. And that is the confidence part. The, I mean, you were your software engineer. You've been a software engineer for long enough to know you're pretty good at it.

You've been doing this for a while. And you were probably pretty confident answering things on Stack Overflow or asking a question and knowing that it's a hard problem. And now going into Embedded, you had to feel insecure about it. Yes. And it was if you take any feeling of starting a new software stack, and once you've done a few software stacks, once you've done a few MVC architecture, you know, web application frameworks, they all kind of do things the same way.

They call them a few different things, but they all do them the same way. So you get really good at knowing, okay, this is where I need to go to do this. But with Embedded, because it was so new, it was really difficult to know where I was in the scheme of things. And because of that, it felt like even 20 times worse than the first time I learned a new web

framework or a new language. I just felt that much more exposed. And, you know, like everyone was going to see me for a charlatan because I had no idea what I was doing. And I'm trying to build this firmware for this new product. Do you think imposter syndrome is more prevalent in embedded systems? I don't have a frame of reference besides my own experience, but there's people doing it, people doing it really well. I'm sure it's there, but I know that jumping the gap between software and hardware, that's definitely, this is a lot more challenging than typical software development. So I really want to tip my hat to everyone out there that does embedded development, because that's really tough.

It is, but I actually, my next question is, what if I wanted, what if I was an embedded engineer, as I am, and I wanted to do application? Where would I fall down? What would be the hard parts for me? What should I start with? What should I learn? Yeah. yeah um so in the embedded world you're kind of stuck because you know you've you're for whatever reason you pick this piece of hardware and so you're stuck with that community with software and when it's just software you get your choice of communities and they all pretty much all the

frameworks pretty much do the same thing and all the languages pretty much do the same thing and so then you really get to choose your community and a community like JavaScript is very open, very welcoming, as is Python. C Sharp and.NET are getting better at that. They're getting much better at that, at embracing open source. So it really all depends upon what community fits your, do you feel like you belong to,

what community fits your style and going from there. And you don't really have that choice in embedded development. You're kind of stuck with whatever community your chip has embraced. Do you think that's a matter of numbers? There's just fewer embedded people and so there's smaller communities and so there aren't as many communities? Maybe.

I mean, I know that there are lots of different chips out there, lots of different vendors. And yeah, maybe there's less people to rally around each one because, you know, it's not, you've got, what, four or five major software platforms that developers use outside the embedded world for web development. And you have, what, five times that many chips alone. So yeah, maybe there's just less opportunity to congregate. Although I don't know where the embedded community writ large hangs out. I just know where people that use different chips hang out on that vendor's forums. I don't know where the embedded community hangs out. I just know where people that use different chips hang out on that vendor's forums.

I don't know where the embedded community hangs out. I know a few of them hang out here, but overall, no, I don't. No. Yeah, I don't know. That was part of my investigation process. So for each stack that I've ever worked in, it's pretty easy to identify the leaders in that stack because, well, the public leaders, because they'll have stuff all over the internet. And that's really how I found you and how I found the bar group. And everything I found was, okay, I'm going to Google embedded development. I'm going to Google this term and see who keeps showing up and then go

ask them for help because clearly they know their way around. And so that's at least that's a constant in software development is you can find people who do this for a living and people who are the leaders and say, hey, do you know more about this? I know you know more about this than I do. Where can I find these watering holes and where can I find these answers? Yeah, I think it is really widely distributed, like you said, and product specific. I don't think there's any, there hasn't been any sort of central, okay, this forum is where to go. If there is one, I don't know of it. And if that's the case, then it's probably not the place. If there is one, could you post our blog and podcast

to it? Because it would be nice. But I was just thinking about what you said about Stack Overflow. I mean, why isn't there an embedded Stack Overflow? There's EE Stack Exchange. There's some on there.

And there used to be Arduino. But they're all product-specific. Right. And product-specific is the wrong way to look look at it because with every project you're going to be i mean there's general principles of embedded software development there's state machines the general principles i mean that was what i tried to do in my book is the the general stuff and then there are all of the tweaky i mean the fact that the nordic the r toss means you can't use your breakpoints that is something you find out when you do nordic i mean that's just yeah but if there was one place you could go to ask those questions where there's a bunch of people

who hang out and some of them have used nordic some of them use this or that and you know sometimes the problem isn't specific to a chip it's like okay i'm using ir and this weird thing's happening with jlink and uh yeah but you do have to list all of those things if you said to me i have an operating i have an r toss and my break points break i could probably i don't care if it's nordic right that's that's a problem set i understand but in order to state that problem you do have to say i'm using this rtos and this chip and this program it's no different than questions and software though i mean it's like okay i've got version 5.1 of framework q and i'm running gcc 4.7 on a mac running, you know, I mean, you have to, if you're stating a problem correctly,

I think you have to list all the, all the, you know, boundary conditions correctly anyway. And you sometimes don't know the boundary conditions. And that's what I kept running into is I didn't know, because there was so much I didn't know about what I was doing, I didn't know if what was important and what wasn't important. So I would try to list the things I thought were important, only to find out they weren't. And there were a few problems I had that I guess are problems every embedded developer has, like Jlink just crashing at random times. And maybe no one else has that issue and it's just me,

but I either am a lightning rod for an embedded development issues, or it's a little rockier than I thought it was. And I don't know which one that is. It's a lot rockier than you think it is. You were describing when you were talking about some of the difficult things as you were learning and first embarking on the project, I was thinking, okay, that was last week and the week before. And this all sounds very familiar.

There's plenty of times when I have hardware and I'll come in in the morning, it was working yesterday, and now it just doesn't. And I'll go through a bunch of debugging steps, I'll ask on HipChat to the rest of the company, hey, what's going on? And they'll say, did you try this? Yep.

Did you try this? Yep. And they'll say, well, I don't know. And then I'll spend another half an hour, and it turns out it just needed to, you know, power cycle, and I needed to remove and reinsert a daughter board for some reason. And now it's fine. Or the time where I lost half a day to the fact that I stripped out our print statements.

We went from using the RTT to using NRF's log library and doing it through serial or through USB at UART. And I ended up losing half a day when I changed this over. And then when I removed some of them, because the state of the system, for whatever reason, it was slowing it down enough for it to not run into a problem because it was trying to print out those debug statements.

And going deep into it, it turned out that our flash storage was interfering with the Bluetooth radio. But because we had those print statements in there, it was taking enough time that they didn't conflict. But once we removed them, they started conflicting. And so it was literally, all right, I'm about to push out this release. Let's strip all the log statements out.

Let's if-det them out or whatever we did. And then everything's broken. And this was like five o'clock on a Friday. And we then sent out these chips for our founder to, or these boards for our founder to take with her. And then she's like, hey, everything's broken. Why? Like, what are you talking about?

It was just working. And then sure enough, that's the problem. Can't actually take out those log statements without realizing what's going on. That happens all the time. And there's no real way to fix that except periodically make sure you

can run without that stuff turned on. And knowing that printf leads to serial, which leads to timing, and it is all a stack of cards that you have to know what the bottom stack is or you run into problems. Do other embedded developers go through this? And that's one of the things that learning is, I don't know the right answer, so I know the good enough answer, but I never end up learning the right answer, so I don't know what other embedded developers do. Do you leave them in? What do you do with all this? There is no right answer. Depends on if there's a cost to leaving them in.

Yeah. You certainly don't want your system to depend on that timing because that's an accident. It's not, you know. It'd be better to have your tasks go at a fixed interval instead of waiting for your printfs. And then if your printfs take too long to set a flag that is an error that says you have too many printfs or take out all the vowels or whatever you do. But your general question of what do other people do, I sort of hate to tell you this, but Chris mentioned it earlier. That was last week. That was the week before. These problems happen all the time. And because we are changing chips and changing operating systems and changing tools, they're new every time.

And sometimes I can come up with a class of problems that it is, like printfs take time. But that doesn't give me an automatic solution. Memory problems. Memory problems. I do remember, though, I remember the day I was talking to George about map files. That was the happiest I had seen him in a long time. That was so magical. Wow.

You have no, that was great because, so I was running into issues where it turned out we were seeing issues with our TWI where it wasn't acknowledging. And for whatever reason, the actual problem that manifested itself is half of our boards would not turn on on battery or they would reboot repeatedly on battery. They did fine through USB. They did fine through USB if you attach the USB first and then attach the battery or letting the battery drain all the way down and doing those steps. But if you try to run them on battery alone, half of our boards would not work. And this is not something that we had done in our board bring up.

I had if-deft out the PMIC and just elected to run it off USB as if USB would always power in because I needed to get the application firmware done. And this was the point where we'd gotten the application firmware done for our initial alpha, and we're about to release them. And then all of a sudden, you know, we got these boards from the manufacturer and half of them don't work. And we don't know why. And so I asked Alicia, I said, Hey, can you help with this? And she's like, sure. And so we pinned it down to a problem inside of the software problems inside of Nordics code. And she's like, Hey, let me show you, you know, pull up the map file. And I'm like the one. And so she takes me through the memory

addresses, the call locations, dumping out everything, you know, doing the annotated build so that everything is in there. And you can see, the instructions and whatnot. And that was amazing because I knew it was there. I just never knew I could get to it that easily. Or that it was actually useful if you got to it. And it was really useful. It showed us that it was, we were using Nordic's TWI library and it showed us that something in the library was causing it not to acknowledge and then we got it we pulled up a logic analyzer and we saw yep sure enough it's not you know it's not acknowledging the signal and we went deeper and deeper and it turned out to be that the PMIC we were using we had an unregulated line and so not enough voltage was getting through to the chip boot up to the PMIC chip it boot up and it was going into a brownout state.

And for this particular chip, they spent all the line telling you about this. And so it was like, well, okay. So we tried to talk to their support, and the first question their support asked us was, how many chips did you buy? And how many do you plan on buying? One billion.

Whatever number gets your help. But it turns out that after that rigmarole and after a few weeks of back and forth with them, they said, hey, yeah, your hardware design is deficient. You shouldn't have had this unregulated line. You should make this new line and this something, something, something that hardware people know. And then it will all work.

And that required a board change at the 11th hour that we weren't expecting to have to make all because we hadn't done a full board bring up because as a software developer, I'm used to software being on a stack that always works. And this is our board. And I did not know that our EE design could be deficient because it had been, like this had been this way since the first prototype of the first board before we had our own EE. So we all took it as gospel.

And it turned out it wasn't. Back to that stack of cards cards looking at the bottom one and noticing it's that's the beauty of this kind of thing is there's just so many things that could be wrong really and they might wait to just really really inconvenient times to manifest so you were a manager before you uh got hoodwinked into embedded software, as well as being an engineer, right? So I started out in the Army as a team lead, then got out of the Army, finished college, and then went to work for a software company as a programmer. And then pretty soon after that, moved up to the Northern Virginia area after the crash in 08. We'd lived in Charlotte at the time, but the banking sector got hit really hard. So we moved up here and I started working for a government contractor, actually doing things

to support the army. And I was like, oh, I know this problem space. This is great. I know exactly all the words they're using. I know the domain. This is awesome. And turns out they needed a team lead. So I stepped back into that role and then was a team lead for there for a year until going to the Motley Fool. And so I bounced between, you know, leading teams and being an individual contributor on teams. So, yeah, it's one of those things that the basics are all the same, but it's different in the software world than it is in the army. Right. What did you do for Motley Fool? I was a programmer.

I was initially worked on. They were starting an MVC application back when ASP.NET MVC was new. And they were starting that, doing that. And they were like, oh, you have experience in this. Come on and join it. So I started with that. But the cool thing about The Motley Fool is they are a technology agnostic environment. So while in the beginning we did a lot of ASP.NET, later on it was Python, it was JavaScript, it was Angular.

It was different software stacks. And so that's really where I first started to branch out and solve problems in other tech stacks on other platforms and gain that confidence of learning new tech stacks quickly. I made a statement a couple of podcasts ago about software stacks, and I want to know if I was lying. I said that it's less likely that a software stack about software stacks, and I want to know if I was lying. I said that it's less likely that a software stack in the web software world

is going to make API-breaking changes from version to version compared to embedded, where they might just throw everything completely out of the... Well, he did see the SDK switch from 6 to 9. Right. So he's seen the embedded switch.

So yeah, what do you think, George? Was I lying? Well, yeah. So there was 6 to 9, then 9 to 10, then 10 to 11, then now 11 to 12. And all of them have breaking API changes. And every time Nordic asks me for feedback, the number one thing I say is, you know, either give me, either tell me what your breaking changes are, or at least release a diff of your old API to your new one and tell me why they changed. And that's not something that happens in the software world. We do have breaking API changes, absolutely.

They're usually small things. And especially in the Microsoft stack, the onus is on Microsoft to say why the change is needed. So very infrequently where you have major breaking changes, for example, from either, I think it's from C sharp three to four, they changed the way that closures worked because you would modify a variable inside of closure and it wouldn't actually do anything. So you would think that, hey, this should work a certain way and it didn't. Well, they actually made that a breaking change because it needed to be fixed because enough people weren't doing it the right way and it was enough of a headache. But there's a whole list of criteria to decide, hey, we're going to make this breaking change. And typically, the more corporate support you have behind a hardware stack

or a software stack, excuse me, the less likely they are to make that change unless they really need to. Maybe when you own the hardware and you own the software, you have a different set of rules. I don't know. I think it's, well, I won't say what I think it is. That's not normal in software, though. I think it's a lack of empathy.

I'm not expecting it. Let's put it that way yeah their goal is not to sell the software their goal is to sell their chips exactly and a lot of embedded software engineers sometimes me included go to one version and stay with that version as long and and grip it as tightly as possible and never, ever, ever change because you end up redoing so much of your system. Or you say, to heck with this, I'm not using all of their system. I'm going to use a little tiny part of their system and keep mine all separate and write it from scratch.

But you know that you're spending money or time you shouldn't be spending. Well, and the difficulty now is some of those companies are making efforts to make things easier and to produce frameworks and make code generating tools i have one in mind but i won't mention it right now is it cube it might be and so they have things like that which which lull you into some confidence that, okay, I just use this. And they're making breaking changes against that now. And that's a little harder

because that's supposed to be leading people to making things easier for people. And instead you get roped in, you do one thing, you get the next version. And now what the hell? It's like they're making things easier to start, but not to continue. It's like, okay, you can get to this point, but like they're making things easier to start but not to continue.

It's like, okay, you can get to this point, but eventually you're going to have to just throw all this out. And software has that too, but it doesn't happen. Like I was heavily, Microsoft is really good about not making breaking changes. Other communities like the open source community around JavaScript, they create new libraries and new frameworks that replace other

ones every six months, it feels like. So you do end up, you know, you get tons of things thrown at you. And you'll say, hey, I'm using, you know, I'm using Grunt. And someone will say, don't use Grunt, use Gulp. And then six months later, no, use NPM scripts, and it'll change and the community is very uh are those all real things those are all real things and i'm sorry those are all task runners and uh yeah and they end up changing and every year it's a new one and a different one but that's a that's a javascript community thing that's what they do uh the c-sharp community is very much into not breaking things and not you know replacing things that work. In other communities, Python is really good about stability

and really good about APIs. It's just now they've been trying to get people to go from Python 2.7 to Python 3 for over 10 years now. It feels like it. Maybe it's not been 10 years, but it feels like it. Hold on to this SDK as long as I can. Pry my fingers from it. When you're new to firmware development and you're like, hey, I need to rely on the manufacturer like I did.

I was like, well, I need to use their peer management library because I don't know how to write my own. And I couldn't figure out how to get it to work. And it was SDK 9. It had just come out. There was no real code around it. And no one was using it. And so I ended up not using it and building my own. And it turns out I really can do that, but I built a really small one.

And then later on, when I figured out more of what I was doing, I went back and have included it for a subsection of our system, but not the underpinnings of the whole system. We don't rely a lot on the traditional BLE because it's slower than what we need. I often forget that a large part of my job is learning. And I get frustrated with myself and sometimes the job when I do have to go learn a whole new RTOS or chip or whatever. Because I'm not producing and so it, because I'm not producing. And so it feels like I'm wasting time. Did you have that feeling? All the time.

As I said, it took me three weeks to go from SDK to 6 to SDK 9, put all the code over. It took another- Which doesn't seem that long, by the way. That is an extraordinary amount of time when you're a startup and your boss looks at you and is like, well, why isn't this done yet? I'm like, well, it turns out it's more difficult than I thought. But the good news is that once you get all the compiler errors figured out, you drop in

the new SDK. They changed the structure of the folder, so I had to remap everything and change everything where everything pointed to. But luckily, from 9 on, they stopped doing as much of that. But once you get the compiler errors fixed, then it's like, well, why isn't this thing actually working on the device? I know the RNO S110 and RSDK6 works. Why isn't this one? And then you turn out things like linker settings, memory settings. The software size has changed.

And at the time, maybe this is unique to them, but at the time, you did those calculations yourself. They didn't have a table and told you, hey, this is all here. You started memory location 1B000, and you go for this long. There was no table, so you did all those calculations yourself. And I'm like, wow, I haven't done this kind of calculation since, literally, since CS101. And it all needs to be done in hex. Yes. But now they have a table for it,

so they've gotten better. But yeah, all embedded developers are just really great at binary math, I guess. I think the hint, you know, looking at SDK 9, 10, 11, the version numbers should be some sort of indicator of trouble is here. Or you think they should have done 9.1 and 9.2?

I mean, it's like, okay, we're at version 9 and still making big changes? What's going on? Well, BLE is still changing quite a lot. Yes, it is. Oh, boy. Just need better abstraction layers. Better abstractions.

Bluetooth is interesting. It's a a i want to say all good things about bluetooth since to work on bluetooth you have to be a member of the bluetooth sig and you have to have their approval to have your name heard their name on your product so you make sure to say good things about them um bluetooth is great they're awesome everything's amazing there are no issues. It's not slow. You don't have to do random weird things to make it work. And yeah, everything's great.

And the 4,000-page specification is not hard at all. It's very readable. Very readable. There is an O'Reilly book looking over getting started with Bluetooth low energy that really helped. But it didn't take me all the way, but it at least got me started enough to understand what I was reading in the spec because the spec was not written for humans. Do you have plans to return to full stack development? I do. So I am ending my time at Jewelbots. Startup life has its own set of issues. And I'm ending my time at Jewelbots and going back to software development full time,

working as an architect on a system that needs to be built and is going to be built over the next few months. So I'm going back to a familiar world. Do you think you're going to have more sympathy for people new to that world? Oh, definitely. Yeah. It's I gave a talk this last week on doing embedded development and a lot of the talk was realizing how much of a beginner i was and you know relaying that and it's a humbling feeling and so having uh compassion and uh sympathy for beginners and helping them through is really important because it's a it's a hard way thing

to feel when you're so used to knowing what you're doing, going into a stack where you don't know anything. And maybe I can make it better for people that are learning. Are you going to miss it? I am going to miss embedded development. Oh my goodness. This has been so much fun. And it's still really cool to know that code that I wrote is going to be on a device that's going to ship, and that's literally the code that powers the device. Yeah, I'll definitely miss that. But when you're

a 10, you've been doing software development for 10 years, and you've only been doing embedded development for a year, and you try to go and say, hey, I'd like to be an embedded developer for you. And they'll say, well, you're a junior embedded developer. And I'm like, well, yes, I am. So we're going to pay you at the junior embedded developer rate. And And they'll say, well, you're a junior embedded developer. I'm like, well, yes, I am. So we're going to pay you at the junior embedded developer rate.

And that's a bit of a change from working in software for 10 years. O'Reilly has been talking for a while about truly full-stack engineers, from hardware to firmware to software to web. And Christopher's noise is exactly how I feel about about it from my ear that that full stack engineer isn't do you think that's possible and if if so do you think it's a good thing it's good that software developers are not siloing themselves to front end or back end, or hopefully, you know, with advances in hardware, we'll get more people dumping it, jumping into embedded development. It's certainly a good thing to learn new and different parts of the system. And that's only going to increase someone's economic liability. But I don't know if it's such a great idea to say, yeah, you should become a professional in this. Um, I would not want me to write mission critical embedded firmware code by myself. Uh, and so there's definitely a sense of, you know, this is going to be a piece of hardware. It's like recently we had the, uh, internet of things, uh, DDoS on the DNS system. And that's, I think, hey, if a jewel bot had Wi-Fi,

you know, I'm looking at the security of the system, would I, you know, possibly contribute to being a part of that as a new embedded firmware developer? So there's those scary thoughts of these devices can interact with the internet in our homes, and maybe we should have experienced people writing the firmware for them. But there's the good part of we need developers to learn different parts of the system

because they all operate in different ways, and the developers need to learn how to solve problems in different parts. So going full stack is a good idea. I just wouldn't say that people should purport themselves to be experts in different parts of the stack. I'm totally on board with that. I think learning all aspects of software development is really useful because you're going to end up working with people. Hardware, too. Well, I mean, now you're talking about learning a lot of stuff.

And mechanical, really. And mechanical. And physics. And chemical. Biochemical. But being able to talk to different teams and communicate and have some empathy back and forth, oftentimes, you know, I've had conversations in the past with other embedded people.

Ah, you know, the web people or, you know, they go off and they write some JavaScript, you know, and people have very, very... They have like six for loops and they think they'll all go wrong. Tribal attitudes about this tool or that tool or this language. And that's all very silly. And I think learning all that is great. What I'm afraid of is companies deciding full stack, meaning everything, is a way to go on the cheap and say,

well, we're just going to hire this person. They say they can design the schematic all the way up to the SQL database and so we'll just hire that person. Yeah, it's definitely not, that definitely shouldn't happen and definitely cannot. I don't

think that's possible because, I mean, even as a small startup, we had three different people. We had an E, an ME, and myself. And the ME dove in on the firmware and really did well in helping me debug it because when you're the sole developer on something, no matter how good you think you are, you're going to get lost in the weeds and having another developer there or someone else that can look over your shoulder and say, you know, you're getting stuck on this or I think I see the problem over here. That's really helpful.

You'll always need a team to build software. And one person doing, saying, hey, yeah, my new software developer, they took a course in schematics. Let's have them design the hardware. That's just a recipe for failure. So hopefully businesses learn that lesson. The businesses that don't learn that lesson will hopefully go out of business. But you'll always need specialized people that understand deeply different parts of the system. more than an eye level perspective of hardware and definitely even on firmware. I'd say, hey, I can do this, but I'm not a firmware person. I'm not someone who would purport to ever be an expert or even a journeyman in firmware.

It sounds like we all agree that you should learn as much breadth as you can, but you should accept some depth, both from yourself and from other people. Being good at something means that you may have a better shot of doing mission-critical things and doing things well and within the idioms of your language and system. Definitely. Which often means more optimal, efficient more maintainable that's the good thing about embedded development that's one of the reasons i'm glad i learned embedded embedded development because you solve problems differently than software developers do so when we needed to

send messages to other devices on a typical bluetooth transmission, you have 20, once you're connected, once you're actually talking with a peer, you have 20 bytes that you can send at a time. And so our first to send buzz messages, Morse code style buzz messages, you know, we literally sent an array of ones and twos across, one for short, two for long. And we sent those across and I picked up or I picked up a book on my shelf that I've had there for years that I've sporadically read and I remembered a problem in it called, the book is called Programming Pearls. And one of the problems in it was, you know, how do you compact that down to a bit array? And so I ended up, you know, instead of sending things across as an array of bytes, I ended up sending it as literally bits.

So one bit would be short, two one bits together would be long. And by doing that, I could compact down a size of 16 to two bytes. And that's a problem that in normal software development, you would just use an array or you would just use a list, and you wouldn't worry about the size of it. But because of the constraints of the system, I had to worry about those things. And it kind of opened me up to solving problems in a different way than I was used to. Yeah. And you might find that useful somewhere else. Or just having that thought pattern there, available.

But maybe we'll never have to use that optimization again. Ah, well. I think we have kept you for long enough. I do really appreciate you talking with us, because I forget what's hard. I try to remember, because I want to teach people because I really do like embedded. There are so many good things about it. But, you know, one time you and I talked about memory and pointers and C. And I'm like, well, I read the C manual in college and I've been using it ever since. I have no idea what you mean.

How is this hard? And that is stuff you have to recall what's easy and what's hard so it's nice to talk to you as somebody i trust as a good software engineer who just didn't know this piece right it's it's been it's been great talking to you and i i really enjoyed my time doing firmware development and unfortunately learning C at the same time. And so the particular problem I had was trying to copy an array by assigning one to the other. And that's

not how you do it in C or an array structure. And that's not how you do it in C. And I was really embarrassed to have made that mistake. Sort of can? Yeah, pointer from one to the other, right? But yeah, if you actually

want to do a deep copy then you should do mem copy and things like that and i had no idea and that was really embarrassing to know that i'd kind of messed up on something so fundamental um and yet so trivial i mean it is well that's a subtle that's a subtle bit of it is a subtle the deep copy versus shallow copy of structs is a pain in the butt george do you have any last thoughts you'd like to leave us with before we start a whole that's a subtle bit of C. It is a subtle, yeah. The deep copy versus shallow copy of Strux is a pain in the butt. George, do you have any last thoughts you'd like to leave us with before we start a whole new podcast about the intricacies of C? Oh man,

I've got to say that the embedded world is amazing and I'm glad to have been a part of it. And that I hope that someone out there listening says, you know what? Our tools could use some improvement and opens them up or makes them better for other people to use or start an open source movement around them. That would be amazing.

Well, it would be amazing. And I know that we do have people listening about tools, so maybe. Our guest has been George Stocker, VP of Software Development at Jewelbots. Thank you so much for being with us. Thank you for having me. Thank you also to Christopher for co-hosting and for producing. And of course, thank you for listening. My book is Making Embedded Systems.

It was published by O'Reilly. If you want a coupon code, you can get, I don't know, 40% off the electronic book and 50% off, I don't know. I can give you a coupon if you email me. I can also send you stickers if you email me. You can email me by hitting the contact link on Embedded FM or by emailing show at embedded.fm. All that, all that. And sign up for the newsletter, subscribe to YouTube, blah, blah, blah. Final thought, a final thought. This one, I had a whole bunch in the notes. And then as we were talking, I realized that there was one that was much better. This one's from A.A. Milne. I don't see much sense in that, said Rabbit. No, said Pooh humbly. There isn't. But there

was going to be when I began it. It's just that something happened to it along the way. Embedded FM is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive any revenue from them. At this time, our sole sponsor remains Logical Elegance.