Embedded - 475: Stuffed Animal or Colleague

Episode Date: April 19, 2024

Chris and Elecia talk about the Embedded Online Conference, their experience learning Zephyr, and some listener questions. Elecia will be presenting on Creating Chaos and Hard Faults at the Embedded Online Conference, Apr 29 - May 3, 2024. Some other talks that look interesting: The Power of a Look-up Table by Nathan Jones Zephyr Tools To Debug Hardware by Chris Gammell Breaking Good: Why Virtual Hardware Prefers Rough Handling by Uri Shaked Beyond Coding: Toward Software Development Expertise by Marian Petre Use the EMBEDDEDFM coupon for a discount (or if your whole team is going, check out the group discounts). Elecia’s book (Making Embedded Systems, 2nd Edition) is shipping (Amazon or Bookshop.org). Zephyr is pretty amazing.  Transcript

Hello and welcome to Embedded. I am Elysia White, here with Christopher White. Hi. How you doing? That's a loaded question. All right. On what dimension? What axis? I just meant in the human...

I'm doing great. How are you? I am doing okay. Feeling a little overwhelmed with things I committed to and not wanting to do anything at all. But overall, fine. Okay. Well, that was our show. Thank you for listening. Thank you for listening. No, seriously, fine. Okay. Well, that was our show.

Thank you for listening. Thank you for listening. No, seriously, seriously. Do you have things you want to talk about today? I have a list, so you don't have to come up with fake things. I think some of the things that I would want to talk about are already on your list. Okay. Start with a Z and end in F-er.

Wow, that, hmm. Anyway, moving right along. Wow. Okay, well, let's get through the self-promotion part of this. Do you mind? I don't mind. Why would I mind? I'm not the audience.

I am speaking at the Embedded Online Conference, which is happening at the end of April, very beginning of May. And you can have a discount coupon, Embedded FM, all one word. That is also, you can get a

discount if your whole team is going there, a group discount. So I'll put a link to where the group discounts are. But for Embedded FM, you have to use that. And what day are you speaking? May 1st at 9.30 Pacific time. Maybe at 10 Pacific time. Maybe they invited me early, but my meeting invite was for 9.30. And what are you talking about?

I am giving a talk about creating chaos and hard faults. The downside of my memory maps talk becomes clear to me now. The downside being? That there's no way I can make a talk of that call ever again. But the creating chaos and hard faults is fun. I start with talking about how to smash the stack intentionally, going through some really bad code. It was fun writing really bad code.

And then a little bit about debugging. And then I get into hard faults, and I actually look at those in cube, STM32 cube IDE, and talk about how they happen, how you can turn them off, why you should turn them on, and how to debug them while you're looking at them, but also how to store them to a bit of RAM and look at them on boot. And then I talk a little bit more about debugging, mostly debugging difficult things, not just how do you use GDB, which I don't go into at all, but how do you think about Heisen bugs and impossible bugs and, oh my God, I'm never going to figure

this out bugs. Which was kind of funny because at the time I was writing my talk, I had a really strange, I mean, it was I2S, which is a, it's like spy, but for audio, it's not a hard protocol. And I could see it on my logic analyzer and it mostly worked. I was getting stereo to one spot and I was getting mono to the other. But under some cases, which took me a long time to understand exactly which left right on both sides, I was getting noise. And there was Bluetooth involved here as well. So there were lots of channels where the noise could come from. And the noise had particular characteristics that weren't there when I generated tones on the processor. So

it must be my codec or the communication to my codec. It wasn't my codec because I could have my codec play things it was receiving and all of those of those stupid wire it was a wire i mean it was had a jumper cable they weren't very long but they were a jumper cable and it was a wire it's a problem with those those things like audio and camera and cameras to video where you've got a serial link that's delivering the data with like spy when you're talking to a device and you're reading stuff that has meaning it's easy to for the code to tell oh this value is out of range or stupid but if you're just streaming audio it can 85 percent work and the thing will receive it and decode it because it doesn't know that it's wrong it's just audio byte it's just

samples so then the same thing happens with cameras if there's a timing thing or some noise okay that shows up in the image not in an error yeah it becomes noise in the data yeah the fact that you can have digital noise i mean it's analog noise at the bottom but the fact that manifests as digital noise that can still be working but not quite is frustrating. Well, and I had so many new pieces. The codec was a new piece of hardware. The two BLE processors I was talking to were new pieces of hardware to me. And the way I was using them was not standard from their dev kits.

And I just kept assuming the problem was processor timing or something in how I set up the streams. And it didn't. So are you going to talk about this sort of thing in your talk? I mean, I'm going to talk about, I mean, I'll talk about the process I used, which is the process I usually use, which is to reproduce the problem with the minimal set. And to keep figuring out what minimal is. And then to also in parallel or back and forth in serial, talk about how to explain the bug. It's not, it's weird how often we just assume that a bug isn't possible and how explaining the bug, I mean, we talk about rubber duck debugging and just talking over your issue with a stuffed animal or whatever, or a colleague, it depends on how much help you need.

The stuffed animal, of course, providing more. But it isn't just the act of hearing yourself say what's supposed to happen versus what is actually happening, which is good practice. There's also this, if I wanted it to happen, how could I possibly create this book? Well, and that's good not just for thinking about causality, but if you've seen things, like if you make, like you're talking about smashing the stack or the other things you're going to demonstrate, if you know what those look like. They become a lot less frightening if you have seen them before.

Yeah, or you know what might be happening the next time you see it for real. Instead of, well, this isn't working. I don't know what that means, you know? Yeah, and that goes back to explaining the difference between what you expect and what you're getting. And with bugs that happen intermittently or are clearly impossible, figuring out those steps necessary to reproduce them and just going back and forth between explaining and reproducing the bug and realizing you're not going to fix it if you can't measure it. That's why you have to reproduce it.

But you're probably not going to fix it if you don't understand it, which is why you have to explain it. Can I explain it not going to fix it if you don't understand it, which is why you have to explain it. Well, let me explain the bug I'm working on. Okay, go ahead. Which you won't be able to. Is this the IMU thing? I helped with that. I did help with that this morning.

I don't know how much to say about it. It involves Zephyr, which we're going to talk about some more. But basically, I have a sensor that i've added to our build uh it's a it's got a standard zephyr driver that can be included although it comes from st's how module but that's it's something you can grab so the the code involved to add it is you know updating a dts and some configs so there's no i'm not writing any driver it's just a DTS and some configs. So I'm not writing any driver. It comes up and I can interrogate it over the shell and so forth.

And you can get the who am I from it. I can get the who am I because I walked in debug and saw that happen. But what happens is when I try to read it, when it tries to read multi-byte things, which are... Like the data. Right, the data. And what happens with that is the device is configured by default to if you do a spy read of one byte from one address,

and then you do another read, it increments the address and gives you the next thing automatically, and you can just stream out the data. What I was seeing is that I could read out one byte fine, but every other byte was subsequently not fine. Either FF or 00, depending on which way I stood on my head.

Which is bad, because I need the other bytes. It's surprising how the first byte of Excel X does you not as much good as you need. So anyway, I was fighting with this and went back and forth on all sorts of things. I can't easily scope it right now. I'm trying to fix that.

But then somebody at the company said, hey, I ran your code and it works fine. Woohoo! Which is, I think I mentioned today somewhere else, that this is the inverse of the it works for me problem. It's a new variant. It doesn't work for me. It works for me problem. It's a new variant. It doesn't work for me. It works for everybody else. But we went back and forth, and we discovered that he wasn't quite running

the same code I was running. He had added some stuff for a completely different peripheral. A completely different bus. Different bus, different bus type, different hardware, on a different board even within the system. And yeah, totally different.

And that makes it work. Woo-hoo, works. Worked for him. I took his code and made it work. So that's disturbing. So we don't know why it doesn't work in some circumstances. And I added the other thing we wanted to have, which I won't talk about too much,

which is an additional config to this peripheral and to stop working again. which I won't talk about too much, which is an additional config to this peripheral, and to stop working again. So with his code, the other item doesn't work. And it goes back to not working. So it's something about power-up sequencing, or power, or something. Timing?

Timing. Interrupts being set or not set? How would you like me to debug that in Zephyr? Okay, so I have more to say about the Embedded Online Conference. Okay, let's finish that, yeah. But let's go to Zephyr. Okay. Because clearly we both want to.

And I was actually, I had this whole plan of how I was going to bring this up, and I was like, so have you been writing much code lately for your work? No. Well, for Zephyr, I did do one thing. I ported an ST driver that didn't exist in Zephyr. And that actually went okay. But did you actually write any code, or did you just copy some files and modify it? I had to write some code to the platform dependent. There's a couple functions I had to write to do the I2C stuff for Zephyr.

But overall? Overall, no. Mostly it was moving files around and editing the DTS and DTSI, which define the devices, and the config, which defines the config, and some other stuff, which defines some other stuff. I think the KConfig, is that the thing other stuff i think the the k config is that the thing where i put sensor in yeah maybe that one oh no i had to do some more because i had to make it conform to the sensory api okay which it didn't so so that was the thing but no yes i have not written a lot of code i have i have i have stared at a lot of things and made small it's

like the moment when you've got a beautiful sculpture and there's like five more chisels you have to make to make it work. That's what I've been doing. Okay, I don't have a beautiful sculpture. But the metaphor is bad. You have a sculpture. But to make the things I've been asked to do work so far it has been mostly

adding three lines, figuring out what three magic lines in a config need to happen. That's the thing with Zephyr. Every time I'm seriously thinking about writing code, I realize I must be doing it wrong. I mean, I did write some code. I added a command so I could test this thing, and I mashed a couple things together.

But overall, if I am writing code, I really should be looking at how to do this differently because it's all there in Zephyr. People have done a lot of work on Zephyr. So that's the thing about Zephyr. I think my... Oh, wait a minute. Zephyr is a real-time operating system and ecosystem.

It's a way of life. I don't know. It really is kind of a way of life. It has its own build system. West. Well, West is a command that drives the build system. Ninja? CMake? CMake and Ninja is part of the build system, but there's a whole structure involved. It has its own config thing, which is...

West even does Git stuff. Yeah, West will do Git stuff, because you'll have to, because submodules are an integral part of Zephyr, and West manages those. See, I'm doing Zephyr, and I haven't had to use West for Git management yet. But I've been really careful not to edit anything. Do you ever type West update? I try not to.

So I had to add a module from somewhere else, and that was adding a few lines in the config file, and then doing West update. I try not to. So, like, I had to add a module from somewhere else, and that was adding a line in the config file, a few lines in the config file, and then doing West update. So the config files have, like, they point to different repositories. My goodness, so many different repositories. Anyway, we're getting into details.

It's very confusing, but yes. So Zephyr's this big thing, and people have put a lot of work into it, and it's really neat. I. And it's really neat. I mean, it's really cool. And I do really appreciate it. And many of the choices they made are the things that I would make. Yeah.

Like the whole macro thing. I probably would have done that. Well, if you're going to use C, what else are you going to do? And even though as a user, I am sometimes baffled by why I can't, but it is pretty amazing. It has. So the things that are good about it, let me start with the things that are good about it before I complain, because I have some complaining to do. Do you? The things that are good about it is people put a ton of work into this.

There are board definitions for so many boards. There's drivers for so many boards. There's drivers for so many devices. And if it doesn't have one of them, it's likely the vendor has another repo somewhere you can point at

to get a Zephyr driver for it. It has, you know, all the basic peripherals of done, done, done, spy, I squared C, all that stuff. I kind of like how, now that I kind of get it,

I kind of like how you can define your board and the pin stuff and how that's easy to change. And you can even, at the build time, you can say, I want to build my code for this board. And then you can say, nope, I want to build my code for this board. And it should work on both boards, as long as everything's

configured correctly. And that is super magical. The idea that you can go from a Nucleo L4 board to a Raspberry Pi, and all you do is change your build. Or a simulator. It has QEMU, so you can just build for a simulator. So that's really cool, and it's kind of the promise of, okay, now we're moving away from the platform dependence that we've really been stuck with with embedded to something that's got a lot of power to abstract that all away in a way

that's not too terrible. Like, the abstraction layer is there, but honestly, it's easier to follow the links. Like, I was having to debug the spy stuff. It was kind of easier to follow than getting into ST's HAL once I did it. But there are some points where they're like quasi-function pointers and things, and debugging gets a little tricky. What else do I like?

I mean, the build system's fine. You don't have to touch it much. If you feel like you need to know CMake, you probably shouldn't be touching anything in CMake when you're working with Zephyr because it's all supposed to be run through the config files and that stuff. I added some files to CMake lists. Oh, yeah, that's one you do have to add.

Yeah, right. So the most you should have to do, I think, is CMake lists. Which is just adding your file name in amongst all of the other files in your directory. And if you're trying to do something tricky, like add a library, there's other stuff where that gets tricky and you have to do things. But the goal with Zephyr is to play in the sandbox. Yeah. Don't get out of the sandbox unless you really have to. And if you think you have to, reconsider about five times because there's a good chance

you don't actually have to leave their world. Okay. Now. So, okay, things I'm not quite, there's some things I'm not quite clear on, so let's start with that before I complain. I mentioned the build system and how it's integrated with Git,

and it pulls in repositories from places. It has modules and stuff, but there's also how you can structure your whole application. This is where it gets really tricky, and I've read a lot about this, and I don't quite understand the right way to do things. We're both working on Zephyr, and we have different ways of doing it, because our vendors set us up with different ways of doing it, and we have different chip vendors. So there's this thing called, so like you can,

the Zephyr repository is the OS repository, right? And you probably don't want your

application to be in there. Right, because I want to commit my application to my code. Right,

and you don't want to fork Zephyr and carry that around, although everybody does that. Apparently everybody

does, but we're trying not to. So there's a bunch of ways to have your application be in another repository and have both of them side by side. So you check out the Zephyr repository as part of West. It also checks out your repository. And then magic happens in the build system that I don't understand.

So you can just type West build and it knows to look in Zephyr for the Zephyr stuff in your repository for your application, plus any stuff that you're kind of adding on top of Zephyr. Like if you want to replace something in Zephyr, like you want to have your own board file. Not even replace. Well, yes. Not replace, but add. Add. So if you have a custom board, you put that in your repo in the same kind of directory structure that the Zephyr one has.

If you want to add a device driver, you do the same thing in the same kind of directory structure that the Zephyr one has. If you want to add a device driver, you do the same thing in the same kind of device in the same structure. You have this, so your, your repo looks like the Zephyr repo just with nothing much in it. So that's super weird. And I don't understand how that works, but there's like three different ways that Zephyr says you can do that. And they call them, I think they call them topologies or something, but it was like, you know, there's hub and spoke kind of thing. And I glossed over at that. So that's confusing.

And if you asked me to make that work from scratch, I would probably, you know, do something that might work but not really understand it after reading the examples. I'm not wild about errors. Like with the config system. So the config system, as you mentioned, is this series of files. There's ones called DTS, which are the device tree specifications. And that's where you put all your peripherals that you define. And you can do the pin control, so you define what pins are for what functions, whether the GPIOs, how the GPIOs are configured, etc. My spy is on pin 0.09.

Yeah, yeah, yeah. And it's a whatever. And if you want to go to sleep, it's on 0.09, and these are its sleep parameters. So there's all that, and it walks all of that and builds the real config that it builds the code out of. The problem with that is if there's a problem in those files, it sometimes is fine. It will give you an error at the line you did it. But sometimes, like, I had one.

It's like old C compilers that would barf at the end of the file, and your error was on line two. Yeah, I had one I was fighting with for a while. It wasn't with the DTS. Maybe it was a consequence of the DTS. It was something where there was a config macro. So all the config macros in the config files

end up being like config underscore spy. If you want spy in your system, you have to do config underscore. That's going to be in the config configs and proj.conf. And some of those end up requiring other configs to exist? Yes, and you

can have it all treed and disable and default if and default else. And sometimes it will decide you need one,

but for some reason there's no default. So it just puts config whatever thing I need equals and then stops. And that's what the actual config file that it builds from has. Config equals blank.

Equals like nothing. It's a new line. Config equals and then new line, the next config. The consequence of that when you build, and people might want to know this, is that everything will build until it gets to an assembler step. And then it will give you an error in the assembler. He was so mad about this. At a particular line in a temporary file. It is.

At a particular line in a temporary file with a random string of letters as its name that no longer exists because once the build fails, it cleans up the temporary files. And so I ended up looking on the internet, and I found somebody who was complaining about something sort of similar, and I looked in that config file and saw the blank thing. And yeah, that's what happens. It turns out it gets blank, and then once it gets, it happily builds that and puts in a line that says, you know, like, if equals nothing somewhere, and then...

But the compiler doesn't give you an error. The assembler gives you an error. It doesn't tell you what it is. It just says expression not complete. I think it said expression blank or expression not complete. Yeah. Something that was really not helpful. And it doesn't tell you what the expression was.

No. And it's deleted, so you can't go look at the online. Right. So, that was from the kconfig. Like, I kind of feel like the build tool, the thing that's parsing the kconfig files and turning it into the config file, should notice when something isn't populated and maybe give you an error there. It doesn't seem like, you know, a difficult task. But there's a lot of things like this where something goes wrong and the system is so complex and it doesn't give you an error where the error actually occurred that it's really hard thing. It's so complex that the

error had nothing to do with what I was doing and didn't make sense. And how do you... I would remove that line to make sure it really was that line causing

this, but then where do you go from there? Well, as I told you, one of my Zephyr debugging techniques is to write the word poop in the middle of config files just to make sure that the config file is being... You know, the rest of us just write like ASDW or some keyboard stuff.

But he actually writes poop. There was a ton of times where something wasn't changing. I don't think I ever solved this. I was changing, was it a GTS file? It might have been a GTS file, and it was having no effect. And I don't know how I got around that. But, oh, no, I was trying to pull in an ST module and trying to figure out how to do that. And I edited, I had a west.yaml.

Ha! Another thing we didn't talk about. The top level thing is the west.yaml, which defines where all your code comes from and stuff. But that had an import to another one. And I changed the contents of the imported one. But West didn't ever notice it. Because you needed to do a West update?

No, it's supposed to notice during West update. That's what I was doing. Because you needed to do a West update? No, it's supposed to notice during West update. That's what I was doing. Oh. I never do a West update. Never ever. But you weren't changing things that required a West update. That's why. I don't know if I ever...

I was adding something. Okay. I had to. There's no other choice. But anyway, so that one is a mystery to me. It doesn't seem to recursively notice dependencies if you do an import in West YAML. So Zephyr is cool, but I feel like it's a really great system.

But I feel like it's the guts of a really great system and that something smoother needs to be on top of it somebody needs to put some buttercream on west or on zephyr that's a metaphor one could use uh but yeah but it's like gosh i don't know i don't know i you know i've only been using it a couple of months so i'm not an expert definitely not an expert uh and you know maybe as i get used to it more it'll be less less uh less frustrating but i do feel like some of the stuff is like okay this is this is the guts of the system and you have to know more than you should like the way they abstracted away the build a little bit, the CMake and the Ninja, so you don't really have to touch that so much.

I want a little bit more of that. Like, the DTS, the KConfig stuff. I mean, there are GUIs for some of that. But then if you change something in the GUI, it goes someplace random and you have to figure out how to put it back in your system, which is really fun. And I don't know what the solution is. I don't even really know what I'm asking for. Something simpler. Something a little easier to understand.

And maybe I'm just asking for better static analysis. Yeah. And I think that's coming because I mean, the macrobatics, the macro stuff that is... Macrobatics? Macrobatics, that was the term that Trant used to describe.

There's a talk about it. And it was really useful in debugging some of those bugs that come up because of the build system. But they are complicated. And it is really overusing the C pre-compiler. Yeah. In amazing and horrible and amazing ways. Yeah, and I don't know what to, I mean, they probably thought about using, making a meta

compiler doing YAC and LEX or something instead, but. I mean, they've still got all these DTS and DTSI and YAML and DEF configs and K configs and so it's not like they aren't using other compiler things. DTS and DTSI and YAML and defconfig and kconfig. So it's not like they aren't using other compiler things. The hierarchy stuff is both a blessing and a curse, I find, because it's really hard to find things. It's so big. That's the thing.

It's like, I've got it in VS Code. If I'm looking for stuff, like there's the board directory and then there's the DTS directory, which is not the same as the DTS side. I deleted all of my extra board files, so it's just the ones I'm using. Yeah, but I can't really do that. Well, I'm not going to commit it. It's not my repo. Oh, I see.

Well, I'll accidentally, yeah. Okay. Well, but there's like a get sparse command that's supposed to help with that. So maybe that's helpful. But that would be nice if that was an option that you could do like with West. Like say, hey, West, here are three boards from Zephyr that I'm going to use. And blow away everything else so that when I search for the things that I want. And blow away, make the Zephyr repo sparse.

That would be nice. But just like the hierarchy, like there's... We're just going to title this one Sigh. I understand this stuff, but on some level I don't. And some of the choices probably were difficult to make. And maybe they all came from Linux where they have a long storied history, which is fine. I mean, honestly, if I was going to start a new project for myself right now, there is nothing I would choose other than Zephyr. Having gone through the pain of this, of the learning curve, it's a nice enough system.

It's modern enough. It's got a lot of features and drivers and stuff that if I was going to build a system, I would probably reach for that immediately because I don't have to screw with Cube or any of that other stuff or weird mobiles. Just pull up, do my pin definitions from some dev kit and go. Because the RTOS itself is nice, too. I mean, the actual API is pretty cool. The kernel. What I would consider the kernel. The scheduler, the muxes. It's got a lot of features.

RTOS part. But Zephyr, when you talk about it, it isn't just the RTOS part. No, no. And that's a mistake. That is one of the most common mistakes is to go in thinking, okay, it's just an RTOS. Yeah, it's like thinking Linux is the kernel. And as the GNU people would like to point out, it is GNU Linux because it includes all the GNU stuff around it. And that's just like Zephyr.

It includes a lot of stuff around it. And people have put a lot of work putting hardware stuff in there that you just don't get with other systems. It's really nice. I have complained on this podcast multiple times about having to write another damn spy driver for an Excel or whatever. And Zephyr really is trying to make it so you don't have to do that. Right.

And even the driver I didn't have, I was able to find an open source driver and it was adaptable very quickly. So it's, I think it's the right direction. And you kind of have to, you know, suffer through a little bit of learning curve. It makes me think about, I read this book about Bluetooth a while ago, and it was awful. And I'm not going to call out what it was because after I learned about Bluetooth, after I got used to it, mostly through trying stuff, but also through like reading documentation from Nordic and TI, but really getting into the whole, what is Bluetooth and how does it work? I ended up going back to this book for some reason.

I think I was trying to find a book to recommend to someone else and I didn't realize I was picking up a book I already had. And it was a great book. It laid it all out very clearly. It made a lot of sense. Because now you knew what it was talking about. And so I think there are instruction materials for here is what you need to know going in and instruction materials for here's the reference and how to balance out what you don't know. You know, there's, you know, the basics. Here's all the stuff that you also can do.

But that book was terrible for a beginner because I didn't need the excessive information. So yeah, so that's part of the problem with me, not with Zephyr, is at least for what I'm doing with it, it was the worst possible scenario for learning because I wasn't taking an ST dev kit and putting some sensors on it and firing up the build system and writing a little example application. I was working with a, I am working with a real system, which is a custom board with many, many, many sensors and buses on it with a custom, not a custom, with a rare MCU, which has its own Zephyr repo because it's from the vendor and it's rare.

You know, I had to, yeah. So I had to do all the hard stuff with Zephyr that you maybe normally don't encounter. And I had to do that first, which is a great way to learn and a terrible way to feel good about yourself um so yeah so even with that my impression is still pretty favorable but man like trying to figure out okay i need to like just even stuff stupid stuff like the driver for the um this is super weird so the driver for

the the imu i was putting in looked like it was in the main zephyr repo but it wouldn't build like if i brought it in it wouldn't build because it said it was missing files well the deal is the way st puts their drivers in, I guess, I don't know, they put half the stuff in there, the non-part-specific stuff, kind of, sort of, but then they have another repo that they have as a Zephyr module that has the other half of the drivers. And if you don't put that into your system, if you don't pull that in in your West YAML, then you don't have the complete driver. Because all the register stuff is off in the ST's repo.

But the driver's in the main Zephyr repo. Why? I don't know. Because they have a directory for every single one of the chips where they could have just put the register files in. But anyway, that was like, what? And so my initial response was, well, they just didn't finish.

And I found the register files and I put them in the Zephyr repo. And I was feeling bad about having to add that to our Zephyr repo. And everything worked. But then I realized, no, that can't be, that can't be how this is supposed to work. So stuff like that is really weird. And I don't understand it to this moment. But yeah, I mean, as soon as you get outside the bounds of just piecing stuff together,

any time you've got custom things going on, when you're going to reach into the DTS and really make some modifications, then things get a little bit trickier for the learning curve, I think. Which I would have appreciated starting with the basics, because I was trying to learn the basics along with the advanced. And that's where pain lives

because you feel completely incompetent about everything and question your understanding of the basics because you're trying to move past that really quickly. But I think that's really common. You get the basics working and you don't necessarily understand them. But you immediately want to change something. Yeah.

Yeah. But it's a small thing, right? I mean, you went full, all out. But it didn't take me long before I needed to change something. But yours is complicated, too. Yours is complicated, too. Mine has a bunch of algorithms. And you have a multi-part system.

So many parts. Yeah, so I think... I spent a lot of time just drawing... Going back to explaining the book. I spent a lot of time just drawing out what the system was and what I wanted it to be. And each one was like a full 8. half by 11 drawing of which parts of the system went where. It was, is horrifically complicated, but. Okay. So. Zebra has a shell. I just want to plug one more thing.

I love the shell. It has a CLI. It's so easy to add stuff to. Never have to write another damn CLI if you use it. Oh, yeah. It's good. It's got history if you want it. It's got all this stuff. Tab complete? Did you set yours up for tab complete?

I don't know if I did. It's got, you know, basic, you can query I squared C bus if you want and scan it and do all this basic stuff just out of the box without doing anything. And you can add commands to it, and it's not that hard. Actually, Chris Gamble is doing a whole talk about Zephyr tools to debug hardware at Embedded Online Conference. Oh, I should watch that. So anyway, just ending on Zephyr, it's very promising. More than promising, I think. It's there.

It's just the learning curve is steep, especially if you're diving in at the deep end. So kind of on the topic of intro versus reference materials. And by the way, nobody likes your Doxygen docs, okay? Nobody even uses your Doxygen docs. They're horrible. I don't care how much time you... Doxygen is an entire system devoted to convincing people they've written documentation. But there's actually... Writing code versus reading code.

Are they different skills? Oh, yeah. I feel like at the beginning of my career... I have an analogy for this. Okay, go ahead. I've been learning Morse code. Mm-hmm. Oh, yeah.

I can send code with no problems at 12, 15 words per minute. I cannot read code at more than about nine words per minute right now. Well, that's because it's got weird things like I's are two E's and C's are two N's. It doesn't matter. But I can send that. Well, yeah, but you. But I can send that. Well, yeah, but you don't have to parse that. Exactly.

You don't have to parse your own code. I think the thing with Zephyr, for me, Zephyr, Vephyr. Vephyr. Vephyr. Vephyr Vephyr. One of the things with Zephyr has really made it clear that while I write code very quickly and usually do a decent job, I need more practice with reading code. I'm not terrible at it, but it's a very different skill. And it isn't even working the way it used to for me, which was, you know, you take a file, you list all of the functions in the

file, you write a little note about what each function is. Because stuff's not structured that way anymore. And then maybe you have a little bit more about what function calls what function. It isn't structured that way anymore. And it isn't just reading the header files or understanding the structures.

That's how modules work. You can't do it that way if everything's going to plug together and link together and in various ways yeah and i think it's the dialect thing right like there is no code that somebody there's no one code language that people write like yes people write c or c plus plus but everyone has their own dialect, whether the way they choose variable names, whether even conforming to a coding standard, how they structure stuff, how much they put in a particular scope. All that stuff matters to how you read it.

And every time I come upon somebody else's code, that stuff's a little bit different. And it takes some time to get used to and it takes some time to get used to. It takes some time to get used to, but I do tend to think, to find that I write in their dialect pretty quickly. Sure, yeah, yeah. But that may just be a consultant thing that I'm just used to being flexible

with matching what's around. I found reading Linux kernel code to be maddening. Like Linux kernel code is extremely difficult to read. There are no comments, usually. There's a lot of meaningless variable names that are very short and lots of logic doing stuff that's hard to understand. I have not read a ton of Zephyr source code, and I don't know if it kind of conforms to that,

but I did find Linux one of the most difficult code bases to read when I was working on it. I guess the code I've been reading is mostly Nordics layers on top of Zephyr, so I don't know that I've gotten down to the Zephyr much. And vendor

code tends to be... Vendor code tends to have a flavor, doesn't it? Oh, each vendor has their own flavor, absolutely. But there's an overarching whiff of vendor flavor to stuff. I can't describe it. I think it's arm flavor, actually. Oh, maybe, maybe. But while we were debugging a DTS problem that I had,

you suggested I use ChatGPT, which was a shock to me when you suggested I use ChatGPT. I think I suggested using Gemini, but that's fine. Well, in the end, we used Gemini, yes. And it was thoroughly unhelpful and very confusing. Yes. On the edge of useful, though. Yes, that has been my experience with trying to ask questions of it about anything. It's almost right and wrong. But it was another instance of I really had to read it very carefully.

Yeah. And in this case, it was an instance of I had to read it very carefully and then throw away everything I read and what I should have been doing was reading very carefully some other document that I don't know where it is or what I was doing before you walked in the room

which was copying files and trying to figure out which ones I could delete, which ones I had key yeah the times I used ChatGPT or Gemini copying files and trying to figure out which ones I could delete, which ones I had key. Yeah. The times I use ChatGPT or Gemini are times of total desperation. Like I've run out of ideas. Fine.

You might come up with something that will jog my memory or point me in a direction, but I don't believe you're going to give me the answer. So, yeah. That's why I suggested it. Might as well see what the robot hallucinates, and maybe it'll point somewhere we haven't looked. It was interesting. I know a lot of people are using ChatGPT and other AI coding tools.

It isn't a tool I... I do not use it. It's not one I reach for. It's not one I would have ever thought of. And as you probably realized, I solved the bug as soon as I stopped looking at that. Yeah. And yet... That's because I told you to put poop in the ATS file. And then I realized it wasn't being compiled.

Yes. Your poop is great poop. Chat GPT will never tell you to put poop in a config file. Let's just keep coming up with titles right now. Okay, let's see. And sometimes that's what you need to do. Put poop in your files. Let's just keep saying poop until you get it.

Anyway, what's the next topic? You know, there are two nine-year-olds listening to the show right now. I'm pretty sure of it. Yeah, maybe. And I'm betting that at least one of them is running around saying, poop, poop, poop, poop, poop, poop, poop. And if they weren't then, now they are. You're welcome.

More talks from the Embedded Online Conference. Breaking good while virtual hardware prefers roughling by Arish Aked, Beyond Coding Toward Software Development Expertise by Marian Petri, and The Power of the Lookup Table by Nathan Jones. tables. That's my number one go-to for all problems that require things to be fast. Math, databases, lookup table. Lookup tables are cool. I must build a lookup table for an entire screen. Like I had to de-warp.

Did I do de-warp or warp? It was something where I had line scan data coming in, but it corresponded to polar coordinates. And so I needed to display it very fast as a donut image instead of the rectangular. And this was like 2007. So computers were fast, but they weren't, you know, 2024 fast. And so I needed to do this at high resolution at 30, 60 frames per second or something. And so I was doing it with, you know,

trig functions and calculating each pixel. And it was working, but very slow. Not very slow, but too slow. And then I hit me, I could just, for every pixel, I have enough memory. I have, you know, whatever 2007 memory was, it was plenty, to store for every pixel, whatever they are, the transformed coordinate. And it just went through and it did a lookup for each one and put it in the right pixel spot. And then I somehow did bilineary filtering on top of that. But that was super fast. But if you have enough memory, lookup tables can solve a lot of problems

that you'd do with computation. Good talk, I'm sure. James Granning and Jack Gansel and Jacob Beningo will also all be giving talks. I mean, there's plenty of stuff there. It's good stuff. So if you are interested,

EmbeddedFM is a coupon you could use. I honestly don't know what the price will be after you put that in. Maybe it'll go up. Maybe it'll go up. Yeah. More self-promotion. My book, Making Embedded Systems, second edition, is really, really, really, really shipping from Amazon and other physical booksellers.

You can get it there or bookshop.org or ebooks.com or Google Play or wherever you want. It does seem to be getting good reviews. The Apple bookstore, somebody wanted it on the Apple bookstore. It should be there. It was not there in their country reviews. The Apple bookstore, somebody wanted it on the Apple bookstore. It should be there. It was not there in their country, I think. Ah, yes. I don't read the reviews, so if you actually want to tell me something from the book that you liked or hated, you'll have to email the show at showatembedded.fm.

I read the reviews. Christopher reads the reviews, and he occasionally tells me that somebody mentioned the chicken button, which, of course, I'm very happy about. Let's see. Nordic Winners. Oh, right. So Nordic was sponsoring the show, for which we thank them very much. They

are giving away three PPK2 power analyzers. It was supposed to be one a month, but apparently I suck at that. So now we're just going to give them all away. Farzad, Julian, and

Adrian, you should all expect emails from me. And then I will say, so you won. Can I give your email to Nordic? And then Nordic will drop ship you stuff. Because we all know I am terrible at shipping things. Let's see. How, how, question, remaining question from Guy Randy. I recall an

episode where you talked about installing an antenna to talk to your dad. How is that going? It's going slowly. But it is going. I have assembled the radiator, my transceiver, which was a much more involved kit than I expected. Ask him about toroids. That took several weekends. I had to wind more toroids than I would like ever. And I tested that with Whisper.

So I just transmitted Whisper with that. I did some scanning around for some CW on 40 meters and heard some stuff. So with just a random wire I had stuck out of it. So that's good. I have a dipole that I need to experiment with. I have antenna tuners. I'm

learning Morse code. Really the only holdup now is getting Morse code to where I the only holdup now is getting Morse code to where I'm feeling comfortable enough to try out some... See, I don't even know the code for talking to someone on CW. Anyway, yeah, it's going well.

My dad's got his radio and stuff. I need to help him with the GPS because he wanted to try Whisper and the GPS assembly for the QRP gets a little weird. It's flexible, but it's unclear how to do some things

without some extra parts. It's happening. It was nice to... The kit was complicated enough that I was so shocked when it just worked the first time. You were pretty shocked.

I'm still not sure it's actually really working because there was a lot of, you have to, there's three trim pots and a trim cap that you have to adjust for the filters and stuff. And it's got internal test tools. So it's got a screen and stuff and some knobs you turn. But those have to be adjusted while transmitting into a dummy load. I guess I did it right, but I think I want to do it again. I mean, you transmitted, you got pings from various people.

You went on the Whisper map and you saw it got here and there. I think a lot of them were for reception, so I might be able to improve the noise floor and stuff if I do that again. But yeah. Cool. What else have you been working on?

Drums, piano, Morse. I haven't played drums in a while. Zelda. Some piano. The dog. I'm learning some, working on some piano stuff. But I don't know. I'm working on nothing.

That is so not true. You're always busy. I know. I probably should stop doing that. I have two clients too. One of them is going to ramp up soon. Okay. Well, I think that covers.

Didn't anybody have any other questions? Oh, right, right. I had a list. What is NP and P? And what do those even mean? We answered that in the last show. With Ina? Yeah.

Did we really answer? Yeah, polynomial and non-polynomial. Polynomials are easier. It's a... It's kind of bounded, where non-polynomials are not really bounded. It's that you can express the... The time it takes to do this algorithm.

To do this algorithm, yeah, in polynomial time, rather than e to the x. Which is more unbounded than polynomial time. Here's the... Yeah, the number of steps required is order of n to the k, where the number of steps is n and k is some number. So k squared,

or n cubed, n to the fourth, n to the sixth. So non-polynomial would be one where that doesn't hold true, which means it could be exponential with the input or worse, doubly exponential. And P bad. And so the idea of P equals NP would be a proof that says, well, all problems that we think are non-polynomial are actually polynomial.

Which would break lots of this. Which would mean there's solutions to lots of problems that are faster. I think I've got that right. If I don't, then I don't care. Then we're revoking your CS degree, which you don't have, so you don't care. Not have a CS degree, yeah. Let's see. What is your favorite computer algorithm?

And which algorithms do you use the most? I can't remember the last time I used an algorithm that's not true i can remember a time i used an algorithm i'm trying to think of the last time i used one i mean you did a big deep dive into the architecture of a machine learning system so hey okay are those algorithms, though? That's just a bunch of linear algebra. All right. My favorite is an FIR filter because I am super lazy. Those are nice.

One that I used and understood extensively that is probably the most CSE algorithm I worked with. I did a lot of stuff like when I was doing networking. Networking, there were a lot of more fun algorithm kind of problems. So one of those was Dijkstra's algorithm, which is a method for finding the shortest path between two points in a graph, which you have to do for certain routing protocols. So for OSPF, which is the open shortest path First routing protocol, which I implemented an implementation of, you have to do that whenever something in the network changes. You have your view of the network and you need to compute the shortest path between various points. And so that's a very efficient algorithm for finding the shortest path in a graph between two points.

So that one was fun to work with. There was other fun stuff there, like ways of doing timers, millions of timers on a system. So we had to have millions. Because there's a lot of timers associated with all sorts of entities and networks, you end up on a route. Packets timing out? Packet timing out, but more usually routes or, because packet timeouts are just counters. Okay.

That they're in the packet. Okay. But stuff like, oh, I need to send an update or I need to time out this route entry and refresh it. But you might have 60 million of those or something. And so you can imagine how do you, you know, you're certainly not going to use hardware timers or a linked list or something like that. So there were a lot of fun algorithms for how do you manage 60 million timers that could time out in an efficient way so you're not constantly scanning. And so you have one master real timer that drives all that.

That was cool. Yeah, I'd have to think about it. Hashtables are another thing I use all the time. Yeah, hashtables are fun. They're weirdly efficient. Laplace versus Fourier. Fourier. Yeah.

I never understood what Laplace was good for transfer functions I guess I don't do a lot of transfer functions I do a lot more signal processing have you seen or read Dune do you have any thoughts about it how much time we have I believe 10 more minutes.

I have read Dune through God Emperor of Dune. That's one and two. No, that's one, two, three, and four. Okay. But you recently re-read Dune and Dune Messiah,

which are the first two. I quite like the Dune series. It's got a lot of thought-provoking stuff in it. It's, you know, of its time, so there's elements of it that are a little squishy. But it ages better than a lot of, in my opinion, it'll age a lot better than a lot of sci-fi from that era. It's more, I found it, I hadn't read it in a while so i read it recently very recently actually um i found it more literary than i remembered it so compared to other sci-fi it was less swashbuckling adventure and more kind of deep so it was cool um it as a series it gets

very weird from each book so the first book is the least weird, and the second book is more weird. The third book is pretty weird, and the fourth book is what? So I kind of crashed after the fourth. I'm not even sure I finished the fourth book. I know I started it, and then I think there's one or two more after that, and they're just out there. And then his son wrote some follow-ons, which I hear are not well-regarded, so I haven't bothered.

I have seen the first, we've seen the first movie that came out a couple years ago. We have not seen the second part yet. I quite, quite liked it. And it's, it's pretty close to the book, but they cut,

their choices were to, sometimes when you're adapting stuff, you change stuff, or you can cut stuff. And they mostly seem to have cut stuff, um, which is, which is fine. A little disappointing,

but you know, movies can't be eight hours long, but yeah, that's cool. And it's, it's, some of it's especially relevant to today. And there's some stuff about AI that's pretty opinionated, which I quite enjoyed.

But without spoiling much, it's set in the year 10,000-something. And this is a human society. Earth is mentioned, so it's not just a random part of the galaxy or something like Star Wars. So it clearly references back to earth history and things but there's very few computers because they had a war over computers because the ai started causing trouble or people people started using ais for trouble i think they made that point very clear it was not the ai ais doing things it was people so yeah so this if you watch the movie it's not really well described in the movie at all but a lot of the technology looks pretty basic like the control panel for the ornithopters and stuff has just switches and dials and

you know analog stuff that looks like it could be from World War II. There's a reason for all of that. It's not because they just wanted to look cool. It was because computers are basically banned. Anyway, I've said things about Dune. Did you have things to say about Dune? Having read, I think, Beyond 4, but not much beyond 4, it definitely had very diminishing returns.

Yes, yes. I haven't read them lately. I thought that the last Dune movie was very pretty and very engaging. I know there's another one out we haven't seen yet, and I don't know anything about that. It's the second half of the book. I'm looking forward to seeing it. But it's very much with both the first book and the movie,

if you don't enjoy the first 10 minutes, you can just leave. Yes, definitely. And it's not a happy story. It isn't a happy story. And the hero is not a hero. And he tries to make that very clear, and I think people still somehow miss it. I think in the movie they did not make it clear that... Oh, they did.

Did they? Yeah. And they're going to make it, from what I've read in the second movie, they make it even more clear that he's not a hero. Okay, so one more... Sorry, there are two more questions, one of which I don't want to but. Okay. So one more, sorry, there are two more questions. One of which I don't want to answer, but this one is from Chris Greenlee. When in your career, did you start feeling like senior engineer roles were the right fit for you?

Five years after I was given them? I don't know. I was given the title of senior engineer four years after starting, which was a bit too soon. I did a lot of senior engineer things at that point and had a lot of responsibility, but I don't think the work I was producing, I hadn't quite figured out how to write code at that level of competency yet. So my code was not good. I think my designs were good and my thoughts about systems were good,

but I don't think my coding skills were that good. Mine was about four years as well. But that was because I went to a startup and I was the only firmware engineer. Yeah, I also went to a startup and I was the only firmware engineer. Yeah, I also went to a startup. I was one of a few same software engineers. But the startup you went to, you were one of several people like you. Yeah, yeah, a few.

And the startup I went to... You were the only person. I was the only one like me, which was like coding without a net. I just remember being jealous of your coworkers. Not that mine weren't great. Mine were great.

My double E and my algorithms guy were both fantastic, but neither were them. There was nobody to bounce ideas off of. There was nobody

to play with. Yeah, yeah. After that job, I felt like I had definitely earned the senior title having put together whole systems

that's where and having been responsible for a large subsystem which you were too yeah and knowing and then having

to clean up your own messes yep make a lot of mistakes to figure out oh no you can't just do it this way

and you can no longer curse the person who did this to you because it was you. Yeah. When you're building code. That was the thing. It was going from maintenance, mostly maintenance with writing a little bit of new code to there's nothing here. Yeah. Blank page. We need to do this.

Like, okay. So I think it, so I had the title after five years and I think I probably felt like I could claim the title in eight to 10. I felt like I grew up fast when I was by myself. I mean, I was close, but somewhere around there. A couple, I mean, it took a couple of years. And I went straight to management after that startup job anyway, so that was kind of weird. You also took time off and got a master's degree. I started it during that period, yeah.

Let's see. Okay, let's go to the question I didn't really want to answer. From John C. No actual experience with Zephyr, but I've seen a lot of progress with POSIX support and linkable, loadable extensions. If you ported a very simple terminal text editor

and a very simple C compiler to Zephyr, provided some AI wrappers, API wrappers Thank God. I was worried about that for a second. For like a standardized file system, you could bootstrap a single user, more traditional Thank God. I was worried about that for a second. For like a standardized file system. Yeah. You could bootstrap a single user, more traditional Unix-ish OS in Zephyr on a microcontroller. I guess.

Obviously, the big question is why would you want to do that? Yes, that is the big question. Yes, it is, in fact. But it's been fun to think about. How feasible do you think this is? John, you need to see Forth. Would there be any actual applications of it? All double E's will tell you that there are many applications of Forth. Why are you talking about Forth?

Because it's a microcontroller compiler on Shell. I mean, no, he's talking about POSIX and Linux. You could totally... The difference between modern microcontrollers now and a general-purpose CPU from the late 90s, early 2000s that you ran Linux on is the MMU. So you just need a Cortex-M4 with an MMU? No, they don't have them. They have MPUs. Oh, right, right.

They don't have any virtual memory. So, I mean, you could run, but there are lots of minimal versions of Unix that ran without MMUs and did stuff. So you could totally do this. Yeah, sure. I'm sorry. The answer is yes, you could do it.

Now you have to figure out why. Obviously, the big question is why you would want to do that. Yeah. Yeah, you could do it. Now you have to figure out why. Obviously the big question is why you would want to do that. Yeah, you could definitely do that. But there are other ways of getting compilers on your system. You said you would reach for Zephyr in the future.

I kind of agree with that, but I might also reach for MicroPython in the future. Yeah, I mean, it depends on what you're doing. It depends on who your audience is. It depends on how fast I want to get something done. Well, MicroPython, if I'm working with clients who aren't quite sure what they want, MicroPython was really beautiful. Yeah, it depends.

So, yeah. But I don't know why you would want to do what he's doing. It almost looks like that already, right? I mean, the build system looks like Linux. It's got a shell that's decent. I'm sure you could port some compiler that ran as a task. File systems are,

I'm sure there are file systems available. Definitely. Everything else is available. There are file systems in Zephyr. Yeah, you can relive recompiling GCC on a 386 in 1990, whatever. Wow, that sounds like something. Hear your hard drive crying for dear life. It disassembles itself.

Oh, sorry, before I just start reading Winnie the Pooh, I probably should say... The show is over. The show is over. The show is over. Thank you to Christopher for co-hosting and for producing. Thank you for listening. Thank you to our Patreon supporters for their support and their questions. If you would like to contact the show, embedded.fm has a contact link, or you can email the show at show at embedded.fm has a contact link. Or you can email the show at show at embedded.fm. All right. Now let's see. It is Eeyore's birthday.

Piglet has popped the balloon that he was bringing Eeyore. And Pooh has eaten the honey he was bringing Eeyore. An owl has written a message on the honeypot that Pooh was bringing to Eeyore that has nothing to do with anything relevant to letters, really. But why should he? He's an owl. Many happy returns of the day, said Piglet, having now gotten closer. Eeyore stopped looking at himself in the stream and turned to stare at Piglet. Just say that again, he said. Many hap- Wait a moment. Balancing on three legs, he began to bring his fourth leg very cautiously up to his ear. I did this yesterday, he explained as he fell down for the third time. It's quite easy. It's so as I can hear better.

There, now that's done. Now then, what were you saying? He pushed his ear forward with his hoof. Many happy returns of the day, said Piglet again. Meaning me? Of course, Eeyore. My birthday? Yes.

Me having a real birthday yes you're and i've brought you a present eeyore took down his right hoof from his right ear turned around and with great difficulty put up his left hoof i must have that in the other ear, he said. Now then. A present, said Piglet very loudly. Meaning me? Yes. My birthday still? Of course, Eeyore. Me going on having a real birthday.

Yes, Eeyore, and I brought you a balloon. Balloon? said Eeyore. You did say balloon, one of those big colored things you blow up, gaiety, song and dance, here we are and there we are. Yes, but I'm afraid. I'm very sorry, Eeyore, but when I was running along to bring it to you, I fell down. Dear, dear, how unlucky. You ran too fast, I expect. You didn't hurt yourself, little piglet. No, but I... Oh, Eeyore, I burst the balloon. There was a very long silence. I ballooned, said Eeyore at last.

Piglet nodded. My birthday balloon? Yes, Eeyore, said Piglet, sniffing a little. Here it is, with many happy returns of the day. And he gave Eeyore the small, damp piece of rag. Is this it, said Eeyore, a little surprised. Piglet nodded. My present? Piglet nodded again. The balloon? Yes. Thank you, Piglet, said Eeyore.

You don't mind my asking, he went on. But what color was the balloon when it was a balloon? Red. I just wondered. Red. He muttered to himself. My favorite color. How big was it? About as big as me. I just wondered. About as big as Piglet, he said to himself sadly. My favorite size. Well, well. Piglet felt very miserable and didn't know what to say. He was still opening his mouth to begin something and then deciding it wasn't going to do any good to

say that when he heard a shout from the other side of the river and there was poo. Let's go with when in your career did you start, did you fart? When in your career did you start feeling like a senior engineer? I can't even get through the question. Okay.