Embedded - 387: Bucket of Spiders

Episode Date: September 23, 2021

Chris and Elecia discuss civic duties, the CAN bus, fulfilling Kickstarter orders, and the answers to a series of questions about embedded systems. Elecia was recently introduced to TRIZ inventive principles (wikipedia page) and started reading And Suddenly the Inventor Appeared: TRIZ: Theory of Inventive Problem Solving by Genrich Altshuller. You can support the show by becoming a patron on Patreon: patreon.com/embedded  Or your company can sponsor a show, see the Sponsor page of embedded.fm

Welcome to Embedded. I am Alicia White, here with Christopher White. This week, it's going to be just us talking about, well, talking mostly about my homework. Okay. It's been a while. I feel like we haven't done this in a while. Well, we haven't had a new show in a while either. I know. I got a complaint recently. A complaint? Oh, good. I like complaints. I'm good at complaints. Apparently, there's not enough origami on the show.

Not enough origami on the show. Okay. I'm not sure the podcast is the right medium for origami. And then you fold it 45 degrees. Yeah. And then look! A swan. Imagine a swan. I mean, part of the problem is that I haven't been doing as much origami. I mean, I've been doing origami

that's not my own pattern, so I don't talk as much about that. Well, I mean, that's how a lot of people's projects work, though. You could still talk about it. Not everyone's doing something from total scratch. Okay, well, I made a bug last week

on Sunday. And Chris was walking by, and I said, did you see the bug? And then he screamed. He didn't scream. It was more of a, you know, a scream is longer. It was more of a, just a yelp.

Yes. Yes. Very good. But the moment you pointed at it and said, did you see the bug? And I knew it was origami. It fell over and it was like 10 feet away from you. So it was a miracle of timing. It was a beetle with super long horns. I think he was just afraid.

I don't think I yelped. I think I said it moved. Did you see it move? No, there was some definite exclamation-type actions. All right. Well, my reputation as a non-bug-afraid person has been ruined. You still have to take out the spiders, though. That makes it sound like there's a bucket of spiders I need to take out somewhere.

Is there something I don't know? I did rearrange my office. There were some things. It's Wednesday. It's time to take out the spiders. You know, they pick them up at Thursday morning. I also haven't been doing origami because I had jury duty. Yeah.

It was a thrilling experience for everyone. Jury duty here is normally you get sent a letter and then you have to call in every day or maybe you have to go in for a couple hours. Maybe at the worst, you have to go in for two days. And they dismiss you.

And then they dismiss you. And even, you were on a trial? How long were you on the trial? Well, so. You have two. Well, it depends on what we're calling here. So, Santa Cruz is here. In San Jose, you had.

In San Jose, which is Santa Clara County, I served on one full trial as a juror, which was probably a week or so. And then I served on another as an alternate, which meant I had to sit as if I were a juror, but then I didn't get to deliberate. That was also a week or so. Really? And then there was maybe longer. I can't remember. They were a while ago. See, I don't remember them being that long.

They were definitely at least a week. Okay. Because, you know, they do the voir dire thing, which takes at least a day, a day and a half. And then there's openings. And then there's, yeah, there were at least a week, those. And there was a third one, which I was in danger of being impaneled on, but it got, there was a plea deal moments before they started jury selection. So in that one, I think I sat around for most of a day and then they hauled us into the court and then the judge lectured us for like two, for like an hour, even though we were dismissed because she wanted us to know all about this interesting case, which I didn't care to hear any of the details about because it was actually quite horrible. Anyway, so yeah, I've been on two and a tenth.

So I've been on the jury selection process before, the voir dire, where they ask you questions and they try to kick you out of the jury if you know too much about it or you've been influenced by news or something. Or you're just not the right kind of person for, they envision making the decision, yes. I always don't want to be the right kind of person. But I ended up on the jury, juror number five, for a two-week case not including jury selection. So, it was more like two and a half. Plus there was a week break? There was a week break, too. In between jury selection. So, was more like two and a half. Plus there was a week break? There was a week break too. In between jury selection. So yeah, it was weird.

And it was in person. And I mean, everybody wore masks when we were in the building, but it was, eh. And it's funny when you're going through jury selection process, they say, have you ever been on a jury? And you say, yes. They say, well, how was it? And everybody, everybody answers, it was interesting. Sometimes there's a pause. It was, hmm, interesting.

It was not interesting. It was awful. It was exhausting. It was emotionally difficult. It was time consuming. It was not interesting. It was awful. It was exhausting. It was emotionally difficult. It was time consuming. It was time wasting. Oh my God, the amount of time spent doing AV. I just, just figure out how to switch screens. It was, God, it was awful. So you're not, you're not hyped up on your civic duty? I mean, I admit parts of it were interesting, but parts of it were just catastrophically irritating. Yeah, that was kind of my impression was that, you know, people, a lot of TV, wow, we're really off topic. A lot of TV is like, you know, NCIS and SVU and all these cop procedural shows and stuff and courtroom dramas. And after serving on a couple of juries, like, well, that's

as realistic as Star Wars. You know, it's all fantasy and the lawyers aren't as smart as they seem on TV and they make all kinds of mistakes and they're not good at public speaking and there's a lot of wasted time and cops lie. And yeah, it's eye-opening. It's good to have gone through, I think. But yeah. The thing that was most eye-opening to me is something you've always said, which is if you are pulled over, well, not necessarily pulled over by a traffic cop, but if something happens, don't say anything.

Yeah. Don't talk to the police. And I admit, I mean, the defendant, if she had said nothing, she probably would have gotten off. She was convicted mainly by her words. But we have this thing called Miranda rights, which we see in the procedurals. You have the right to remain silent. Anything you say can and will be used against you in a court of law.

You have the right to an attorney. If you cannot afford an attorney, an attorney will be appointed, something like that, yeah. But they tell you that when they arrest you, because that's when they're supposed to read their Miranda rights. What they don't tell you is that now with body-worn cameras and cameras inside the police cars with microphones they wear, everything you say from the moment you talk to them is being recorded, can and will be used against you in a court of law. Yeah.

And so that was really eye-opening. I mean, it was good because we saw the raw footage. We didn't have to deal with the reports. So-and-so said this, yeah. And I mean, we saw a lot of the reports too, but we had all the raw footage of as soon as the officers got there. I was really surprised at how that whole you have the right to remain silent thing really doesn't apply anymore.

I mean, you still have the right to remain silent. You have the rights. It's that they don't have to tell you about your rights in a timely manner. So I have been very cranky because of jury duty because I was completely stressed out by it. And so I have so much work. I should not have taken on the third client, which.

And the fourth client, boy, that was a real mistake. Yeah, the third client I'll get to talk about next month. It's not so much a client. Well, that's true. It's a project of your own. It's a project. I'm doing for a company though. Yeah.

But it's not our, it's not my usual type of thing. It's your thing. Yeah. It's, it's not doing something. Somebody, it's not a product that somebody has asked you to work on.

It's, it's somebody else. Yeah. So that's cool. Um, so yeah, I shouldn't have accepted that because I had too much work to do this summer. And then I took a forced and stressful two and a half week vacation that is led me to much demotivation just because I felt. Tired. Too much.

Yeah. Yeah. We have gotten some good guest suggestions in the mail recently, and I'll be following up. In the mail? In the emails. Sorry. That's fine. I'll be following up soon on those because I am a little behind on my scheduling.

Okay, so now there are three topics. Three topics. Four. Four topics. Five. I'm going to say there's seven topics. Probably. Okay.

How is your Kickstarter going? The Kickstarter is over and I am shipping records and I'm doing it myself. So I'm going to hopefully finish up shipping everybody's stuff this weekend or the latest early next weekend. Early next week, not early next weekend. You did this thing called Backerkit. Yeah, yeah. So with Kickstarter, the fulfillment thing is always sort of nebulous. And it wasn't clear to me what Kickstarter did for you.

And going through it, kind of figured that out over time. There's a lot that needs to be done when you're finished because you have, you know, some number of people who've ordered things. And they've all ordered different amounts of things. Maybe one record and some stickers or a record, a different variant of a record and something else. Maybe one record and nothing. So that there's everybody's, you know, ship list.

That's different. Um, there's everybody's addresses. And so at the end of it, you don't have any of that information like addresses and stuff. You just know the country and the person's name and their email. So even though I've told Kickstarter my address, they have not shared that with anybody.

Yeah. So in order to get that, you have to do surveys, which entails sending an email to everyone and saying please tell me your stuff um which so that's that's one thing that would be difficult to manage on your own kickstarter will help with that but um and then there's you know mundane stuff like sending everybody a digital download code or getting everybody's address and buying postage for everything, dealing with tracking numbers. So what Backerkit does is for a percentage of your take, they will manage a lot of that stuff

for you automatically. So they import everything from Kickstarter. They know who your backers are. They send the surveys out. They automatically remind people if they haven't responded to the surveys. Yes, I know it's annoying. I'm sorry. And they manage everybody's stuff. And then at the end, you can buy postage for every grouping of different items. And then you just print that label out and slap it on there and then they they will send the tracking stuff so it handles all this stuff that would be a nightmare of spreadsheets and mistake making for somebody to do manually mail merging and the other cool kind of cool feature is they kind of had a last minute changes feature where you could say add stickers or something if you decided at the end oh oh, I really did want stickers or I really did want a different record or an additional one, which was kind of nice because we got enough out of that to actually pay the backer kit fee.

Nice. So we made it totally worth it to do it because it was paid for by last-minute additions from people, and it saved just a huge amount of time. It was kind of tricky. I think, I don't know if it could have been made easier. There was some weird stuff like, I really wanted to send the download codes before people paid us because I promised. And there was no good way to do that with Backerkit.

So I ended up doing that manually. But then Backerkit sent them out again once people did pay. And then a bunch of people were like, my code doesn't work. I was like, well, you already used it. It's like, oh, that's the code for the album. Not that. Yeah. Anyway. So there's stuff like that. And, and it was tricky because the add-ons were tricky and there was various things that there were things that could have been made easier and that that took hours of time for me to figure out, but far fewer hours than sitting in front of Google Sheets and trying to maintain, you know. We didn't have that many people. It was like 100 people or so. So it didn't seem unmanageable, but...

You could have done it all yourself. I could have done it all myself, but it would have been painful. And, you know, just using the USPS website to buy postage, that would have taken forever. And, you know, just using the USPS website to buy postage, that would have taken forever. So, yeah. So it's going. And you've been complaining a lot about CAN in Python.

Do you want to share your pain or just whimper? I can share my pain. I don't know that anybody will be able to fix it for me. So here's what I'm doing. I have a device. I can say what it is. It's a radar. And it's an automotive radar.

And the way it reports things is it has a different CAN message. I'm assuming some knowledge of CAN for this discussion, so I'm not going to explain it all again uh it sends a different can message id for every track so if it's tracking 64 objects there'll be 64 different can messages you're looking at me funny no oh okay there's 64 different can messages uh one for each track and those come in sequence. But then after that sequence, there's 10 additional messages that are packed with additional information about each track. And inside each one is seven or eight.

Seven or eight. Wait, it's 10 messages. It's okay. You can totally lie. It's okay, you can totally lie. There's some amount that divides evenly of additional information about each track packed into each of those 10 messages, and the last one just has one for the remainder because it doesn't divide evenly. And those come right after. So I need to...

I have some other information coming in in a timely manner and I want them to be synchronized and so I'm using that other information that's non-can to drive the read of the can information so I get the other information and then I say okay time to go look for my tracks and so I sit and wait for the first track to come around because it's continuously sending and I don't want to start in the middle so once I get first track, I do a continuous read using the socket interface to get all my messages. What's happening is that periodically, and often enough to be completely annoying, I'll get to, like, say, track number 40, and then I'll get track number zero again. And then it'll start over from there.

And at that point, the entire sequence is botched, and I can't do anything with the data that was corresponding to the other thing that I downloaded. Do you get socket errors? No, there's no errors. If I look at the CAN interface statistics in Linux, there's zero errors of any kind. It's all perfect. I had some inefficiencies in my code that I removed, and so now it's just doing a tight loop with receive from gathering all the messages contiguously, and then I process them all later, which is the right thing

to do if you're trying to do something in a timely manner. But even then, it's still happening pretty frequently. The weird thing is there's a can dump command on Linux that just spits out hex of everything it's getting on the CLI. If I use that, this never happens. So it's not that the radar is actually deciding, hey, I'm going to drop a bunch of stuff and start over. It's not that the CAN interface is dropping a bunch of stuff and I'm just missing it. It's got to be something like my code just goes out to launch for long enough that it's missing a bunch of transactions. Your other asynchronous input. Mm-hmm.

If you just say that's always there and don't pay attention to whether or not it really is, can you keep up with the sockets? That was my next thing I was going to try, was to just take that out and read continuously. But I took a lot of that out wrong triggering i'm triggering off of a signal i get for that thing and then i execute a callback function that i get that i provided and i took out everything related to that in the callback function so it was saving a file before doing the can read i took all that out so that doesn't work um so the next thing is just take that thing out entirely the additional

information and just do a tight loop and see if it ever happens but then okay say it goes away well that's informative doesn't make a lot of sense but um it doesn't really lead me to a way that i can solve the problem because it does Sure it does. Well, I don't want two things running simultaneously because synchronizing the timestamps will be very difficult. But it'll be more precise. I don't care about precision. I just don't want it to stop. The thing that's frustrating is I'm not doing anything hard.

It's like 10 lines of code. It's nothing. And it can't keep up. And this is not a slow processor. It's like 10 lines of code. It's nothing. And it can't keep up. And this is not a slow processor. It's a Xavier. I've got it cranked up to max power. I've turned everything on that, you know, turns the CPU up. So it's very confusing.

I don't really understand it. And I actually did some things today where I was like, okay, let's see if it's going out to lunch. I took a timestamp every time I did receive from, and then got an array at the end with the deltas. And then I looked at the deltas versus where it started over. And there's maybe taking a little bit longer, but we're talking a fraction of a millisecond. So it doesn't make any sense. It's not like it's saying for a quarter of a second, I'm taking a break. So it's an interesting, one of those embedded systems problems that's just stupid. And I should probably go rewrite this in C and never think about it again. But there's reasons to do it in Python that I'm rapidly thinking about ignoring. You know, everybody's going to ask you questions like, have you tried the can raw FD parameter in the socket library? I don't understand the question well the socket library since i have

googled it for you yeah thanks i've spent many many many hours reading all that documentation but go ahead no it's it yeah it's just i've tried the wait all notification so that i could what i really wanted to do was read a batch without doing receive from. So the thing with CAN in Linux is, apparently, and people can correct me if I'm wrong, but I'm not wrong, it's a datagram-oriented protocol. And so the way the socket implementation works is you can ask for as much as you want, but you're still getting the 16 bytes of a CAN message only once a read. So it would be nice if I could say,

hey, read a whole block of this stuff and give it to me as a chunk. But it won't do that. It only returns one canned message at a time. So there's time in between each read for something bad to happen, which is what I suspect is happening. I tried, there was a socket option that said wait all. That sort of indicates that give me everything I asked for the number of bytes I asked for and just sit around until you get it.

That does nothing. Uh, yeah, I'm using the can raw interface. There's another can module for Python. That's not built in called socket can, I have considered trying that, but that's built on all the same stuff. So I don't understand how it would be any better, but maybe it is for reasons I don't understand. So I might try that. I don't know. If somebody's had good experience with that, they can let me know. But I doubt people are doing a lot of Python can. We should probably sit down together and look at it.

I mean, I did some of the can can i don't remember having problems keeping up in python yeah but i did it a totally different way so okay um now now for my homework okay are you ready so i have been asked to answer a bunch of questions. Let's see, there are 17 of them. We don't have to do them all. Okay. And I figure I'll ask you the questions and then I can answer them on my own later, having cribbed your questions, cribbed your answers. I'm supposed to answer as you? No, for yourself. Oh. and you're going to use my answers

in your... Only the parts that are relevant. Okay, all right. I mean, if I was copying your English essay, I wouldn't do it word for word. You wouldn't be able to read my writing. That's true. What do you wish you had known about embedded systems when you were first starting out? This is hard to just answer on the fly. What do I wish? I'm going to repeat it back because that's the technique for stalling. And then say that's a good question. Don't forget that.

That's a really deep and interesting question. I think I wish I had known more about electronics to start with. And I think I, you know, this isn't a technical thing necessarily, but I think I wish I had known how messy everything is, mostly because there were a lot of frustrations that coming from i was coming from a development environment that was technically embedded but it was writing stuff for routers and so i wasn't dealing with hardware very often i was at the protocol level so i was talking to networking device drivers and things and writing networking device drivers. And so, it was, I think, let me turn the question around,

what things did I learn quickly that were important, that became apparent quickly were things like, wow, cables are really evil. And if you don't have the right electrical signal integrity, your communications don't work. Or timing, you know, timing, if you have an electrical timing issue, even if that's controlled by software, you can have intermittent strange problems that take a lot of time to figure out. So I think, I think, yeah, I mean, I think it's back to what I said originally, the electrical part of it was stuff I didn't know that would have helped me go a lot faster if I'd had a background and somebody saying, here's how these communication protocols work at the wire level, and here's why you can get into trouble.

But, I mean, I knew some of that stuff, so I don't know. Sorry, there's the sound of me typing. The sound of you typing. Okay, so what I got from your answer is embedded systems exist, which I didn't know starting out. I knew that. And are everywhere. What? Wait a minute. I mean, when I first started out after school, I didn't know there was...

You're now answering the question. Yes, I'm paraphrasing what you said. Okay. I didn't know how to work in an interdisciplinary team. That's fair. I didn't know how many different applications there were and how my specific knowledge could be used. And that was one of the great things about embedded systems for me was that I went from having a software education and a bunch of foyer to actually having a career in signal processing. I didn't know that I could do both. Yeah, yeah. I think, I wish I had known not more about electronics, but more about debugging hardware,

about the simple stuff of checking cables and looking up information in data sheets and understanding the logic analyzer output. Actually, at that time, lifting the logic analyzer would have been a process. Or using it, period, since I'd never come across one before. Yeah. Okay. What factor or set of factors is key for someone to start a successful career in embedded systems programming?

Patience. Patience or persistence? What's the difference? I guess persistence is when you're patient but also mad about it i don't know that i would have termed it that way no i mean patience is waiting for something to happen persistence is keep trying until you figure it out it's both okay there's a lot of waiting around for something to happen and about it. Have you ever tried to download an image to an old slow device? Key factors. I think having an understanding of how computers work

is really key. And not necessarily like my previous answer, not necessarily electrically, but really having an understanding of what a microarchitecture for a CPU, what that consists of, what the parts of the CPU are, what things you have access to and what they control, how program flow works, where memories, what the different kinds of memories on a device are and how they are used.

Because with embedded, at least until very recently, if you're debugging problems or you're trying to optimize your code, knowing those things is hugely useful because you'll find yourself in various situations that you just can't solve if you don't know what a stack is or a program counter or the return register or you know all these intricate little parts of a device that you have access to doesn't mean you need to know assembly language necessarily but i think having an understanding of computer architecture is pretty key. Okay, so a general understanding of microprocessor architecture, patience and persistence,

minor willingness to learn, puzzle abilities, and creative thinking. Puzzle abilities. I'm not sure about that. Word, but being able to deal with resource constraints and trading off RAM versus speed and that sort of thing. Yeah. I don't want people to feel like Google interview questions are necessary to be answerable as an embedded engineer. The goat and the lettuce or whatever is not going to help you. But you're talking about optimization and, yeah, fitting things, rearranging.

All right, I need a different word. All right, I'll think about that. What is the biggest frustration that people go through when trying to learn embedded systems and what are our recommendations to overcome them uh well we'd have to lean on what people have told us i don't know know a lot yeah but my frustrations are not going to be somebody else's frustrations i guarantee it my frustrations are that i have to sit in a desk in a chair typing staring at a screen not moving I mean you could this desk works as a stand up desk

I don't use it I also stand in frustration frustrating things I think one of the frustrating things for me was coming to terms with realizing that there are going to be problems, two kinds of things. A, there were going to be problems that took weeks to figure out, and at the end of the two weeks was going to be one line of code or a change in a configuration register. Because of that microarchitecture thing, even if you have a good understanding of it, every chip is different, every system is different, and the kinds of issues you're going to have, there's always going to be something that's

surprising about any system you develop. No matter how genius your software development methodology is, you're still going to have bugs. Your programming language is not going to prevent you from writing code that has bugs. Your hardware will have bugs. And so there's always, in the development of any device, at least a couple times where you're like, this doesn't make any sense, and I can't figure it out, and it takes weeks to solve. Especially if you're an interdisciplinary team or a large software team, and there's different parts of the system, and somebody off in a corner just happens to write something that occasionally writes a byte that is over in your software, in your part of the code's memory, and you can't, you know, you have no responsibility for that so you wouldn't go think to check the other person's code. So there's a lot of stuff like that

where there's just these mysteries and they take a while to solve. The other frustration, which is worse, is there are occasionally problems like that that you will not solve. And there's plenty of times that I've shipped stuff and it's got a bug and we can't figure it out and that's just the way it goes and the best you can do is have a workaround

because the trade-off is sometimes well we could spend six months trying to figure this out or we can accept it and move on if it's not around if it's not a critical issue yeah um and that's really frustrating because you feel like you've completely failed or you're an idiot. But sometimes the systems are just so complex that it's like, well, I don't know where this is coming from. And that's usually true when something's really intermittent too because it's really hard to solve problems that you can't reproduce. And that's one of the differences from embedded. And I am talking and talking and boring everybody, but one of the big differences from embedded is your visibility into things varies wildly compared to when you're writing application code.

When you're writing application code, oftentimes you have these big debug suites that will graph things for you and show you memory leaks and put up things and say i detected this and that and we don't have that a lot in embedded so a lot of the times uh problems just don't get noticed um or the system is so complex that it's very difficult to have a problem be reproducible because it might happen in a customer's premises but not in your lab for some subtle, tiny difference. And you can't recreate the scenario. So, yeah. I mean, one of the reasons for that is embedded systems are supposed to be cheap and lean and lightweight and not have a lot of extraneous structure and things.

And that limits what you can do in terms of logging and putting in extra stuff for debugging. I would add that our compilers still suck, which is kind of part of your visibility and IDE and debugging suites. Okay. I mean, some compilers suck. Compared to a software suite of tools, our tool chains are not great. The compilers are fine, sorry. The whole tool chain is kind of weak and fragile.

Anybody who's had to use GDB servers and had it crash and wondered why you were debugging your system and it just wasn't working. I also think that one of the difficulties is having to navigate hardware and software when you've probably only really learned one in the past. And then the thing that was so frustrating for me, or things like STM32, F03, blah, blah, blah, blah, blah, blah, blah. I mean, it's just impenetrable jargon. Well, it's not jargon. It's just junk. But no, I mean, you need to know. Okay, so I said F0 instead of F3 or F4, so that indicates something important.

Spy and I2C, we just throw these terms around because, of course, we all understand. And you are a new start. Does it matter if I say the S or not? Yeah, I don't think any of that's unique to Embedded, though. It's just, there is a lot of impenetrable jargon. I mean, even looking up I2C, does I2C and I2C mean the same thing? Yes. Yes, they do, but... Ah, but I3C.

But Googling it isn't as easy as it should be. I think one of the differences, too, is that you mentioned the compilers and stuff, but it's not just that. It's the languages. And I know people are going to shout rust at their earbuds or whatever, but a lot of, a lot of stuff still done in C. And if you're coming from learning software in college, you probably haven't done a lot of C.

And so it's, it's a bit of a brain shift to something that's ancient and brittle, but also powerful in what it allows you to do, but also gives you enough cliff to jump off of. I don't know. That's not a good metaphor. A cliff to jump off of. So that's a hard thing.

I think getting more so because modern languages have moved so far beyond C that if you've cut your teeth on software development and that stuff, coming to C is like, what is happening? Or more likely, what isn't happening? It's not helping you about anything. Let's see. These two I think will combine into one. What's the biggest opportunity for people getting into embedded systems and why should anyone choose to pursue a career in embedded systems? Biggest opportunity?

I don't know. I don't know. I wasn't quite sure how to take that. I think they're asking what's the best sub area to focus on. Oh. Like IOT light bulbs or automotive stuff. I don't know what's hot. I mean, everything's hot.

I think there's enough available to do in Embedded that you can kind of seek out things that interest you. Yeah, I don't know where to go with that precisely. What was the second part? Why should someone choose to pursue a career in embedded systems? Well, you get to see a lot of different things. You get to learn a lot of different things. You're not limited in the kinds of products that you might work on. You do get to work in a more interdisciplinary environment where you might regularly talk to electrical engineers

and mechanical engineers and even optical engineers and system engineers and all kinds of people and learn some things that aren't necessarily, you know, software development. I think that's the answer I'd stick with. I wouldn't go much deeper than that. What should people do to ensure they don't get stuck in their career? People do to ensure they don't get stuck in their career. Quit their job? Okay, I'll write that down as a suggested answer. I had learn.

Whether it's reading or experimenting on their own or teaching and sharing their knowledge with others, it's just important to keep learning. Yeah. Okay. Okay, what's your favorite embedded system development board and why? I pass. That's unfair. I can't choose. I guess it's the one that is on my desk because it's on my desk. What programming languages are used most by industry? Go on, shout Rust again. We don't mind.

Wait, what was the question? What programming languages are most used by industry? C. Followed by C++. Followed by assembly language. Possibly followed by obscure weird stuff like Ada. I think I'm going to go with C, C++, and that knowing Python is helpful for making additional tools.

Yes. Rust is up and coming, but I don't think it's a high volume of stuff right now. Just let me know when that bounds checker works for you. I'm not going to. I'm not doing it. I'm not saying you're doing it. Okay. It's the borrow checker.

See. Is it the borrow checker? It's the borrow checker. Oh, man. I'm going to get my burns right. I have to get them right. Yeah, see, now people are going to really yell at you. Yeah, I deserve it too.

Let's see. What is the state of art in terms of embedded software development today? How can you answer that question? Why would you ask that question? I don't know. The state of the art is GCC plus Clang and VS Code. I don't know. What were they asking?

The tools? I don't know. But I'm totally stealing your answer on that one. What is the best way to prepare for an embedded engineer hardware interview? Wait a minute. Best way to prepare for an embedded engineer interview? Get a good night's sleep. Make sure you eat breakfast.

Shower. Go for a nice walk first and review the basics of some things like real-time operating systems, communication interfaces like UArts and I2C and Spy, and do some bit-twiddling problems. I don't know. It depends on what people ask. I don't really like interviews. I think they're useless for the most part. So I think, you know, people do hang their hats a lot on technical questions. So the kinds of things you're going to be asked

by a competent embedded engineer asking you questions is the things I mentioned. How do you, you know, how how do you what is a thread how do how do how do you synchronize between uh you know two threads how do you read data off and of spy bus blah blah blah what's dma that kind of stuff just all those three letter acron all of those three-letter acronyms, three- and four-letter acronyms. It's tough to know all of those. It's tough to know all of those? It's half a day to know all of them to a level where you can say what they are.

That's true. And if you don't know, I would say if you don't know what an RTOS is or I2C or SPI and you're interviewing for an embedded interview. You may not be in the right place. YouI and you're interviewing for an embedded interview. You may not be in the right place. You probably shouldn't be interviewing for an embedded interview. I had my recommendation as the best way to prepare is to consider doing a portfolio project, something you can hold in your hands, something you enjoy talking about and understand. Yeah. Because you can derail the interview by talking about something you know instead of answering questions about things you don't know.

It's not derail. It's... Manipulate? Not manipulate. It's guiding the interview toward things you know. Okay. Sure. Sure, sure.

It's being in control of the flow of the interview to stay away from things like bit twiddling. I love bit twiddling. Yeah, but you can look it up. You can look up what RTOSs are too, but it takes longer to understand the concepts behind the RTOSs. It's like saying you can look up physics. Well, yeah. Sure you can't, but if I'm looking at how to, you know, I don't know,

do one of those weird XOR to count the bits things. Knowing that offhand saves two minutes. Knowing what an RTOS is saves you a month. Yeah, exactly. What do you wish your students knew about you? I don't have students. Fair enough. My answer to that was kind of backwards.

How about what I wish they never found out? I don't know what I'm doing. I have to look up a lot of stuff. Sure, I can translate between binary and hex. I can explain how direct memory access works in theory. But when I need to implement it on a project, it's a matter of looking at the user manual for the particular chip, sorting out what bits need to be set on which registers.

I'd really rather students thought I knew everything. But I don't, and that's okay. Let's see. A funny story slash memory of your career. See, I can't steal these from you. Well, you know, there was the time that the guy was arrested. No. Okay, what funny stories from my career that don't involve me blowing up boards, which would ruin my credibility?

Well, there was the time you melted your shoes in the desert. There was the time... That was fun. I was on a bomb range. There was a time you were on an active bomb range with signs saying things are going to explode. There was a time that... The same time... Turns out the same time, all two of those things, there was the rattlesnakes everywhere. I had a snake bite, Ken.

Well, that's great. That's fine then. You need to let them bite you. And boots. Well, until I had to snake bite, Kent. Well, that's great. That's fine then. You don't have to let them bite you. And boots. Well, until I had to put them together back with that. There was the time you got to drive, not drive, you got to ride in a stock car at 200 miles an hour. There was...

And I left you that voicemail where the car was coming into range and it just got louder and louder and louder and louder until it was just screaming as it drove by at 200 miles an hour. I don't remember that exactly. It was a good voicemail. Yeah, those were a couple of good ones. What work are you most proud of and why? I think the work I'm most proud of, although the company was kind of a mess, was the stuff we did at a company called Avenger. We made imaging and interventional catheters

to address peripheral artery disease in people. Long story short, it saved people from having amputations because that's the usual treatment once it's gone far enough. And using our device, they could actually repair people's arteries so that their legs healed. And this is a big problem. Peripheral artery disease is a big problem for people with diabetes and older people. And so even though the company, it still exists and they're still doing stuff, but, you know, financially the company didn't do well and it was hard to work for, for various reasons. 7 a.m meetings and you know actually having to go to hospitals and observe stuff and animal studies and other unpleasant things um you know knowing that uh that uh you know something i had a big hand in probably saved a few people's lives and quite a few people's ability to to you know have

a good quality of life but it's pretty good. Yeah. Will you answer that question for me? Cause I don't have as good an answer. Toys. You made thousands, tens of thousands, hundreds of thousands of children's playtime, fun and educational. Well, I mean, there are, yes, I am actually really pleased with that because I know that there are, yes, I am actually really pleased with that. Because I know that there are, I know there's at least one boy who probably would have had to go through kindergarten again, if not for a toy that I made. And I don't know who else were affected by them.

But I'd like to believe there were people out there who had an easier time learning to read. I was thinking about saying I might say I'm most proud of the podcast for all that that's weird. Oh, that's a good answer. Because I know a lot of people— I don't consider that part of our embedded career, which it totally is. So, yeah. I mean, getting to talk to all the people we've gotten to talk to and having people say that it helps them stay engaged in their career. I'm pretty pleased by that because it was hard. I'm glad they didn't ask least proud of because I got two competing answers for that.

One is working on the internet, which I think may have been a gigantic error. And I apologize for any hand I have had in making the internet work better or faster or any of that. The second one is kind of the dermatology medical thing. I think it did help quite a few people, so I'm not so down on it, but there was a fair amount of... It was hard because the product did help people who needed help, but it also had a large application space that was purely vanity. So it was kind of a... You got into it could address those. But the main use case was people with a lot of money

who wanted to smooth out their face and stuff. So it was a toss-up. So that's how the other medical device was more clear-cut. It was pretty obvious the application there. And the thing I'm most sad about that is it uh you know the market is weird and things don't take off the way you think they're going to especially in health care okay i think that covers my homework there were a couple more but i can answer them the other least proud thing is the uh apocalypse machine i made um And I think I got,

know how I say never work on time. I think I may have screwed up the clock on that. So, sorry. Well, it was set to go off in a thousand years, but one of the important things about embedded systems is knowing the difference between milliseconds, seconds, and those things. And UN8 versus UN32.

Yes, yes. There may have been a rollover bug there, but since it's already shipped, I can't. We didn't do OTA for that because we did that in the early 2000s. Something to look forward to. What's the next topic? I found out about this thing called TRIZ. T-R-I-Z. And it's from, it's old.

It's a way of solving problems. It's from, let's see, Zendrik Altshuler, who is a science fiction writer as well as a Soviet engineer and inventor. And the idea is that there are methods to doing invention. And he had they did magazines and lots of instructions, uh, in the Soviet Union, um, about around this, but I just discovered it recently. And I wonder if other people out there

have heard about it. It's, it's not forced creativity, but it's thinking about problems in a more structured manner. And when I say thinking about problems, it's things that require inventions. And if we want to think about how we're going to solve big problems in the future, this has a methodology to go through. And they start talking about different tiers of problem solving and how most of us are in trial and error. If I have a problem, I think about what I've done in the past and then I try something. And if that doesn't work, I try something else that's similar. But if you have a problem that's really different, that's going to get to a non-optimal solution because you're going to get to a solution that is close to what you know. But if it's a new problem, being close to what you know isn't the right choice. And so... I'm thinking of examples of that right now from companies i was at where it's like

they knew how to do lab stuff and so they built this deeply embedded system with national instruments boards doing everything like gpios it's been two thousand dollars on a gpio board so they knew how to build lab equipment and buy these parts from national instruments but they didn't realize you could do stuff with a microcontroller because that was just not in their brain. Yeah. Yeah, exactly. And then they go through, like, Tier 2 is searching for a solution kind of outside yourself. Tier 3 is using multiple disciplines as a way, searching through multiple disciplines. But there's this whole process that I've never really heard of before.

And I found it pretty interesting. I'll put a link in the show notes. The original one is in Russian, but there are a couple of different books and there are a couple of translations. Yeah, never heard of that. It's just been kind of interesting to think about. Apparently there are like 40 steps, different problems you can solve using different ways. One of the examples given in Alt Shuler's book was a simple, okay, so how do they put the raspberry goo inside the little bottles of chocolate? So you have a candy that is...

Oh, the bottle-shaped candy. The bottle-shaped candy, and then inside they have liquid... Sorry, that didn't make any sense. I'm sorry, that was totally... I was in my own world. How does that work? Or the ones that are filled with alcohol. Alcohol, okay, yeah. So do you have any idea how they build those?

How they make the candy? How they make a liquid inside the solid chocolate. I had assumed that they make hollow chocolate and inject it. But if it's the kind of gooey raspberry jam consistency, then you'd have to heat the jam to be able to inject it in. And that would heat the chocolate. What if I... Okay, I'm not going to solve this problem. No, no, no.

This is, by the way, is not an interview question. Do not ask this as an interview question. But one of the first tenets of this whole process is to think about how to change the state, the physical state of the problem you're trying to solve. And so the solution with the candy is you freeze the jammy stuff in the middle and then you dip it in, which as soon as I said it, you were like, of course that's what you did. And I was off on the alcohol, which you can't do that with alcohol. Right.

Right. So there are these, okay, so if you have a problem, one of the first things to do is, can you change the state of it? And another one is, can you do the opposite of what you expect? Okay. Which was, I mean, in the candy one, that is, instead of heating the jam, you're going to cool it. Yeah. And so you end up with a better solution. Well, when a problem comes along, you must whip it.

It reminds me, it's about time for me to have tea and whipped cream, which is what I put in my tea. That sounds decadent. It's delicious. Do you have anything else you'd like to talk about? No, I should probably get back to can. I had one more topic, but I think I'm going to save it. What is it? Just to tease people? I had this idea about how to introduce people to hardware, how to introduce them to schematics and reading data sheets.

And I think the way to do it is maybe to use the Arduino board, to actually take off the hood and look at the board and what you could do with it if you weren't in Arduino land. That's certainly something that's available. And I'm hoping it's a way that people who are a little concerned about learning hardware would be willing to follow, because it's something they have familiarity with. Okay. Oh, and Remotecon. Hackaday is having the Remotecon in November 19, 20th. Okay.

And their call for proposals is open for some period into October. I assume everybody can look up Hackaday Remote at Con. Okay. Okay. All right. All right. Thank you for listening. Thank you to Christopher for co-hosting and doing at least part of my homework for me.

Now we have some Winnie the Pooh. Thank you to the Patreon. Patreon. Patreon. Patrons. Thank you to the Calzones patrons. Our supporters on Patreon. Thank you to you who have supported us for supporting us. Yes.

We really do appreciate you. And should anyone want a sponsorship, they should email us. Email sponsorship at embedded.fm. Okay, now for Pooh. Sure. Okay, at this point, Eeyore has lost his tail and been informed of such. You must have left it somewhere, said Winnie the Pooh. Someone must have taken it, said Eeyore. How like them, he added after a long silence. Pooh felt that he

ought to say something helpful about it, but he didn't quite know what. So he decided to do something helpful instead. Eeyore, he said solemnly, I, Winnie the Pooh, will find your tail for you. Thank you, Pooh, answered Eeyore. You're a real friend, said he. Not like some, he said. So Winnie the Pooh went off to find Eeyore's tail.