Embedded - 203: Save My Board

Episode Date: June 8, 2017

Charlie Ladd (@csladd) joined us to give an overview of good hardware practices. The oil quality sensor is from VSI Oil. Recent fiction included Ready Player One, John Scalzi, and Matthew Mather. To stay current, Charlie reviews the trade magazines: EEWeb.com, EDN, ECN, and EETimes. A junior engineer's tale of woe.

Hello and welcome to Embedded. I'm Elysia White alongside Christopher White. I know that some of you completely understand how critical the bond is between a software engineer just starting out in Embedded and the electrical engineer explaining the mysteries of hardware. You will understand my excitement when I say our guest this week is my first electrical engineer. Please welcome Charlie Ladd. Hi, Charlie. Thanks for being here. Good morning, Alan. Chris? Could you tell us about yourself? Well, I'm an athlete and an electrical engineer. Been an electrical engineer for way too long and started out in Embedded and then met Elle

and then got really serious about some other Embedded stuff. Did a bunch of sensor things and here I am. Where are you now? Well, I'm headed toward, well, I've been working with large devices, and now I'm actually headed toward very small devices, which will be fun. In between, I had been doing a little bit of small devices with the contract manufacturing company and doing project management, which was something else that was really fun for me to do. To be a double E and a project manager at the same time is a challenge and it

keeps me busy. I have a tendency to be a very quick, fast designer. And as most people have noticed, I get things done quickly, not always correctly the first time, but I think quick is important so that you can get to the point where you can have your firmware guy start working on the hardware. And if the hardware's wrong, we'll find out sooner than later. And you've had a pretty varied career between medical and FDA certified products and FAA certified products. And also lots of consumer products that need to be done, oh my god, right away. Yeah.

Well, I started in life sciences, which was an interesting beginning. It got me into, well, I've got to design something that's really going to work right and perform well. But it doesn't have to be certified like an FDA certified thing. But it's almost. It's got to be approved. And these guys are doing DNA sequencing on something or making peptides in the lab. They want it to be approved and these guys are are doing dna sequencing on something or making peptides in the in the lab but they want it to be right so our hardware really has to work right all the time so that was a good segue into the faa and the fun stuff that we ended up doing after we did the dna

stuff um and that was an interesting challenge i had never had a design that needed to be qualified as heavily as the crossbow stuff that when we were developing the FAA heads up display or heads up AHARS. I got into a manufacturing company that was working with medical devices. The interesting thing about that was that I didn't have to actually go through the FDA approval process because it was our customer's job to do that. But I had to design so that it was going to meet those FDA approvals. And so we had lots of documentation and redundancies and risk analysis and all that stuff that goes with FDA stuff. And then, you know, I've done commercial stuff. It's always fun to look up at the bus that's at the San Jose airport, the shuttle bus that goes between the parking lot and say, yeah, I designed that piece of electronics there and worked with the hardware guys and worked with the, the customers firmware guys.

And that's my product up there and it's kind of pretty cool. So yeah, done lots of stuff like that all over the place right well i plan on asking you lots of career questions so we'll get to more of that okay uh but first i want to do lightning round you don't seem that excited about this i had to think about some when you prompted some of the questions, like, oh, geez, I got to think of what I'm going to say. So you're going to get me off guard here,

I'm sure, but that's all right. Go for it. It's true because I only give the guests a few of the sample questions. Uh-oh. Chris, do you want to start? All right, this should be easy. Okay.

Favorite movie or book, fiction, that you encountered in the last year? Ernest Cline, Ready Player One. Okay. Oh, yes. movie or book fiction that you uh encountered in the last year ernest klein ready player one okay oh yeah see i had but i had to make sure so but yeah i've been reading actually a lot of um sci-fi fantasy matthew mather and john scalzi are just wonderful so in fact you introduced me to somebody else when one of your blogs when you wrote it in the summary part and i can't remember who it was, but anyway. A favorite fictional robot. Oh, holy moly. Will Robinson. Will Robinson.

Danger. Danger. Okay. Preferred voltage. Ooh. Believe it or not, 24 volts. And that's because I love working with high power stuff and making things move massive quantities of of weight so yes um that was a hard one because i you know all the voltages are good of course well that's a that's a lower voltage than we got from a recent podcast that's going to go out of order.

So it's not going to make a lot of sense to the listeners. There's a larger voltage coming up. There's a larger voltage, yes. Well, you know, apparently automotive is going to go to 36 or 48 one of these days because all the darn electronics in there. Would you rather explain ground loops or I2C protocol? Ooh, ground loops. Yeah. What are more important to grow at home, hot peppers or tomatoes?

Green beans. Oh, sorry, tomatoes. Oh, I guess it could have been any vegetable. Beach or mountains? Beach. Although I grew up in Boulder, Colorado. Maker, hacker, tinker, or engineer? Engineer.

Yeah, that one doesn't shock me. But you do, so now we're done with lightning round, so you can answer longer. We need like a little audio cue. Yeah, that's true. That was a good enough cue. Go ahead. If only we had somebody who wrote music and can make lightning sounds. Yeah, I'll have to talk to that person.

So do you do things outside work? Yeah, actually a lot of things. A lot of sports things. I try to play racquetball a lot, try to play volleyball, though it's been a while since I played. Who knows, we might go down to the beach and play a little bit. I've been hiking a lot try to play volleyball though it's been a while since i played who knows we might go down to the beach and play a little bit um i've been hiking a lot lately boy the bay area

has got some of the best hiking in the world it's amazing i've been to zion and grand canyon and a bunch of other places like that and we've taken hikes in fact the most famous hike that as far as i know in all of the united states is going up the narrows at zion and it's a river hike where you're in the water the whole time and you're walking on boulders. It's like, how can you possibly do it? But when you get the right kind of sticks and the right kind of shoes, it's really fun to do.

And so we did that two or three years ago. But I've been doing a lot of hiking in the Bay Area. East Bay, which is where I am, has a whole bunch of wonderful vegetated area that you're up in the Redwoods, you're up in the hills, and you're literally a mile from downtown Oakland, or two miles from it. And you look over the hill and there it is. There's the whole Bay area.

So it's gorgeous. So that keeps me busy. That wasn't at all what I meant. I meant you do technical work outside. Why would I do it? Yes, I do. That's not entirely true. That's true.

I've been with a company that is trying, trying, trying, trying, and has been for 20 plus years to get a device to market that will make him some money. I'm actually working on the board this week, and he says this is the last one. Can we talk about the specific company? Do you think he'd mind? Because, I mean, I did a little work yes you did vsi oil right yeah okay yeah vsi oil um uh joe's the joe joe runs it he's been the only one that has been running it for the past 10 years and it's a little oil sensor that goes it goes in various types of places um the drain plug is usually a good place to put it because there's a little tiny PC board that sticks into the oil and then you've got just a couple wires that come out and it goes to this little processor that then processes the signal. And it's actually not

just one signal. There's three or four signals. There's a viscosity signal. There's a humidity signal. There's a temperature signal. There's a shoot. What else was he working on? So there's a temperature signal there's uh shoot what else was he working on so there's three or four different types of sensing elements on the little board that go back to the processor and then that's what helps determine whether that oil needs to be changed or not and of course oil can can go bad in many different ways typically it's water getting introduced and so water is a very important sensing element but it also can just be wear and tear. All the additives that are in oil eventually do decay, and that's when you really need to change your oil. So, yeah, that's a fascinating sensor. He's been working on it a while, vsioil.com.

But you usually have a full-time regular job and then this has been hobby level ish yeah hobby level it was funny of course i was uh between jobs at one point in time and actually worked in his garage we literally went over to his garage in his garage he had stacks and stacks of equipment and we were building the first 100 of these sensor modules and and the associated uh device that actually converts the signal that would have the the user interface that a little red light that would blink or whatever and uh yeah built those up that was a while ago but yeah so yes i did have a kind of full-time after work time type of job with VSI. But you don't do all these maker things where you make a fair and building silly robotic arms.

No, I've just not gotten into it as I wandered off. Because you have other things. Like going to the beach. Yeah. So, yeah, I've done a little bit of other moonlighting but it's only been one of my projects that's right one or two places here and there yeah you have to know charlie to get him to work on your stuff yes how do you square that with your full-time job you when you're changing jobs you say well i've got this other

long-term thing that i've been doing and that this part of my contract you should know this or I've never considered it yeah um it's such a not um uh no uh whatever you want to call it there's no no reason for anybody it doesn't interfere with my yeah yeah so um it's just a matter of of I don't think there'll ever be a conflict of interest, so that's the key. Okay, getting back to the, you were my first electrical engineer, what is important for a software engineer to know about getting down into embedded systems and touching the hardware, I mean, in addition to don't put the electronics directly on a metal surface? Yeah, well, the first thing is to make sure that the EE is there when you turn it on as opposed to you turning it on by yourself,

especially if it's the first board. Of course, I'd make sure that I don't let the firmware person even touch the board until I've done a bunch of testing on it or at least a little bit of testing. But the key for a firmware person to understand when they're first bringing up a brand new board is that they're going to need something to just blink a light. Well, wait, that means that I better put a light on the board that they can blink.

I try to put test points that are of interest to the firmware person, whether it be some sort of communication bus that they can look at the signal on or at least i can look at it um providing software test points that don't go anywhere they're just little iopins that the firmware person can say hey look at that it's yeah that's a big one yeah so uh and and recently one of my firmware engines has actually asked for like five of them it's like what do you need five for i want to kind of make it into a little state machine so I know which state

it's set. Oh, okay. So there you go. Five times two to the fifth. That's a lot of states. That's good. That's important. But the other half

for a firmware engineer, a good one is to know how to use a DVM and a scope. And I mean, really know how to use a DVM and understand

that you're not measuring AC signals. You're measuring just a DVM and understand that you're not measuring AC signals you're measuring just voltages and a scope so that you can at least get one or two probes on a board and look at what's happening to those signals and see that in fact you are going tri-state when you're supposed to go tri-state and not grounding a pin when it's supposed to be tri-state

and understanding what the scope is going to show when you do such a thing. So, those types of aspects of debugging hardware that a firmware engineer can help out with are really important. Is there anything that you do specifically to protect the board from firmware engineers? I'm reminded, because I read an article that's been going around the internet this week where a junior software engineer came to a new company which was a web database company and they gave him some instructions to set up his dev environment and uh in the instructions were specific things

that were like super user and then they gave him like a post-it note saying oh here are the real things you should use and of course he got flustered and just did the instructions as as written and as written it destroyed the entire company and so he he got flustered and just did the instructions as as written and as written it destroyed the entire company and so he he was asking for advice saying they're threatening his legal action and all this and everybody's saying that company is stupid it's not your fault but but this happens i mean on a smaller scale with junior engineers where they come in and you know i've done it you've done it and you get a piece of of electronics and we're flyby, we know what we're doing, and suddenly it's on fire.

Completely too embarrassed to ask. I don't think there's been one firmware engineer I've worked with that has taken a motor board and not blown up the motor board. Yeah, motors are seriously. And every single one of them have been blown up, every one of them that I've designed.

And as I've designed more and more of them, I've realized, God, do I really have to protect the firmware engineer from breaking my hardware? And it's just like, no. In fact, if they break it, they learn something. And that's important. And so what the heck? Am I going to prevent them from doing something?

I might. And it depends on if it's because what they're doing might happen, not because they're doing it, but because some other thing would happen and then blow up my board and that would be bad but yeah they're always um it's it's a challenge but you know you've got to assume that the the processor that they're working with is going to have some level of correct initiation code that it doesn't blow up the board and am i going to tell them exactly what pins to to program you bet one of my more useful documents for bringing up a processor is what i call the pick io since i use microchip picks and i put every single io pin on an excel spreadsheet

in one column another column has the pin number another column has well what is it doing from a hardware point of view another column has well what state is the active state another column is well what do i want it to be when you first turn it on or when it what the code first comes up and then the last column is for those guys the firmware guy to say well i'm going to call it this not that dorky hardware name i've never been given one of those i wish i had you gotta be kidding phil so charlie gave me those and phil gave me those and some of the other electrical engineers i've worked with the leapfrog people i guess i had something similar from one person but when i went to some place and said why aren't you giving me this the electrical engineer was like that's for you to make and i knew at that minute that this company was not going to work out.

And this electrical engineer could just go, whatever. I don't care. Yeah, no, I mean, because I have found it is better when the electrical engineer starts that. Because you read the data sheets before I do. And you know what those have to be. And I am now, I have this board. I have a processor manual that's 1,000 pages. I have hundreds of pages of data sheets and I'm trying to get them to go

in order so that I'm not doing everything at once. And that layout gives me this step up so that when I have, okay, now I need to work on this part and I only have this much I need to read. I don't need to know everything at once. Well, and you know, we're often given the schematic and said, well, here it is, you know, comment on this. And you know, it's got the things like you say, active low, active high, whatever, but it's distributed over multi pages with interconnects and things. It's not organized in a way that you can just go down and say, Oh, okay,

here's this group of these important signals. Here's this next group. And they're, they're organized this way and configured. You have to go through, and pins are pins. They're all over the place. And some chips, it matters what ports they go to, so you can get to low power, and if it's in an Excel sheet, it makes so much more sense. You spoiled me. I'm sorry. We're just going to go with that. Well, the other document, just want to make sure that there's two documents that I consider very, very key. That would be the PIC guide, the IO document. The other one is I have a tendency to put together what I call a theory of ops. I'll take my schematic, which I'll put usually on the first, the title page, I'll put a block diagram of what the board does. I love it when you guys do that and it's so nice and usually we'll hierarchical do the the schematic so that down the bottom would be all the pages of the

schematic and the pins that go between them so when you see on page five that there's there's this signal then you go back to page one and say oh well where does it go oh there it is oh it goes to these two pages and you go to those two pages see. So I do the next thing is the theory of ops. And it can be very general or it can be very specific, but I'll break apart the schematic into block diagram sections. And each block diagram I'll have, here's what this is supposed to do. And here's the key chips. And here's cut and paste from the data sheet sheet the important things that you need to know. Like, by the way, here are how you set up those four mode switches in order to get this chip to work in the right mode. And I usually make those software configurable

because who knows, we might want to change what type of mode this particular chip is going to run on. So I'll put that in the theory op. So you've got it in the I.O., but you don't necessarily know what those mean. So I'll put that in the theory ops. So you've got it in the IO, but you don't necessarily know what those mean. So I'll cut and paste the data sheet information there. And then you don't have to search through a 10 page data sheet for that important information. Because you, again, you have already done all of this work. Why would you make me do it all over again from the beginning?

Well, I don't know. Yeah. Yeah, yeah. And when you use, just to kind of bag on an electrical engineering experience recently, when you use the hierarchical views, do you name the variables the same on every page? Because he tried to explain to me how this was object-oriented, and I just was, yeah, okay. Remember that time we blew up all the capacitors on a board because I put them on incorrectly? I blame you for that.

Yep. One of the important things, though, was that you didn't panic. And maybe it's just you. Yeah, capacitors are cheap. Well, I mean, while we were launching them over the cubicle walls, it was really fun. And they popped, popped, popped, popped. There was like 15 of them. It was great.

But being laid back, is this, you introduced yourself as an athlete. Do you think that goes with that? Or do you think it's just you? Yeah. Well, it's a little bit just me. I've certainly met a bunch of hyper EEs. In fact, kind of working with one right now that's driving me up the wall, and that's why I'm leaving. And hopefully this podcast will never get to him, but who knows.

So there's just as many flavors of EEs as there are firmware engineers. Although I've found that most firmware engineers are a little bit higher strung because they are just that nature. And EE's have a tendency to be a little less high strung because, well, stuff happens. And we just know that it's going to happen. And we just get another board and say, don't do that. And so I've learned that, you know, things are going to happen. It's always a learning curve. And we always buy three extra boards because that's what's going to happen.

So when you pop all the capacitors, it's like, well, yeah, that was my fault. I probably should have looked at the board before I let you turn it on. Yes, well, it definitely was a learning experience. This laid-backness and it's okay to make mistakes, that seems very at odds working with things that are medical or FAA, where hardware that can kill people. Yeah. How do you balance that?

Well, you start realizing that it'd be better to have the board fail on the bench in some way than it would be to have it fail where it kills somebody. So the more ways I can destroy the board, the better the design is going to end up being. Now, am I purposely putting a stupid design together? Obviously not. purposely putting a stupid design together obviously not but i'm i'm trying hard to put enough stuff on the board to make sure that it does what it's supposed to do and when it fails i want to fail nicely and so i'll put redundancies or i'll put backups and make sure that okay i don't want to give anybody wrong information if if it's an faa type of device I don't want to give anybody wrong information if it's an FAA type of device. I don't want to move something incorrectly if it's an FDA type of device or give the wrong

amount of drugs. I mean, I'd rather not do something than to do something wrong. And so using the fact that I want to have the failure happen on the bench, I take advantage of that, relax, think about, well, what could go wrong? And then work with that as being the key to getting the design to work right and safely. Do you have any tactical tips and tricks for doing some of that? Well, as I alluded to, redundancy is one of the keys. If you can find a way of redundantly having some aspect of the design work when one part of it doesn't, or at least have a way of understanding that it's not going to work properly, giving the firmware engineer a voltage that should always be there, and then realizing

that that voltage is powering the circuit that's for the sensor. You want to make sure that, you know, they look at that voltage, and if that voltage is wrong, then you better not be reading the sensor because it's not going to be the correct sensing information. So that's the level of redundancy that's kind of right down at the hardware. Things like battery backup. You don't do that with a commercial device, right? What do you need battery backup for unless it's critical? And so you've got the primary power and you want to build in a backup supply. It might be a backup supply that powers the whole circuit

or it might just be a backup supply for SRAM to make sure that it keeps going. But you might do the same thing for a critical circuit. Fuses, resettable fuses are cool. Man, you go out there, something happens that's out external to my board and it fails. I want to be able to save my board, not have it go down just because something external to me failed. So I put a fuse, preferably a resettable fuse that then, you know, the software is going to say, you know, that voltage doesn't exist out there. What's wrong? And so you can flag an error and then you've got a safe

device again. It doesn't blow up anything and you don't do something that you shouldn't do if you don't have that information. So those are the simple tips and tricks that I've used through most of my FDA designs. And it sounds like you start from a principle of designing for failure. You kind of think of what can go wrong and what could I do to mitigate that? So it doesn't sound like you're flying by the seat of your pants. In fact, it sounds like you're thinking ahead and saying, well, this may not be right on the first time. Okay, what are the kinds of things that can go wrong? Okay, what can I put in now to ameliorate that? Which is funny because, you know, FDA wants

you to formalize risk analysis. And it's like, well, yeah, but I just do that naturally. That's the things that I do when I'm designing something is I say, geez, what happens when this fails? And so now you're right. When we get that formalized analysis of what's going to fail, my circuit should already have the mitigation for when it fails. And not always, but sometimes during the analysis of the risk that we put down on paper, we'll notice that, you know, that circuit isn't, you know, it's a high-risk circuit. What will we do? And every now and then you'll have to change the design because of that analysis. And it's good because that's what the FDA wants us to do is to make a safe product and go through that analysis and make sure that we don't harm patients. Think it through seems to be one of the things they want.

Some of the things you said, like resettable fuses, can get expensive, which is one of the reasons medical things can cost a lot, is you need those safety nets. But routing voltages so that the software can read them, that's cheap. I mean, that's just information software is free sort of it does consume an iopin and it's an analog iopin that might be limited in numbers so yeah you got to be careful you can't give them everything and it's kind of silly to give them the 3.3 volts that the processor is running on because if that goes away whatever but you could give them other voltages that are and it's only two three or four here it's not a lot of analog. But you could give them other voltages that are and it's only two, three, or four here. It's not a lot of analog voltage. And you could even put it into a comparator and it becomes an IO input instead of an analog input so that they could

at least have a yes-no answer as opposed to, oh, it's out of range. So, yeah, they're cheap. You're right. I'm surprised by that, that more people don't do it. But on that, sort of on that topic, one of the projects at Crossbow that you worked on, that I didn't, because it didn't have a lot of firmware, was creating an alarm in cherry pickers, a tilt alarm. And so that if it was going to dump you out of the bucket that the operator is in, it would scream.

And you visited the customer and got in the bucket with your hardware. Were you nervous? Yeah, because I'm afraid of heights, but not because I thought the hardware would fail. In fact, I knew the hardware wouldn't fail because we had already tested it. We knew what was the problem, or at least we're pretty sure what the problem was and so it was just a matter of going up and confirming that yeah that that thing is not reacting properly because of and there was an oscillation and i finally saw it on the scope and was able to tell the customer it looks like you've got an oscillation at this frequency and he was doing a whole bunch of really good things with the information but he just wasn't able to see the actual the real information that i was able to look at record on a scope

show it to the the firmware guys that were writing the code for the sensor input and say hey this is what will trigger something that you should probably be looking at and here's about the frequency and here's the about the amplitude and so now you can do a little signal analysis and when that happens you should probably quit doing what you're doing and make sure you don't dump the the person and so yeah yeah so we rocked the boat a little bit up there up at 100 feet. And I got the information and came back down and had a glass of water and that was it. So on the other end of the spectrum from medical and safety critical things, you've worked on consumer products that had to be designed quickly with tiny bombs. How is electrical engineering different for that? Are these the same skills or is this just

almost totally a different career? want the best quality, best functionality, the thing that'll operate over the widest range of operational modes. While with consumer, you want, well, I wouldn't call it the cheapest, but the least expensive solution for that little design. And if that means that you actually put a half a dozen resistors in the passers down because they all cost a tenth of a cent each, as opposed to an integrated circuit that might cost a dollar, then that's what you do. But if space is a premium, you may have to find the cheapest little small grid device that will fit into the hardware space that you have. And the premium will be difficult to manufacture.

It'll be difficult to even design and debug. But since it's consumer and it needs to be inexpensive and small, that's what you're going to do. So it's just a different type of analysis of the same thing that I'm doing, where instead of quality or the flexibility of the design is taken into consideration for the medical type of design. It does it, it gets the job done, and it's fairly inexpensive. So similar process, but different parameters? That's right. Really that is it. Cool. You've been an engineer, and you've been a project manager. And I know you kind of enjoyed the project

management part uh but now you're mostly an engineer again yeah um i can do the little story um way back when i was at the life sciences company applied biosystems i worked with a chemist and we came up with a cool idea for a new product and we worked on it and realized that we're going to do a product development so i'm the ee he's the chemist we've got a mechanical couple mechanical guys and i said well i wouldn't mind leading the project here because the electronics is pretty much done all we have to do is make sure it all get coordinated. So they had me be project lead. And I was put in front of 300 chemists to present the project. And there I was standing up on a pseudo stage, and they would do this quarterly, and they'd finally pick my project. And so I realized that I can do this. I can actually

lead a project, stand in front of my peers, explain what we're doing, and I enjoyed it. I didn't get to do it until a lot later when I had been dismissed from Crossbow and had to find a new job. We got laid off.

We finished the product. It was all ready to go. A bow was on the paperwork and then we got laid off. Thanks, Crossbow. 9-11. Yeah. Well, anyway. And I got a job at Flextronics, which is a contract manufacturer. And I was hired to just do double E stuff.

And these were where the little projects, like the commercial projects, came along. But they had a need for the customer interaction manager, the project manager, the person who actually built the customer. And the project that I was hired on, first, the RF engineer got dismissed because of lying about his resume. Okay, that's going to be something. And then the project manager that just hired got something on his bicycle. He had an accident and was in the hospital for like two months. And so now we're missing an engineer and a project manager. I said, well, look, it looks like this electrical engineering stuff is going to be pretty easy to do.

I'll take over the project management. I've kind of done it before. And this is the video cam, drive cam that's now at San Jose Airport. I took over the project management of that design as well as the electrical design of it and worked with a wonderful mechanical engineer who packaged it up. And we did it and the customer kept changing his mind and I thought, oh, holy crap, he's going to hate me gonna we're over running over budget we're running over time and years later i had a connection on linkedin with him and he says you know you were one of the best project managers i've ever had and i just you gotta be kidding me i thought it was just awful because i was you know always late and over budget and no no everything that i had done was exactly

what he was expecting and i also realized i really this. It's fun to interact with the customer, especially at a technical level. I don't have to manage people. I'm just managing the process of the project and I can do it. And since I'm fairly good at double E stuff, I can do that at the same time. In fact, it keeps me in the loop of the technical side of the project. And so, yeah, I really enjoy the project management. And that's where I'm headed back at my next job. We'll see. Does it give you a different perspective when you go back to engineering? I mean, if you are just being an engineer, and yet you can kind of, now you know how it's all supposed to go.

What would you tell us engineers who kind of wonder, what the hell is a project manager doing? And what is this for? Why do they keep asking me to hurry up? Good project managers will provide information to the engineer that says, here is what you are going to need to be doing for the next, say, two weeks. And I want you to do this because that, it turns out, is part of the critical path of the project. You can postpone the other parts that you're working on because those aren't the critical path. And that's my job as a project manager is to say, here's the critical path. In fact, the real job of a good project manager is to make everybody on the critical path.

Yes, yes, indeed. And then you make it into a little horse race, but that's another thing. But the good project manager will provide all the engineers the right information so that they can actually get their part of the project done. And provide the connections between the mechanical and the firmware and the electrical. Because nobody else is providing that conduit between the engineers other than the project manager. He'll be very good at trying to tell you what the milestone is, not what the milestone, when it is. That's easy. You just say, hey, it's due in two weeks. But you want to tell all the engineers that milestone represents whatever it is. It's a beta product and it has to have this functionality. And we've all agreed on

that because that's what needs to be done to get to the next step stage of the project. And so yeah, project manager can be kind of a pain in the butt because he's telling you, you should have had it done yesterday. But if he's having to tell you that it's because either you're not doing the right thing or you don't have the right information to do what you need to do. So. What's the difference between a product manager and a project manager? Because I get confused between those things. The companies I've been at, the product manager is the person who, usually in marketing, who is helping the project team build the right product.

Specifically, in medical, and I don't know if you've ever talked about it i think you have a medical product has an intended use and that's a very important part of medical designs and the product manager is usually who's spelling out like black and white what the intended use is and if you want to put other bells and whistles as an engineer, as we all love to do for that product, that's fine, but that's not what the product is supposed to be doing. It's supposed to have this intended use and to put a little light on there that says, oh, this is really cool. That's fine, but that's not necessarily what the product needs to do. And that's what the product manager, at least in my experience, would do is keep us focused on what the product is supposed to be in the end.

And the project manager is just trying to get you to that end by way of organizing how the tasks of the project are supposed to be done. You mentioned explaining milestones and telling people about critical paths. Most of my goals with project management are not to say you need it done by next Tuesday, but to let them figure that out themselves.

I know that that's some of my style. It's cool. I don't need to tell you the date, just we have a milestone going in and maybe you can work it out yourself that so-and-so needs it before that date and the mechanical people need to test. How do you figure out how to give them the information they need from their peers without drowning them in so much information that they can't use. Yeah. I've worked with some pretty good engineers like yourself that really do understand the big picture.

And so if I have to explain to an engineer what the big picture is so that they can understand what they need to do, I probably have the wrong engineer on the project or um he needs to work with somebody else um so yeah you want to be able to rely on the engineer as a project manager that that the engineer is going to do his job the best as he can and understand that the milestone is this and that means that he has and understand that the milestone is this. And that means that he has to do this, this, this, and this without having to spell it out exactly what his daily job is going to be doing for the next two weeks. I've been lucky.

I've been working with some pretty darn good people. And the few times that I've worked with junior firmware engineers, for instance, it's usually been in tandem with a senior. And the senior engineer is the one who helps guide the firmware engineer. As a double E, I've worked with firmware engineers and have had the pleasure of letting them know that this is probably the important thing that you should be working on now because that's what our milestone is. And as a double E, I've been able to guide the junior engineer as well. And that's always fun to do because it's nice to work with somebody and be a peer to them and have them really enjoy working with you because you taught them how to use an

oscilloscope. That's cool. The mentoring part is a big part for you. I mean, that was always true. And yet, how did it start and we're all on ancient IBM PCs that really don't do anything very well from an engineering point of view. And he had an Apple Macintosh at his work. And he would organize wireless on his Apple Macintosh because so much easier to do using the little database program. So he was what I would call a head of the curve and everybody kind of knew it. And so, man, I just sucked up next to him and he just grabbed my arm around, put his arm around me and mentored me and I could see how he did it. And I learned how to mentor and I learned from my father how to teach because he was a teacher. So, you know, it came naturally when I finally had enough knowledge that I should be parting some of that to others as well. It was easy for me to figure out how to do that and to mentor my colleagues that were younger and less experienced. It's fun, too, experienced. It's fun too, actually.

It's very rewarding. It keeps me engaged even when I wouldn't otherwise be. I completely agree. But it's not something you always know. No. And I also agree with the stand next to the guy who's really smart or woman who's really smart. Stand next to the person who's really smart or woman who's really smart. Stand next to the person who's really smart.

Let's go with that. And soak up what they're have to offer. Yeah, yeah, yeah. When I was at Flex, there was a guy. Well, and you remember Ken at HP. Boy, now they're just. He's a listener. Be careful.

Hi, Ken. Oh, man. I was just about to compliment him immensely. I mean... So smart. He is. And I feel like I'm almost as smart as him now. It's taken me 10, 15, 20 years to become as smart. You know, but anytime you can let a guy fly you upside down in his airplane...

No upside down for me, I promise, Christopher. So, yes, he was an impressive engineer to be mentored by, and we absorbed things. And he had a lot of respect for what I did, too, and that was a key to it because in mentoring me, he told me that I was good at what I was doing. And without actually physically saying it, we're on a patent

together for goodness sakes. Right. So, you know, he knew that I was capable of doing some stuff too. So, yeah. You're on a fair number of patents. No, not that many. Just four. Just four. Yeah. Come on. It's the Bay Area. Every, every engineer in the Bay Area has at least one, if not two. And some of them have. No, I, I mean, but it's,, but it's not true that everybody gets them.

Oh. Especially now. Especially now. I think they aren't as prevalent, and the software patenting is starting to go away. Thank God. Yeah, you're right. I've had the pleasure of working on some pretty interesting

electromechanical devices that have firmware in them that make for a patentable situation. And, you know, come up with some neat little algorithm and you better believe you'll get a patent. Yeah, let's see. There's been, I think it's four. And I'm just now, yeah. And I just was told I'm on another patent application

that we're going to be putting in. And it's cool. It's neat. Used to be you'd get a big bonus when you got a patent 20 years ago. Now it's nothing. Yep. And I won't get that bonus because I'm not going to be there when it gets issued if it does. So we'll see.

I have a listener question. And for one of the reasons that I finally nagged you into joining us, we have had shows. One of the favorite shows from our listeners is Jack Gansel's Being a Grown-Up Engineer, Being a Grown-Up Firmware Engineer. And Ken wanted to know, as somebody who's recently gotten into embedded systems as a hobby, are there any people or blogs or books that have the advice to go from, here's a resistor, here's a capacitor, here's a data sheet, to here's how you'd be good at this? Yeah. Obviously, having gotten a degree, a college degree in engineering,

that's where I learned what a resistor and capacitor and all those things do. But those are not difficult things to do. In fact, my mom is just now taking a class, a video class, online video class on electronics. And she had a question for me the other day when I had dinner with her. I don't understand how this resistor divider works. And I said, well, you know, and we went through it. And so, there are lots of classes that you can take that are online. I wouldn't know of them in particular, but there's other places where I've learned a lot of my current trade. eeweb.com is an interesting one that is always got information about the latest stuff that's out there.

There are the trade magazines, EDN and ECN and EE Design are the three that are the big ones that I'm aware of. What else is there? I can't think of anything. There's enough stuff. And like I said, I alluded to the technical magazines. I actually get a physical technical magazine, two of them sent to my home,

so that I can take them to work and peruse them at lunch. And I'll take one of them and at lunch just go through the whole thing, zip through every page. And then when I see something that's interesting, an article that might be interesting, I'll rip it out and then read it at another time. Same thing with the ones that are online. If it's an interesting article, I might click on it and read some of it. A lot of times they'll be introducing a new something or other, a new IC or a new type of board that does something special that I might be interested in.

Of course, the Internet of Things is just going berserk, and I haven't gotten into that yet, but there's a whole bunch of stuff about those. And then the other thing I was thinking about and just remembered is that you can go to Newark or Digikey, and there's a whole bunch of development boards. And some of them are really, really inexpensive. And they are, one of the ones that I remember seeing had like five or six different sensors on the board that would allow you to play with all kinds of different inputs and outputs on the board without having to do much more than just turn it on

and connect it to your pc using a usb port and all you'll you'll see on those development boards the electronics that is used or needed to connect those sensors to that i i um to that microprocessor and what better way to learn than to copy somebody else's working circuit and then see why it works the way it is you'll get the ic and you can go to the data sheet and they usually have app notes about it and so you you can go to a an ic manufacturer's website and look for their development boards they'll usually have them in one big section. Or you can go to DigiKey or Newark, which I find both Newark and DigiKey be just a plethora of information because they even have all kinds of videos on how to use their various parts that they're selling. And there you go. You've got an already designed circuit

that you can then snarf up and play with. And it's a tested circuit. It is, yeah. Which is kind of nice. It may not be the most efficient one, although some of the vendors have boards where they show off their power efficiencies. Okay, well, then they're going to design their hardware

the way they want it designed. And a lot of those dev boards have a bunch of test points and a bunch of ways of modifying them that they explain very well in there and say, I know you don't want this voltage, but if you want to change it to this voltage, here's what you do. I know you don't want that frequency of low-pass filter. You can just change these two components and you've got a different frequency. And here's the computations form. Many of the things you listed are, I mean, they're trade magazines, they're mostly ads.

Well, yeah. And that's important because you want to look at that ad and say, well, am I interested? And if you're not, don't worry about it. But yes, they're ads for something that's now available. I don't know how else to keep up with all the plethora of new stuff that comes out every single day in electronics, other than to look at a bunch of ads and say, here's my new IC, here's my new inductor, here's my new PC board, buy me. And it's like, well, why do I want to buy you? Okay, I'll look at the information and if it intrigues me then I will go to the website and and actually look at the data sheet or or

look at the notes for how it's used and and yeah it is a bunch of ads there's some interesting articles once in a while in there that are actually useful. But those ads are equally as important. How has electrical engineering changed in the last 20 years? Well, as I was just alluding to, things like the plethora of different components has just gotten astronomical. Processors, there used to be 20, 30, 40 different processors and that's all you had to choose from.

Pick a Motorola or you pick a TI or or you pick somebody else, and that was it. Now we've got microchip must have a couple hundred versions of each of their microprocessors that have all kinds of different embedded parts to it. And I don't know how, you know, that's just something that has changed drastically in the last 20 years is the capabilities of ICs. What we were just talking about, development boards. 20 years ago,

they didn't have them. They did. They were about $5,000. Yeah. And it's like, well, finally, they figured out the dev board

is how you get an engineer to buy their product because now they've given the engineer a way of developing their circuit using their product. And even then also, I don't think the field engineers were as capable 20 years ago as they are now. I was working on a battery charge circuit just the other day and was having a heck of a time. Physically went to LinearTech and walked in the door where the FAE was after having contacted him and said, here's my board. Tell me what's wrong. Two days later, he handed it back to me and this, this, and this is what's wrong i was like whoa okay so that i don't think was the type of service that ic vendors were using were providing 20 years ago and to be able to to communicate physically with somebody who's going to help me improve my circuit is very helpful and even if

it's over the phone and you you know you just have to talk to somebody as opposed to try to decipher a data sheet or an app note. I think that's changed. And more and more people are calling up microchip and saying, what's wrong with this? Point me to the errata sheet that tells me why I can't do that. Because errata sheets can be hard to find and you can do a search and you get a bunch of hits that don't have anything to do with what's wrong with your circuit. Why doesn't my UART work at 256k baud? Well, because you're using this pin and this pin for that other reason, and that's why it doesn't work because we messed up or whatever. And so that's what the Erratisheet will tell you.

If you're using it in this mode, you might not be able to use this thing in that mode. And it's so nice to be able to call somebody up and say, what's wrong with this design? And they'll point you to the errata sheet. That's funny. I never think of actually calling someone. I have a hard time even emailing my FAEs. I spend so much time on forums and looking around and figuring it's my fault. And forums can be useful. And that's certainly the first place to start because you don't want to necessarily be calling your AFI every time you have a problem.

But 20 years ago, you couldn't. They really didn't exist as being a resource. And forums didn't exist either. And so you were stuck with trying to find the Rata data sheet that would tell you what was wrong. And that's definitely a big difference. There was an internet there then, but there was no YouTube. So you couldn't just have somebody explain it to you.

Of course, 20 years ago, I could actually see the resistors and capacitors too. So that's a little difference that's occurred in the past 20 years. Is it the resistor and capacitors fault or is it Charlie's fault? My eyes are not getting that bad. Well, I want to talk more, but I think I want to talk more without microphones in our face. And maybe we can do that at the beach. Oh, that sounds like a good idea. Chris, do you have any questions before we go?

So you just went on a long description of what has changed. What hasn't changed? You still have to breadboard the device, still have to test it out, still have to put it through its paces. And even though you can't see the resistors and capacitors, you probably have to be able to probe right down into that resistor capacitor net and somehow figure out what's wrong

because there's always something wrong. And so that really hasn't changed. the whole idea of breadboarding something testing it out playing with the parameters looking at the signals whether it be a through hole power supply or a integrated circuit with tiny little devices on it and then iterate from there and improve from there think about what could go wrong so the whole process of of designing a board is still the same you still have to get your hands dirty and and play with the scope and that skill set of debugging is probably the difference between a really good engineer and a great engineer. Because a really good engineer will be able to debug, but it'll take him a little longer. A great engineer, he'll be able to come in, look at the circuit after he's seen how it reacts and say,

oh, that's really stupid. I got to change that. That brings a question up for me. You were, you're older than I am. And I think on the show, we've talked about age and all of that. But I have some engineering friends who have left engineering for one reason or another, career changes, and now seem intimidated, like they can't get back into it. Do you ever feel like it's too hard, or do you have any advice for people who think they can't do it? So when I told one of my friends that I was looking for a new job, he says, aren't you old?

How are you ever going to find a new job at your age? He says, don't you find yourself not being able to do what you were doing when you were younger? And I had to look him in the eye and say, no, in fact, I'm actually probably better at doing what I'm doing now. I think I probably have gotten better partly because I do keep track of what's new and coming along partly because i work with kids like you who keep me keep me fresh um and no i i for whatever reason i still find engineering to be the fun challenge and i get intimidated once in a while when i've got a very complicated design that's got a highly integrated chip like this power supply that I had to go to Linear Tech with. It was a very integrated chip, and I needed a little help with it, which I don't usually.

But I took up the challenge, went and figured out what I needed to ask and how I needed to ask it, and felt comfortable that I didn't understand everything about it. Well, that's just fine. That's what those guys are all about. And I guess, like I said, the key is I'm an engineer at heart. I love solving problems and I love creating things that, well, in my case, usually move as opposed to blink. But anyway, and that's why I am always going to be an engineer and probably never be a manager because I don't want to manage people. I want to manage things. I feel like I should say, was that your last closing thought? But I know you have another. So yes, do you have any thoughts you'd like to leave us with? So, we had a pretty long career together.

You were at HP Agilent and then we moved on to Crossbow for, it must have been six or home, somebody reminded me that I should probably wear clean underwear every day because you never know when you might get hit by a bus. And sure enough, I got hit by a bus on the way home and survived to tell. On your motorcycle? On my motorcycle. Don't leave that part out. It's one of the exciting parts. Yeah. I was wearing a helmet and doing all the right things. Scratches and bruises and a few broken bones and nothing big. Just a few

broken bones. I see you for a little while. Yeah. Both the lungs collapsed. It was not a big deal. And eight weeks later, I was skiing in Vail, Colorado. So, you know, bones, they heal in eight weeks. That's good in vail colorado so you know bones they heal in eight weeks that's good enough so that's that's my parting thought make sure that you know every time you leave home leave work you've done everything you think you should do you've got

a nice organized desk that helps you the next day start your job up again. And because you never know when you might get hit by a bus. And now you've got a reason to be offended when people say they've thrown somebody under the bus. I've been there, done that. This is why I don't say in case I get hit by a bus. I say in case I win the lottery and blow this popsicle stand. Because, yeah, we stopped using that phrase at HP. Sorry.

All right. Our guest has been Charlie Ladd, principal electrical engineer. Thank you for being with us, Charlie. My pleasure. Also, thank you to Christopher for producing and co-hosting. Thank you to Digilent for their 15% off coupon. Embedded FM, good for the month of June. Thank you to Patreon supporters for your support

and letting me email mics and stickers around the world. And of course, thank you for listening. My final thought this week comes from EB White. That's Charlotte's web. Why did you do all this for me? He asked. I don't deserve it. I've never done anything for you. You've been my friend, replied Charlotte. That in itself is a tremendous thing. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.