Embedded - 53: Being a Grownup Engineer

Episode Date: May 28, 2014

Jack Ganssle shared his wisdom on being a good embedded software engineer (hint: it takes discipline).  Jack's website is filled with great essays and new videos. He's also written the Art of Designing Embedded Systems and The Embedded Systems Dictionary (with Michael Barr). We covered a lot of ground, here are some of the highlights: Spark language Capers Jones on high quality software and associated statistics Joel on Software test for good teams LDRA unit test tool James Grenning's Test Driven Development for Embedded C

Welcome to Embedded, the show for people who love gadgets. This is Elysia White, and I'm quite pleased that our guest this week is author, speaker, and extraordinary embedded systems engineer, Jack Gansel. Hi, Jack. Thank you for chatting with me. Hi, Elysia. How are you doing today? Just fine. It's a beautiful day and a long weekend. How about yourself? Oh, great. It's beautiful here. As soon as we're done, I'm leaving and we're going off on our sailboat for a few days. Excellent. My co-host Chris is here too. Good afternoon, everybody.

Hey, Chris. So, Jack, I first learned of you from reading your column on the last page of Circuit Seller Inc. each month. It was a very formative experience for me. But for folks who don't know you, could you introduce yourself? Sure. Actually, that would have been in Embedded Systems Design Magazine, which is long defunct, unfortunately. Well, Circuit Seller lives on, and I enjoy every month's issue. Really? I got that wrong?

I'm so sorry. Well, you know, print has been dead for so long that I totally understand. It's not like there's so many embedded magazines to choose from. You know, Circuit Seller is great because it's a real embedded systems magazine for people who actually want to build stuff. It's not so hoity-toity, not so theoretical. It's all about building the hardware and the software and making things work. And I think that's really cool. But, I mean, I guess I probably did get both magazines and

didn't notice that they were different. But Embedded was great. Yeah. Yeah, I mean, I thought Embedded was fantastic and it lives on as Embedded.com, the website. It went through a lot of, like all the

print stuff, it went through a lot of, like all the print stuff, it went through a lot of tribulations over the years. And it just is so expensive to produce a printed magazine that has to get mailed out that they couldn't really justify it. And I don't have any insight into the machinations that went on in the decision process. But I certainly understand it. And we've seen it everywhere. EDN. EDN's got to be my all-time favorite magazine. And even EDN is dead.

Yeah. Well, getting on to more positive things. As we talked about the show, you suggested a topic, that firmware people need to act like grown-ups. And I completely agree, but I also want to know what you-ups. And I completely agree, but I also want to know what you mean by that before I completely, completely agree. Well, okay, I'm sure I'll change your mind here. In the 40 years that I've been in the embedded world,

I've seen this industry morph from basically a lot of wild-eyed hackers, of which I was one once upon a time, back when the industry first started. We were building fairly small bits of firmware, and you could get away with the heroics. So, heroics really worked. Today, when we're building these enormously complicated things, I mean, geez, a cell phone can have 15 million lines of code in it. A car can have 100 million lines of code in it. And I think any rational person would realize that the old hacking sort of technologies just don't scale to these larger applications. The thing that's frustrating to me in all my travels around the embedded space is that so many of us are still working on using heroics rather than engineering.

And I guess what I've thought about a lot, especially in the last four or five years, is that we use the term software engineering, but we're not doing engineering. I mean, engineering is about taking a known art and then using that to create a design, create an architecture, to design an implementation, to implement it, to predict what its behavior is going to be, and then to measure that behavior to see if it matches up with our predictions.

And if it doesn't, as it often won't, then we take some corrective action. And what I see, and I hate to generalize, but in general, what I see in the embedded firmware space is that it's mostly about as quickly as you can get in there and start writing some code and then try to make the thing work and i i it's it's clear to me the evidence is pretty clear i mean geez look at the disasters we've seen like this toyota thing recently uh that that just doesn't work and yet we're still by and large not adopting new processes to uh to be grown-up disciplined engineers that so it's i've raised kids you know and one of the hardest things with raising kids is teaching them things like discipline like you know you're going to get

up and go to school you're going to do your homework after school you're going to go to bed turn off the tv that kind of stuff. And that's part of the process you go through to become a grown-up individual. And so, you know, as grown-ups, we do things like pay our taxes reluctantly, pay our bills, take care of the people we are supposed to take care of. That's all part of the discipline of being a grown-up. And I really passionately believe that we need disciplines in order to effectively create reliable and safe firmware.

I definitely agree with that wholeheartedly. Did you catch Michael Barr's keynote at EE Live? Yes, I did. I thought it was really interesting. I thought he did a great presentation. There was a, for people who are listening who don't, didn't see it, Mike Barr was the lead expert witness on the Toyota unintended acceleration case. And which was settled, or I shouldn't say settled, there was a fine levied against Toyota for $1.2 billion by the government. And there's still several hundred cases being adjudicated in civil courts.

So Mike has more insight into what's going on than anyone else. He had to, as part of his expert witness gig, go through the code and figure out what was wrong with the code. And what he found was basically appalling. The code was, well, you were there, you heard the stuff, you know, 11,000 global variables and a lot of stuff that he's not allowed to talk about. It was pretty appalling. And that's not how you should write code if lives are on the line. It's really not how you should write code, even if lives aren't on the line, because that's just bad. But I've done FDA and FAA certifications, and that seems more grown up. I was a little disappointed in Michael's talk, because he talked about all the things we do wrong and he didn't really talk about how to do it right. I think it'd be difficult

for him to have talked about doing it right in the short time that he had for the talk. He did a great job, but now we should talk about more. Everybody should be talking about more. And hey, you've got the insight. If you've done the FAA stuff, I guess

178B or C type work is a very different kind of environment than obviously that practiced by some of the automakers. But it is a lot of documentation. And a lot of the documentation is proving that you sat down and did the design and you thought it through, which was similar to the FDA documentation. It wasn't write things because writing is fun. It was to prove that you thought these things through. Right.

And I do believe, though, that there's no one-size-fits-all. And the FAA stuff is very interesting because the costs are sort of irrelevant. The risk factors are enormous if they get them wrong. And the interesting thing is it works. That has worked for that particular environment. Well, as an engineer, I knew what the costs were. I mean, I knew that my device could make somebody crash and not only crash and lose their life

but crash and kill other people and there was no way I was going to let anything shoddy through. You know, there's a suspicion, or let's put it a different way.

You made a really interesting point. There's all this documentation which is required, but it's more than documentation. There are things you have to do, like the testing, the MCDC testing and all this kind of stuff. So that's process things, although you then have to document it like crazy. And a question has been raised in the academic literature,

which is, are these processes what leads to this great code? Or is it a culture that believes in writing great code and is willing to or is forced to go through all of these documentation and other processes? And there's no real evidence for one way or the other. But it is interesting. And in my travels, I have seen organizations that have a safety or reliability mindset tend to produce safe and reliable code, whether they use the 178 or some, you know, maybe even their own process. Having done FDA stuff in the past and been in a couple of companies, a handful of companies that follow FDA processes, you know, it's interesting because it's a lot of what the company says, right? There's a process you follow and you do your documentation and you claim to have performed these tests, but it's all what you're claiming and there's ways to game the system. And, you know, I've seen pretty terrible code come into places that had pretty terrible code that followed FDA processes to the

letter and been at places that had fantastic code that weren't so adept at navigating the FDA process. So, what you're saying is right, the culture versus the process. It's an interesting balance, and it's hard to decide. I tend to fall on the side of it's a great team with conscientious engineers is the important thing, and the process will help you get along, but it's not going to save you from disaster necessarily. Well, I'd like to come back to that comment. That's a really interesting comment.

I'll go a little off on a tangent here. The FDA processes such as they are that you talk about are sort of interesting. FAA is freaking out. Matter of fact, I had a meeting with them just two weeks ago.

They're freaking out because basically these devices, so because basically these devices, so many of these devices, the software is just crap. There's no pretty word for it. I mean, in many cases, like you say, the stuff is great. But the 510K process, which is sort of how you squirrel your way into FDA's good graces, is, in my opinion, having been through this sort of a joke.

I mean, they- Yeah, I'd agree. Yeah. And I saw one case where they file a 510K, and then the instrument they delivered wasn't anything like the 510K. But there's no loop closure, you know? So they can get away with that. Yeah, the loop closure was usually something goes wrong,

and now you're being audited. But that's way too late. That's way too late. It really is. And so if we go back to what you said about good processes and teams, man, the team is hugely important. I totally agree. And one thing that I have tried to push in my current work and for all those years when I was a manager of teams was, again, this discipline.

And I tell you, there are people who will not follow the rules. And it's more than just being a great team. It's more than being just great buddies. It's more than being great, really smart people. It's people who are working together towards a common goal. And I've, on occasion, had to use dramatic firings to show that this is not behavior that we will tolerate, that this team is only going to produce great code. And then we have to follow some processes. And to give you some examples of things that I really believe are necessary,

it's a very disciplined test regimen, although test sort of doesn't work. Test is what proves that what you built is correct. Unfortunately, what we use test for mostly is to find the bugs, which is exactly the opposite of what the quality movement has taught us. Tests should be just to prove that there are no bugs or that there are very, very few bugs. The quality movement, which revolutionized manufacturing, showed that get it right is the right way to do it. You can't bolt quality onto a product. So when it comes to, I mean, in an hour,

we can't talk a whole lot about processes, but there are some that I would pick that I think are really important, and I'm sure it will tick off lots of your audience. Requirements. I think requirements gathering is really important because if you don't know what you're building, you're not going to get a happy solution. And everybody says, yeah, yeah, yeah, but requirements elicitation is hard. And they're absolutely right.

It is really hard. But that doesn't mean that we skip it you know that means that we uh we're grown-ups we practice awesome engineering and we do the work and get the requirements they'll never be 100 if you're building a web page or something like that which i know nothing about i presume getting good requirements is very hard because you have everyone wanting to tweak with colors and stuff like that but in my experience in the embedded world, mostly we have a pretty good idea of what it is we're trying to build. And since we have to do hardware, we have to have some idea.

You have those constraints very early. How much processor power, how much memory, all that kind of stuff. What your IO is, that's all very well defined. Very, very early in the program. And I've done so many surveys about this. Nobody, or I shouldn't say nobody, the number of developers using any kind of formal requirements tools is essentially zero.

It's in the noise. People will do it using tools like Word and stuff, and that's great. That's awesome. As long as it's done with discipline and it's a very, very carefully elicited set of requirements, very carefully thought through. And my friends in the agile movement would, or I shouldn't say would be, we have this debate constantly, totally disagree with me. Which is, of course, they're right. And we've had some, do you know James Grenning? I do.

It's one of my questions to ask you about. Okay, great guy. He's a great guy. He was on the show a few months back. Oh, was he? Okay. He's a great guy. Well, he's the test-driven development

in the Embedded World Guru. And he and I have had some fantastic times together. We did a presentation where he was in front of this big group. He was pushing test-driven development and he was supposed to be the engineer convincing the boss, the CEO,

to use TDD. And I was the somewhat skeptical CEO. And of course, I got a top hat and a big cigar and all this kind of stuff to play the role. It was gobs of fun. But where I disagree with a lot of the agile TDD folks is when they say things like, well, if you write enough code,

a design will emerge from the code. Oh, yes. I totally disagree with that. And literally, those are quotes by the Illuminati in that field. Agile does make me cringe because they don't do any requirements sometimes. Well, and especially applied to hardware. I can't square that circle. You can't make all that stuff up as you go along.

And we'll just ship whatever is ready when it's time to ship that makes me absolutely cringe as somebody who's been you know started and run businesses when this whole notion of is it'll be done when it's done it's just it's just poor business it's not a business like business like behavior because there's a the olden days would use a perch chart because there's more than just the engineering. There's the advertising campaign, the marketing literature, all kinds of stuff all has to come together with the engineering. And for the software group to say, hey, don't bother us. We'll let you know when we're ready.

That's just extremely poor business. Well, it seems to me that there's a whole notion that because schedules and plans often aren't correct, or it's impossible to accurately predict things down to a certain period of time, you might as well just not do it. Yeah. And that doesn't necessarily really follow for me, because you still have to plan inputs and outputs and where things line up. And yeah, you might be wrong, but that's okay. You can be wrong a little bit, but not doing any planning, you're likely to be much more wrong than you would have been. You know, that's what I hear from people all the time. It's so hard to get this right, and it is hard to get it right.

That doesn't mean we abdicate our responsibility. And the other thing we have to recognize is that we will never get it right. That doesn't mean we abdicate our responsibility. And the other thing we have to recognize is that we will never get it right. But, you know, if you're in school and if you get a 90%, that's an A. An A is a B. You know, maybe if we aim for 90, we'll be doing an A. That's not bad. It's a whole lot better than the Ds and Fs that most of us are getting right now. Well, and I think doing a little better today, this week, this year, than we did last is the way to go. I mean, and then tomorrow we'll get a little better.

So let's at least get 80s now, and then the next year we'll do 90s, and then maybe someday we really can have safe code that doesn't cost a fortune. I mean, some FDA and FAA products do tend to cost a lot more because there are these processes that imply grown-upness. Yes and no. And boy, several things spring to mind here. Let me start with the first one, which is you've made a really good point. We'll do a little bit better this year and a little bit better next year. That's absolutely right. With one caveat, which is we're engineers.

Measure it. If you can't give me a number, it doesn't mean anything. And so when I hear teams saying, well, we're using this technique because we all like it, I say, well, that's ridiculous. I mean, give me numbers. So if we say we're going to do a little better this year, measure it. Now, what is your metric? Maybe it's released bugs. Maybe it's schedule.

Maybe it's all of the above. But until we start taking numbers, we're not doing engineering. We're basically liberal art majors. Metrics are hard, though. How do you compare apples to oranges? Because the products aren't always the same. They are never the same. And you're not trying to get 100%. Right now, we have nothing. And if we can measure something, we'll start with what I look at as an impressionistic painting. We might not be able to actually see the outlines of that barn in the distance there. What we see is sort of a blur. But if we start taking metrics today, we'll at least start to see that blur of the barn.

And as we improve, we'll see that in sharper and sharper focus. And they'll never be perfect. All the academics say you don't dare measure anything in lines of code and they've got some really good justifications for it. But lines of code will get you to 80% correctness for things like bug rates, productivity and stuff like that. And you have to average it over tens of thousands of lines of code. And that's not bad. That's a good place to start. The academics want to see these function points.

Well, we're not going to do that. You know, it's stuff we don't, it's not the way we think. But we have to start measuring something. And then what was the other thing I was thinking? Oh, well, I asked about

the price of being a grown-up. Oh, thank you. That's what happens as you get older, the short-term memory. Yeah, well, you know, there are some really cool studies on this that have shown that safety-critical code doesn't cost anything more. As a matter of fact, they've got one here, which is really cool. Using the 6-1508, the European safety-critical standard, they've shown that if you follow discipline processes, and they use the capability maturity model as a process, not the canonical process, but a process, you can get safety critical code for the same price as the old stuff. Now, that doesn't include all the extra documentation and certification, because those are additional requirements put on the thing.

But you can get that great code for no additional cost just by changing the way you do things. And I'll give you some numbers. And you may have heard some of this because I think you sat in on at least one of my talks at the show. The average

firmware project spends half the schedule in debugging, which is ridiculous. I mean, it's crazy. That means the other half of the schedule, I call that bugging. Writing the bugs, yeah. In no other endeavor would anybody tolerate that. I mean, can you imagine if you were throwing printed circuit boards away that fast? You know, oh, sorry, boss, I need four more revs today.

I mean, it's crazy. And so by using the right processes, we can usually reduce that. Give you another example. Oh, here's some good stuff. Everyone knows Ada, or I shouldn't say everyone knows Ada. Everyone knows of Ada, which of course was designed from the outset to lead to really high quality code. There's another language called Spark, which is actually Ada. Any Ada compiler will compile it. It's a subset of Ada. And they've done some stuff where basically in the comments, with formal notation, you write the intent of what the code is. And there are these tools, which are all free and open source, that check those annotations against the code and flag if there's a problem. And the thing that's interesting about Spark is that it's creating a problem for people because the bug rates

are so low, they can't measure them. And one of the guys I know who's involved with this said, oh, he's completely forgotten how to use a debugger. I was looking at the project, it's online, this is a big bad NSA project called Toconeer, where they actually compared that to a traditional, using a traditional language and sloppy processes. And they delivered a system with one, they did find one bug in something like two-thirds of the time that it would take with a traditional process. And so we often tend to delude ourselves into thinking, well, getting it right is going to cost us time and money and stuff, when really, quite the opposite is true. And that's what we found in the quality movement.

Hey, all the car manufacturers, except Toyota. Oh, I shouldn't say that. I have two Toyotas. I love them. But all the car manufacturers found that by building them right and not trying to fix all the dents and scratches at the far end of the production line, they save money. I used to have a joke at one of my previous companies

that we don't have time to do it right, but we have time to do it twice. Yeah. What was it? One of the famous guys, Barry Beam maybe, one of these guys said, bill one and throw it away, you will anyway.

No, it's not right. No, no, and I don't advocate for that, but it's interesting. It does, yeah, lots of people seem to. Okay, but how do we do it right? I mean, is it code inspection? Is it writing test-driven development, but maybe not doing the agile part? I think, okay, from the very short summary view, I would start with get the requirements right,

or at least as close to right as is possible. Get a design which is correct. And when I was a young engineer doing hardware design, everything went through a design review. Everything. And even on space missions, there are critical design reviews that should be reviewed.

This is the hardware and the software design should be reviewed. The interfaces should all be specified extremely carefully because that's how everybody talks to each other. The code, every bit of code should go through a code review.

We have so much data on this, it's ridiculous. But code reviews lead to usually better code at a fraction of the cost of doing debugging. I think we should aggressively use the tools, the static analyzers. Oh, man, there's some cool stuff out there now. And they're not cheap. You're talking $10,000, $20,000. But compare that to a cost of an engineer, a loaded engineer's cost.

It's not unattractive. A couple of weeks of debugging. Yeah, exactly. And there are other things like lint. I talk to so many people who won't use lint because it's a pain. It takes a while to learn how to tame it. It is a lint. I talk to so many people who won't use lint because it's a pain. It takes a while to learn how to tame it. It is a nuisance. It'll print out gobs and gobs of false positives until you learn how to deal with it.

Until you learn how to write really clean code. Yeah. So these tools are basically saying, you know, grow up. Okay, do it right, grow up. Use those extra parentheses. I'm not quite right, grow up. Use those extra parentheses. I'm not quite sure what this means. Use those extra parentheses. And I think it's awesome.

And you said earlier, Alicia, that we can do a little better this year, a little better next year. I really advocate this notion of personal improvement, that every year we should be trying individually to be getting better

to write cleaner code, to write code with fewer bugs. In terms of the process, we should be measuring conformance to requirements and bugs at every stage. I view each of these stages as filters. And you're going to remove defects at inspections. You're going to remove defects from the tools, finding them.

You're going to remove defects when you do start debugging. And we need to know where we stand, what the numbers are. Because we have the numbers. We know the numbers are because we have the numbers. We know the numbers. What's his name? What? What? No, no.

Capers Jones. I don't know why I'm having so much trouble pulling stuff out of the old memory banks today, but Capers Jones has got gobs of data, which he's been kind enough to share with the industry, and he's actually shared all to share with the industry, and he's actually shared all the raw data with me, which shows where companies fall into a malpractice based upon how many bugs are found at each of these filter levels, and where companies are in

the top 2%. And the interesting thing to me is that nobody knows where they stand or very few people know where they stand where their organizations stand and in in business they call this benchmarking you look at how your financials are compared to other uh other companies in the same industry they call it benchmarking so you can see roughly you know are you totally out of whack with the industry are you a star performer or what? And I think engineering groups need to do the same thing. This data is really not that hard to gather.

It means changing how we do things, writing a few more things down. There are some great tools for it. Oh, other tools. I've got this tool here, which I've been playing with. It's insanely cool. People don't believe me when I tell them what this does. It's LDRA unit. And what this thing does is it looks at your code and generates all the test code.

Now, you have to help it along. It's not totally automatic. I mean, there's some things it can figure out by itself. There's some things that you have to act as an oracle for. But, whoa, I mean, I play with one thing. I fed it a 10,000 line program. I got 50,000 lines of test code. But if you think about

that, suppose you had not, and there are other companies that do this too. If you had written the test code manually, how many lines do you think you would have written? I bet you it wouldn't be anything. 10,000 lines, maybe. How many lines do you think you would have written? I bet you it wouldn't be anything A thousand lines maybe Okay, so I agree with you

But some of the places that I have worked definitely wouldn't How do I, as a consultant How do I try to gently herd my clients in the right direction? Or as an employee at a big company, how do I get my manager to agree? I mean, design reviews done by yourself, I've actually done that. It isn't as bad as it sounds, as long as you let a week go by and forget it all. Exactly. But how do you persuade the machine to let you start doing a better job? I don't think it's a simple answer.

I think in one case, if you can find an advocate in some organizations, you have an evangelist who can convince people that this is something we should try. I have seen that work. In other organizations who are very resistant to it, I throw out a challenge saying, fine, okay, don't do it. But just to pretend that you're an engineering organization, start collecting metrics. And then we sit down and talk about the metrics. Just collect the metrics. Don't change anything. Just start collecting metrics. And then we sit down and talk about the metrics. Just collect the metrics. Don't change anything.

Just collect the metrics. And then compare those to the rest of the industry. And maybe you'll see that, oh, my God, you're a top performer. In that case, maybe you don't want to change anything. In the cases where I've been involved, that has not been what we've done. And to tell you the truth, at that point, it's still not easy to get the message across. But at least then you're starting with numbers and you start to say, okay, these are your numbers. Now, if we change the numbers to this, which is, say, oh, maybe the top 10%, you would save this much money and ship this much better code.

Are you interested in that or not? Yeah, once you start being able to hit them in the pocketbook, it does get a lot easier. What's fascinating to me is everybody who's an engineer or is in a technical field is supposed to have a science background. Most of us have come through an engineering degree, which is science-based, you know, taking those core classes, and somehow we've lost that science sense of things need to be tested and measured and challenged to see if what you're doing is

actually doing what you think it does. And I don't know at what point we lose that sense. Maybe it's when we have to ship a product in a big hurry and then things go out the window, but it is kind of sad. Yeah, and certainly time to market has always been a problem, and it's worse every year. I keep thinking that if you could build a perfect system with a million lines of code in a month, then the next one, they'll want you to do it in two weeks. We're chasing our tail.

It's really very much of a problem. But that doesn't mean we shouldn't be trying to do the right thing. I think it's a matter of ethics, to tell you the truth. I really believe this. As engineers, we know better than anyone else how things should be done in our particular domain. Of course, we think we know it for all domains. And I think an ethical engineer is going to stand up for doing things the right way.

I agree with that. And I have been accused of being unyielding in some ethical things. And to me, that's quite the compliment. Thank you. So, going about it a different way, when I'm looking for new employment, say full-time employment, what questions do you ask to figure out if the team you're joining is full of grownups? I mean, Chris, you said that the team is really important and I agree with that. So how do we stop going to companies where it's all hack it, fake it?

You know, Joel Spolsky, the guy who did the Joel on Software blog for a long time, he has a test exactly for this. And he lists about a dozen or 13 things you should look at for an organization. And I would argue that a lot of what he is looking at are sort of high levels, things like do you use version control and things like that. And those are all necessary conditions, but they're not adequate. I think we have to ask the hard questions like, describe your life cycle. How do you create software? What happens if the time pressure is on and we want to skip step A, code inspections, whatever that might be. What happens?

And I think you have to ask that if you're the prospective employee. Oh man, I haven't been one of those in like 35 years. I think you have to ask the people who are interviewing you, who are largely the boss to be and whatnot. But also, if it's a good company, they're going to let you talk to their regular engineers

so you get a sense of what the team really is like. And you have to ask that of them. Joe, okay, what happens? The ship day is two weeks. We know we have this thing we have to do. Are they going to allow us to do this? Or are they going to say, stop doing all the good stuff, just ship it, damn it?

Just ship it. I've heard that before. I hate that phrase. I'll tell you, I don't want to pretend that any of this is easy. in a way I hope that this Toyota double call is a it won't be a wake up call

but it's a first step into a wake up call because if we just talk about that one settlement

1.2 billion dollars we know it was around a million lines of code plus or minus that's a 1,200 dollars per line of code

just for the fine. What do you think? Does that include comments? That's now the most expensive code ever written. And that should make people think. So I've got this other notion I've been ticking around in my head. I haven't figured out how to make this happen. But if you look at any public company, look at the annual report, there's a

risks section. Well, you know, we may not be able to buy these parts, there might be labor unrest, and all these things could cause our stock to go down. What happens when the CEO has to put in there saying, well, we have decided to use poor software engineering processes, so we may get sued out the door or all of our customers may rebel or the airlines might collapse and millions of people are killed or something like that. When the high, high-level boss,

when people start to recognize the impacts of these problems and the high-level boss is forced by investors to disclose uh these risks then action is going to be taken and we have to be ready to tell them what that action is going to be i think we can't wait for regulatory agencies to kind of help with that because i mean going back to the fda for a second when i was working with them uh a decade or so ago they didn't really understand software they were afraid of it but they didn't really have a good sense of what it meant and then the risks involved with it versus hardware and in fact we could take things that were basically pure software and put them in an fpga and suddenly they were happy even though it was you know the same stuff it's just we moved it

to vhdl and so it's that kind of thing that you know the same stuff it's just we moved it to vhdl and so it's that kind of thing that you know it's going to take them a lag of five to ten years to really come up to speed on things and understand them and maybe longer and it's really up to us as engineers to to do the right thing not wait for somebody else to tell us to you know when i get pushback on this um engineers, I then challenge them and I say, okay, you tell me your boss won't let you do this. I totally understand. That's a valid concern. But in two, four, five years, you're going to be the boss.

What are you going to do? And this gets back to the ethics discussion, I guess, too. Well, you know, I've done that. I went from being an engineer to a boss to a director. And at each point, it almost gets harder to stand up and say, no, we're doing this right, because the pressure from both sides becomes hideous. And you do have to hold on to those values and say, no, this is important.

This is more important than shipping on time because I'm going to save you money. And if we do this really right, we're going to ship on time instead of your plan, which is going to take longer, and it's going to be buggy. Everyone loves to complain about bosses. So bossing is the hardest job in the world in many ways. And yeah, boy, the pressure from on high because the first time I became a boss, started running an engineering group back in the 70s. My boss, we were in a little tiny 100-person company, way undercapitalized, always struggling for money. Paychecks would bounce. It was horrible. And my boss, the VP, would come in and say, how long is this going to take? And I would tell him.

He said, well, if you can't get it done by this date, much shorter date, you have to lay off three people. I mean, that's a lot of pressure. That's a lot of pressure. And I don't think there's any secret sauce or magic fairy dust. I mean, we have to do the best we can do to withstand that pressure and to protect our team, to keep our team doing the right things. And it's better. It really is better. Once you get into the habit, it's not harder either. Right. Once you've started. But that hurdle of, oh, I don't want to count how many lines of code I have. I don't even want to think about the bugs I have. It's hard to get over that. Some of the more successful teams I've seen, and man, I hate to say this, but what they do is they only hire new college grads who haven't been polluted. And the funny thing about this is these new college grads work for them for a number of years and then inevitably move on. And so many of them report being appalled when they go to another organization that the other organization is such chaos. When I left HP and went to a little startup and they didn't have version control, I spent a whole week wondering what in the world had I gotten myself into?

How could they not have version control, let alone everything else they needed to have? So you were at HP. Which HP? I did a stint at NetServer Division and then I went to HP Labs and then we got spun off into Agilent. Oh, through that transition.

Yeah. Agilent's a great company. I have gobs of test equipment from them. It's great stuff. Probably not any of

mine since DNA scanners aren't really electrical engineering things. Yeah, I agree. It's interesting to see how the different organizations, how different they are. So, one of the things that I've heard you rant upon is how much you dislike the word programmer.

And I agree, because engineering is just better better because it indicates all of these other things, this careful planning and thoroughness. But what else do you hate about the word programmer? And then let's go into hacker because I'm sure that's going to be fun too. Yeah. And you have to also understand the fact that I'm an old guy and you get old guy habits I think I hate the word

programmer because any idiot can write code and it's really true any idiot can write code hey basic is 50 years old this year right and anyone can learn to write a few lines of basic. That's programming. That means, by definition, that you're a programmer. Whereas what we do, I believe, is software engineering, which is, yeah, programming is part of it, and a hugely important part.

So is typing. Good point. Right, exactly. I hadn't thought about it that way. That's excellent. And so I prefer software engineer, and I want us to live up to that. I think engineering is a noble profession, and I think if we're going to call ourselves software engineers, then let's live up to the nobility of that moniker. So there's some identity there with it being a profession that's, I mean,

it's been around hundreds, probably thousands of years. Software engineering? Engineering. Well, you know, like Chris said, it's the great teams that do great software, and this profession is also sort of a team. We are

engineers, because we worked hard to become engineers, and then we can't call ourselves engineer anymore in so many states because of the regulations. But we still choose to do so, thankfully. And that's sort of segregating us from the rest of the population. Okay, we are in this discipline and we practice this discipline in a certain way and i think that's awesome and programming is like you said typing is perfect i really like that if you want to go into hacker and maker uh well first of all hacker is a word i detest and i know that the definition

is changing and you're a hackaday judge where they're all... Yes. And yes, it kills me, the hackaday website, the hacker hardware. Like, I hate that word. Hack, to me, has this connotation of just throw something against the wall and see what sticks. That's everything I don't want to do. Right.

However, and I'll sort of segue towards maker because i'm not sure what this the difference is when we talk about things like arduino arduino is way cool and originally it came out as a way for artists to build little systems that do some like smart art which is really cool and uh then it sort of got confused and conflated with this maker or hacker thing where every time some 16-year-old makes an LED blink, he's on the news, it's going to be the next Albert Einstein or something. And I think the maker movement is, at one level, a fabulous thing, getting people into this cool feel. I sort of got into it, at least in part, through the ham radio.

And in ham radio, you could get in as a novice, knowing very, very little, but you could get in. But with ham radio, there is this progression. You then study for the next test, and you get more benefits, and you're constantly improving your skills. And I don't see that in the maker-hacker movement. They're just more blinking lights. Different blinking lights and motors.

Playing. Playing is great. If that's what they want to do, play. That's fantastic. But I think that's miles away from the stuff that we do. I agree. I'm astonished at how complex some of the devices

are. And I love that they share the enthusiasm with so many people. And it makes engineering cool. But I wish more people understood that there is a difference. Yeah, well that there is a difference. Yeah, well, there's a difference between,

this comes down to also the difference between like a prototype and a product. I mean, there's some separation there between, yeah, I made this thing that does this once. Right. That does this once, exactly. And the hackers slash makers who get into this just to do some playing, I admire them. The ones that get into it and use it to progress along a path of further knowledge, I admire them. The confusion exists with the ones that are sort of, like you say, creating products that are really just prototypes.

I mean, a prototype is 10% of a product, basically. Yeah. Although I sometimes feel a lot of pressure towards being a maker artist sort, having my code up on GitHub and expectation to do makery things in my free time all the time and to be technical all the time, except not like professionally technical,

just kind of poke at it technical. And I need downtime, too. Well, and you need a reason to do that, other than just being part of a movement. Yeah, like I said, I'm going sailing here in a little bit. Exactly. I'm not going to judge people who are compelled to do this 24-7. I mean, that's great if that's what makes them happy. Yeah.

But I do think that it is a really big, fascinating world out there. And I, for one, wouldn't want to be just throwing slinging bits around. Fair enough. And Linux in embedded systems. I keep hearing more and more about that. And part of my reaction is, well, once you put Linux on it, it's not an embedded system anymore. I gather I'm not in the majority with that view. Not even in the room.

One thing I've done is to give a talk at a Linux convention and start off by saying Linux sucks. I'll agree with that. Well, and my point is not that Linux sucks. It's that Linux is a pile of bits. And it's totally a perfect solution for a lot of problems. And it's a lousy solution for a lot of other problems. It's just important to make intelligent decisions rather than crowdsourcing

knowledge and seeing what everyone else is saying. I get literally every day email from people saying things like, well, when should I use an RTOS and when should I use Linux? That's sort of like saying, is this a lake or is this the Atlantic Ocean? I have some data on why people use Linux and embedded systems.

And the top three reasons are so I can further my professional career, which is not a very good engineering reason. Because I've always wanted to learn about it. It's the same as the first. Yes. And the third one is because of the memory requirements, which tells you that they're totally clueless about what's going on.

Wait a minute. Was it because they want, I mean, is it because they secretly work for a memory company? Maybe that's it. Wow. Hey, I've seen some great,

great Linux-based embedded systems, and I would call them embedded systems like your cable box, your set-top box. That is almost certainly running Linux. I mean, most of them do today. And it only does one thing. It's just a cable box.

I would call that an embedded system. Well, there are a lot of routers that do it, too. And routers. Those are definitely embedded. Yeah. And those are probably pretty good applications for it. One of the coolest technologies to come out in a while, I think of the ARM Cortex-M series parts.

Yeah. I mean, they're cheap. Some of them are half a buck. And they'll never run Linux. They will never run Linux. And they are truly deeply embedded systems. And they're pretty cool. They don't have the MMU. So they can run the micro-C Linux, but that's a

small cut down. People forced Linux onto them with great pain. And then that's all they do. Great pain. Right. I mean, if people are putting external memory on these MCUs, they might as well go to a Cortex-A9 or something anyway. So,

how do you evaluate if something is a fad? I kind of feel like Linux and embedded systems right now is sort of a fad, and we'll figure it out and there'll be things that Linux is good for and things that it isn't, but it's such a everybody wants to do it right now sort of thing.

Well, there's a fellow named Steve Leipson, who's a good friend and a sort of well-known analyst. He postulated what's called Leipson's Law, which is that any new technology really takes about 10 years before we really understand it to apply it well. I think there's a lot of wisdom there. I think Linux is sort of over the fad hump in the embedded world. I think that, although a lot of people are still confused by it, it's being applied mostly

in more cases than not, I think in reasonable fashions. People have gotten over the thought that they could put it on in 8051. Literally, I've heard that question. And so, like you said, routers and stuff like that, those are pretty good applications.

I think there's still a lot of confusion. People are saying, well, I'm going to use Linux because of security, which is nonsense because we know the numbers. Linux is what's called evaluation assurance level 4+, which is exactly the same level as Windows. And what that means is that it's perfectly secure if nobody tries to break in.

So, I mean, people are a little confused about some of those things. What I have a little trouble with is that there are a lot of zealots, whether they're Linux zealots or zealots for whatever it may be, who tend to propagate an idea just because they like the idea

instead of holding it to the cold truth of engineering analysis. Right, yeah. And it goes back to requirements, you know. Does Linux meet your requirements better than anything else? Okay, that's what you should use. But you shouldn't just pick it first

and then write your requirements around it or try to mold it to fit Absolutely And embedded Linux is very much free as in puppies Yeah, that's a good way of putting it Have you heard that? Free as in beer Free as in freedom and then free as in puppies I hadn't heard the puppies one

Yeah, you know, you get a free puppy And it was not what you should have done It was. Yeah, you know, you get a free puppy, and it was not what you should have done. It was so much more work than if you'd bought a puppy. Right. Or even better, bought the goldfish. A free puppy is a pretty lousy gift. Exactly.

So what else have you been up to? Well, I do a lot of writing. I do a lot of seminars about embedded systems. And I have my own research projects, things I'm interested in. I've been working on, well, one of the fallacies you'll see advertised everywhere today is a lot of these MCU vendors are saying that their processors can run off a little coin cell for decades. And that struck me as sort of silly. And so I actually built some test systems. I've spent a year, I've collected millions of data points and have characterized how these batteries work in those sorts of ultra low power up situations. And I've published a number of articles about it. I've got a system in back of me right now, which is taking new data.

It will be for the next couple of months. And it's interesting. It again goes against the grain. I guess what I'm trying to do is tilt at windmills, where people say things like, sure, this battery will last for a real long time because we do this facile bit of math on it, instead of really doing the engineering analysis. And so that's what I'm trying

to do with the batteries. I've done that with debouncing contacts. Everyone says, one of the questions I love to ask, how long do you wait for a contact to bounce?

And people give me numbers, 10 milliseconds or whatever. And I say, well, where did that come from? And the answer is either well that's what everyone does or joe told me or something empirically determined because i tried it three times on my desk with two switches right so i took a 50 pairs of switches and developed all kinds of data and i published it on my website it's very it's interesting and uh i i and I'm enjoying shining a spotlight on some of these corners of the field where we really don't have the data that we need.

There's nothing worse than conventional wisdom. Yeah. Your website has a lot of information. I went and bookmarked a bunch of the articles that I hadn't seen before, and I'm looking forward to going back to reading those. But there were, yeah, you had a ton of stuff on there, and it's all free. But you also have a book, more than one. I have one on my shelf that I know is yours,

The Art of Designing Embedded Systems. Yes, that's my second oldest. I've done six books. You mentioned Mike Barr. Mike Barr and I did the Embedded Systems Dictionary. Oh, I think in 2003 that came out. And I'm thinking that if another hundred years go by, we may earn out our advance on that one. That's been a real flop.

But yeah, I publish a out our advance on that one. That's been a real flop. But yeah, I publish a lot of stuff on the website, and actually we'll be launching a whole new website next week, and I'm going to start doing weekly videos on that site. Because, although I'm sort of resistant to it, I personally

don't care for watching videos on my computer. I spend enough time on my computer. But I've gotten a whole lot of requests for it. And I know that's how people communicate and learn today. So it's cool. I've got a few in the can. It's been a lot of fun to put them together.

Why do you do this? I mean, you probably could make more money, if that's the goal, if you did more seminars or did more code. Why all the writing? Why all of the information dissemination and the videos, too? I had a tools company. We made debuggers, in-circuit emulators, which I sold in 1997.

And I retired for a day. I was bored to tears. And I thought about this. I thought, you know, I'd like to do something that I felt would be worthwhile for something. And all I know is engineering. And so I thought, well, I had been doing all the the writing which I really enjoyed growing up as a Catholic school they beat that into us and I thought well I'll just continue to do this I mean as you can tell I think I'm very passionate about the subject I want to help engineers

improve their ability to build these products because they are getting so much more complicated and it is a cool feel I'd be miserable if I was divorced from it. I agree with that. I mean, I would be too. So do you have suggestions for college? I mean, college students mostly teach coding. And if you do get to go someplace that has a good process and helps you grow as an engineer, but what about the people who are just getting out of college and aren't

necessarily in those places that teach them right? How can people grow up? How can they learn as they're just getting into the field, or is this something that takes a while? You know, I do think it takes a while because in sales, they say you look for the pain and then you try to sell them something to fix the pain. And I think until you feel the pain, the pain of missed deliveries and bugs

and all this kind of stuff, you don't really understand that there is a problem or what the problem is. So the number one bit of advice that I always give to, well, two things I give to young engineers. First is save money every week. Use the power of compound interest. You'll be 40 before you know it.

But the second bit of advice is constantly study this field. There's a lot of stuff going on in this field, cool stuff. Some of it's wacky stuff, but you learn from all of it. And it is interesting. My data suggests that the average firmware person reads one technical book a year. I don't see how you can really stay connected with what these processes are, what the ideas are, what the exciting concepts are, if that's your level of engagement outside of the office. Well, we read a lot of data sheets, too. A lot of websites now. Yeah.

A lot of websites. Stack Overflow, Stack Exchange. But, you know, those are great, but the level of misinformation is pretty incredible too. There is something really nice about a book that someone took the time to think out an organization for and to say this is a good way of presenting the information and a good way of learning it. So yeah, I like books. And academic articles. And they can be very frustrating because sample sets are small and things. But some of them have some fabulous ideas in them.

And we have to read widely to sift the wheat from the chaff. And the web is awesome. I mean, I'm on the web constantly. But an awful lot of the stuff is vendor biased and all of that. You have to look at the bylines. Who's writing the stuff? Why are they writing it? Do they have some hidden agenda?

And evaluate the information you're getting that way. Yeah, exactly. I'm thinking about the things I've been writing and what agendas I have and I try to show them I try to show my cards with those because that's not my primary goal in life I like to share the information but yeah sometimes people have asked me to have agendas and to not show them and I know that's no I'll write for you if you pay me but I'm not going to tell anybody you're not paying me exactly well I think we are about out of time um do you have any else anything else we should talk about I can't think of anything.

I mean, I think it's great that you're doing this stuff and advancing and disseminating the knowledge base. That's fantastic. Well, thank you. I like you. I enjoy this very much. Well, you know, I've been doing this professionally for 40 years. And when I talk to young people considering a career, subject to their abilities, I often recommend engineering. I think it's a fantastic career.

And it has never been boring for me. And today, engineering, even in terms of pay, is pretty good compared to an awful lot of other career choices. An engineering degree will get you into a lot of those other careers, too. Mm-hmm. Oh, yeah. All right. My guest has been Jack Gansel, author, engineer, speaker, and all-around embedded systems hero.

Thank you for taking the time to speak with us. Well, thank you, folks. I really enjoyed it. Thanks, Jack. And thank you to Christopher for holding down the other mic and for producing the show. And thank you for listening.

Hit the contact link at embedded.fm or email us at show at embedded.fm. And Jack also has a contact on his webpage. I believe it is jack at gansell.com. That's correct. And let's see, a final thought for this week from Ted Nelson. I don't know who that is, but oh well. He had good things to say.

I believe that good design is magical and not to be lightly tinkered with. The difference between a great design and a lousy one is in the meshing of a thousand details that either fit or don't.