Embedded - 53: Being a Grownup Engineer (Repeat)

Episode Date: September 28, 2016

After a few new announcements, we replayed the episode where Jack Ganssle shared his wisdom on being a good embedded software engineer (hint: it takes discipline).  The new announcements include: Book giveaway contest deadline Oct 1st ARM's mbed Connect conference is Oct 24th IEEE CS talk by Elecia iRobot has internships (and other jobs), check their job site and if you want to apply, email csvec. Jack's website is filled with great essays and new videos. He's also written the Art of Designing Embedded Systems and The Embedded Systems Dictionary (with Michael Barr). We covered a lot of ground, here are some of the highlights: Spark language Capers Jones on high quality software and associated statistics Joel on Software test for good teams LDRA unit test tool James Grenning's Test Driven Development for Embedded C

Hello, I'm Alicia. This week I want to share with you one of my favorite episodes of our show, the one with Jack Gansel. I know some of you may have already heard it, but it's been over two years and it's worth a re-listen. Before we play that, I want to remind you to send in your numbers to win books. Deadline is October 1st. Choose a number and email show at embedded.fm. The charge of a cool-down is already taken, as is Brian's phone number. That was a little odd, Brian, but okay. In other news, Arms Embed is having a conference in Santa Clara on October 24th. I have tickets to give away, so if you'd like to attend... Well, I don't have a plan for how to distribute them yet. If you'd like a ticket, listen next

week. If you'd really like a ticket, they're $25 and the link is in the show notes. Chris and I do plan to attend, but we aren't speaking or anything. Phew. Though I will be speaking October 11th in San Jose at IEEE's Computer Society meeting. It is in the evening and free. My talk is titled Embedded Software, the Tricky Parts. I'll be giving out Embedded FM stickers, so say hello if you come. Listener Megan had a suggestion that we ask companies that are hiring about internships. That leads to two things. First, now is the time to request funding for 2017 summer interns. I know it seems really early, but you have to be ready. Second, the Keeners iRobot have intern positions already available in software engineering and electrical engineering. They're looking for embedded mobile cloud test and robots.

Wait, robotic algorithms. For more details, check out iRobot.com slash careers. For bonus points, you can email our blog contributor, occasional guest and friend, Chris Feck, at csvec at iRobot.com. Please include your resume, which particular position you're interested in, and a brief intro. And with noedded, the show for people who love gadgets. This is Elysia White, and I'm quite pleased that our guest this week is author, speaker, and extraordinary embedded systems engineer, Jack Gansel. Hi, Jack. Thank you for chatting with me. Hi, Elysia. How are you doing today?

Just fine. It's a beautiful day and a long weekend. How about yourself? Oh, great. It's beautiful here. As soon as we're done, I'm leaving and we're going off on our sailboat for a few days. Excellent. My co-host Chris is here too. Good afternoon, everybody. Hey, Chris. So, Jack, I first learned of you from reading your column on the last page of Circuit Seller Inc. each month. It was a very formative experience for me. But for folks who don't know you, could you introduce yourself?

Sure. Actually, that would have been in Embedded Systems Design Magazine, which is long defunct, unfortunately. Well, Circuit Seller lives on, and I enjoy every month's issue. Really? I got that wrong? I'm so sorry. Well, you know, print has been dead for so long that I totally understand. It's not like there's so many embedded magazines to choose from.

You know, Circuit Seller is great because it's a real embedded systems magazine for people who actually want to build stuff. It's not so hoity-toity, not so theoretical. It's all about building the hardware and the software and making things work. And I think that's really cool. But, I mean, I guess I probably did get both magazines and didn't notice they were different um but embedded was great yeah i mean yeah i mean i thought embedded was fantastic and um you know it lives on as embedded.com the website it went through a lot of, like all the print stuff, it went through a lot of tribulations over the years. And it just is so expensive to produce a printed magazine that has to get mailed out that they couldn't really justify it.

And I don't have any insight into the machinations that went on in the decision process. But I certainly understand it. And we've seen it everywhere. EDN. EDN's got to be my all-time favorite magazine. And even EDN is dead. Yeah. Well, getting on to more positive things.

As we talked about the show, you suggested a topic, that firmware people need to act like grownups. And I completely agree, but I also want to know what you mean by that before I completely, completely agree. Well, okay. I'm sure I'll change your mind here. In the 40 years that I've been in the embedded world. I've seen this industry morph from basically a lot of wild-eyed hackers, of which I was one once upon a time,

back when the industry first started. We were building fairly small bits of firmware, and you could get away with the heroics. So heroics really worked. Today, when we're building these enormously complicated things, I mean, geez, a cell phone can have 15 million lines of code in it. A car can have 100 million lines of code in it. And I think any rational person would realize that the old hacking sort of technologies just don't scale to these larger applications. The thing that's frustrating to me in all my travels around the embedded space is that so many of us are still working on using

heroics rather than engineering. And I guess what I've thought about a lot, especially in the last four or five years, is that we use the term software engineering but we're not doing engineering i mean engineering is about taking a known art and then using that to create a design create an architecture to design an implementation to implement it to predict what its behavior is going to be, and then to measure that behavior to see if it matches up with our predictions. And if it doesn't, as it often won't, then we take some corrective action. And what I see, and I hate to generalize, but in general, what I see in the embedded

firmware space is that it's mostly about as quickly as you can get in there and start writing some code and then try to make the thing work. And it's clear to me, the evidence is pretty clear. I mean, geez, look at the disasters we've seen, like this Toyota thing recently, that that just doesn't work.

And yet we're still, by and large, not adopting new processes that that just doesn't work. And yet we're still, by and large, not adopting new processes to be grown-up, disciplined engineers. I've raised kids, and one of the hardest things with raising kids is teaching them things like discipline, like you're going to get up and go to school, you're going to do your homework after school,

you're going to go to bed, turn off the TV, that kind of stuff. And that's part of the process you go through to become a grown-up individual. And so, you know, as grown-ups, we do things like pay our taxes reluctantly, pay our bills, take care of the people we are supposed to take care of. That's all part of the discipline of being a grown-up. And I really passionately believe that we need disciplines in order to effectively create reliable and safe firmware. I definitely agree with that wholeheartedly. Did you catch Michael Barr's keynote at EE Live? Yes, I did. I thought it was really interesting. I thought he did a great presentation.

For people who are listening who didn't see it, Mike Barr was the lead expert witness on the Toyota unintended acceleration case, which was settled, or I shouldn't say settled. There was a fine levied against Toyota for $1.2 billion by the government. And there's still several hundred cases being adjudicated in civil courts. So Mike has more insight into what's going on than anyone else. He had to, as part of his expert witness gig, go through the code and figure out what was wrong with the code. And what he found was basically appalling. The code was, well, you were there.

You heard the stuff, you know, 11,000 global variables and a lot of stuff that he's not allowed to talk about. It was pretty appalling. And that's not how you should write code if lives are on the line. It's really not how you should write code, even if lives aren't on the line, because that's just bad. But I've done FDA and FAA certifications, and that seems more grown up. I was a little disappointed in Michael's talk, because he talked about all the things we do wrong, and he didn't really talk about how to do it right. I think it'd be difficult for him to have talked about doing it right in the short time that he had for the talk.

Oh, yeah, no. It wasn't. It was just, you know, he did a great job, but now we should talk about more. I hear you. Everybody should be talking about more. Well, and hey, you've got the insight. If you've done the FAA stuff, I guess 178B or C type work is a very different kind of environment than obviously with that practice by some of the automakers. But it is a lot of documentation, and a lot of the documentation is proving that you sat down and did the design and you thought it through, which was similar to the FDA documentation. It wasn't write things because writing is fun.

It was to prove that you thought these things through. Right. And I do believe, though, that there's no one-size-fits-all. And the FAA stuff is very interesting because the costs are sort of irrelevant. The risk factors are enormous if they get them wrong.

And the interesting thing is it works. That has worked for that particular environment. Well, as an engineer, I knew what the costs were. I mean, I knew that my device could make somebody crash and not only crash and lose their life but crash and kill other people and there was no way i was going to let anything shoddy through you know there's um there's a

suspicion or let's put it a different way you made a really interesting point there's a suspicion, or let's put it a different way. You made a really interesting point. There's all this documentation which is required, but it's more than documentation. There are things you have to do, like the testing, the MCDC testing and all this kind of stuff. So that's process things, although you then have to document it like crazy. And a question has been raised in the academic literature, which is, are these processes what leads to this great code? Or is it a culture that believes in writing great code and is willing to or is forced to go through all of these documentation and other processes? And there's no real evidence for one way or the other. But it is interesting.

And in my travels, I have seen organizations that have a safety or reliability mindset tend to produce safe and reliable code, whether they use the 178 or some, you know, maybe even their own process. Having done FDA stuff in the past and been in a couple of companies, a handful of companies that follow FDA processes, it's interesting because it's a lot of what the company says. There's a process you follow and you do your documentation and you claim to have performed these tests, but it's all what you're claiming and there's ways to game the system.

And, you know, I've seen pretty terrible code come into places that had pretty terrible code that followed FDA processes to the letter and been at places that had fantastic code that weren't so adept at navigating the FDA process. So what you're saying is right, the culture versus the process. It's an interesting balance, and it's hard to decide. I tend to fall on the side of it's a great team with conscientious engineers is the important thing. And the process will help you get along, but it's not going to save you from disaster necessarily.

Well, I'd like to come back to that comment. That's a really interesting comment. I'll go a little off on a tangent here. The FDA processes, such as they are, that you're talking about, are sort of interesting. FAA is freaking out. As a matter of fact, I had a meeting with them just two weeks ago. They're freaking out because basically these devices, so many of these devices, the software is just crap.

There's no pretty word for it. I mean, in many cases, like you say, the stuff is great. But the 510K process, which is sort of how you squirrel your way in through FDA's good graces, is, in my opinion, having been through this sort of a joke. I mean, they- Yeah, I'd agree. Yeah. And I saw one case where they file a 510K, and then the instrument they delivered wasn't anything like the 510K.

But there's no loop closure, you know, so they can get away with that. Yeah, the loop closure was usually something goes wrong and now you're being audited. But that's way too late. That's way too late. It really is. And so if we go back to what you said about good processes and teams, man, the team is hugely important. I totally agree. And one thing that I have tried to push in my current work and for all those years when I was a manager of teams was, again, this discipline. And I tell you, there are people who will not follow the rules. And part of this part, it's more than just being great team. It's more than being just great buddies. It's more than being great, really smart people. It's people who are working together towards a common goal. And I've, on occasion, had to use dramatic firings to show that this is not behavior that we will tolerate, that this team is only going to produce great code. And then we have to follow some processes. And I mean, to give you some examples of things

that I really believe are necessary, it's a very disciplined test regimen. Although test sort of doesn't work. Test is what proves that what you built is correct. Unfortunately, what we use tests for mostly is to find the bugs, which is exactly the opposite of what the quality movement has taught us.

Tests should be just to prove that there are no bugs or that there are very, very few bugs. The quality movement, which revolutionized manufacturing, showed that get it right is the right way to do it. You can't bolt quality onto a product. So when it comes to, I mean, in an hour, we can't talk about a whole lot about processes. But there are some that I would pick that I think are really important.

And I'm sure it will tick off lots of your audience, requirements. I think requirements gathering is really important because if you don't know what you're building, you're not going to get a happy solution. And everybody says, yeah, yeah, yeah, but requirements elicitation is hard, and they're absolutely right.

It is really hard. But that doesn't mean that we skip it. That means that we're grownups. We practice awesome engineering and we do the work and get the requirements. They'll never be 100%. If you're building a webpage or something like that, which I know nothing about, I presume getting good requirements is very hard because you have everyone wanting to tweak with colors and stuff like that. But in my experience in the embedded world, mostly we have a pretty good idea of what

it is we're trying to build. And we often shortchange that. Well, and since we have to do hardware, we have to have some idea. You have those constraints very early. How much processor power, how much memory, all that kind of stuff. What your IO is, that's all very well defined very, very early in the program. And I've done so many surveys about this.

Nobody, or I shouldn't say nobody, the number of developers using any kind of formal requirements tools is essentially zero. It's in the noise. People will do it using tools like Word and stuff, and that's great. That's awesome. As long as it's done with discipline

and it's a very carefully elicited set of requirements, very carefully thought through. And my friends in the agile movement would, or I shouldn't say would be, we have this debate constantly, totally disagree with me. Which is, of course, they're right. And we've had some, do you know James Grenning?

I do, it's one of my questions to ask you about. Okay, great guy. He's a great guy. He was on the show a few months back. Oh, was he? Okay, he's a great guy. Well, he's the test-driven development in the Embedded World guru.

And he and I have had some fantastic times together. We did a presentation where he was in front of this big group. He was pushing test-driven development. And he was supposed to be the engineer convincing the boss, the CEO, to use TDD. And I was the somewhat skeptical CEO. And, of course, I got a top hat and a big cigar and all this kind of stuff to play the role. It was gobs of fun.

But where I disagree with a lot of the agile TDD folks is when they say things like, well, if you write enough code, a design will emerge from the code. Yes, I totally agree with that. And literally, those are quotes by the Illuminati in that field. Agile does make me cringe because they don't do any requirements sometimes. Well, and especially applied to hardware. I can't square that circle.

You can't make all that stuff up as you go along and and we'll just ship whatever is ready when it's time to ship that makes me absolutely cringe as somebody who's been you know started and run businesses when this whole notion of is it'll be done when it's done it's just it's just poor business it's not a business like business like behavior because there's a the olden days's not a business-like behavior. Because the olden days would use a perch chart, because there's more than just the engineering. There's the advertising campaign, the marketing literature, all kinds of stuff all has to come together with the engineering. And for the software group to say, hey, don't bother us, we'll let you know when we're ready. That's just extremely poor business. Well, it seems to me that there's a whole notion that because schedules and plans often aren't correct or it's impossible to accurately predict things down to a certain period of time, you might as well just not do it.

Yeah. And that doesn't necessarily really follow for me because you still have to plan inputs and outputs and where things line up. And yeah, you might be wrong, but that's okay. You can be wrong a little bit, but not doing any planning, you're likely to be much more wrong than you would have been. You know, that's what I hear from people all the time. It's so hard to get this right. And it is hard to get it right. That doesn't mean we abdicate our responsibility. And the other thing we have to recognize is that we will never get it right. But if you're in school and if you get a 90%, that's an A. An A is a B. Maybe if we aim

for 90, we'll be doing an A. That's not bad. It's a whole lot better than the Ds and Fs that most of us are getting right now. Well, and I think doing a little better today, this week, this year, than we did last is the way to go. I mean, and then tomorrow we'll get a little better. So let's at least get 80s now and then the next year we'll do 90s and then maybe someday we really can have safe code that doesn't cost a fortune.

I mean, some FDA and FAA products do tend to cost a lot more because there are these processes that imply grown-upness. Yes and no. And, boy, several things spring to mind here. Let me start with the first one, which is you've made a really good point. We'll do a little bit better this year and a little bit better next year. That's absolutely right. With one caveat, which is we're engineers. Measure it. If you can't give me a number, it doesn't mean anything. And so when I hear teams saying, well, we're using this technique because we all like it, I say, well, that's ridiculous. I mean, give me numbers. So if we say we're going to do a little better this year, measure it. Now, what is your metric? Maybe it's released bugs. Maybe it's schedule.

Maybe it's all of the above. But until we start taking numbers, we're not doing engineering. We're basically liberal art majors. Metrics are hard, though. How do you compare apples to oranges? Because the products aren't always the same. They are never the same. And you're not trying to get 100%.

Right now, we have nothing. And if we can measure something, we will start with what i look at as an impressionistic painting right we might not be able to actually see the outlines of that barn in the distance there but we see sort of a blur but starting if we start taking metrics today we'll at least start to see that blur of the barn and as we improve we'll see that in sharper and sharper focus and they'll never be perfect. You know, all the academics say you don't dare measure anything in lines of code, and they've got some really good justifications for it.

But lines of code will get you to 80% correctness for things like bug rates, productivity, and stuff like that. And you have to average it over tens of thousands of lines of code. And that's not bad. That's a good place to start. The academics want to see its function points.

Well, we're not going to do that. You know, it's stuff we don't, it's not the way we think. But we have to start measuring something. And then what was the other thing I was thinking? Oh, well, I asked about the price of being a grown-up. Oh, thank you. That's what happens as you get older, short-term memory, I guess. Yeah, but there are some really cool studies on this that have shown that safety-critical

code doesn't cost anything more. As a matter of fact, they've got one here, which is really cool, using the 6-1508, the European safety-critical standard. They've shown that if you follow discipline processes, and they use the capability maturity model as a process, not the canonical process, but a process, you can get safety-critical code for the same price as the old stuff. Now, that doesn't include all the extra documentation and certification

because those are additional requirements put on the thing. But you can get that great code for no additional cost just by changing the way you do things. And I'll give you some numbers. And you may have heard some of this because I think you sat in on at least one of my talks at the show. The average firmware project spends half the schedule in debugging, which is ridiculous. I mean, it's crazy. That means the other half of the

schedule, I call that bugging. Writing the bugs, yeah. In no other endeavor would anybody tolerate that. I mean, can you imagine if you were throwing printed circuit boards away that fast? You know, oh, sorry, boss, I need four more revs today. I mean, it's crazy. And so by using the right processes, we can usually reduce that.

Give you another example. Oh, here's some good stuff. Everyone knows Ada. Or I shouldn't say everyone knows Ada. Everyone knows of Ada, which, of course, was designed from the outset to lead to really high-quality code. There's another language called Spark, which is actually Ada. Any Ada compiler will compile it. It's a subset of Ada.

And they've done some stuff where basically in the comments, with formal notation, you write the intent of what the code is. And there are these tools, which are all free and open source, that check those annotations against the code and flag if there's a problem. And the thing that's interesting about Spark is that it's creating a problem for people because the bug rates are so low, they can't measure them. And one of the guys I know who's involved with this said, oh, where they actually compared that to using a traditional

language and sloppy processes. And they delivered a system with one, they did find one bug, in something like two-thirds of the time that it would take with a traditional process. And so we often tend to delude ourselves into thinking, well, getting it right is going to cost us time and money and stuff, when really, quite the opposite is true. And that's what we found in the quality movement. Hey, all the car manufacturers, except Toyota. Oh, I shouldn't say that. I have two Toyotas. I love them. But all the car manufacturers found that by building them right and not trying to fix all the dents and scratches at the far end of the production line, they save money.

I used to have a joke at one of my previous companies that we don't have time to do it right, but we have time to do it twice. Yeah. What was it? One of the famous guys, Barry Beam maybe, one of these guys said, build one and throw it away, you will anyway. No, it's not right. No, no, and I don't advocate for that, but it's interesting. It does, yeah, lots of people seem to.

Okay, but how do we do it right? I mean, is it code inspection? Is it writing test-driven development, but maybe not doing the agile part? I think, okay, from the very short summary view, I would start with get the requirements right, or at least as close to right as is possible. Get a design which is correct. And when I was a young engineer doing hardware design,

everything went through a design review. Everything. And even on space missions, there are critical design reviews that should be reviewed. The hardware and the software design should be reviewed. The interfaces should all be specified extremely carefully because that's how everybody talks to each other. The code, every bit of code should go through a code review.

We have so much data on this, it's ridiculous. But code reviews lead to usually better code at a fraction of the cost of doing debugging. I think we should aggressively use the tools, the static analyzers. Oh, man, there's some cool stuff out there now. And they're not cheap. You're talking $10,000, $20,000. But compare that to a cost of an

engineer, a loaded engineer's cost. It's not unattractive. A couple of weeks of debugging. Yeah, exactly. And there are other things like lint. I talk to so many people who won't use lint because it's a pain. It takes a while to learn how to tame it.

It is a nuisance. It'll print out gobs and gobs of false positives until you learn how to deal with it. Until you learn how to write really clean code. Yeah. So these tools are basically saying, grow up. Do it right, grow up. Use those extra parentheses. I'm not quite sure what this means.

Use those extra parentheses. And I think it's awesome. And you said earlier, Alicia, that we can do should be trying individually to be getting better to write cleaner code, to write code with fewer bugs. In terms of the process, we should be measuring conformance to requirements and bugs at every stage. I view each of these stages as filters. And you're going to remove defects at inspections. You're going to remove defects from the tools, finding them. You're going to remove defects when you do start debugging. And we need to know where we stand, what the numbers are.

Because we have the numbers. We know the numbers. What's his name? What? No, no. Capers Jones. I don't know why I'm having so much trouble pulling stuff out of the old memory banks today, but Capers Jones has got gobs of data,

which he's been kind enough to share with the industry, and he's actually shared all the raw data with me, which shows where companies fall into a malpractice based upon how many bugs are found at each of these filter levels and where companies are in the top 2%. And the interesting thing to me is that nobody knows where they stand, or very few people know where they stand or where their organizations stand. And in business, they call this benchmarking. You look at how your financials are compared to other companies in the same industry. They call it benchmarking. So you can see roughly, are you totally out of whack with the industry? Are you a star performer or what? And I think engineering groups need to do the same thing. This data is really not that hard to gather. It means changing how we do things, writing a few more things down. There are some great tools for it.

Oh, other tools. I've got this tool here, which I've been playing with. It's insanely cool. People don't believe me when I tell them what this does. It's Oh, other tools. I've got this tool here, which I've been playing with. It's insanely cool. People don't believe me when I tell them what this does. It's LDRA unit. And what this thing does is it looks at your code and generates all the test code. Now, you have to help it along. It's not totally automatic.

I mean, there's some things it can figure out by itself. There's some things that you have to act as an oracle for. But, whoa, I mean, I play with one thing. I fed it a 10,000-line program. I got 50,000 lines of test code out of it. But if you think about that, suppose you had not, and there are other companies that do this too. If you had written the test code manually, how many lines do you think you would have written?

I bet you it wouldn't be anything. 10,000 lines, maybe. had written the test code manually, how many lines do you think you would have written? I bet you it wouldn't be anything. Maybe. Okay, so I agree with you, but some of the places that I have worked definitely wouldn't. How do I, as a consultant, how do I try to gently herd my clients in the right direction? Or as an employee at a big company, how do I get my manager to agree? I mean, design reviews done by yourself, I've actually done that. It isn't as bad as it sounds, as long as you let a week go by and forget it all exactly but um how do you how do you persuade the machine to let you start doing a better job

i don't think it's a simple answer i i think uh in one case if you can find an advocate in some organizations you have an evangelist who can convince people that this is something we should try. I have seen that work. In other organizations who are very resistant to it, I throw out a challenge saying, fine, okay, don't do it.

But just to pretend that you're an engineering organization, start collecting metrics. And then we sit down and talk about the metrics. Just collect the metrics. Don't change anything. Just collect the metrics. And then compare those to the rest of the industry.

And maybe you'll see that, oh, my God, you're a top performer. In that case, maybe you don't want to change anything. In the cases where I've been involved, that has not been. And to tell you the truth, at that point, it's still not easy to get the message across. But at least then you're starting with numbers and you start to say, okay, these are your numbers. Now, if we change the numbers to this, which is say, oh, maybe the top 10%, you would save this much money and ship this much better code. Are you interested in that or not?

Yeah, once you start being able to hit them in the pocketbook, it does get a lot easier. What's fascinating to me is everybody who's an engineer or is in a technical field is supposed to have a science background. You know, most of us have come through an engineering degree, which is science-based, you know, taking those core classes. And somehow we've lost that science sense of things need to be tested and measured and challenged to see if what you're doing is actually doing what you think it does. And I don't know at what point we lose that sense. Maybe it's when we have to ship a product in a big hurry and then things go out the window,

but it is kind of sad. Yeah, and certainly time to market has always been a problem, and it's worse every year. I keep thinking that if you could build a perfect system with a million lines of code in a month, then the next one, they'll want you to do it in two weeks. We're chasing our tail. It's really very much of a problem. But that doesn't mean we shouldn't be trying to do the right thing. I think it's a matter of ethics, to tell you the truth. I really believe this.

As engineers, we know better than anyone else how things should be done in our particular domain. Of course, we think we know it for all domains. And I think an ethical engineer is going to stand up for doing things the right way. I agree with that. And I have been accused of being unyielding in some ethical things. And to me, that's quite the compliment.

Thank you. So going about it a different way, when I'm looking for new employment, say full-time employment, what questions do you ask to figure out if the team you're joining is full of grown-ups? I mean, Chris, you said that the team is really important, and I agree with that. So how do we stop going to companies where it's all hack it, fake it? You know, Joel Spolsky, the guy who did the Joel on software blog for a long time, he has a test for exactly for this.

And he lists about a dozen or 13 things you should look at for an organization. And I would argue that a lot of what he is looking at are sort of high levels, things like do you use version control and things like that. use version control and things like that.

And those are all necessary conditions, but they're not adequate. I think we have to ask the hard questions like, describe your lifecycle. How do you create software? You know, what happens if the time pressure is on and we want to skip step A, code inspections, whatever that might be? What happens? And I think you have to ask that if you're the prospective employee. Oh, man, I haven't been one of those in 35 years. I think you have to ask the people who are interviewing you, who are largely the boss to be and whatnot. But also, if it's a good company,

they're going to let you talk to their regular engineers so you get a sense of what the team really is like. And you have to ask that of them. Joe, okay, what happens? The ship day is two weeks. We know we have this thing we have to do. Are they going to allow us to do this? Or are they going to say,

stop doing all the good stuff, just ship it, damn it? Just ship it. I've heard that before. I hate that phrase. I'll tell you, I don't want to pretend that any of this is easy. I think, in a way, I hope that this Toyota devil call is a, it won't be a wake-up call, but it's a first step into a wake-up call.

Because if we just talk about that one settlement, $1.2 billion, we know it was around a million lines of code, plus or minus. That's $1,200 per lines of code plus or minus that's a twelve hundred dollars per line of code just for the fine that is comments that's now the most expensive code

ever written and um that should make people think so i'm i've got this other notion i've been ticking around in my head i haven't figured out how to make this happen. But if you look at any public company, look at the annual report, there's a risks section. Well, you know, we may not be able to buy these parts. There might be labor unrest. And all these things could cause our stock to go down. What happens when the CEO has to put in there saying, well, we have decided to use poor software engineering processes so we may get sued out the door or all of our customers may rebel

or the airlines might collapse and millions of people are killed or something like that. When the high, high-level boss, when people start to recognize the impacts of these problems and the high-level boss is forced people start to recognize the impacts of these problems, and the high level boss is forced by investors to disclose these risks, then action is going to be taken.

And we have to be ready to tell them what that action is going to be. I think we can't wait for regulatory agencies to kind of help with that. Because, I mean, going back to the fda for a second when i was working with them uh a decade or so ago they didn't really understand software they were afraid of it but they didn't really have a good sense of what it meant and then the risks involved with it versus hardware and in fact we could take things that were basically pure software and put them in an fpga and suddenly they were happy. Even though it was the same stuff, it's just we moved it to VHDL. And so it's that kind of thing.

It's going to take them a lag of five to ten years to really come up to speed on things and understand them, and maybe longer. And it's really up to us as engineers to do the right thing, not wait for somebody else to tell us to. You know, when I get pushback on this um from engineers i then challenge them and i say okay you tell me your boss won't let you do this i i totally

understand that's a valid concern but in two four or five years you're going to be the boss what are you going to do and this gets back to the ethics discussion i I guess, too. Well, I've done that. I went from being an engineer to a boss to a director. And at each point, it almost gets harder to stand up and say, no, we're doing this right, because the pressure from both sides becomes hideous. And you do have to hold on to those those values and say no this is important this is more important than shipping on time because i'm going to save you money and if we

do this really right we're going to ship on time instead of your plan which is going to take longer and it's going to be buggy. Everyone loves to complain about bosses. Bossing is the hardest job in the world in many ways. And yeah, boy, the pressure from on high. Because the first time I became a boss, started running an engineering group back in the 70s. My boss, we were in a little tiny 100-person company, way undercapitalized, always struggling for money. Paychecks would bounce. It was horrible.

And my boss, the VP, would come in and say, how long is this going to take? And I would tell him. He said, well, if you can't get it done by this date, much shorter date, you have to lay off three people. I mean, that's a lot of pressure. That's a lot of pressure. And I don't think there's any secret sauce or magic fairy dust. I mean, we have to do the best we can do to withstand that pressure and to protect our team, to keep our team doing the right things.

And it's not easy. No, it's not easy, but it's better. It really is better. Once you get into the habit, it's not easy but it's better it really is better once you get into the habit it's not harder either right once you you're you've started but that that hurdle of oh i don't want to count how many lines of code i have i don't even want to think about the bugs i have it's hard to get over that some of the more successful teams i've seen a man i hate to say this but what they do is they only hire new college grads who haven't been right polluted right and the funny thing about this is you know

these college new college grads work for them for a number of years and then inevitably move on and so many of them report being appalled when they go to another organization that the other organization is such chaos. When I left HP and went to a little startup and they didn't have version control, I spent a whole week wondering what in the world had I gotten myself into? How could they not have version control,

let alone everything else they needed to have? So you were at HP. Which HP? I did a stint at NetServer Division, and then I went to HP Labs, and then we got spun off into Agilent. Oh, through that transition. Yeah. Agilent's a great company. I have gobs of test equipment from them. It's great stuff. Probably not any of mine, since DNA scanners aren't really electrical engineering things.

Yeah, I agree. It's interesting to see how the different organizations, how different they are. So one of the things that I've heard you rant upon is how much you dislike the word programmer. And I agree because engineering is just better because it indicates all of these other things, this careful planning and thoroughness. But what else do you hate about the word programmer?

And then let's go into hacker because I'm sure that that's going to be fun too. Yeah. And you have to also understand the fact that i'm an old guy and you get old guy habits um i think i hate the word programmer because any idiot can write code and it's really true any idiot can write code hey basic is 50 years old this year right yeah and uh anyone can learn to write a few lines of basic. That's programming.

That means, by definition, that you're a programmer. Whereas what we do, I believe, is software engineering, which is, yeah, programming is part of it, a hugely important part. So is typing. Good point. Right, exactly. I hadn't thought about it that way.

That's excellent. And so I prefer software engineer, and I want us to live up to that. I think engineering is a noble profession, and I think if we're going to call ourselves software engineers, then let's live up to the nobility of that moniker. So there's some identity there with it being a profession that's, I mean, it's been around hundreds, probably thousands of years. Software engineering? Engineering. Well, you know, like Chris said,

it's the great teams that do great software, and this profession is also sort of a team. We are engineers because we worked hard to become engineers. And then we can, well, we can't call ourselves engineer anymore in so many states because of the regulations. But we still choose to do so, thankfully. And that's sort of segregating us from the rest of the population.

Okay, we are in this discipline and we practice this discipline in a certain way and i think that's awesome and programming is like you said typing is perfect i really like that if you want to go into hacker and maker uh well first of all hacker is a word i detest and i know that the definition is changing and you're a Hackaday judge where they're all... Yes. And yes, it kills me, the Hackaday website, the hacker hardware, like I hate that word. Hack to me has this connotation of just throw something

against the wall and see what sticks. That's everything I don't want to do. Right. However, and I'll sort of segue towards Maker because I'm not sure what the difference is. When we talk about things like Arduino, Arduino is way cool. And originally it came out as a way for artists to build little systems that do some like smart art, which is really cool. And then it sort of got confused and conflated with this maker or hacker thing where every time some 16-year-old makes an LED blink, he's on the news, it's going to be the next Albert Einstein or something.

And I think the maker movement is, at one level, a fabulous thing, getting people into this cool feel. I sort of got into it, at least in part, through the ham radio. And in ham radio, you could get in as a novice, knowing very, very little. But you could get in. But with ham radio, there is this progression. You then study for the next test, and you get more benefits, and you're constantly improving your skills. And I don't see that in the maker-hacker movement.

They're just more blinking lights, different blinking lights and motors. Playing. Playing is great. If that's what they want to do, play. That's fantastic. But I think that's miles away from the stuff that we do. I agree. I'm astonished at how complex some of the devices are.

And I love that they share the enthusiasm with so many people. And it makes engineering cool. But I wish more people understood that there is a difference. Yeah, well, there's a difference between this comes down to also the difference between a prototype and a product. I mean, there's some separation

there between, yeah, I made this thing that does this once. Right. Just this once, exactly. And the hackers slash makers who get into this uh just to do some playing i admire them the ones that get into it and use it to progress along a path of of further knowledge i admire them the uh the confusion exists with the ones that are that are sort of

like you say creating products that are really just prototypes. And I mean, a prototype is 10% of a product, basically. Yeah. Although I sometimes feel a lot of pressure towards being a maker artist sort. Having my code up on GitHub and expectation to do makery things in my free time all the time. And to be technical all the time except not like professionally technical just kind of poke at it technical and i need downtime too well and you need a reason to do that other than just being part of a movement i mean yeah like i said i'm going sailing here and exactly i'm not going to judge people who

are compelled to do this 24-7 I mean that's great if that's what makes them happy but I do think that it is a really big fascinating world out there and I for one wouldn't want to be just throwing slinging bits around

Fair enough and Linux in embedded systems. I keep hearing more and more about that. And part of my reaction is, well, once you put Linux on it, it's not an embedded system anymore. But I gather I'm not in the majority with that view. Not even in the room.

One thing I've done is to give a talk at a Linux convention and start off by saying Linux sucks. I'll agree with that. Well, and my point is not that Linux sucks. It's that Linux is a pile of bits. And it's totally a perfect solution for a lot of problems. And it's a lousy solution for a lot of other problems. And it's just important to make intelligent decisions rather than crowdsourcing knowledge

and seeing what everyone else is saying. I get literally every day email from people saying things like, well, when should I use an RTOS and when should I use Linux? And that's sort of like saying, is this a lake or is this the Atlantic Ocean? I have some data on why people use Linux and embedded systems. And the top three reasons are,

so I can further my professional career, which is not a very good engineering reason, because I've always wanted to learn about it. It's the same as the first. Yes. And the third one is because of the memory requirements, because I've always wanted to learn about it. It's the same as the first. Yes. And the third one is because of the memory requirements, which tells you that they're totally clueless about what's going on.

Wait a minute. Is it because they secretly work for a memory company? Maybe that's it. Wow. Hey, I've seen some great Linux-based embedded systems, and I would call them embedded systems like your cable box, your set-top box. That is almost certainly running Linux. I mean, most of them do today.

And it only does one thing. It's just a cable box. I would call that an embedded system. Well, there are a lot of routers that do it too. And routers. Those are definitely embedded. Yeah. And those are probably

pretty good applications for it uh one of the one of the more one of the coolest technologies to come out in a while i think of the arm cortex m series parts i mean they're cheap some of them are half a buck and uh they'll never run linux they will never run linux and they're they are truly deeply embedded systems and they and they're pretty cool. They don't have the MMU. So they can run the micro-C Linux, but that's a small cut down. People forced Linux onto them with great pain. Yes, and then that's all they do.

Great pain, right. I mean, if people are putting external memory on these MCUs, they might as well go to a Cortex-A9 or something anyway. So how do you evaluate if something is a fad? I kind of feel like Linux and embedded systems right now is sort of a fad, and we'll figure it out, and there'll be things that Linux is good for and things that it isn't.

But it's such a everybody-wants-to-do-it-right-now sort of thing. Well, there's a fellow named Steve Leipson who's a good friend and a sort of well-known analyst and he postulated what's called leapson's law which is that any new technology really takes about 10 years before we really understand it to apply it well and i think there's a lot of wisdom there and i think linux is sort of over the fad hump in the embedded world. I think that, although a lot of people are still confused by it, it's being applied

mostly in more cases than not, I think, in reasonable fashions. People have gotten over the thought that they could put it on an 8051. Literally, I've heard that question. Like you said, routers and stuff like that, are pretty good applications i think there's still a lot of confusion

people are saying well i'm going to use linux because of security which is nonsense because we know the numbers linux is what's called evaluation assurance level four plus which is exactly the same level as windows and what that means is that it's perfectly secure if nobody tries to break in. So, I mean, people are a little confused about some of those things. What I have a little trouble with is that there are a lot of

zealots, whether they're Linux zealots or zealots for whatever it may be, who tend to propagate an idea just because they like the idea instead of holding it to the cold truth of engineering analysis. Right, yeah, and it goes back to requirements, you know. Does Linux meet your requirements better than anything else? Okay, that's what you should use.

But you shouldn't just pick it first and then write your requirements around it or try to mold it to fit. Absolutely. And embedded Linux is very much free as in puppies. Yeah, that's a good way of putting it. Have you heard that? Free as in beer and then free as in freedom and then free as in puppies. I hadn't heard the puppies one.

Yeah, you know, you get a free puppy and it was not what you should have done. It was so much more work than if you'd bought a puppy. Right. Or even better, bought the goldfish. A free puppy is a pretty lousy gift. Exactly. So what else have you been up to? Well, I do a lot of writing.

I do a lot of seminars about embedded systems. And I have my own research projects, things I'm interested in. I've been working on, well, one of the fallacies you'll see advertised everywhere today is a lot of these MCU vendors are saying that their processors can run off a little coin cell for decades. And that struck me as sort of silly. And so I actually built some test systems. I've spent a year, I've collected millions of data points and have characterized how these batteries work in those sorts of ultra low power up situations. And I've published a number of articles about it. I've got a system in back of me right now, which is taking new data, will be for the next couple of months. And it's interesting. It again goes against the grain. I guess what I'm trying to do is tilt at windmills

where people say things like, sure, this battery will last for a real long time because we do this facile bit of math on it instead of really doing the engineering analysis. And so that's what I'm trying to do with the batteries. I've done that with debouncing contacts. Everyone says, one of the questions I love to ask, how long do you wait for a contact to bounce? And people will give me numbers, 10 milliseconds or whatever.

And I say, well, where did that come from? And the answer is either, well, that's what everyone does or Joe told me or something. Empirically determined because I tried it three times on my desk with two switches. Right. So I took 50 pairs of switches and developed all kinds of data and I published it on my website.

It's very, it's interesting, and I'm enjoying shining a spotlight on some of these corners of the field where we really don't have the data that we need. There's nothing worse than conventional wisdom. Yeah. Yeah, your website has a lot of information. I went and booked Mark a bunch of the articles that I hadn't seen before, and I'm looking forward to going back to reading those. But there were, yeah, you had a ton of stuff on there, and it's all free. But you also have a book, more than one.

I have one on my shelf that I know is yours, The Art of Designing Embedded Systems. Yes, that's my second oldest. I've done six books. You mentioned Mike Barr. Mike Barr and I did the Embedded Systems Dictionary. Oh, I think in 2003 that came out. And I'm thinking that if another hundred years go by,

we may earn out our advance on that one. That's been a real flop. But yeah, I publish a lot of stuff on the website. And actually, we'll be launching a whole new website next week. And I'm going to start doing weekly videos on that site. Because although I'm sort of resistant to it, I personally don't care for watching videos on my computer. I spend enough time on my computer. But I've gotten a whole lot of requests for it,

and I know that's how people communicate and learn today. So it's cool. I've got a few in the can. It's been a lot of fun to put them together. Why do you do this? I mean, you probably could make more money, if that's the goal, if you did more seminars or did more code. Why all the writing?

Why all of the information dissemination and the videos too? I had a tools company. We made debuggers, in-circuit emulators, which I sold in 1997. And I retired for a day. I was bored to tears. And I thought about this. I thought, you know, I'd like to do something that I felt would be worthwhile for something. And all I know is engineering. And so I thought, well, I had been doing all the writing, which I really enjoyed growing up as a Catholic school. They beat that into us.

And I thought, well, I'll just continue to do this. I mean, as you can tell, I think I'm very passionate about the subject. I want to help engineers improve their ability to build these products because they are getting so much more complicated. And it is a cool feel. I'd be miserable if I was divorced from it. I agree with that. I mean, I would be too. So do you have suggestions for college?

I mean, college students mostly teach coding. And if you do get to go someplace that has a good process and helps you grow as an engineer. But what about the people who are just getting out of college and aren't necessarily in those places that teach them right? How can people grow up? How can they learn as they're just getting into the field? Or is this something that takes a while? You know, I do think it takes a while because in sales, they say you look for the pain. And then you try to sell them something to fix the pain. And I think until you feel the pain, the pain of missed deliveries and bugs and all this kind of stuff, you don't really understand that there is a problem or what the problem is.

So the number one bit of advice that I always give to, well, two things I give to young engineers. First is save money every week. Use the power of compound interest. You'll be 40 before you know it. But the second bit of advice is constantly study this field. There's a lot of stuff going on in this field, cool stuff. Some of it's wacky stuff, but you learn from all of it. And it is interesting.

My data suggests that the average firmware person reads one technical book a year. I don't see how you can really stay connected with what these processes are, what the ideas are, what the exciting concepts are, if that's your level of engagement outside of the office. Well, we read a lot of data sheets, too. Well, a lot of websites now. Yeah. A lot of websites. Stack Overflow, Stack Exchange.

But you know, those are great, but the level of misinformation is incredible too. There is something really nice about a book that someone took the time to think out an organization for and to say this is a good way of presenting the information and a good way of learning it. So yeah, I like books. I look in academic articles and they can be very frustrating because sample sets are small and things, but some of them have some fabulous ideas on them. And we have to read widely to sift the wheat from the chaff. The web is awesome. I mean, I'm on the web constantly. But an awful lot of the stuff is vendor biased and all of that. You have to look at the bylines. Who's writing this stuff?

Why are they writing it? Do they have some hidden agenda? And evaluate the information you're getting that way. Yeah, exactly. I'm thinking about the things I've been writing and what agendas I have. And I try to show them. I try to show my cards with those because that's not my primary goal in life. I like to share the information.

But yeah, sometimes people have asked me to have agendas and to not show them. No. Sometimes people have asked me to have agendas and to not show them. And I know that's. No. All right for you if you pay me, but I'm not going to tell anybody you're not paying me. Exactly. Well, I think we are about out of time. Do you have any else, anything else we should talk about? I can't think of anything.

I mean, I think it's great that you're doing this stuff and advancing and disseminating the knowledge base. That's fantastic. Well, thank you. I like you. I enjoy this very much. Well, you know, I've been doing this professionally for 40 years, and when I talk to young people considering a career subject to their abilities, I often recommend engineering. I think it's a fantastic career, and it has never been boring for me. And today, engineering, even in terms of pay, is pretty good compared to an awful lot of

other career choices. An engineering degree will get you into a lot of those other careers too. Oh, yeah. All right. My guest has been Jack Gansel, author, engineer, speaker, and all-around embedded systems hero. Thank you for taking the time to speak with us. Well, thank you, folks. I really enjoyed it. Thanks, Jack. And thank you to Christopher for holding down the other mic and for producing the show. And thank you for listening.

Hit the contact link at embedded.fm or email us at show at embedded.fm. And Jack also has a contact on his webpage. I believe it is jack at cancel.com that's correct and let's see a final thought for this week from ted nelson i don't know who that is but oh well he had good things to say i believe that good design is magical and not to be lightly tinkered with the difference between a great design and a lousy one is in the meshing of a thousand details that either fit or don't. Embedded FM is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California.

If there are advertisements in the show, we did not put them there and do not receive any revenue from them. At this time, our sole sponsor remains Logical Elegance.