Embedded - 183: Robots Having Nervous Breakdowns

Episode Date: January 11, 2017

Philip Koopman (@BetterEmbSW) spoke with us about making better embedded software. His Better Embedded Systems Software blog has lots of great information including links to his growing video library. Two posts noted in the show: Recording Peer Reviews Spreadsheet Code Review Checklist His company, Edge Case Research, performs design and code reviews and teaches how to do them. You can find out more about his course and background on his Carnegie Mellon University staff page. That also leads to the pretty amazing Vintage Aero paper airplanes. Phil’s book is Better Embedded Software, available via koopman.us and (more expensively) Amazon. Fagan Inspection Videos of robots being stressed   Also: Embedded.fm Jan 28th Party RSVPs on Eventbrite

Hello and welcome to Embedded. I'm Elysia White, joined by Christopher White. I'm pleased to have Professor Philip Kopman on the show this week to talk about making embedded system software better. Before we talk to him, I need to remind you that we're having a party. January 28th, the last Saturday in January. This is 2017. And it will be in Aptos, California, 2 to 5 p.m.

Bring a hat or a hack or both. More details and RSVP instructions in the show notes. Hi, Philip. Thanks for being on the show today. Hi, Ellen, Chris. I'm really delighted to be here. Could you tell us about yourself? I've been designing embedded systems for about 35 years, but I did not take the standard

professor path. I spent time as a U.S. Navy submarine officer in the Cold War. I worked for Harris Semiconductor as a CPU designer. I worked at United Technologies doing all the things because they own Otis and Carrier and Pratt & Whitney and folks like that. Then I switched career paths to be a professor at Carnegie Mellon, and I've been there about 20 years now

teaching embedded system hardware and software. I've done a few startup companies, and really my big push now is trying to get embedded system knowledge out to folks in industry. So this is where we throw out the outline and just talk about submarines for two hours. Well, if you've seen Hunt for Red October, that's more or less what I was doing and when I was doing it.

That's terrifying. So were you Jack Ryan or somebody else? Every night between midnight and 6 a.m., I was in charge of the submarine on the periscope while the captain was asleep. That was my job. Well, something you don't really want to mess up, I suppose. No, you don't really want to mess up. See, I wanted to ask more about being a professor

after so long in industry. It seems like it's a very, I mean, that's two paths that don't usually cross. Sometimes the professors come over to industry, but usually it's very separate. How did you look around and say, you know what I want to do? I want to go back to school for a really long time, and then I want to teach. Yeah, well, it was certainly an interesting path. So I did my time in the Navy, and I decided that I really liked research. I really enjoyed it.

I did a startup company while I was in the Navy. I built a CPU in my spare bedroom. I literally did this out of TTL and wire wrap and stuff. And I loved it so much, but I didn't know how to write a paper. So I said, all right, I'm going to go back. I already had a master's at that point. I'm going to go back, get a PhD, and learn how to actually do research. And so that's why I went back to get a PhD.

And after that, I wanted to do real stuff. So I went to Harris, and I ended up building a chip based on that CPU design. And also, I helped them with their other chips. And that was a blast. But right around then, the submicron geometries hit, and there used to be like 10 or 20 companies with a CPU, and it went down to three or four or some numbers like that. And Harris was one of the casualties.

Our submicron fab just wasn't going to do it. So they got out of the CPU business more or less, and I went to United Technologies. And I loved being there. I was at the Central Research Center. So I got to play with the guys at Otis and Carrier and Pratt & Whitney. And I did some automotive work that was deployed in cars for 10 years. It was a lot of fun. But then the number of research publications

sort of went to zero and the patents went up and it became more product development than 50-50 research. So I said, okay, I'll go to Carnegie Mellon. I was fortunate enough that they were able to take me mid-career. And it's tough because the universities are good at, okay, you have a fresh PhD, we know what to do with you.

They're good at, you've been in industry forever, you're in the National Academy of Engineers, we'd love to have you as a professor. But mid-career, it's really tough to make the switch. Not many people do it, and there's a reason. But I was fortunate enough, I had a lot of support, and I was able to make the switch and get on the tenure track at Carnegie Mellon. Cool. Alright, before we ask you questions about submarines and paper airplanes, we do this thing called lightning. You have been looking at my website. I see that. Those are my dad's airplanes. I just put them up there because I think they're cool. round where we ask you short questions and we want short answers and if we are behaving ourselves we don't ask you why and how and all of the additional questions that come up which we

usually do anyway go uh what is your favorite embedded compiler i love code warrior which tells you how old i am that's an editor but that's okay no no it's an idea it's an idea it was a whole idea i still use it in teaching well probably not anymore but i've been teaching hc12 for about 10 years and it's it's great stuff favorite processor living or dead rtx 4000 why because i designed it but they actually made a chip and it worked for silicon worked. It was great. Favorite fictional computer. I'm going to go with Rosie from the Jetsons. Because when I was a kid, that was really cool. And I was thinking about this earlier today. I have a dishwasher. I have a microwave oven.

I have two Roombas. That's about 80% of Rosie, right? So things have come along. What is one question that you wish students would ask more? I wish that they would, it's not quite a question as a mindset, but I wish they would think about when they get out in the real world, what are they going to use this stuff for? I mean, it's nice to have the theory, that's essential. You have to have that. But it's also important to think about how this stuff translates into the real world. And it's tough because students mostly don't have that much experience. So, they don't have a lot of pegs to hang things on. Wow. I don't know very many CS profs that would say that. It seemed like so much of what I learned in college was more, you should learn this because it's beautiful or interesting or optimal and not because it's useful.

I think useful is probably a bad word. There's a lot of room for the elegance. And absolutely, there's plenty of that. But I start every one of my undergrad lectures with a story about a product that I've done a review on. And most of the times I'm able to say, and we're going to learn something today that these guys got wrong, and here's how to get it right. What language should be taught in the first university course that teaches programming? I'm going to go with that's the wrong question, but let me explain why.

I don't think we should be teaching university students programming. I think we should be teaching them software engineering. Not hardcore heavy-duty software engineering, but basic software engineering literacy. And the programming language kind of doesn't matter. I think I kind of agree with that. And we've asked this question of a number of people and gotten various answers and sometimes hinting toward that. But it is kind of a strange question because it's very specific. And yet, like you say, it probably doesn't matter. Well, I think when I was asked that question, was it last week or the week before, I said you shouldn't learn in university.

You should learn much younger than that. Yeah, well, it's sort of different. I totally agree. Okay, then a different question. I was kind of going with they probably already know a language anyway. Right. Yeah, there's that too. Yeah.

What is your favorite language to work in? Oh, I had such a blast working in Forth. Yeah, that one is fun. Because my processor was a stack processor. But that's still around. You can still find it. But I do most of my work now in C++ and really the C-ish subsets of C++

because that's where the mainstream of embedded is. And so I want to keep current in what's going on there. Is a mobile phone an embedded system? And so I want to keep current in what's going on there. Is a mobile phone an embedded system? Which CPU inside the mobile phone are we talking about? I guess that's an answer of sorts. I'm just going to accept that. Yeah, okay.

It was intended to be, yes. What is the worst formal version control system you've ever used? Okay, I've got a great story on this one. Well, I have two stories. I was doing a design review and there was this old gray metal filing cabinet and there was a label on top that said software. So I took a photo of that and I show that in my class when we start talking about version control. But that's not the worst one I used. The worst one I used was there was a product I worked on, and I'm not going to name the company. There's a product I worked on,

and the way you committed code to production was you printed it out on 8.5 by 11, and each piece of page got an engineering document number. That reminds me of the time when I was working at a company doing FDA stuff, and the regulatory person wanted me to print out the software source code, which included VxWorks,

all of our code, so they could put it in the file for the FDA. And I explained to them it would be six feet high. And they said, bring it on, right? We eventually talked them down to a CD. See, these things make Dropbox or copying the folder look good. But yeah, all right. Christopher, do you have one more question? Sure. What science fiction or technology concept do you think will be real in our lifetime that is not already, I guess, stipulation?

Oh, you know, I didn't expect the not already because I'm a nonconformist. My answer is, I'm in Europe and I'm video chatting with my wife on a phone I can carry in my pocket. When I was a kid, that was science fiction. Yeah. It's amazing how much stuff we take for granted that's just totally magic. If you want the looking forward answer, self-driving cars will eventually be here for real. That's going to take longer than anyone really wants it to. And flying cars, that's the technology of the future. It will be for a while. Why bother with flying cars? Well, there's that question of the future. It will be for a while. Why bother with flying cars? Well, there's that question. I don't want to digress into that.

All right. So let's digress into paper airplanes. You have on your site a number of very detailed, cut out and make beautiful, not fold and fly paper airplane. Tell us about your paper airplanes. Well, that's all work done by my dad. And actually, every one of them does fly, as it turns out, even though they're 3D and rounded and all that. I've built several of them and did it with my boys, and they fly great.

My dad was really into that. My dad ran a model airplane company when I was a kid called Vintage Aero. And he specialized in recreating old balsa wood and paper models and things like that. But as part of that, he also did some paper airplanes. Back in the World War II era, there were some paper airplane designs by a gentleman named Rigby. And what my dad did was got hold of them and re-engineered them to be more modern and put them on computer files and things like that. So as a tribute to my dad, there's a website where you can download these models and print them out in color and on pretty much any paper, cut them out, fold them, glue them, and every one of them flies 10, 15 feet. Usually, if you're indoors, the wall is is the problem not how far it flies that sounds like fun it also sounds like i want a laser printer so i don't have to cut them out laser cutter not printer all right the cutting is the cutting isn't that bad there's a lot of

fine points so you really need an exacto knife to to do it well but uh it doesn't take that long they're a lot of fun well they were beautiful so I wanted to make sure you pointed them out. But let's get back on to topics that I invited you to talk about, which is you have a blog. It's called Better Embedded Software,

which seems like something we should all want. Could you tell us about your blog? Sure. There's sort of a story that goes around it and if you don't mind i'll tell the slightly long version but i had done maybe 70 80 90 design reviews in industry get on an airplane take a look at somebody's code or design or whatever see if there's something they really need to fix tell them what to fix And I had taken a sabbatical to write a book on embedded systems, and I tried to write a standard academic book, and it did not work out for many reasons. It just

wasn't the right thing to do. But then I got to thinking, I said, you know, I have all this experience. I've seen all these things. Let's go that way instead. So I just went through all my archives and figured out what people got wrong, made up a tally. And each chapter in the book is something that a real team got wrong on a real design someplace, although I don doing with the blog is I would do a design review, and it would be something that a lot of times is in the book, but sometimes something kind of different or much more specific. And so I'd write a blog post, and the blog post would be, let's suppose you're a team that got this wrong. Here's how you'd get it right. And then I would send a pointer to the blog post of the team who got it wrong. And that's sort of how I've been building pieces after the book came out.

So for a long time, the blog was mostly just sort of recording additional things I learned. Could you give us some examples of topics you cover there? Oh, it is all over the map. I'm going to have to pull it up to check. I remember you posting a question about if you have an error variable. So you have functions that return errors and you have error equal did something. How do you then handle the errors in a graceful way that puts you at the end of the function for returns

and yet doesn't mean you have to have if error everywhere. Oh yeah, I remember that post. There's a lot of controversy about whether early returns are good or not. So sometimes that was a very specific question. There was a group that had an issue with what they should do about that. And so I did some research and said, okay, there are a bunch of ways. Here's the one I like. And maybe there's not even a right answer.

It's a trade-off. I have a lot of posts on peer reviews, on some on embedded networking, some on, there's a couple pretty popular ones on how to not corrupt your eSquare because there was a company that was having product problems because of corrupted eSquare.

So I did some digging into that. Real-time requirements, it's all over the map. Basically, all these different things that eventually you're probably going to have to get right in your embedded system. What do you mean by eSquare? EEPROM? eSquare PROM, yeah. Okay. Sorry. I just wanted to make sure we clarified that before I got lost again. This is very tactical from a CS professor, and I think you've sort of explained that coming from industry

means you do have a different perspective. But do you also need to do original research into, I don't know, interdimensional wormhole recursion sorts of things? Well, at the risk of splitting hairs, I'm not actually a computer science professor principally. I'm a computer engineer. And at Carnegie Mellon, there are actually different deans. There's a dean of engineering and a dean of computer science. So my home appointment and my courses are all taught in the electrical and computer engineering departments. We're engineers.

Now, I actually do have a courtesy appointment in the software engineering department and I have a courtesy appointment in the robotics institute. So I have all the business cards. That's not a problem. But from my point of view, I'm looking at it and saying, you know, as an engineer, theory is fantastic. If you can get the theory, that's great. If you know the math, that's great. But if the problem has to get solved and you don't know the theory, you don't know the

math because they don't exist, that doesn't mean you walk away from the problem. Then you have to get empirical, right? And so a lot of the stuff I do is, here's how to solve it. And one of the issues I see in embedded is, there's not a lot of theory for a lot of embedded. I spent, I don't know, quite a bit of time on cyclic redundancy codes, CRCs, and digging into what's a good one. And it turned out that no one really knew.

I was astonished. This is ancient technology. And the advice that was out there was all ad hoc, and a lot of it was wrong. And then you can ask, okay, what about watchdog timers? Well, just try and find a theoretical description of how to use a watchdog timer, and you're not going to find one. So a lot of embedded systems isn't really about the theory. It's about accumulating good practices and just knowing how the pieces fit together.

So embedded systems is, to use your word, it's kind of a tactical profession. It's about breadth. It's about understanding best practices. There are some parts like real-time scheduling where the math is great, but a lot of it is you just have to know what the best thing breadth. It's about understanding best practices. There are some parts like real-time scheduling where the math is great, but a lot of it is you just have to know what the best thing to do is. And that isn't interdimensional wormholes. That's, hey, here's how you do good engineering. Here are what the steps are. So I can see a theoretical underpinning for things like CRCs where you could, I don't know, mathematically analyze their coverage or how often they, you know, identical things map to each other and that kind of thing.

Do you think there's a basis for those, for theoretical underpinnings of other things like watchdogs? I'm trying to formulate in my mind what that would even look like. Well, for CRCs, it turns out that brute force evaluation is what you need to do. The math doesn't actually tell you the good ones. There are a few special cases, but in the general case, the math doesn't tell you. And that's why the mathematician struggled with it so long. For the other things, for watchdogs, sure, there is some math you can do analysis.

But let's talk about peer reviews. Where's the math on peer reviews? It's about humans and code. And sure, you can get some tool support to help you. But they're just all these topics, all these things you sort of have to know to be able to be an embedded software expert. And that corresponds. My book was, the chapters in my books were basically, I guess you could call it grounded research, where you go out, you see what's going on, and you sort of build something on top of that. And so the approach I'm taking is trying to observe what I see wrong and respond to that, not because that still might be something there, but apparently everyone figures out what's wrong for themselves.

So there's no point in me spending time on it. I should worry about the things that people are getting wrong. You mentioned your book. You have a couple of times. We should say what the book is. Sure. So it's Better Embedded System Software. And there'll be a link in the show notes notes but it's a little cheaper on your website

yeah if you buy it through coatman.us you don't have to pay the amazon tax basically so you can buy it from via paypal it's a hardcover book yeah and my canadian friend suggests that it is very expensive in Canada. But, you know, why do they need good software? Well, the website used to ship overseas and the postal service basically doubled the cost of shipping overseas. It's just gotten insane. So it's actually cheaper now on Amazon because they have more efficient shipping infrastructure than it would be to ship it to Canada. That's unfortunate, but that's the way it goes. You might want to recheck that, but shipping costs is not what we want to talk about. It is intended as a textbook, right? Or a professional guide? It mostly was developed as a professional guide. Strictly speaking, it's not a textbook because there aren't homework questions at the end of every chapter.

But I've used it as a text in my class for a number of years, and a few other universities have as well. So it's designed to be the kind of thing you put on your shelf, and five years later when you need an answer to a question, there it is, and there's a chapter there that answers your question. And that's how most of the people who buy it use it from what I've heard. I was very pleased to discover that a lot of people with my book were actually using it the way I intended, which is to sit down and read it at the beach or to sit down somewhere on vacation and read it and not use it as a reference. I mean, you know, there are good reference books, and then there are books, they're all different purposes, and I like that there are different ones

and that people are obeying our wishes. Well, I'm certainly not going to suggest mine is a B-tree. Yours may be too heavy. Yours is more like that, yeah. So, you've mentioned reviews a few times and i know that it is something that is very near and dear to your heart when you say design review what does that mean well i have a whole rant that i can do for half an hour but i'm not going to go there let's keep it 25 minutes that's one of the videos you can do for half an hour, but I'm not going to go there. Let's keep it at 25 minutes. That's one of the videos you can get for free on my blog is the full-length peer review rant.

So I'll put a plug for that because it's free. But this is not you take a look at your code. It's not your buddy takes a look at your code. It's not you email it and somebody spends however much time they spend and says, looks okay to me. I'm actually talking about, traditionally they're called Fagan style inspections. You sit down, two, three, four people in a room, three is usually a good number, and you walk through the code a few lines at a time with a checklist in your hand looking for problems. And when people hear that, they say, oh, that sounds like a lot of work.

Surely the 80-20 rule applies that I can do 20% of the work by informal review and find 80% of the bugs. And data since the 1980s shows that that's not true. And I've had several experiences of my own where I've run the experiment in my classroom or in industry, and the 80-20 rule works backwards for peer reviews. The last 80% of the benefit is with the last 20% of the formality. You really have to be formal to get value out of these things. The checklist strikes me as something that could be automated, or a portion of it. I mean, it sounds like Lint to me, to a certain degree.

So there must be more human stuff in the checklist than Lint can do. Well, it depends. So some teams aren't using static analysis at all. And for those teams, having Lint-like rules in the checklist is not a bad thing just to get them warmed up. But if the first item,

if the entry requirement is you're not allowed to peer review until you went through, pick your favorite static analysis tool with a lot of warnings, and that compiles clean. Then you go into review, fine, get rid of all those in the checklist because you automated that, good for you.

But there are going to be things like, do you count cyclomatic complexity as static analysis? Some people are going to fix that before the review, some people aren't. Things like variable naming, does the variable make sense? Is there a logic error? Pull out the flowchart or state chart, does the code actually match it?

Did you forget exceptions? Things like that. At some point, your automation runs out of steam and you need a person to get in there and find out if the code's good or not. I'm not sure we should be using humans as lint checkers because the cost-benefit analysis there is painful given that the computer is really good at doing lint and humans aren't, but the rest of it makes a lot of sense. But this is a little on the hard side. We have two or three people plus the author and you have, I think you had a guideline. Was it 150 lines per meeting?

150 lines per hour. The author doesn't have to be there. The author actually, there's no reason to have them there unless they want to watch and learn. Yeah, it sounds expensive, but I've done the math, and the math is when I go to industry, now this is high-quality embedded systems where if this software doesn't work, maybe nobody dies, but the company has to recall a million things and goes out of business. And this is high-stakes stuff. And this stuff is all over the place, more often than you might think. Oh, we think it's all over the place. Yeah, it's all over the place. Yeah, it's all over the place. And so those kind

of systems, you know, it's worth spending some time to get it right. But I've also found, talking to industry, they're averaging maybe two lines of code per hour. Now that's everything. That's all in. That's the testing, the requirements, all the way through the process. Two lines of code per hour is actually pretty good. Sometimes I see three. When I see 10, usually the code's horrible. So at two lines of code per hour, if you can peer review 150 lines per hour, even if there's three people doing it

and you're reviewing other things other than the code, you're coming in at 5% or 10% of the total product cost. And in classrooms, which backs up the industry data, once I get the students to actually do peer reviews the right way, they're finding two-thirds, half, three-quarters of their bugs in peer review. So if you can find two-thirds of your bugs for 10% of your project cost, that sounds like a good deal to me. It does.

It really does. And I'm glad you mentioned that this is for the more safety critical things because when you talk about consumer products consumer products i've worked on children's toys and they actually had the best quality assurance system that i've ever seen because one mistake may cost millions of dollars on On the other hand, they weren't doing these sorts of reviews because, well, one mistake may cost a million dollars, but no kid will be hurt. Well, but the question is if you have a recall. So I don't know about the toy market.

I could see that going either way. But let's look at a thermostat. If you have a thermostat that locks up and freezes pipes in a house and you have to recall them all, thermostats aren't extremely life-critical. It's not like a self-driving car, but that's a lot of money. So, one of the things that's a tension with this is the desire to have everything connected now. And so the notion of a recall becomes, oh, we'll just push a patch, which I think encourages companies to be

not necessarily explicitly less careful, but it is certainly kind of a subconscious, well, we don't need to, we can go faster because we'll just ship an update if there's a problem. And so I think that's intention with this kind of rigor. And I don't know where we should fall on that. Certainly for things that are less safety critical, maybe that's okay.

So, you know, we don't have a recall and we can go faster. But yeah, there's all these axes of competition and the other guys aren't being as careful. And so it's all very difficult. But certainly for safety critical, yeah, there shouldn't be any other way. Well, I think it matters beyond safety critical because there's also mission critical

where the company can die. Right, right. Right, you know, that's an issue. And a lot of this is tricky because what happens is a company will cut corners and get away with it and get away with it until they don't. But then the company's dead.

That's a fatal event for the company. And so you have all these companies taking these chances, and they are not necessarily weighing their risk appropriately. They're sort of taking the chance without knowing that they're taking the chance. The tough part for me is if you're in an industry where there's three or four players and one of them takes the risk and wins while you're being careful. You see where I'm headed with that? I'm not saying it's good to not be careful,

but it's very hard. It's very hard to make that decision and decide, no, we're going to do it the right way. I see where you're heading, but maybe there's a false premise here. Okay. So I've run this experiment any number of times in my class, my grad class, where we do a boot camp all semester long.

And what I found and what the data from many other projects supports is just on peer reviews. If you're doing good peer reviews, the total effort comes out to be exactly the same or cheaper. So all the students who did peer reviews reported the same number of total hours to do the same project, except there were two differences. The first difference was without peer reviews, their quality was horrible

and the project barely worked or sometimes didn't work. With peer reviews, every one of them was excellent. And the other thing was without peer reviews, they were pulling all-nighters the other thing was, without peer reviews, they were pulling all-nighters by the deadline. And with peer reviews, the last couple of weeks, they were just letting their integration tests cook along

and weren't spending any time at all. So really, this is a case where the data shows, I've had several years of data, lots of folks have this data. The quality is basically free, except you actually get better quality and it doesn't cost you anything more. And these are students who are learning peer reviews on the fly as they do this. And once you know them, actually the total cost goes down.

Because you're finding the bugs up front where they're cheap instead of killing yourself on test, finding a bug, and then having to figure out where the heck it came from. Why isn't this industry standard? Why don't we do this? Why don't we have, no, sorry. I was going to ask similar things we don't have. You're supposed to be doing it. Okay. People don't. And it's a cultural failure. What I see is managers think that when you have the code written, you're making progress. Writing code is making progress, right? And there's this notion of a thing called code complete in a lot of industries, where that means the code compiles without errors the first time or something like that. And they're all pushing to code complete.

But that's the worst possible thing you can do because you're shortcutting. You're not doing design. You're just slopping down some awful code and then beating on it to get the bugs out. Well, that's the most expensive way to write code there is. That's just crazy. But why do people do it? Think about the embedded industry. Now, the aerospace part is a little different, but the bread and butter embedded industry, who are the managers? Do they have the word computer in their degree? Yes, usually. Do the people who are writing the code have the word computer in their degree? Usually.

They took that programming class when they were in college, and the rest they've had to learn on their own. Oh, you're saying that they haven't taken, that they don't have computer in their degree? They don't. Mostly they don't. So the industries I work with, it's mostly mechanical engineers, electrical engineers, industrial engineers, physicists. I see where you're going with that. But I do work in industries where most of the managers and engineers do have computer in their degree and they still don't. It's still a mess. Do any of this yeah well there's that i mean um i was just pointing out there's more to it than than a lot

of the listeners probably work in in areas where there are a lot of computer folks around uh but a lot of the software depending on for your life and and every day both life critical and just everyday things working, is written by domain experts. They're an expert in water heaters or microwave ovens or whatever. Why on earth would they have the word computer in their degree? That's not what it's about, but they're writing the code. And you're right.

Even the folks who were trained in computers, think about it. If you go to an electrical and computer engineering department, how many courses do you take in software testing? Certainly, having reviews as part of the curriculum is a new thing to me. I don't remember we did that. We barely covered version control. We spent a lot more time on O notation than cyclomatic complexity. Guess which one I am way more interested in? Yeah, well, it's even in computer science. Plain old everyday computer science isn't really about software engineering, is it? It's so people don't have the exposure to these concepts. And to a large degree, they're just not taught in most universities unless you get a specialized degree. If you get a software engineering degree, sure,

you learn all this stuff, but that's a tiny, tiny fraction of the people coming out in the computer area. Okay, I'm going to bring up the elephant in the boardroom. I'm going to ask you... Do you mean Agilephant? I'm going to ask you in the most neutral way possible. And I've forgotten how to do it. Wait, I had it in my head.

How do you feel about Agile development? Well, that was not as neutral as I was going to make it. Does this square with the current Agile thing that's taking over the industry? Okay, well, I'll be as neutral as I can. It depends what you mean by agile. So there's some folks who use the word agile, and what they mean by agile is excuse to do whatever the heck they want, all right?

And let's just set that aside, because that's just smoke screen for doing a bad job at software and having an excuse to do it, all right? But I don't seriously believe those people are agile. They're just using it as a buzzword. But I've worked with teams that actually do agile. You know, here's the flavor of agile we're using. Here's the book we're following.

It's a methodology. And I think that's great. I'm more interested in, there's a bunch of things you have to do to get good software. For example, you have to have some design before you start writing code. You have to have some requirements.

You have to have some testing. You have to have peer reviews. You have to do all that stuff to get good code. Now, if you want to call your requirements a requirements document, or you want to call your requirements user stories, I don't really care what you call it. I also don't care what shape you draw the picture

that the steps link together in, but they all have to be there. I'm nodding so hard my neck hurts. Yeah, I think that's a lot of what's missing is that in my experience, part of the Agile manifesto is explicitly about less documentation, more working code. And I think that gets taken as an excuse to not do design sometimes.

Well, I love less documentation. So my research team who does robot stress testing runs Agile, and they're great at it. Okay, so that's not a problem for me. Now, they're doing research code. It's not production code, so the bar is set a little differently. But if you're doing production code, what I'm saying is not that you have to have

a thousand pages of design documents, right? What you need is something that gets the job done, and the more clever you are, the simpler that paper can be. So for peer reviews, I could not get my students to do peer reviews to save my life. I just couldn't do it. Then I finally figured out that if I gave them a very simple spreadsheet that was very freeform, and the only data I collected was, tell me how many bugs you found. And that was the only paperwork they really had to hand in that I paid attention to all of a sudden they started working but it was the fact that they had to give

me a piece of paper that made it work not that it was a 20-page document because it wasn't it was just one line a week how many you found so the art in getting a good process for embedded is not to go to a software engineering book and copy out the 30-page form or whatever, making that up, but you know what I mean. I worked in the military, believe me. I've seen the 2,000 pages of papers to install a diode. That's a true story. That's not what you want.

You could install it backwards. You've got to be careful with those things. But what you need is a non-zero amount of documentation. A very wise manager I met in one of the companies I did a review for said, you know, if it's not written down, it didn't happen. And there's a lot of truth to that. So you need a piece of paper you can audit to make sure it happened. And people hate the word audit.

It sounds so stuffy. But, you know, if nobody's looking, it doesn't get done. And the only way you can make sure it got done is it's written down and you should be as clever as you can possibly be to make it easy frictionless make sure the document has some value but zero is the wrong amount okay so there's somebody out there who's nodding along thinking well it sure would be nice not to pull all nighters it sure would be nice to have more confidence in my code, but this isn't my corporate culture. How does this person explain both to his colleagues, her colleagues and manager that this is worth it? How do we go about changing

this process? I don't believe it's laziness because people care and they want to do a good job, but it is hard to talk your colleagues into changing their patterns. So what do we do? It can't just be research says because that clearly didn't work. That didn't work for open offices. Yeah, I know. God, I hate open offices. So, I'll give you three reasons. And the third one I'm serious about.

The first two are just for context, right? The most effective way to introduce cultural change into a dysfunctional software development process is a near-death experience. There's nothing to get management's attention like being on the front page of the New York Times and the Wall Street Journal for a software fiasco. Okay, but do you really want to wait for that to happen? Well, I've seen cases where that's literally what it took and it is what it is. Okay, there's another one, which is to force them to go through a boot camp. Now, I have the luxury as a professor of saying, if you want to sign up for my course, you're going to do what I say, and you have to do it or you fail. And so I take, I guess it's about 60, 70, 80 students a semester every time I teach

my grad class, and I give them a hard problem. And I say, I want you to do peer reviews, but I'm not going to be too serious about checking. And they do it, and it almost kills them because I make sure to give them a hard problem. Then I make them do peer reviews. And by the end of the semester, I'd say maybe a quarter of them went in doing it all the while because they believed. More than half of them at the end of the semester are standing up in their final presentation saying, you know, peer reviews are the greatest thing ever.

I'm so glad that he made me do them, basically. It's sort of like a parenting exercise. And there's a few that still don't do them. And they are all up there saying, you know, that was stupid. We should have done them. But it's hard for someone in industry to force them to go through a boot camp.

You can't do that. Not realistically. And those are group projects, right? This is a group of four. And it's, you know, 80 students, groups of four. It's a lot of students. But we very repeatedly, very regularly, this is what happens. You know, some of them believed the whole time.

Most of them sort of got converted by trial by fire. And there's a few that never believed. But, you know, they can always serve a purpose by being an example to the rest right because their projects are the ones that don't work that's how it turns out but if you're an industry you don't have that luxury so the one thing i've done that seems effective but i'm still working on this because it's a really hard problem is i um i was with a team in china and we had an american and we had a bunch of Chinese developers, and they're very smart, very hardworking, but there's not a lot of old guys around to

sort of give them wisdom, right? And so they're trying to still find their way. So the American said, all right, here's what I'm going to do. I'm going to show you some of my code that I think is pretty good, so no one in the room has to feel uncomfortable with a code being criticized, and we're going to do. I'm going to show you some of my code that I think is pretty good, so no one in the room has to feel uncomfortable with a code being criticized. And we're going to do a peer review. And so we sat down and we did teams of people, but we had three roles. We used a checklist. We walked through five lines at a time. And the American said, you know, all right, there's probably one or two bugs in here. But we went through methodically very slowly, not, oh, I don't think I see anything, let's move on,

but made them look at every line, made them use the checklist. And in 35 lines of code, we found like a dozen and a half bugs. And one of them was a system killer. And this was production code, and no one had any idea that problem was there. So I think the only way to get people to do peer reviews is for them to live and experience where they see the value. And so at some point, you just have to have someone buy into, look, you're going to give me a half day of your life to have the experience and then you can decide. I don't know another way to do it. Sometimes you can get a team to say try it that that ends up being

if you don't it's kind of like gambling if you don't win on that first time you may lose them for a long time but yeah yeah you may and and you know this is is um well i'll admit this is slightly an advertisement you know one of the things my company does is we're the folks who come in and make sure that exercise succeeds. That's one of the things we do. Although, in the case of the story I gave you, after that senior developer saw how we did it, he went off and did it with a bunch of other teams and had great success. So part of it is you just sort of have to learn how to make that exercise work. But I still have, I'm still thinking about our listener out there who's still trying to figure out how to do it. And maybe getting management and some engineers to sit down is a good way. But then a lot of people we talk to, and I certainly have been in this boat where I'm the only embedded person at the company or in the team. And there's a mechanical engineer, an electrical engineer, and maybe an application software engineer, maybe a scientist and a manager, maybe even a team of 20 people. But I'm the only one who has any knowledge of a microcontroller. How do I get my code reviewed then? I see that all the time.

My recommendation is try and find a buddy in the same building or down the block who is in a non-competing company, execute a mutual NDA, and help each other by reviewing the other guy's code. I mean, that isn't obviously simple, but that could work because you don't need to know the product to review the code. Not really. If you have a design and you have some code, you can do a lot of the review just by looking at the design, looking at the code, seeing if they match. But you said informal reviews don't work as well. Well, ideally, you have a review club of three people, four people, and you take turns, right? Although having an independent set of eyes is way better than not. But yes, in fact, the value is there for

probably want three people in the room. And it's a trade. I mean, it sounds like you're asking someone to do work and they're going to say, well, you should pay me then. But the goal is really to trade over time. Yeah. If it's like a book club, you each take turns, right? Say, okay, we have a hopper full of reviews and we each throw stuff into the hopper and we take turns. And so, it's a mutual benefit society. And then there is the, you, there's the tactical part of reviews. And I used to hate reviews until I learned about, uh, classifying bugs into, uh, real bugs, ought to, major, minor, and nits. Because 95% of what everybody tells you is a nit. It's something they feel compelled to tell you,

but it shouldn't compel you to change your code. And we all like being right. We all like pointing out our favorite little things. Well, it's the easiest way to score, right? It is the easiest way to score. Yes, that's it. Ah, I've found all the braces I want added or moved. All the parentheses, which really should be there.

And I agree they should be there, but do we have to spend 10 minutes in a meeting on this? No, we should just mark it and knit and go on. Well, I sort of solve that a number of ways. One is that if you use a checklist, a good checklist that you've morphed into something that works for you, everyone's checklist sort of needs

to be specific to their situation. And yeah, you ought to be using Lint. And if you are using Lint, don't put that stuff in the checklist. That's a waste of time. But get the checklist of the things you think are high value and concentrate on stuff in the checklist. If it's not on the checklist, why are you giving me a hard time about it? Well, maybe it should be added. Let's take that offline. That's a pizza lunch. We'll hash it out. But also things like, I see a missing brace fine that counts as didn't follow the house style guideline that counts as one we're not allowed to mention that at all again for the rest of the review because the author has to go back and square it against

the style guideline let's move on so that's a way to contain that yes and you move you move arguments about style guideline to a different meeting oh you have to because there's nothing more fun than wasting time arguing about where

the where the curly braces go when you should be doing reviews feels like work feels like you accomplished something and so there there are a bunch

of rules and you know there's rules like don't attack the author and right everyone makes mistakes get over it and don't talk about style because everyone makes mistakes, get over it. And don't talk about style because that's not what you're there for.

I have suggested and have conducted reviews of online code so that people can get comfortable with the process without feeling attacked. And poor Adafruit, I usually use their sensor code because it has many things I love and a few things I hate. Do you have good code that you suggest people look at? Do you have checklists? Do you have really tactical advice for us? Well, there's a checklist on my website that was developed in response to a handful of design reviews I did with Gautam Katak, who works for one of the companies I work with. And so there's actually a checklist up there. It's really old.

I wouldn't necessarily use it now. I'm still working on a new one. But it's a starting point. And it isn't, where is the curly brace? It's, gee, does this code actually do what it's supposed to? Is it more complex than it needs to be? So a good checklist doesn't actually tell you what to look for. It asks a question to put your mind in the right spot to look at the code and see if something's wrong.

So the checklist isn't curly braces in the right spot. Checklist is, does it follow the house style guideline? Are the variables named appropriately? So it doesn't tell you the rule. It asks a question to get your head in the right space so you don't forget to look at an aspect of the code. You mentioned complexity, and that leads me to think about cleverness. Do you ever find that you're working with companies where they have kind of the rock

star culture where there's one or two star engineers? That seems like it would be difficult to imprint this collaborative approach onto if there's one or two big egos, especially in meetings. How do you deal with that? I'll bet we've all taken our turn being that ego, so I'll confess. Sure, everyone has that issue. The issue is the code isn't supposed to prove how smart you are. And it takes a while to get into this, but the best code is code that looks stupid simple because you can't make a mistake you know so if you have code that's clever or code that is clearly correct because there's no way there can be a mistake because it's so obvious and so straightforward the person who's smarter was able to write the simpler code and it takes a

while to see that and it takes a while to be comfortable with that. And if you have a rock star and their code is impenetrable, then that's got to be fixed because it is a team effort. And they'll say, I never make a mistake. I was in a review where we had one of those and said, there are no mistakes in our code. So I flipped

through and in about 30 seconds, I found a bug. And I got kicked out. Jeez. So, that approach does not work. Oh, yeah? I've learned a lot since then. That was an early review.

But that's the way it goes, you know? So, part of reviewing is learning how to handle situations like that. And there's no easy formula. I mean, it's really, a lot of that boils down to a people problem. And, you know, they don't teach us that much about people problems in engineering school. They're getting better these days at it, much better than they used to be. But, you know, ultimately, a lot of this is a people problem. Very, very much a people problem. So back to tactical advice. Do we review code after it compiles,

after it works from unit tests, or after it works in integration testing, or before any of those? It depends on your process. I would say that you need to draw a bright line where you start counting bugs. Because at some point you have to count bugs or you don't know if your reviews are working. Your reviews aren't finding half the bugs.

They're broken, pure and simple. That means you have to count. That means you have to draw a line. Ideally, and I think Jack Gansel gives this advice, ideally you actually review the code before you do unit test because economically the value's there. But a lot of places when they're writing code,

they're co-evolving the code in the unit test. So it's really hard to say when it's ready. And so if you want to say, I have the basic developer written unit test done and then I draw the line and I start doing peer-reviewed start counting bugs. I mean, that's defensible too. But at some point you draw a line

and a lot of places what they point, you draw a line. And a lot of places, what they do is they draw the line as when you do the first commit. And when you're done messing with the code, you commit it back into the code base, you draw the line, every bug counts, and you do a review basically at the time of commit. So a lot of places, I submit a commit candidate and it doesn't get committed until it comes out the other side of review. That's what I would recommend. Okay. I tend to want to do reviews after unit tests because I want to know that the basic functionality works, but my unit tests are usually smallish.

Yeah, and that's a really common situation, and that's perfectly fine. So, cool. Let's see. I want to move on a little bit and ask you about staying current. It is tough to – there is no magazine anymore. There's no book publisher that I trust, although, of course, I love O'Reilly because that's where my book came from. I would never diss them. That worked well.

How do we stay current? What do you suggest? Are there industry publications or academic publications we should be looking at? Embedded system programming, which became embedded system design, was great. I'm sorry to see it go. I suspect that's what you were thinking of. I knew several of the editors. I know Mike Barr pretty well.

It was a great pub, but it's gone. Embedded.com has stuff. There's a few blogs. Partly, I keep current because I'm doing these reviews, and they provoke me to go digging and learn new things. I'm still learning new things all the time. And I also work with a bunch of really smart folks at the university. And a lot of times, I just learn what's hot by assigning them problems and seeing what happens. So, I can cheat.

I can farm that out a little bit. But you're right. It is very tough staying current. And I don't have a good answer other than subscribe to some good blogs and try and keep up that way. It's a tough question. And of course, listening to his podcast. Of course. Of course. And there's this great podcast and so on. Your blog is one of those that I tend to subscribe to for the purpose of staying more And so on. good things. I appreciate that and I should mention that I'm finally joining you know, I'm an old guy

that's just the way it is, but I just set up my Twitter account, that was my resolution for the new year, so I'll be sending out updates when I do a blog post and things like that. Twitter's for jokes. Tell us you're going to tweet jokes and we'll follow you.

Okay, I'll follow you. It'll be out of business by April anyway. Twitter, yeah, maybe. That tells you how behind I am. But I'm trying to push my blog out to try and help that in the way I can. But it's tough. There's no one-stop shopping anymore. And your blog does sometimes point to your videos, which you mentioned, and your videos and your design and peer review, those fall under edge case research. Is that right?

That's right. So that's my startup company. The startup company does a couple things that seem unrelated at first we do robot stress testing and we also do sorry i just had this horrible horrible image of a poor robot in like a steam room or something or well it's being yelled at continuously a robot being yelled at it's not that far from that if you go to the astaa automated, Automated Stress Testing of Autonomy Systems, there's a couple of videos of robots having nervous breakdowns. So, they actually do do that, so to speak. They're not sentient. It's okay. So, the startup company started to commercialize that technology.

But what we found was, a lot of times, the problem wasn't, oh, well, if you throw a not a number, it destabilizes the control loop. That's one of the videos. But rather that you look down inside and the code's a mess and it needs to be redone. Sometimes that's what we find. And so we've also broadened to general embedded systems. But a lot of it is at some point we'll go in and do a design review. At this point, I've done about 200 design reviews. The book was at 80 or 90.

Now I'm up to 200. I've learned a bunch more things. And we'll go in and say, right, you need to fix these three things. And so the video series is there because now instead of us having to explain the three things, which takes a lot of time and detracts from the time we can spend

looking for more things, we can say, all right, go watch these three videos. That's going to tell you how to fix things. So we're working, mostly me, I'm working on building up a pretty big library of training materials so that when folks have problems, we can say, all right, this video has what you're looking for. I do that with blog posts sometimes. I'll write a blog post so that I never have to explain this again. Just go read the blog post. Yeah. And that's a lot of what my blog was. And I sat back and said, you know, I've done a couple hundred design reviews.

Let me take another shot at what that orange book from 2010 would be now if I did it today. And there's a lot more chapters in it because I've seen a lot more stuff. And so that's what the videos are. There's the one minute videos, which are the quick, here's the idea. Teasers. But there's 15 or 20 minute. Yeah, those are the videos are. There's the one-minute videos, which are the quick, here's the idea. Teasers. But there's 15 or 20-minute, yeah, those are the previews. And there's the 15 or 20-minute full videos, and there's a few of them up on the website already, which are sort of the full treatment. Here are all the things you should worry about.

And more of those are being made all the time. The one-minute previews, all of those will be freely available. We're still working on a business model for the larger videos, but the idea is eventually all this stuff will end up in videos, and on a good day, there'll be a book that comes with it where each video is a chapter. So those are plans. We're not there yet. It'll take a while, but that's kind of the direction I'm going. That's pretty cool. That'd be a great resource. That's the idea. I noticed when I looked around edge case research that you're hiring. What are you hiring for?

We're hiring in two different ways. One is we do a lot of development work. So we're looking for really sharp top 1% folks who can do embedded systems. So we do a review. We say, all right, here's your problems. And then we send one of our guys in to help them fix it. But it's not a work for hire kind of shop. It's more of a mentoring and a, we have one specific problem, help us with that and just make sure we're on track after that. We also do some internal tool development.

But the other category we're hiring, which is a lot harder to find, is someone who's really senior, who's really seen a lot of stuff, who could probably, if you give them one of these random videos, they would say, oh yeah, I can tell you all about that. Because we're also looking for people who have the ability to go into a shop that's having a problem and figure out what's going wrong and help them. So we're also looking for lead assessors as well. And those are typically going to be pretty senior people who are looking for something different to do. Cool. Sounds like a cool job.

Yeah, but it's on site. You do have some emphasis on being there in person. Is that right? What we've seen is that you have to be in there in person for like a day to really size things up. You know, part of it is you look at the code, you look at the design documents, or lack thereof, you have a pretty good idea which way it's going to go. But there's nothing like talking to the team in person and finding out what's really going on. Because these folks are usually really smart. They know what's broken. They may not realize they know it. They may not have a way

to articulate it. And sometimes you ask a few questions, I go, oh, that's why that's such a problem. But that just works so much better in person. I've done reviews over video chat with Asia, although usually I already know someone in the room, and so that helps a lot. But a lot of this is you really just need a day of hands-on to really figure out what's going on. So this is not a peer review. This is we're in the middle of the project. We want to make sure everything's right. Kind of an external sanity check.

Yes. Yes. Wow. I do those occasionally or the last minute, oh, my God, we're shipping and we just realized that we have a horrible bug. Come in and fix it. Oh, I get those too, yeah. Those are really fun, but if you're going to come to me with that, be sure to realize that it's going to be expensive because I don't like to work all night.

Really don't like to work all night. A lot of what we get is that the companies that realize they have a lot on the line when they ship a product. There's a lot of issue with brand tarnish. What's the worst that can happen? If you're a big company, brand tarnish can be crazy bad. Hello, Samsung. I'm not, oh, I have more stories than we have time for, okay? Sure, you can pick that recent one. There's many more oh yes right and um and so a lot

of them the the big companies will do it as insurance they'll say come take a look tell us what you find and almost always we find something you know usually it's not something big but but you never know often you find something and then you also say and here's a bunch of stuff to work on the next one but don't sweat it for this one. The smaller companies, usually something has to go wrong. That's when you've got a field failure and they need immediate help and stuff like that. So it depends, but it's tough because it's an insurance problem. And people don't like to buy insurance if they don't think they're going to have a loss, right? And that's why a lot of this is so hard. And yes, insurance is very difficult because it's also saying you don't trust your engineers.

If you phrase it badly, that is what it can sound like. And it is something only other people should need because, of course, my quality is fantastic. And yet, really, everybody needs at least another pair of eyes. Well, there's the Lake Wobegon School of Programming where all the programmers are above average. Okay. But you know what I found is that I had some code that I had written 10 years ago, and then I pulled it back up. Every once in a while, I just do this to keep my technical chops up. And I found a bunch of bugs in it. That's just the way it is.

Nobody is perfect. Everyone makes mistakes. Get over it. You need a second set of eyes, and it's that simple. It doesn't matter how smart you are. Everyone needs a second set of eyes. And it is worth dealing with the people problem to get that second set of eyes. And it is worth dealing with the people problem to get that second set. It really, really is worth it. As hard as it is, and as seldom as I succeed, I still 100% believe it's worth it. Yeah, I believe that too. You know, there's

another analogy I wanted to throw in just to get people thinking about being serious about a process, okay? All right, so let's say you're going to build a house. Now, my goal in life is to never build a house because I know it's stressful. But let's say you're going to build a house, okay? And you hire a carpenter who knows how to nail boards together. And you hire a plumber who knows how to connect pipes with a torch, okay? And you order a pile of lumber and you order a pile of pipes until I make a house.

Does that sound sane? Okay. No. That was a rhetorical question, right? It wasn't. All right. I'm going to build myself a 100,000 line program.

I'm going to hire a bunch of folks who know how to write code and I'm going to turn them loose. Why is that any different than what I just said about the house? You still need the GM. You need somebody who can look at it from a systems perspective. So let's be more specific. If you're doing a house, you don't hire a carpenter and plumber.

You hire a general contractor, right? Okay, but that's not all you do. Even a general contractor, probably that's not enough. What else do you have? Architect. You have an architect. And a city permitter who audits it all. And you have a building code.

Yeah, a building code. You have a building code. And the building code is the rule. So, if you want to make the software analogy, the building code is the checklist for the peer review. The city inspector is performing the job of that peer review. The architect is the guy who has the big picture, right? And so when you turn a bunch of people loose to hack out some code, you're building a skyscraper with somebody who knows how to hammer wood together. That's not going to turn out.

And I don't understand why people think it will turn out in software when they do the same thing. Because it's cheap and we need the skyscraper tomorrow. Yeah, well, I don't want to be in that skyscraper. Exactly. Yes.

All right. It is a weekend, even though this will go up on a Wednesday and we have kept you from your Saturday for long enough. Phil, do you have any last thoughts you'd like to leave us with? Sure. Let me give you a last parting shot. I truly believe, and this is amazing. I mean, I used to be a CPU designer. I never thought in my wildest dreams I'd be advocating

methodical rigor and process, but here I am, because I've just been convinced by my life experience. Software quality pays for itself. But to get there, it takes a lot of discipline. And discipline is hard. So what I would recommend to all the listeners, I mean, this is the New Year's, we're recording this on New Year's Eve. So, you know, it's a New Year's resolution thing. Hopefully this one sticks. Take some time to reflect on how you can get better. Find something small you can change. Maybe just get one other person to review.

I'd rather you did three, but someone else looking at your code is better than nothing. Find something you can change that's going to have payback by the next time you have a deliverable. Because if it doesn't have payback right away, you're not going to do it. Do it. Stick to it. Commit to it. And if it works out, when you're comfortable with that, pick something else small. Try and bootstrap your way up. Because if you have a big New Year's resolution, I'm going to completely change how I do software. We all know how that turns out.

Pick something small and stick to it. I think that's a great message for many things, and certainly one we've heard on this show. If you need to go listen to Jack Gansel's episode, I will link it in the show notes, because, like Phil, the goal is to do one thing at a time, to start small, win that, and go on. I love that episode. That's my favorite episode. So, everyone who hasn't listened to Jack should listen to him too. We see eye to eye on a whole lot of things.

Our guest has been Philip Kopman, professor at Carnegie Mellon University and co-founder of Edge Case Research. His book is Better Embedded System Software. You can find it on Amazon or kopman.us. You can also find his quite useful blog at http://betterembsoftware.blogspot.com or you can just search for Better Embedded Software Blog. And it will, of course, be in the show notes, so find it there. It links to many of Phil's videos and his Edge Case Research site also links to his videos,

so check those out. Phil, thank you so much for being with us. Thanks for having me. It was a blast. And thank you to Christopher for producing and co-hosting. And of course, thank you for listening. Consider coming to our party. That will be in the show notes. Go to Embedded FM if you'd like to read our blog, contact us, and or subscribe to the YouTube video channel. And now a final thought for you from Martin Luther King Jr. If you can't fly,

then run. If you can't run, then walk. If you can't walk, then crawl. But whatever you do, you have to keep moving forward. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.