Embedded - 243: Pick a Good One

Episode Date: April 27, 2018

We spoke with Michael Barr (@embeddedbarr) about the Barr Group embedded systems survey. You can download the 2018 survey at the Barr Group survey page. The Barr Group Embedded C Coding Standardis also free to download (with registration). You can buy a paper copy on Amazon. Programming Embedded Systems in C and C++ 1st Editionby Michael Barr, also available for free in HTML on the Barr Group site. The second edition is Programming Embedded Systems: With C and GNU Development Tools, 2nd Editionby Michael Barr and Anthony Massa. The second book was Embedded Systems Dictionaryby Jack Ganssle and Michael Barr Elecia’s book is Making Embedded Systems: Design Patterns for Great Software.

Hello and welcome to Embedded. I am Alicia White here with Christopher White. Bar Group conducted a survey of the embedded systems industry. A total of 1,703 survey responses from active professional embedded systems designers were received from engineers with an average of over 16 years of paid experience. What did we get from that? Our guest this week is Michael Barr of Barr Group to tell us about the results. Hi, Michael. Thanks for joining us.

Hey, thanks for having me. Now, I know you have been on many technical panels and been introduced in many different conferences, but how would you introduce yourself? Oh, to do it concisely, let's say I am a former professor adjunct at the University of Maryland and a consultant and trainer and expert witness, and as well an author and former editor-in-chief of books and a magazine about embedded systems design, of all things. That seems quite relevant. We are going to mostly ask you about the survey, although there are some other stories we'd like to hear. But before we do that, we want to ask you lightning round questions.

These are short questions and we want short answers. But sometimes that doesn't always work out. Are you ready? Okay. Favorite animal? Labrador retriever. What kind of car do you drive? I don't

favorite tapas ah favorite tapas uh those from the uh the bass country where they actually call them pinchos favorite processor favorite processor. Or least favorite. Least favorite, the old microchip picks with all sorts of odd features and memory regions and banks and things of that sort. We're going to get along just fine. Hardware or software? Software. Worst or most common mistake you've seen? Worst mistake I've seen? I think the Patriot missile bug that resulted in deaths in Saudi Arabia. Yeah, that would be pretty bad. Pretty bad. One more, maybe this is a longer question anyway. How do you stay current in embedded systems?

I write about it. Okay, and that wasn't very long at all. Okay, so tell us about this survey. Okay, well, so one of the things we do at Bar Group is we conduct an annual survey of embedded systems designers. And this is our fourth year of doing that. And in order to aid our community in understanding what's going on with the design of embedded systems, with best practices and other aspects that I'm sure we'll talk about, we not only conduct the survey, but we also publish our results. The first few years, we did a webinar about the results. And last year and this year, we actually put together a formal written report and make it available on our website for free.

And when you say best practices, what do you mean by that? Well, the survey talks with designers about how they're designing systems. So, for example, one thing that you might do if you're writing software is perform pure code reviews, where you either in pairs or in larger groups look at the source code and look for bugs and look for areas where it's unclear how it works, And then as a result, take action to improve the code. So we ask developers who take the survey whether they do those. And if they do, we might have some follow-up questions about certain practices. So it's not multiple choice. This is a long answer kind of thing?

No, it is not multiple choice. This is causes me to rephrase slightly to try to capture more of them in the multiple choice answers, or if a lot of people are saying the same thing that wasn't in the list, then I'll add it as a choice in the following year. Got it. And that we also do sort of select all that apply sort of choices. Asking questions the right way is tough. I remember many things from psych class about how surveys needed to be designed. Do you worry about that? Absolutely. I worry a lot about it. And I've no formal training in that area. But this isn't the first survey that I've been involved with. When I was editor-in-chief of Embedded Systems Programming Magazine, which used to be affiliated with Embedded.com, the magazine is no longer in print, we did an annual survey. And that survey predated my involvement with the magazine, but I then ultimately had responsibility for it for a number of years. And our survey is similar, and I have studied the results and the phrasing of those surveys.

And some of the questions, in fact, are purposely phrased in the same way so we can compare the old data and the new data. Yeah, that was one of the things I liked about the results was the longitudinal aspects where you could look at the respondents who preferred languages over years. And maybe I should ask you, which ones of those did you like? Well, the two that we really did longitudinal like that, looking back about 15 years in each case, were with respect to the choice of operating system and with respect to the primary programming language. And that's because that's the data that I had going back to that earlier survey, both in the years I was involved as well as more recent years. I was able to compare the answers across that time frame.

I think I had about one gap year between their survey and ours. What did you find for languages? I found some very interesting stuff with languages. And in particular, it's that not only is C still the most popular choice for the primary language for writing embedded software, but also that C in recent years has actually still taken share from not only assembly, which we would expect, but also from C++. Yeah, there was a peak of C++ a few years ago that actually it went up and now it's going down in embedded systems. Yeah. And I actually blogged about that specifically and it's an interesting finding. It is. More about the survey. How much work goes into it?

It's about, it consumes basically a month of my year professionally. And that's getting respondents and reviewing the data and analyzing it and writing the report. Is there more to it? That's pretty much it. It does begin with the step of reviewing the prior year's results. For example, like I mentioned, looking at the other answers. How do you get respondents? Gathering a large sample must be, I don't know if that's a challenge or not, but how do you go about that? Getting respondents was the most difficult in the first year that we did it as Bar Group. And in fact, we rented a number of mailing lists

in the embedded systems industry, as well as asked a number of partners with their own mailing lists, friends and such, to pass along links to the survey. But we now are able to do it through the outreach primarily. We still rent one list, but primarily we're doing this through social media, particularly LinkedIn, through our own mailing list, and also through several partners who do it for, they're spreading the word for us for free. In 1,703 participants, do you have an idea of how much that represents the industry as a whole? If you're asking how many embedded systems developers there are in the world, I don't know the answer.

All right, me neither. Probably more. Probably more. But that sounds like a good size sample. Yes. We actually get more responses than we want. We had about, I think it was about 2,100 this year. But sometimes you get people who are students, maybe they're grad students, maybe they're undergrads, who hear about

the survey, maybe hear about the fact that we're giving away, this year we were giving away a couple of fluke multimeters. And you sometimes also hear from professors who are interesting folks to talk to. But we're really interested in people who are working on real designs, who are being paid to do it. And those designs are actually going to be used by real people rather than academic projects or people who are trainers, things of that sort. So why do you do the survey? Do you have specific goals for it? We do. Specifically in our business, one of the things that we do in consulting is to help companies make their products safer and or more secure. Sometimes safer simply means more reliable because no one's life is on the line. But improving the safety of products is one of our consulting focuses. And similarly,

security is an issue, obviously, with more and more embedded systems being put onto the internet. It's important to get security right. And obviously there's some overlap sometimes between safety and security. But we ask questions that include some overlap, like what programming language are you using, with other industry surveys that are more vendor-focused or put out by a magazine so that it can gather demographic data on its readership so that it can sell advertising. What we do is dig deeper beyond those issues. And we don't really ask about brands that people are, you know, what brand of chip are you using? You know, what brand of real-time operating system? We're more interested in what are they doing in terms of their practices?

Are they doing pure code reviews, for example? Do they have a coding standard? What are they doing on the security front in order to secure their product, if anything? And why do you publish it? I understand now why you take it, but why are you doing the effort of writing or doing the webinars? Well, as a company, our mission is to help as many engineers as we can make their products safer more reliable and more secure and as in all businesses we can't reach everybody as a paying customer so we do consult with a number of companies that are building products and that's

one way that we help engineers. But we also are interested in improving the safety and security of lots of designs that we can't be personally involved with. And one of the ways that we hope to do that is by publishing this data back to the community so that people will look at the data and see perhaps that there are opportunities to improve their own practices. And hopefully this makes for a world where it's safer for me to use embedded products and for you and for all of us. What are some of the interesting results? Well, we put together a couple of infographics that summarize some of our interesting findings related to safety and another infographic related to security. And I guess I'll start with

security first. So to look at how secure embedded systems are, we looked at the subset of the designers who said that their product was going to be on the internet in some way, directly connected or indirectly connected. For example, it could be a product that's going to be on a network in a car, but that car has a cellular connection and therefore is connected up into the internet more broadly. Or it could be something that's directly connected up to the internet. And we found actually, first of all, that about a quarter of those internet-connected devices actually are also dangerous devices. That is, they're devices that could injure or kill one or more people. And so you have a large number of embedded systems that

are on the internet and a subset of those that are also dangerous. And by the way, I refer to this as the internet of dangerous things. But one of the most shocking findings is that nearly a quarter, about 22% of the engineers who said they were working on systems that were going to be on the internet said that on their project to-do list, security was not even on the list. There was no sense asking them what security practices they put into place because they simply didn't have as a requirement even to pay attention to security. And do you have a list of the those

products very specific names and companies that's the one thing we do sell but then furthermore for those who did say you know yes security is something that's important on our project we found that they weren't really doing a lot of different things that you might do if you cared about security. And of course, there's no one-size-fits-all answer to securing embedded systems. But one thing that you might do if you were going to put your product onto the internet

and you wanted to secure it is encrypt the data and commands that you're sending over the internet and you wanted to secure it, is encrypt the data and commands that you're sending over the internet. We actually found that less than 50% said they were encrypting their data. We found also that just 49% or actually just over 50% are using static analysis tools,

which can automatically search the source code for things like buffer overflow potential attacks. And even more than that, 54% aren't performing source code reviews where they have humans looking at the code to look for bugs and weaknesses. And these are both safety and security related. But they are in the sense that 25% of these, here we were looking at the internet connected devices, and about a quarter of them are devices that could kill or injure. I don't have the statistics in front of me, but I can tell you that I did drill down further when I was preparing my report into that 25% specifically with a theory that, well, you know, maybe those who are designing embedded systems that are going on the internet and also could injure or kill people, maybe they're doing these practices more or better. And it's true,

they are doing them more, but not significantly more. Not 100%. and not even close to 100%. Why do you think this, and this is totally speculative, but this is a surprising result to me because I have worked on all manner of things, internet connected and not internet connected and things that can kill you or things that can maim you. And my primary goal always when starting a project was to protect the customers. And I don't understand how somebody wouldn't be thinking about that. And, you know, when the choice became for some medical devices, oh, should we have this have a network port? It was always easier to say, no, just let's not do that at all. So it's surprising to me that people are being this lax.

Why do you think that is? Why are we so bad at this? Well, this is obviously outside of the survey. And as you say, it's speculative. Or in my case, it's fine. a focus and you're always thinking about safety. But when you're starting out making the latest and greatest Internet of Things product, your primary concern actually, or really those who are funding you and your engineering, their primary concern is, is there a market here at all? How big is it? How much of it can we access? And so when they don't have customers, they don't care

about security. And then they do have customers. If they get traction, then they come to us sometimes and they say, we're having this problem. Our devices are being attacked or we suddenly care about security for the next version, things of that sort. And unfortunately, as you guys would be familiar with, very often it's hard to retrofit security into a device. It might be different if you're doing a completely new hardware design, a completely new software design. But if you just want to cram security in, it's like building a home that has a number of weak spots and then deciding that you want to make it into a bank. That's a really good scenario for the startups because it makes sense.

It wasn't that someone was negligent or malicious or even incompetent. It was they were trying to build something for cheap and then it got out of hand. That's negligent? I think it's kind of negligent, but on the other hand, I have a lot of startup companies and they don't have a lot of money and they're trying to see if there is a market. Do you remind them that FDA auditors are authorized to carry sidearms?

I had that conversation on Friday, yes. But actually, the reason I had that conversation was because the startup had gotten some reasonably good advice for consumer devices, and he didn't understand why it was not good advice for FDA. And he's an entrepreneur, and I help him with the software, but he doesn't know it all, and he's not going to know it all. And he's far more focused on his application than on the software or but he doesn't know it all. And he's not going to know it all. And he's far more focused on his application than on the software or even the hardware. So is it wrong? I mean, it is wrong. Clearly, it's wrong. But how do you fix that? That's not a question I have an answer for. I'll tell you about another scenario that I encounter and I think explains some of this. And that is one type of internet-connected devices are coming from companies that are existing or established companies that make non-internet-connected products. But are pressured by their marketing departments or their sales departments into

branching out. So maybe they used to make a medical device that's used in hospitals, and it was never connected to any sort of network at all. But for one reason or another, now they need to add an internet connection to it. And in that scenario, unless it is specifically called out from the beginning that security is an important thing, security may not have been an important concept in the design of that medical product before. And there may be no one working inside that company who, at least on the design of those products, who has security in mind.

Sure. It was a black box. you couldn't open it and why bother with security if you can't get in or out and then they come along and add a little ws28 whatever that's an led controller yes two million letters numbers and. Add a little Wi-Fi chip on there and suddenly it's not a locked box anymore. It's a box open to anybody who has the knowledge. You just carved a door inside and opened it. Yeah. It's a scary world we live in, huh? How do we get better? Well, I don't know how we get better. And I can tell you that across these years of survey that we've made, there's not been any significant improvement

visible. The most noticeable improvement is that this year, there was a trend towards fewer people saying that they didn't care about security at all. That had not been present until this year. And I speculate that one thing that's going on is a feedback cycle between the news about various types of botnet attacks and the denial of service attacks and all the different things that are going on badly in security is making its way into the engineering departments and their requirements lists. But that's obviously a slow cycle, a multi-year cycle. Yeah, that fits with the things that I've heard about. I don't want

my face on the cover of Wired. The survey is good for telling us things are happening. How do you dig deeper for figuring out the why and how to fix the things that shouldn't be happening? In a survey, it's a bit difficult to do that, as you imagine. First of all, surveys have the problem that you only have a few minutes of someone's attention. I certainly have been invited to take surveys that very quickly appear to be heading towards 15 or 30 minutes of survey, and it's easy to lose interest. And if you want to get the real facts, you need to gather them quickly. And then there are things that are subjective and things that are objective.

It's obviously easy to say, what programming language are you using? But it is subjective to say, why are you using that programming language? And in fact, three people working on the same team might give different answers for that. Do you think our continued use of C is a defect in the industry? Yes. Why? What would you rather we use? I won't answer that, what would be a better choice. But obviously C is a language that has a lot of holes in it, has a lot of places where things can go wrong. As one author referred to it in his book,

C gives you enough rope to shoot yourself in the foot. Yes, yes. But you don't have any advice for better? I think that the problem is that embedded design is by its nature diverse and by its nature cheap. The reason we have Internet of Things devices is because the cost of processing power has come down. in software that have zero, once you have a processor in there, that have zero weight and zero added bomb cost. And in that sense, it is magical to take a cheap processor and put it into, could be anything, could be a chair, a smart chair, if you will. And once you put that processor in there, you are able to do something that allows you to sell more than your competitor or more than you used to. The problem is that the only reason you're putting a processor in that chair is because you can do it for a dollar. And so that brings with it a lot of problems. One of them is, you know, you don't necessarily have a lot of resources in terms of the processing but there's just so many different uh not only families of processors and architecture sets instruction sets but also different models with different peripherals and other things that it is very difficult to

suddenly introduce a new programming language and have it, let's suppose that tomorrow you, Alicia, invent the next greatest, perfectly secure, perfectly safe embedded programming language. How do you get all the compilers to be available for all the chips from all the different vendors? And how do you get all the programmers to suddenly know that new language? So it's a very difficult chicken and egg problem. Yeah, and it really is. That's why we keep using C, is because we know it. And C++, not as many people know it.

And I don't know that it is always better. I know that it is not always better. And I have been using Python a lot, but I don't know that I'd use it in an embedded system because there are times I do slices and sorts and things that I know must be super slow. And I just do them because I'm on my computer and it's got a fast processor or I'm on a Jetson and it's got a fast processor or a bunch of fast processors.

And so I don't know what language I would use for embedded. And everybody who's shouting Rust at their audio device, yeah, I've heard of it, and it has the problem that Mike was saying. It doesn't have enough compilers and it doesn't have enough users. And it is a chicken and egg problem to get that started. Also, you go people. I don't need email about that either. Sorry, Mike.

It's interesting. You bring up Rust and you bring up Go. You bring up Python. These are all languages that in the language area of our survey, they all show up occasionally in the other answers. But the number is countable on two hands, for example, for Rust out of 1,700 surveyed. Not even enough for me to say, you know, I think Rust should be on the short list. Yeah. And I don't know if that's going to change. Switching subjects, though, from languages, one of the things that I was kind of shocked and appalled reading your survey was version control. Hundreds of your respondents,

hundreds are not using version control. Do you remember what the percentage is? I calculated it, but I think it was six or eight. It was less than 20, which was good. Well, that's all right. Six or 8% think the earth is flat. Yes. You're right.

It's in the 90% are using it overall. And it does go up if you subset just the ones who are building, you know, safer systems that could kill or injure, but not to 100%. And that's obviously appalling, but there are different ways you can track the versions of your software. The one that bothers me even more is actually the lack of a formal list of bugs or known issues. Because we didn't ask whether you're using a specific tool like Bugzilla or some other bug tracking tool. We asked, do you have a way that you track the bugs in your system or the issues that have been observed?

Wait a minute, they're not even using Excel? It sounds like they're not even using a Post-it note. Exactly. I mean, it could be a Word document, it could be Excel, right? And that is actually more appalling to me. I mean, the chutzpah it takes to say essentially that there's no bugs in my system

or there's no issues or to believe that the ones that actually exist that you know about you can keep track of in your head or on some post-it notes or something and that and keeping aside the whole issue that improving self-improving as an organization should include a feedback loop that you're creating more bugs or creating more bugs of a certain type or you're detecting them at a certain stage or a particular programmer needs some training because he's creating more of the bugs or there's a lot of opportunities to do self-improvement with that feedback loop of here's the bugs we had this is the facts can we learn from them

i've been thinking about how to get people to use version control and how to make GitHub simple enough that a solo developer or a pair of developers can work with it without having to worry about rebranching and confusing things, rebasing. But I guess now I really should be saying, look, if you use GitHub, you get bug tracking for free. You don't have to put any code in there. You can just have a bug tracker or use Excel. Anything where you write down, this is the bug, this is the severity, this is how you reproduce it, and whether or not it's fixed so you don't drop things through cracks. It suggests to me that there's a fair percentage of people who are coming into this industry who don't, okay, for lack of a better way to say it, don't know what they're doing and have no

experience and weren't trained formally in any way and just picked up a computer and a compiler and said, I can, okay, I can do this. I mean, they're... Is this a failure of our education system? Well, it seems like they're just not exposed to fundamental concepts. They know Dijkstra's algorithm, but not how to use version control or bug tracking? I wasn't going to give them that much credit.

Oh, why? I don't know. It seems odd to me, because some of these concepts are things that once you just encounter them, it's like, okay, that's obvious. I should be doing that. Oh, my God.

Version control is control Z for my code? And I've had that experience, right? Where you say, oh, you're using version control? What's that? And it does this and this and this. What? That's amazing. Backups?

Offsite? So Chris has a point. I do. Is there a method people get to embedded software where they don't get exposed to these concepts? I think there is. There's a method by which individuals come to it, and there's also a method by which organizations come to it many times that result in the software side software side at least quality control aspect being lost the first of those with respect to individuals is you have engineers many of them uh possibly the majority of them i believe it used to be 75 but i don't know currently uh who come to embedded programming from the hardware training side.

That's how I came to it with degrees in electrical engineering. And that was the norm, I know, for 75% about 15 years ago based on survey data. As recently as 15 years ago? Wow. Yeah. Wow. See, we did get version control in one class, although never bug tracking. Yes, the software engineering class, right?

I don't know what we did for bug tracking. You're right. I don't think we ever saw bug tracking in college. But I saw it immediately in my first job. Well, it's because we worked at big companies who had big processes. So with respect to individuals, I think you get a lot of people who are hardware trained and end up writing software because they're the most qualified or because they like it or because they do both the hardware and the software. Because they're the ones who read the data sheet.

Exactly. At least at the lowest level, at least at that boot up code, at least at the driver level. And sometimes, obviously, the whole application is small enough that one person can create the hardware and the software, or one person does the hardware and one does the software. So with the individuals, I think, yes, there's a gap. If you look at the Venn diagram of an electrical engineering curriculum and a computer science curriculum, you find an overlap. And in that overlap, there is some of the stuff that you need to know to develop embedded software. But there's also stuff you need to know to develop embedded software well that lies outside of both

the computer science curriculum, because it doesn't talk about embedded systems or real-time systems or safety-critical systems very often. And also outside of the electrical engineering realm where software might be a Fortran class or a Java class and not really be focused on the methodology of software or software quality. Hopefully no one's studying fortran anymore well there's still a lot of fortran in the world right all those scientists still haven't gotten to python yet and i alluded to this well i alluded to this other path for organizations and i think it's worth talking about which is that many companies become in the same way that a product maker that say makes a medical device might

suddenly feel the pressure to put it on the internet another thing happens too or it has happened a lot in the last 20 years which is companies that make stuff out of metal or metal plus electronics and maybe are good at making their trunk lids identical time after time after time, or their circuitry have a low mean time between failure, end up in the software business in order to make their system a little smarter. And there might initially just be a handful of people in that company that are doing software. And they might also come to it from this background of mechanical or electrical engineering. And so they don't bring processes to it. And the organization doesn't care about software processes. And in fact, the ISO 9001 docs that the organization has are all about making trunk lids or all about making electronics. And so when they adapt those processes to software, they miss the mark.

I'd say anecdotally from my own experience that even... I've been in organizations that start the way you say. It's a mechanical vision. They come from a history of products that were purely mechanical, decide they want to add some computing aspect to it. But in my case, they rightly hired people with computing experience. But it still was a huge struggle to explain to them how it was done right in software, because they kept falling back to that other experience and it didn't make sense to them. Why is this taking so long? Or why do we have to do these steps? Why

are you doing all this process? So even in cases where the right people are in place, if the company culture is toward that more making space leaf rockets, it can be a challenge just to convince people to put the force behind quality in software. Absolutely. And when the teams are, as they are in embedded systems design, predominantly less than 10 people, the survey shows that about 85% of software design teams will never be bigger than nine people. Yes. Survey shows that about 85% of software design teams will never be bigger than nine people. Then big process is difficult to implement unless you have a company where there's a lot of those teams.

They're making a lot of similar products. And then there's that person who refuses to use whatever you implement. And that's really hard to get buy-in if everybody's using version control except for one person who wants to use the shared Dropbox. Yeah. Seems to be a common personality trait among engineers. Stubbornness is good.

It is good, but they're taking it too far. I'll do anything as long as you didn't tell me to do it. There's a lot of that, yes. I wonder if there's a way to help people understand the benefits. Because I have heard plenty of stories of, oh, I used SVN, I messed up the merge, and it's all bad, and I've just gone back to tar balls with versions. I did wear a seatbelt and I got ejected from my car and it saved my life. Sorry. Thank you.

Yeah. I take your survey and I'm like, okay, there's a lot of things wrong. Let's fix all this. And yet it's insurmountable in parts. Are there parts you wish people would take action on first? First? Soonest?

Well, with respect to security that we've been talking about, I think the number one thing that people need to do is actually put it on the list of requirements. And I think there's a reason why that needs to occur, and there's good evidence for that in the ACM and IEEE Codes of Ethics, which have amongst their very first rules, in the ACM Code of Ethics, it's Rule 1.2, and in the IEEE Code of Ethics, it's Rule 1. And. And in the IEEE code of ethics, it's rule 1. And I'll just paraphrase the IEEE rule, accept responsibility in making decisions consistent

with the safety, health, and welfare of the public. And I think that that includes, if you're putting advice on the internet at least, that you have an ethical duty to not ignore security. That's the number one thing you need to do. And obviously I have some other recommendations as well on security, but if you are continuing to stick your head in the sand, you're not going to go anywhere. Fair enough. What about the other side, the safety side?

On the safety side, there is a lot more with security, adopting best practices, for example, you know, having a coding standard in place and following it and enforcing it, doing peer reviews, doing regression testing. Those are all good practices, and they will, by making the software more reliable, have a security impact. But they're obviously not everything, and they're not cure-alls on the security side. On the safety side, we can get a lot closer to safe products through process. And it's one of the reasons, for example, that you don't see very many planes falling out of the sky, is because the rules for developing safe hardware and redundant systems and safe software for commercial and military aircraft are encoded and followed widely. I'm referring to DO-178 in particular. And in the FDA side, we do see some guidelines and recommendations for medical devices that are in part based on DL178, but probably don't go far enough.

And certainly don't go far enough in breach the safety of your device and getting better and smarter with more resources over time. And in that respect, it is a more solvable problem. We have talked about the IEEE ethics list before, but we will link to that in the show notes, because both the safety and security fall from these. It's not that we are saying you should use version control and test-driven development and security and bug tracking for no reason. And ethics alone are not a reason. They are the uber reason. They're the start of it. I don't know where I was going with this.

Do you know where I was going with this? Did I have my blinker on? I think you were saying that these things provide a foundation from which good software can be produced, but they are not sufficient? The ethics guidelines don't tell you what to do. Right. They tell you why to do it.

Yeah. And we still need a lot of what to do. Okay. And the survey is useful for figuring out what to do of the things that we should do, what aren't we doing. Right. Okay. I had someone, by the way, take the survey this year and email me right afterwards and says, can I get a copy of the questions?

Because while I was taking the survey, I realized two or three things we're not doing here. Yeah. Wow. And so partly it's an awareness issue, right? Yes. Yes. And that's what this outreach, I mean, you do a lot of outreach for this, and we do too, where we try to make sure everybody knows this stuff. And anybody wanting to know more about version control, we've decided our next Basics episode will be about version control. It'll be so exciting. I'm on vacation that week.

One of the things you mentioned earlier were coding guidelines, and I understand you have some. We do. The Bar Group publishes a set of free coding standard rules that are not like other coding standard rules. If you've worked in a company where you have an internal company standard, you find that often the way it was formed is, well, John worked here 20 years ago and he wrote the standard. Or when we were debating whether to put the curly brackets on the same line as the if or on the next line, Bob really insisted that it go on the same line as the if and no one else really wanted to fight about it. And so that's why we do it that way. And when I was running another company before Bar Group, we were, of course, like everybody else having the same problems with our coding standard, which was people had differences of opinions and things. And religious beliefs. And religious beliefs. And so one of the things I

said when there were disagreements about it was, okay, whoever can convince me that their version of this rule is going to reduce bugs more than the other. Or as a backup, maybe it reduces bugs the same, but it's statically enforceable through a static analysis tool and the other one's not. Then you win. And so this internal company standard we started using with consulting clients. And then I got convinced to write a third book, which I said I would never do, which was the coding standard in a book form for external use. And that's been now about 10 years and one company ago. And now it's actually turning up as followed as the basis, primary basis of coding standards for more than one in 10 of the survey respondents. I think it's 12% this year.

And a big part of that is probably that within the last year, we made it completely free not just in html but also in a pdf document but for some of that increase but it essentially has become the second most followed coding standard the first being the mizra guidelines the mizra c guidelines for essentially a subset of the c language that is safer for use. Initially, it was created for automotive devices, but being used in other domains as well. And our standard actually doesn't contradict to MISRA. And so it can be used as a stepping stone towards MISRA or as a complement to MISRA

because they don't address issues of style and we do. And so you do address things like where to put the curly brace. Do you also address things like how to use dynamic memory in a resource-constrained system? Don't. Okay. We don't actually address architectural issues like that in our standard. So it's more focused on the names of things, the placements of things,

and how to reduce bugs through the use of the language without getting into the architectural choices. One of the things that's been happening with some coding standards is they've been, at least for the style kind of things, is they've been embedded into the IDs themselves

for like VS Code, you can choose, oh, I'm going to use Google or the other one's escaping me. There's a Linux one. Like a new one. Is that something that you've been looking at? Because that would be nice, too. Or in any of the static analyzers.

Yeah. We actually have static analysis configurations for a couple of the tools, including PCLint and a static analysis tool from LDRA. And that's actually an area that we're working on actively is having more of those relationships in place so that developers don't have to write their own scripts for their own tools.

Okay. Right. And one of the things I like about your coding guidelines is you two have the reasoning behind them. It was one of the reasons I like about your coding guidelines is you two have the reasoning behind them. It was one of the reasons I like the Google ones is because there were descriptions of, okay, this is why we're doing it this way. And so that when the intransigent engineer says, I've always put it here, I'm always going to, you can say, look, no, there's actually a reason. I'm not just doing this randomly. As if logic would convince them. We can always try. We can start with logic. Start with logic, end with beers. No, we actually find that many people who follow our standard appreciate that about it and are convinced by those reasons and code examples and other things.

And we do have one rule that a lot of people complain about and that we're actually withdrawing from standard because it's not worth the effort. It wasn't one of our bug killers. It was just a technique that we preferred. And yes, but generally speaking, those rules do convince people. Those statements do convince people. Which one are you withdrawing? We have a rule in there

that was based on a style that I had used for a long time personally, which was not to do pound includes inside header files. Oh, yeah. Yeah. No, that style has changed. Everybody files. Oh, yeah. Yeah. No, that style has changed. Everybody hates me for including that.

Yeah. I mean, I used to follow that too, and I don't so much anymore. I have come to the other side. That ship has sailed. Yeah. You know, it strikes me that there's kind of a two-level thing with with coding standards

a you should have one yes and it doesn't matter what it is but that that's kind of the first order thing you've got to have one and just pick one and the second is well if you really care pick a good one well i would say even more importantly is that whatever you have, you need to have an enforcement mechanism. Exactly, yes. So we actually follow up with questions. We begin in the survey, do you have a written coding standard that applies to your current project? And then we ask them, okay, so you do. What's the primary basis of it?

MISRA, Bar Group, Linux kernel standard, et cetera. And then we ask them how they enforce it. And even with the people who do have a coding standard, you find that there is a sizable percentage that have no enforcement mechanism, that it's essentially, some say this thing is not, it's just a, somebody wrote this once and we do have it, but nobody does anything with it. Nobody follows it.

And another common answer is that, even more common, is it's up to the individual programmer to enforce it. There's no static analysis enforcement. There's no pure code review. There's nothing. Well, this goes back to small teams with little process who is going to set up the static analysis tool and then who's going to set it up to be this on the other hand a small team is easier to keep everybody honest if you're doing code reviews and stuff they'll catch stuff larger teams is where it gets out of hand even when you have a coding standard because then you

have a hundred people you know making making small variants in each file. That's when you really need something that says, no, this commit is not allowed because you did something that doesn't comply. But it is really hard to have a small team and to set all this up. And then you're spending all your time chasing the tools and you're not writing the software you want to be writing. I wish there was a box you could get. It all had it all set up.

I don't know. It's a metaphorical box. Moving on. You said you had three books. And I'm familiar with your Programming Embedded Systems book. It came out. It's an O'Reilly book. It came out a decade before mine, and I remember was I worked for a year, about 2003, I think, with Jack Gansel to write a, it was titled, the book is The Embedded Systems Dictionary. And we were familiar, in fact, working with the publisher of Newton's Telecom Dictionary, which many of your listeners may have heard of, which was at that time, I think, in its 17th edition.

And so both Jack and I being published authors before, we felt at the outset it was very important to negotiate with the publisher all kinds of rights in the future editions, including if one of us died, what would happen to our share, the proportionality in the 17th edition and things of that sort. And for our work, we received, I think it was about $1,500 each in advances. And I spent that throwing a book publishing party. And I'm really glad I did because that's the only thing that ever came with that work. Dictionaries are hard. Well, it came out about the time that, it came out about the same time as Wikipedia really took off.

Oh. What is a week in your life like? Do you do much programming and do you mostly teach or run the business? What does a CTO really do? CTO does whatever it takes to make the company run that the CEO doesn't. So in our business, I have a partner, Andrew Gerson, who's the CEO, who oversees most of the business aspects.

And I get involved in all manner of different things from writing newsletters and developing surveys. And our marketing team sometimes is asking me for input on this language on the website or this language on a product. And then working with engineers on the individual projects that they're working with with clients. New courses, because a part of our business is training engineers. And although I don't teach those courses anymore or create those courses, I guide the engineers who do. And I also, we have some new products in the work. And I kind of get involved in all of it to whatever degree is necessary to make things

happen. You went from developing engineer to adjunct professor to book author. And how did you move to speaker and teacher full time? That's a great question. I don't know, one step at a time, I guess. The book actually came first. I had someone ask me once why I thought I could write a book about a topic like embedded programming in my early 20s, which is when I wrote that book,

the O'Reilly book you mentioned. And I guess no one ever told me I couldn't. And I like to write things down so I don't have to rediscover or relearn them. I guess I'm lazy like that. Some might say writing is not being lazy, but for me, it's not having to figure the same thing out twice. And so I wrote a first book, and before I knew it, I had a proposal to a contract with O'Reilly. And then I wrote a few articles and spoke at a conference about one of the topics that I had written an article about. And next thing I knew, I was editor-in-chief of a magazine somehow.

Somehow, yeah. No, I mean, I understand after my book came out, I, I was offered to teach some classes and I did a little bit of teaching, but I mean, my heart is in programming. I, I love building things. I love the design parts and teaching was all a drain for me. It was not fun. I mean, it was fun. It was fun teaching people things, but it wasn't energizing. I don't know. I'm an introvert. Let's just go with that. Well, I can tell you why I wrote that first book, which was that I kept encountering from my own learning and then watching others learned that when you were learning embedded C, programming C for embedded systems,

that all of the books about C at that era in the late 90s began with Hello World. And then you built up from there. And so I've written the only book about C that ends with Hello World. That makes, all right, yes. I like that. And there are two versions of your book. I don't know if you want to talk about that. I will say that I liked the first version best.

I'll just say that I liked the first version best as well. Okay. And that will lead people to find whichever version they want. The first version is available for free in HTML on Fargrip's website. Oh, cool. I didn't realize that. That's awesome. Okay, one more question, I think, and then we'll let you go about your weekend. What did you want to be when you grew up, when you were little?

At what age? Whatever you remember most strongly, you know, something between four and ten. I can tell you it wasn't an embedded software. And I can tell you that even as I graduated from college with a degree in electrical engineering, I had never heard the words embedded systems, or at least those words put together. Yeah, no, I didn't know what it was until somebody was like, you should work with hardware. And I'm like, okay. Yeah, I'm not sure when I heard that term first.

And it was firmware first for me. Yeah, I mean, but I wasn't. I used to describe it as programming with a data book, that that's what I enjoyed. That I had taken computer science classes because I enjoyed programming and I'd worked as a programmer during my years in college. But I was coming out with a degree in electrical engineering.

And so I knew how to read those data books and I knew how to interface to the hardware. When they were real books. Used to have shelves full of them. Yeah. A lot of yellow and blue. Exactly. The big yellow books. Do you play much now with Arduino or any of the systems, or does it mostly work for you?

I get my programming thrill through watching my 14-year-old learning Python and other languages and trying to help him out when he gets stuck. Cool. Christopher, do you have any other questions or should we let Mike go? I could, no. Mike, do you have any thoughts you'd like to leave us with? No, I don't think so. I think we've talked about a lot of different things and I really appreciate the opportunity to meet you both and to be on your show. Our guest has been Michael Barr, CTO of Barr Group.

You can find his survey, blog posts, and many things on http://barrgroup.com. There will, of course, be a link in the show notes. Mike, thanks so much for being with us. Thanks. I have a couple of announcements. Dave and Brett Smith won the Tindy soldering kits from Jasmine. Congrats. And Jack Cancel had a survey too. He wrote about the many things he didn't like about

online embedded materials. He wrote about many things in his survey, but one thing was related to us. He didn't like the online embedded materials had so many advertisements and there's a huge quest for page views and spam email and junk comments and press releases disguised as user reviews. And none of that applies to the embedded blog. Also, we've redesigned the front page to help you find posts or series you're looking for. So please, come on over and check out our blog. There's a lot of good stuff there. Now we're painted in a corner.

Now we can never have ads or spam or clickbait. Well, I didn't say there was never clickbait. I mean, I did write a post about octopuses. How can people not click on that? Thank you to Christopher for producing and co-hosting. Thank you for listening. And you can contact us at show at embedded.fm or hit the contact link on embedded.fm.

I'm not going to leave you with a quote this week. You had enough with that whole Jack and winning tindy things. Have a good one. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.