Embedded - 99: You Can Say a Boat (Repeat)

Episode Date: February 2, 2017

While we planned to ask Andrei Chichak to podcast when he was in town for the Embedded.fm party, we spent too much time goofing off. So we are replaying Andrei's first appearance on the show where he spoke with us about MISRA-C and ethics.  (Note that this is the same Andrei who writes the STM32 Embedded Wednesday posts for the Embedded.fm blog.) Linker post: It's dangerous to go alone! Take MISRA-C Andrei's has personal website (we failed to talk about his kite aerial photography, it is really neat though) and his company is CBF Systems. Plum Hall C Compiler Validation PC Lint JPL Coding Standards for C (and the mentioned video discussing Mars Code) ISO 26262 Automobile software standard Cortex-R for high reliability systems (ARM's description) National Society of Professional Engineers code of ethics and Canadian Engineering Guidelines on the Code of Ethics Offline, Andrei recommended two books and another podcast about MISRA: C Traps and Pitfalls Safer C MISRA with Johan Bezem (podcast)

Even when Michael Barr says go to fail, whenever he hears me say it, this is Embedded.fm. Welcome to Embedded. I'm Elysia White, alongside Christopher White. For our 99th episode, our guest is Andrei Chichak. We are going to talk about miseracy, but I'm expecting a few giggles, so hold on to your hats and glasses. Before we get to you, before we get started, I put together a little survey to find out about our audience, and I would appreciate it if you would take a minute and tell us what you think. The link will be in the show notes. You went off script.

Your script and my script no longer match. What? Well, hi, Andre. Welcome to the show. I can't tell how this is going to go today. Do you realize that 99 and Wayne Gretzky's number? I did not know that. One day Canada's going to get a hockey player that good.

You just wait. I thought he played cricket. No, it's the game with the girls in the short skirts with the curved sticks and the hockey ball. The cross. So, we have mentioned Andre on the show before, and he's always been Andre from the Great White North. And that is because you live literally in the middle of nowhere. Well, close. You can see it from here. I mean, I found Edmonton on the map, but I had to go to the center of the continent and then up by 50%.

Yeah. Nobody knows where the hell anything is in Canada. So I just say that I'm six hours north of Montana. I always think of Montana and North Dakota as the northernmost wasteland. And then I look and I'm right, all of Canada is above there. Oh, yeah. And it's a lot bigger than you, a lot bigger than you think. Uh, yeah, it's going to be one of those shows.

Andre, could you tell us a bit about yourself? Sure. I've got a BSc in Computing Science from the University of Alberta, 1983. Yes, I've used punch cards, nine-track tapes, ate 300-baud modems, hand-toggled programs, used paper tape, wire wrap, got my first internet email account in 1983 when you had to use exclamation marks to tell the mail routers how to get to your machine. Let's see, my first experience in embedded systems was after I graduated when I got a job making a controller for highway truck way scales. I didn't get paid for that gig and I ended up in employee standards hearings.

Then at Dow Chemical. Wait a minute. What are employee standards hearings? What? Oh. That's when you take somebody to court, except it's sort of more along the lines of employee-employer relations rather than I sue you. So they have umpires. Arbitration.

Yeah, yeah. Okay, so Dow Chemicals? Dow Chemical. Doing operating systems with some hugely talented people. I wrote my first sort of web spider in 1984 way before the web so go for Archie

no before that this is like we had a 1200 baud line between Dow Chemical in Ontario and Michigan. And it's all sort of private lines. But you had things like deck net. So you had a bunch of vaxes strung together.

And they had their own Ethernet with the cable that's three quarters of an inch thick. I remember. You remember. And it was very cool because we could install operating systems down on machines in Texas from Canada. And this is way before, like there was maybe 20, 30 machines on the internet and we were going across the, we were already going across the continent. If I recall correctly, that was back when ethernet had vampire taps. If you wanted to add a new machine, you literally screwed a thing into the cable that pierced it. Well, before that, you actually brought out a drill with a plastic bit and bored through the sheath yeah it was bizarre

and uh we actually came up with this thing of course everything in those lines was um a command line oh yeah and all of our customers were like chemical engineers so we came up with this kind of a menu system where you had lines of text up on your screen and you could say i want to do number three and it would say oh number three said this and that links to this file therefore you want to do this it's very much like hyperlinks and then they said hey we should put keywords on this crap so instead of saying number three you could say do foo and it would go to its database look up foo and and and actually do that so uh this was sort of spread out all across north america

and they wanted to uh sort of put together this keyword database by going through all the pages, picking up the keywords, picking up all the links, putting them into a data structure so that when you say do, it looks and does it. Kids don't know how good they have it these days yeah but it was all done actually i wrote that in past uh previously where they would do it all in fortran and uh it needed to do recursion so so we were actually doing recursive web crawling

way back before the internet was a thing. When the dinosaurs roamed and the tubes were full. I think in Canada up there the dinosaurs do roam. Well, you know, strange. We just got our first 9600 baud modem last year in Canada. What? No, now you're just yanking a chain. Would I?

No. Okay. So anyway, one of my buddies from university said, hey, I'll give you a 33% wage increase if you come and work for me. So I buggered off and sorry, I'll give you a 33% wage increase if you come and work for me. So I buggered off and, sorry, I left and we made a hydraulic pump that rammed varsol and mercury through solid rock. That was pretty cool. What does that give you? Is that like fracking?

Well, first of all, you got to take a core sample to see if we got to check the porosity. And if the porosity is too low, then you frack. I see. So you do things like, uh, at one point they would dump liquid oxygen down an oil well. That seems safe. And then flare it. That seems safe. That seems fun.

I worked with a guy. He was the last one alive of his crew. Isn't that roughly turning the Earth into a rocket engine? Does it push us out of our orbit if we do that enough? No, you make sure you do it on opposite sides of the planet. At the same time. Yes. That's fine.

But, yeah, fracking's been done here since the 40s or 50s. I really have no idea why people bitch about it. It's like you just don't understand what's happening daily. And then? So anyway, then I ended up as a tech support guy. We're going to go from blowing things up to a bank a bank oh it's cool because uh they had uh staff that they got from the computer career institute because they were cheap and i'd be debugging their programs and it's like, ah, this should be an and instead of an or. Like, no, no, that's supposed to be an or.

Like, no, no. And I'd look at their cubicle and they'd have these little cheat sheets up of the truth tables for ands and ors. They were wrong. Oh, my God. And this is at a bank. They also didn't calculate interest properly. It's like, come on, dudes, you're at a bank. They also didn't calculate interest properly. It's like, come on, dudes, you're at a bank. Yeah, I got to go for a minute.

I got to go bury all my money in the backyard. I was just wondering which bank, and could he tell us exactly what was wrong? Because I figure we can game this system. Canadian Bank, probably. Oh, it was even crazier than that. They probably used Canadian money, too. Yeah, everything's backwards. Well, when you sell mutual funds, the guy who sells it gets a portion of the commission.

Right. And his boss gets a portion of the commission, and his boss gets a portion of the commission. So, when they're doing the calculations to disperse all these commissions, you always ended up with parts of cents left over? Sounds like the Superman plot. Yeah. Well, all those parts of sense ended up in account zero, which was owned by the guy who wrote the program. Yes. I dreamed of doing this before I realized that being a criminal might involve jail and

that would be bad. It was the plot of both Superman and and uh superman the one with um forget it and also office space they ended up shutting down the bank yeah so after the crash of the bank i ended up uh eight years doing tech support at a hospital which was pretty cool because they were on fault-tolerant computers made just up the road from you guys. And it had no unscheduled downtime for 24 years. That's impressive.

We were allowed four hours of downtime a year to do all of our maintenance and operating systems and crap like that. Wow. That's a lot of nines after that. Yes. So, what's it say here? Then I went to a high-tech startup that used electrical stimulation to help people walk

again. It's an amazing product. Therefore, the company had to die. Another medical startup making a device that helped people with their damaged lungs breathe again apparently it saved lots of lives therefore the company had to die yeah seems very familiar to me too uh-huh and we'll talk about my current gig in the third half all right uh that's quite a career quite Quite a career, yes.

Yeah. What was your favorite job? You know, they've all had their ups and downs. Working at Dow was actually pretty cool, but the level of bureaucracy was just so oppressive that I just had to get out of there. And the bank... Oh, God. That was just such a gong show. No, I had to get out of there. That sounds like something that would just make for good dinner conversation.

I mean, all the things that could go wrong. Actually, I'd have to say my current gig, because at a lot of these high-tech startups, you end up working on the same damn project for two or three years, and you just get sick of it. And what we're doing now, it's you work on a project for six months, and they go away, and you get to move on to something else that's completely orthogonal. And it's cool again. And you're a consultant.

You work for a professional services company. We own a professional service. Well, we own a product development company, electronic product development company. And do you want to get into that now? Sure. Okay. So 10 years ago, after the last thing cratered, we took the engineering team from the previous place and said, you know, we could go off and all get new jobs or we could start our own company. So we just hung up a shingle and called ourselves CBF Systems. And the basic idea is if you have an idea for a product and some

funding, but you don't want to hire staff, do the product, and then lay everybody off you call us so we tend to do stuff in regulated industries like medical devices and stuff that lives in uh explosive atmospheres around drilling of rigs and stuff like that but you know also we've done stuff like a USB device for the computer gaming industry and a little organization called DARPA. And so, like, we had our guest, Michael Worre from Nuvation, and that's what his company does, although his company is a bit bigger. Yeah. Okay, so similar. Yeah, well, of course, right when we started was right before the big crash. So, we've stayed small and busy, and we've gotten all through the last eight years of being busy, and things are kind of ramping up now. and as we are consultants although not part of a firm like that where there are multiple uh certainly i understand moving from company moving from client to client and getting to work on new

things and learn new things all the time yeah yeah it's always been something i realized early on in my career was i'm not very good at maintenance not getting bored at a company when you're working on you know the second third fourth version of the same thing oh exactly i'm good with the first and the second good with a couple of versions yeah but then it also gets depressing when you know you do the first and then they say oh hey we're shutting it down it's just business and you never get around to the second and the second is always the best one because the first one, it seems like you always make a lot of compromises

to get something out. The second one is where you feel like, ah, this is my opportunity to right all the wrongs and correct all my mistakes and do it the right way this time. Then you get maybe 50% of the way there, but it's still kind of satisfying. Yeah, and then you start at a new place

and you do it again. Yep. And so I'm going to bring this around to our stated topic, which is MISRA C. And that is a certification sort of standard. MISRA are these people who said, you know, if you only use C properly, it's not as dangerous as it is all the time. So here are some standards. And do you use it? I've had clients who wanted to use it, but I never

have had anybody actually in the end say, okay, we're really doing it. We talked about using it at one place, I remember. Well, that would be okay, too, even if they were to take the essence of it. That would probably help a lot. How about if I give you some background? Sure, please. Okay. Back in the 1980s, 1990s. We're talking Britain here

and the British motor industry. This is not the paragon of best practice. So, apparently, unlike General Motors, they actually recognized that they had problems and a

bunch of people from the British motor industry, lucas and ford and lotus and jaguar and people like that their software people got together in a big group and decided to discuss the problems that they'd been seeing in their software look at the failures and you know share even though these are different companies they decided okay we're gonna we're gonna let you see our code base take a look at the failures that they've had and dig back to the root cause of the failures and from the root causes they figured out that a lot of it was because people were getting really tricky or they were they didn't quite understand how the language worked or you know they used a particular facet of C that just everybody was consistently getting wrong.

So they came up with 127 rules for the Use of the C-Language in Vehicle-Based Software by the Motor Industry Software Reliability Association. That sounds thrilling. It's actually pretty good. I mean, this is very much like what Jack Gansel was talking about. Let's be adults here. Let's be adults here. Let's be, yeah. And most of these are checkable by static program analysis. I mean, it's not like they're all rules that are like, don't be clever,

which is hard to make tactical. These are all things, I mean, almost all, not all of them, but the 143 static rules are things like don't use malloc and don't use even int. Instead, specify with the size of your integer. Use int 16 underscore T so that you can. It's worse than that. It's test time!

Chris, what's the signedness of an int? I'm going to answer the correct answer. The signedness, it is signed. No. It depends on the compiler. An int itself could be signed or unsigned depending on who wrote the compiler.

I've never encountered a compiler where it was unsigned. Oh, hell. On a microchip PIC-18, how many bits are in a float? On a PIC-18? Cricket, cricket, cricket. Yeah, I've never used one, but I would assume there's 16, maybe 32. If I ever met a PIC-18 in the world, I'd smash it with a hammer. Smash it with a hammer.

So, actually, dear listener, if you read Elle's book, you'd know the answer to these questions. No, there's no way I put something about a PIC-18 in my book, because I've never really spent time with one. Ah, but the essence of the question is it depends on the compiler. Yeah, definitely. And I hate it when people define their own integers, integer sizes, like capital int 16 or my company name int 16. There is a standard C99 standard. Use standard int.

Then you won't have this problem. Didn't you work on a TI chip that had a really weird... Yeah, the TI, actually it's still a very prevalent processor, the F2812s, the C2000s. They're byte size, if you want a character string, everything in that string is 16 bits. So their byte is always 16 bits. Cool.

And you can pack it, but don't ever assume all you're getting at this time. I mean, if you have an array and you walk through it, you're 16 bits in a byte it's really magical the moral of the story is make sure everything is explicitly specified yes yes and well something like on a pic processor they're nominally an 8-bit processor but you know the instruction size is 12 14 but that really doesn't matter. But in the compilers, an int is declared as being a 16, even though it's an 8-bit processor. Sure, sure. Wow.

As a compiler writer, I could see making that decision because it's so much more common than an int being 8. It depends. When you're down at the low end, like when you're dealing with 8051s and stuff like that, a lot of people, you know, it's an 8-bit processor, therefore an int is 8. But we should

never talk about this. We should never have to, because we should all be just using the C99 standard ints. Ask for what you want in your type. Actually, I think that that came out of the MISRA rules. Because over the years, they've overhauled the document twice. The latest version is the 2012. And instead of saying, okay, we're going to use C89, they said, okay, you can use C90 or C99.

And a lot of it is for C99. And they say, use stdint. Don't bother declaring your own unless your compiler is C90 and you don't have it. Make your own. C90, you should go find one, somebody. Yeah. Anyway. you don't have it make your own c90 you should go find one somebody yeah anyway uh so but there so there are now 143 i had the wiki page up here a second ago yeah of these these static program analysis checkable things and there are also 16 that aren't they they're more like complexity limits.

I'm not buying the paper, not buying the book. What are those? Okay, there's basically two classes of rules. You have directives and rules. The directives are not something that can be statically checked so the directives are like number one uh blah blah blah blah blah yeah that one's boring we'll go for number two All source files shall compile without any compilation errors. How can you not agree with that?

Wow. I mean, I was expecting you to say without warnings, which some people seem to think is okay. But errors, yeah. What's the point? Number three, all code shall be traceable to documented requirements. That's FDA. Yeah, it's FDA. It's probably DOT. So, you know, just be an adult and do this.

And the static ones, the ones that are easy, I mean, that aren't process related, are things like sections of code shall not be commented out. That's actually a directive. It's a required rule. Yeah. There's three levels of rules. Let's see levels of rules. Let's see. Come on.

Okay. Mandatory, required, and advisory. So let's take a look at some of the mandatory ones. 9-1. 9-1. 15, 10, 10, 10. 9-1. All automatic variables shall have been assigned a value before being used. Oh, come on.

No, really. If the stuff is declared static, so it's allocated on the heap, it's zeroed out at boot time, right? Yeah. zeroed out at uh at boot time right yeah but if you call a subroutine and you have stuff that's allocated on the stack there is nothing in the c language that tells you that that is going to be zeroed out but people will merrily start using it well my come on was not that shouldn't be a rule it was that should be a rule for everyone all the time. It sounds like you guys want your programs to run the same way every time. Where's the fun in that? Yeah.

I've been bitten by that one in the past. Oh, yeah. And because you've been bitten by it, it's time to grow up and start always doing it. For some reason, it was a little worse in C++. There was some easier way to shoot yourself in the foot with C++ that way. All right. What do you guys think about number 15.1, the

go-to statement should not be used? Oh, wow. That's a way to make, you know, if you ever have like a team meeting. After this, we can talk about vi versus emacs yeah well that is one i love it when people like take that take that oh my god go to is a necessary thing for error handling or something i've heard cogent and reasonable arguments in favor of go to but i just would prefer that they don't exist okay let's put it this way. Back in the 1960s, we'll go way back, probably before you were born,

in the psychiatric industry, they used to give people... I don't know where we're going with this. Hey, just wait. Okay, they used to give people... They used to give people insulin and put them into insulin shock to make them comatose. Okay. Now they look back at the sixties and say, Oh God, we would never do that. I was talking to my brother the other day. He's a civil engineer and he does bridges. And he said back in the sixties, they would take a bulldozer and run it into a river and rearrange the riverbed.

And nobody would ever consider doing that. Of course not. It wouldn't be part of your 600-page environmental impact study. Well, exactly. But 1969, you had a guy by the name of John Backus, also known for writing the Fortran language, who wrote this nice little article that got a lot of airplay that says, go-to's considered harmful. So we've known about this since the 60s. So the computing industry is still putting people into insulin coma. And, sorry, rant over it.

So I guess you're in favor of this, which, by the way, I am too, because while I have seen Go 2s used effectively, the possibility of them used incorrectly is just way too high. You know, the folks these days who I hear arguing in favor of Go 2 tend to be people who came from the Linux kernel development community. Which is a bastion of really high quality code. High, jokingly quality code. But, you know, it's used a lot there for error handling and single point of exit from functions and things.

Ooh, good segue. Shall we go into Apple's goto fail? Exactly. Right. Okay, this is... But is that a go to fail or is that a brace fail?

I think their failure to use braces was if statements without braces They worked together. Well this code you can actually look at look up the original code and it basically said

if D go to fail if E go to fail somebody deleted a line so it ended up saying if D go to fail. If E, go to fail. Somebody deleted a line. So it ended up saying, if D, go to fail, go to fail. Yep. Okay, let's see.

First of all, let's see. This should have been caught in a few ways. First, don't use go-tos. Second, scan the warnings of your compiler and it would have shown you that you've got unreachable code. Yep. Right? Third, use braces. So deleting the if would have left a open brace, and it wouldn't have compiled.

Fourth, look at your diffs. And, you know, you've got this weird, why did you make a change there? How did this get through a code review? And of course, fifth, why didn't it get caught in regression testing? If we were grown-ups, it would have been better, yes. So, you know well you mentioned warnings i mean that one easy way to deal with warnings is to turn warnings into errors which yes most places i'm at do and that's quite nice because it doesn't allow for uh getting away with stuff and it it means your code can cleanly compiles well and the mizra c guidelines have been used by they've been

adopted by obviously the automotive industry as we saw did did you hear about michael bar and his analysis of uh the toyota code yes we did yeah uh they were claiming to be MISRA compliant. Wait a minute. What I heard about that code was that it was horrible and had go-tos. Yep. But one of the things MISRA allows you to do is to say, we're not using this rule. What did they do? Say, okay, there are 143 rules.

We're only using one of them. Five. Seriously? Five, yes. And two of them actually had obvious violations in the code. So three and a half. Yeah. I mean, I understand being able to say our company really believes in go-tos. We're going to use them.

Here's the rule we're deviating from and why. If that's the way you're going to do it, fine, do it. And now you've at least educated everybody in your company that you're breaking the rules but i guess we should explain for a hundred of them yeah we should explain one thing for the listeners uh you've got these rules and they say things like thou shalt not coerce an integer into a pointer. To which we all say, how the hell are we supposed to get at our, dude, we got to put a deviation in on this MISRA rule. You document it fully, and you're good to go. It's not a problem. Yeah, it's kind of like, you know, we used to use Lint extensively at one company, and there were

times when Lint would complain about something, and'd just say, no, that's fine. And you'd put a comment in which would shut Lint up. And this is a more formal process of the same kind of thing. Well, and I'm reading these, since I don't have the book, I'm actually reading these from IAR, where I have the option to enable MISRA-C and then I can check and uncheck as many of these little guys as I want. Oh, I didn't realize you could pick and choose the individual rules.

Yeah, Andre mentioned 5.1, but I'm looking at 6.5, which I can check. It's required, it tells me. And bit fields of signed types should be at least two bits long. Okay. Well, you use one bit for the signed bit and one bit for your value. Yeah. And so I can check that or uncheck it, and then it will give me an error if I fail.

Yep. And that seems, I mean, these are all seem like really good ideas. Yeah. It's kind of like the more of the bringing together of a bunch of good information rather than brutal rules that, you know, rule restrict you. And they're, yeah, they're good rules. You may not need all of them, but people sometimes write to the show and say, okay, you talk a lot about being a grown-up engineer

or doing things properly. Yep. This is one of the things we mean. Even if you aren't using MISRC, maybe you should... Look through the rules and see if there's any that jump out at you as, oh, I didn't know that. And why is that a good idea? Oh. Yeah. Although, to be fair, I have a client that wants to use MISRA-C. We're going to be moving

to it soon. And I had compiled their whole project. And since I had IAR up, previously their project compiled without warnings. And since I turned on miserate now i have 1500 errors and i suspect that's uh mostly because oh look it does say error limit reached yes i was wondering if it gave up at some point and some of these you know i'll look at them type def names shall not be reused yeah yeah that seems like a good idea let's go look at those maybe my def names shall not be reused. Yeah. Yeah. That seems like a good idea. Let's go look at those. Maybe my type defs are in two places. I'm going to take a step back here. A lot of people would look at this in the modern age, people who aren't necessarily in the embedded world, but who write software elsewhere. And they'd look at this and say, my God, you have

all of these rules that tell you how to use this language, which has all of these pitfalls. Why don't you just use a real language that protects you from all use this language, which has all of these pitfalls, why don't you just use a real language that protects you from all of this? Like Python. I wasn't going to say Python, but... Java? JavaScript? Oh, wait. Oh, wait. Didn't Java take away your ability to do pointers?

Not exactly. I've never actually used it. But there are newer things out there and they aren't necessarily being used and embedded but building there's an argument to say minimize your exposure to C and build other stuff

on top of it in something safer I don't know that I believe that I mean yeah because I've seen so much worse code written in other languages I don't know that I believe that I mean yeah because I've seen so much worse code written in other languages worse code but safer code certainly things like

buffer overflows and things like that are more easily protected against in managed languages yes so I'm not necessarily arguing this position I'm just saying it's easy to look at this and go, oh, wait, this is scary. It's scary you need these rules.

I'm thinking that one of the problems here is that we don't tell anybody which processor to use. It's sort of imputed from upon high where they want to use the minimal processor. And, well, we don't have Ada or Modula 3 for an 8051. Right. So, there's no economic backing outside of the PC world to come up with really cool languages. So, we're sort of stuck with the leftovers.

There are some embedded processors that are getting overhauled to use different languages. I know I saw on Hackaday they were doing a library for all the Python stuff, and it would run some sort of Python on embedded, and I didn't I didn't really want it but I'm sure somebody would. But that's what you're used to.

Yeah, that's very much what it means to me. And you are competent at using it. But it does strike me that maybe we've been a little insular for the last 50 years or however long C's been around, 30 years, 40 years. C's like a really great chef's knife.

And yet you could totally cut your arm off with a chef's knife, especially if you try. And yet there are ways of competently handling this if you're aware that there's a pointy end and a sharp end. Yeah, but you wouldn't use a chef's knife to make a chocolate malt. And I wouldn't give a chef's knife to a five-year-old, which sometimes people who've only seen straws... You give basic and logo to five-year-olds. Yeah, exactly.

How many years did you go to school? Too many. Chris has a master's degree. I do not. Because I spent four, sort of five years in computing science. And when I left, I started learning. Yeah. And like the people who come out of the tech schools, they feed them the latest current stuff in two years. And they do it resume builder wise, which

means you've seen it, but you don't really understand it. When you say tech school, you mean like vocational? Yeah. Not a college, but Not a university. I don't know. You guys go to college, we go to university. Okay, sure, not a university. Yeah, and you go to college at a university.

Well, there are actually differences here. We went to a college. not a university. Yeah, and you go to college at a university. Well, there are actually differences here. We went to a college. Yeah. Not a university. Because it didn't give PhDs. Ah. And that's just a detail. And it only had five majors.

Anyway. What? Yeah, we only had five majors. And you call yourself an engineer. Well, engineering was one of the majors. Engineering, math, chem, bio. CS. CS and physics.

Oh, they added one. Six majors. Six majors. Six. Cool. Yeah, that was fun. Where were we? I don't know.

Let's start again. Welcome to Embedded. Okay, so MISRA says that their goals are safety, which is one we've been talking about, and portability and reliability. Safety and reliability, I'll put those in the same bucket and say, those are great. We should talk more about those. But portability is not something we talk about as much it was something i felt like i used to do and now people choose a processor and it's an arm core and we don't talk about portability so much always a goal at the start of my projects

that ended up going by the wayside portability yeah um we still one client that i've worked for for many many years i used to work for them full-time, we started with that in mind. We started on Windows, it was a medical device. All the engineers were like, well, we're just doing this because it's expedient, but we have a goal to move to some other operating system, Linux, Unix, whatever, that we have more control over.

And it never happened. But still, when I talk to to them there's conversations about well we can't do this exactly because that would break our portability story which is ridiculous because they're never going to port it to anything else but um well back in the bad old days you used to have you know on linux boxes you you had this very rich standard library and you could take code that was compiled for a VAX and throw it up against a Sun workstation. And there might be a few little deviations, but you could get it going pretty quick.

I think Microsoft sort of screwed that all up. Yes, yes. And on the embedded side, God, every time you get a new processor they hire somebody to put together a compiler for them and you've got all these deviations like i at one point we were doing a medical device and you know one of the things that you'd have to do is justify all of your compilers. So I tried to find a C compiler for a PIC that had gone through the Plum Hall conformance tests. And I called up the companies that made the compilers, you know, due to the structure of PIC processors, there is actually no way to have a real C compiler.

So what we do is compile something that looks like C and it tends to do what you asked it to. But as for portability, no. That's the weirdest thing I've ever heard. Yeah. What possibly could prevent them from making a C compiler? PICs don't have a stack. I was thinking it was their interrupt handling.

You can't actually do things like, well, you've got eight registers that are effectively the stack. So you can go eight levels deep with one parameter. Yeah. So good luck with trying to get something compliant. Thanks for telling me that. Now I have more arguments against people choosing picks. Oh, do not. Confirming your confirmation bias.

Yep, that's why it's a confirmation bias. We'll talk offline. Okay, and then, you know, you're not going to get a SPI implementation on a cold fire that is portable to an arm. It's sort of similar, but you're going to have to put a layer in there to make it portable. Oh, yeah. I mean, the drivers aren't going to be portable. No. Because the registers aren't. Well, even beyond being in different places, the cold fires has an awesome SPI where you set up a queue of 16 requests

and you say, go do that and let me know when you're done. And it's a whole bunch different from bitbanging on an 8051. Yeah, so portability in that sense, I wouldn't expect that to mean your entire code base must be portable, but to mean if you strive for portability in that sense, I wouldn't expect that to mean your entire code base must be portable, but to mean if you strive for portability, you make things like hardware abstraction layers.

Yeah. Okay. And, I mean, portability goes back to using the standard int.h. Yes. Yes, at the most minimal level. Oh, that's one thing. MISRA has absolutely nothing to do with design. Right.

This is an implementation tool that should be, like, if you're using something like a Keel compiler that has MISRA checking built in, huzzah. Other than that, get yourself a copy of PC Lint. That's PC Lint. It's just awesome. And use it thoroughly. Yep, I agree with that. Although... I think it's not very expensive either.

No, not really. Not for what you get out of it. I hate to say this, but I have tried to use PC Lint a number of times, and each time the setup has been too much. And I'm embarrassed because I am one of the people who's saying, you know, fight through these things. It's worth it. And yet, that one, it's so difficult to get started that I have trouble with it. Maybe now it's not, I don't know. I actually haven't used it in a long, long while. I haven't, yeah.

It is. It's still like that. And it's brutal. Which sort of gets to the point of your upcoming person needing to or wanting to use misra if they are in a uh an environment that requires it then well okay there you go if they want to use it for bragging rights well great but i'd say use the force anyways, because it's a good thing. But, you know, if you don't happen to reach conformance, so be it. The people at the Jet Propulsion Laboratory, it's very cool. They have a set of, let's see if I can find this.

What is this? While he's looking, I will say that PC Lint by Gimble, which is the normal one, is about $400 per seat. I thought it was cheaper. There may be other Lints. There are, and much of PC lint has been pulled into gcc okay yes if you put all warnings on okay so at the um at the jet propulsion laboratory these are the guys that uh build satellites and rovers those neat little robots that are roaming around Mars. They sort of went through the same effort as MISRA did.

They took a look at their code base, their failures, what they did wrong, and they tried to do something about it. And they came up with a set of rules. There's only 10 of them. And they're in four levels of conformance. Sorry. There's six levels of conformance. Level one, language compliance, which basically means use a C compiler.

And all of your code has to compile without warnings and it has to lint clean. Yeah. The second level they call predictable execution. The third level is called defensive coding. The fourth level is called Code Clarity. The fifth level is called MISRA 2004 Shell. And their highest level of conformance is MISRA 2004 Full Compliance. So the JPL has MISRA Compl compliance as their highest level of code quality.

Cool, huh? That is pretty cool. I didn't see it. I didn't hear an entry for make sure kilometers and miles are not used interchangeably. But that's me. Are you going to make me look it up? No, no, no. Well, no, because that's a design in human thing. That wouldn't be a

code thing. Right. And that's the point you made, but maybe we need to make it again, is these are things that static checkers are not going to make you into a good coder. You can still write the worst program in the world and have it be lint clean and

MISRA compliant. On the other hand, when people have truly, oh my god, I don't know what to do next, it keeps crashing bugs, that is a place where after spending a couple of days thinking about it and banging your head against it and wishing it all got better, that might be a place where PC Lint or MISRA, you turn it on and you see this file that I'm having so much trouble with.

Is there anything weird about it? Maybe I should turn all the warnings on high. And that's where you find things like, I didn't initialize a variable or my memory's overlapped. It wouldn't have helped with your particular problem. Or I don't think there's a C warning for you left thelapped. It wouldn't have helped with your particular problem. Or don't think there's a C warning for you left the alpha blender on

when you were trying to write to a RAM. It's called a blender. What did you expect? It was going to blend. Yeah. I expected it to treat the TFT and the RAM bank separately. But something like lint will pick up off by one errors. See, now that's nice.

Yes. I sat there and I stared at this error for a while and it finally, it dawned on me, it's talking about something that I passed in and went back up the code and it's like, holy crap, it does have a value of nine instead of eight. And it's like, ah, you moron. Yeah.

I mean, even if you aren't willing to do this all the time, it is a debugging tool. Oh, yeah. Run it once a week against the code. That's what we did at Reliant, where we used Lint extensively. It was one of the guys just kind of, every week, he'd run it against the entire code base. And then he'd come over to you and say, well, you see this here?

What are you doing here? And you'd look at it and go, oops. That's one way to do it. I get an email from a company in Orange County that is looking for embedded software engineers to work on some neat medical neuro thing, and my response to them was, oh, oh i'm gonna forward your email to the guy who used to do that the pc land guy like yes you want software quality that's that's the guy

it was fantastic especially well i've done the muscle stimulator stuff and um you don't want to burn people. Yeah. That's a whole other show that I that company's gone. I probably can't talk about it. Yeah. Well, remember the product was designed to burn people. It's just you don't want

to burn them too much. Wrinkles. Wow. People are willing to go through a lot to get rid of wrinkles so including my software you did say that you don't use this for every product i always think it's a great idea and yet don't actually it's like unit tests it's motherhood and apple pie. Yeah. Well rights it's pretty much it's not worth getting uh pedantic about the whole thing and let you know if if this is something that somebody's going to pay you extra because it's you're it's time okay it is it is and And the way I see it, it's sort of like the, it's the difference between being wise and being clever. Do you know what the difference is between being wise and being clever? Wise knows when to stop being clever.

Yes. I was headed towards some variant of that, yes. Yeah. And the thing about C is it has very few guardrails built into it. And people are forever doing really clever things to show how leet they are. And, you know, the poor bastard who has to maintain this stuff six months down the line he's just going to delete this stuff anyways and do it in a clean fashion because you know even if it it it might be you yeah yeah that's that's my first instinct when i come across legacy code that i do not understand is to just throw my hands up and say,

well, I know how I would do this, so why don't I just do that? It would take me less time. Writing a different set of bugs. Well, it is, but it's at least a set of codes you can continue to maintain. Yeah, and there's two people that are going to be reading this thing.

The compiler, who doesn't care, and the guy who has to maintain to maintain it so but you have used msrc in projects yeah fda projects yeah any others uh well it's one of these things where you finally get the thing working and then you start linting the code. And it starts throwing a whole bunch of flags and you say, ah, yes, yes, yes. I forgot about that rule. Yes, you're right. And you fix your code.

Or you just say, I'm going to give myself a deviation and turn it off for that structure. Like, sometimes you just want to use unions. I use them all the time. Yeah, that's

in the rules. Don't use unions. They are easy to missuse. I'm not Mr. Compliant. Well, you could be if you decide that this is a good thing. You write a deviation. Write a deviation and you're done. And you're compliant again. This is the time I get to make jokes about deviants.

Oh, yeah. So I had a friend, I had a dinner with a friend last night, and he was working on code that during development they believed it was going to go into consumer products. And that was fine. That was how they wrote the code. They're happy with the code. It's pretty effective. They have a nice test suite. Good process.

Good process. But he just found out that they may be going into automotive and probably not the mission critical stuff but now with all of the self-driving cars coming and this is a long-term sort of thing he's concerned that his code might kill someone. And I know I've been through FDA, I've been through FAA, and those are definitely, I was sat down by one of the FAA guys and said, you understand why we're so serious about this. And it was really brought home that killing people is bad, and if your software does it, it's still bad.

Doing it accidentally doesn't make you feel any better. Is Misra the way... I was trying to figure out how to tell him. And I babbled some stuff, but I couldn't figure out how to tell him how to do it right. Well, I was watching a video the other day from the oh i can probably even give you a name it's from the uh jpl and he no i don't have his name anyway there's a video on the tubes that uh the guy's explaining their um uh uh software development process at the JPL and how they go through all these incredible amount of static testers to the point where they have nothing that they can detect anymore before this stuff is declared ready to go on a rocket to Mars. And given their new script on how to write code, they don't know of any bugs on the you know this is this is the uh mars rover that did the sky crane

thing yeah that worked they could not test that on earth they just had models and they knew that the code as far as they were concerned had no bugs bugs because they had beaten the snot out of this thing. And there was, like, you take a look at the code, and yes, this is obvious what it does. And they went through all of their code. And yet, I'm sure they still had bugs. And I know of several, because I follow one of them on Twitter who worked on that.

And they mention issues from time to time and they come across things. So nothing's ever perfect. Yeah. And yet they knew they had to take all of these things because they couldn't go touch it when it went wrong. Yeah. And even to update the code, which they can do, it's a, what is it, 16-minute round trip or something like that? Or more? I think it's more.

It depends on which side of the sun we're both on, but yeah. Wouldn't that be kind of cool to say that about your code? I'm sorry, it just depends on which side of the sun. So, I guess it's the difference between hubris and humility. Don't be too proud of your

code and you've got a lot to learn. Don't be too proud of the technological terror you've constructed, Lord Vader. I probably got that wrong. So, we mentioned ethics towards the end of last week's show. chris said it would have to be a seven show and so you're gonna bring it up now at the one hour mark of this show yeah i figured

we'll do five minutes every week for a few months and we'll total it up okay go do you have uh so you were one of the people who fussed that we sort of just left it at nothing. Where would you start with that? Well, the question that you were supposed to ask me is, are you a software engineer? Right. It's in the notes. I see it right here. It's in the notes, yes. I'm sorry.

Let's just go back and leave all that in. Are you a software engineer? Absolutely not. Yes, I'm sorry. Let's just go back and leave all that in. Are you cannot call yourself an engineer unless you have like straight out of university, you take an extra class in ethics, and then you write the ethics exam. At that point, you become a professional engineer. And you get the you get the pinky ring and the groupies and the big bucks. If you somehow you went and got your MBA and you screwed around in small business for a while and then you decided, oh, I'm going to go back and get my P-Eng. What they do is they take a look, say you're a mechanical engineer. They might say, okay, you have to redo statics and dynamics and strengths of materials and stuff like that, and then write a technical competency exam and the ethics exam we can make you a certified Microsoft systems engineer.

The Canadian Engineering Society went ballistic and took them to court. And now it's a systems expert or something like that. Is that just for Canada? Because I know Cisco has a certified Cisco internet engineer, similar thing. Not in Canada. Okay. Not a chance. So I don't know, didn't you guys take ethics in your last year of uni?

Nope. No. So you couldn't call yourself an engineer in Canada? No, there is a professional engineering thing in the United States, but it's pretty much civil engineers only who take that. Some mechanical, but mostly civil. But there's no legal sanction against using the term engineer. And I don't think the PE exam is really, I think there is an ethics, but it's a lot more civil engineering questions. In Canada, it's uh ethics so in canada i could i could actually go and

become an engineer but uh i would have something called a restricted practice so in my case it to probably software and embedded software. So I couldn't go and stamp drawings for bridges and stuff like that. Which would be okay. Yep. And most of the ethics, I mean, hold paramount the safety, health, and welfare of the public, avoid deceptive acts. I'm sorry, but these are sort of like the miserable rules.

These seem like good ideas all the time. In Canada, an engineer can walk by a construction site and say, that thing isn't designed properly, and call up the appropriate authorities, and they have to shut down the site oh this sounds like fun we need this for software well it's not just they can do that if they see it they they must they have to do i want to find a bug in my iphone and shut apple down you might be able to let me touch it i'll crash it for you

might be able to complain to the fcc touch it. I'll crash it for you. Might be able to complain to the FCC. So this is interesting, though. They have this term engineered in Canada that has legal power behind it, and an exam, an ethics exam. You are not constrained from working on all kinds of things that require or should require an understanding of ethics. But if I do stuff that is, if somebody determines that I am doing engineering,

I can be taken to court and fined. So what's their definition of engineering? Probably design that has something to, designing something that can affect public safety interesting so even in our little business the engineers that i work with uh sometimes they say hmm we're gonna have to make that determination because you don't have the little pinky ring and you might be construed as doing engineering. But didn't you mention working on truck scales? Yep. Wouldn't, I mean, if you did that incorrectly or you did that deceptively with a backdoor to, I don't know, make it so that your friends don't weigh anything.

Yeah. That would affect public safety as it might cause the collapse of a bridge. Yes. Yes, in that case, you're correct. I was thinking, eh, it's pretty benign, but no. And I was actually working for an engineering company at the time, so they would take the responsibility for my actions. So aside from the legal point, it does seem interesting to me that

there is no, at least a college requirement, if you're going to be entering the design and engineering world. I mean, I had nothing. I mean, I had a philosophy course in freshman year. But I don't think Kant had a lot to say about the college we went to was humanities I mean we had a third humanities

and I have to I mean a lot of that was kind of talked about and we talked about the honor code and we talked about but it was nothing specific how about conflict of interest or taking money to sign off on plans that you know weren't correct? Well, that's, I mean...

Doesn't that fall under career limiting? Only if you get caught. Jeez, of course you're going to get caught. You are going to get caught someday. But, you know, I didn't graduate with an engineering degree. I have a math degree. So if it was an engineering requirement, I wouldn't have had it. Nobody gives a mathematician

an ethics exam. I went through almost all of the engineering curriculum and didn't have it. There was nothing. No. I also do not have an engineering degree, but I went through most of the curriculum. I think the hard thing about ethics is not these clear-cut things

like bribery or turning a blind eye to safety issues. There's a lot of gray areas. I was listening to a podcast about lying and Enron and how that happened, and it wasn't one giant lie, like it appears. It was everybody not even, I mean, they were being too, everybody was too optimistic and ever more so. Nobody was the skeptic who reined everybody in. There were times where we were asked at a medical device company to slightly cut corners. Can't you just do this? Can't you just do that?

And that is kind of the thing where if you're a young person in an engineering organization and you're being asked by upper management to make it a little cheaper, or get that done a little faster, and if you just do this, it's not a big deal. Your unit test doesn't need to check the error cases they'll never be seen we we just did this release i know you just fixed something can we just push it out really quickly can we just hex edit the version numbers well no but you know my let's just i've been asked that let's skip through this test yeah those are the kinds of things that you might not realize are an ethical question, and they are.

And in that circumstance, we always pushed back extremely hard and said, no, we're not going to do that. We had a little joke that none of us looked good in orange. Well, in that case, it would have been the boss that would go to jail, though. That is one thing about the FDA. But the consequence would have been to a patient, which none of us wanted to live with either. It's all well and good that, well, we're not going to go to jail, but

I don't want to hurt somebody. But that's an ethical thing. And this is my work. I want to stand behind it. Or moral. I guess that's moral. Well, there's kind of an overlap. One of the companies that I worked for previously, one of the medical companies, we were working with a gas known as nitric oxide.

This is N O. Yes. It's a, um, it's not the laughing gas. It's, it's cousin. It's smog.

Um, that's N O two. Damn. Okay. Uh, what N O is it's, it's really cool. It's the stuff that gets, um, gets, you make it yourself for healing.

So when you get a cut, your body naturally generates nitric oxide. But you can also get nitric oxide from sticking a nitroglycerin tablet underneath your tongue or by taking viagra it's all the same thing one of the side effects of it is that it relaxes your lungs so if you have a baby that's born prematurely they have uh i think they call it blue babies, because one of the last things that happens when a child is being produced is that their lungs are formed. So if you're born premature, you've got these premature lungs, and they don't really transfer oxygen awfully well. So what you can do is give them a shot of nitric oxide, and it relaxes their lungs, and their oxygen uptake goes through the roof. And if you are a little old guy who's smoked your whole life,

your lungs don't work anymore, and they give them nitric oxide just to, you know, in some cases, if you turn off the nitric oxide, these people will crash and die. So our device was dispensing minute amounts of nitric oxide into the breathing air of these people who couldn't even breathe for themselves anymore. And the people on our board at the beginning of the board meetings would say, according to the reports this month, we have saved 23 people just to set the context for why they were doing this. So when you're writing code for that, you just sort of keep in the back of your mind

little babies and somebody's grandfather. If I don't do my job properly, they could die. I think that was what bothered my friend, was that he didn't write it with that intention, that thought. And his whole team sort of feels this way. And now it may be in that circumstance, and they don't know what they would have done differently if they had that frame of mind, that idea that their code was important, critical. So do they have the opportunity to do a second pass to clean it up? I hope so. I think so. That was what we talked about. What would he do

on the second pass? What changes would he make? And we talked about the FDA process and calling all of his current code prototype and then using a good process to go from that to something he felt more comfortable with. And how these processes are not only annoying documentation fest, they are also a way of building more confidence that what you're doing is the right thing

and that you aren't alone in your bugs, that you have not distributed the blame, but really recognized that more than one pair of eyes is a good thing when it's important. Yes. I could see that being a tough sell, though, because if you're in an organization that's not used to doing that, it's the low-level people are the people who realize we have to take a second pass at this.

It doesn't mean management does. So now the rank and file have to explain, you know what, we're going to have to take another year to get to this point. Well, one of the things that you have to do with all of these projects is look at who the stakeholders are. And in automotive, you would have to comply to ISO 26262, the functional safety standard, that spells out a lot of this stuff. So, like, automotive is particularly weird because the power is just crap.

Like, you got plus 200 volt spikes, you got negative 200 volt spikes, and your processor has to keep on running. You probably heard about the R-series processors from ARM. Yes? Let's pretend we have, but we haven't. Okay. Well, something like the A's, you got the A's, the R's and the M's ARM. Okay. The A's you end up with in things like your iPhone. Yeah. The M's are the stuff that we deal with. The R's are the highly reliable ones. Oh, okay. okay and what what they do is they take the basic

processor and they double it so you've got two cores running in lockstep and then they have a checking mechanism that checks to see if both of them agree on the calculation but what they do is they take one of those dies and they turn it upside down and they rotate it by 90 degrees very clever so if you get a uh electro uh electromagnetic pulse coming through you might uh well if both of them were the same and the same pulse comes through you might end up with the same error in both cores. But if they're upside down and rotated, one will get upset and the other one might get upset, but it'll be in a different manner. And the lockstep mechanism will throw a flag and you shut the system down. So this is the kind of crap that they're moving to in automotive because they've made the effort of making their stuff more reliable. Now they're

sort of stepping up their game a little bit. And one of the things is that this stuff is filtering down into things that you wouldn't really expect, like hot tub controllers. Okay. So if you upset a processor in a hot tub controller, you end up with really hot water that people jump into in their bathing suits. Really, people, put your into in their bathing suits. Really, people, put your fingers in first.

Exactly. But it's the sort of thing that you want to filter down. I mean, I remember reading about servers with ECC RAM and wondering, well, why don't I have that on my computer? I don't want bit errors. But ECC RAM is not one of those things that is filtered down to consumer devices. It was there. PCs used to have that a while ago. Yeah.

And then they declared it to be too expensive. That was sort of one of the bragging things that PCs had that Macs didn't. And then eventually they were going for know, going for the bottom. So they jettisoned the ninth bit. Like there's, I'm sure that there's a whole bunch of protocols there that don't use any error checking. Just, you know, you pass a message back and forth and, well, we've never seen any errors. So therefore it must be okay. Ow. That's a be okay. Wow.

That's a design thing. Yes. Well, we've kept you for over an hour at this point, so I only have a couple of questions left. I do want to say that you were one of the first listeners who wrote in to the show that I didn't personally know already. And it was really neat and very nice. Well, I heard your podcast get mentioned on that other podcast. The Amp Hour.

And I said, you know, hold on. I got that book. I should really read it. So, you know, I got that book. I should really read it. So I go through it and I figured that was the first book that I've ever written notes in the margins. Books are amazing. You can't write in them. At the end of it, I did the Sheldon Cooper thing and I sent you in a rattle list and you ended up getting that on your birthday in 2013.

Yes. But in with the errata was lots of compliments. Well, yeah. So, you know, it's always easier to take. And you'd seen most of it before. So what should we highlight for the 100th show? I liked show number eight with the Ballisticats. Who are those guys?

Well, show number eight was Chris on drums. Chris is a drummer? That was my band with my brother. I thought he was a musician. Oh. I also play bass and piano, so that covers the musician part. And by the way, we do

have a little bit of intro music, and hopefully soon we'll have a little bit of outro music. You keep putting the pressure on me. Yeah. The Ballistic Cats are my band and my brother, and we were around for quite a while. I ended up leaving a few years ago, and he carried on. But I think they're on hiatus permanently now, or semi-permanently. So what is your intro music?

It's just some weird stuff I cooked up with a synthesizer. Oh, that's cool. I really like it. There's something like the Engineering Commons. Yeah. Their music just bums me out completely.

Yours is cool. We may have some more stuff soon. All right. I'm not supposed to pressure about that.

As soon as my back stops hurting. Come on, man. Synth. Moog. So the eighth with the Ballistic Cats.

We had two with the Ballistic Cats, I think. Yeah, I wasn't on the other record. Yeah, the other one you weren't on. All right, well, we will mention that one. With this survey, I'm finding out that half of the listeners hated the cat episode and the other half just loved it. So that's just bizarre. I did listen to the whole thing.

I was vacuuming. Yeah. I can't believe anybody listened to all of it. But I know some of you did. More work than I've ever put into a podcast episode. And poking the cat over and over.

That was her job. No, now I've taught the cat to come to me whenever she wants a snack and meow, which was not the smartest thing to teach the cat. We've trained our cat to be extremely loud over the past couple of months. It worked out well for everybody. Except at four in the morning. Yeah, exactly. Usually she waits until five.

So, Christopher, do you have any more questions? Why do you do this to me? I never think about it. You know, you do this every time. You think I'd be prepared. You never think about it? That's true. We were going to ask you to say about, weren't we?

He did a few times. I got something for you. He also said processor and the JPL. Sorry. Right. You have that thing, those things that go in the water, not ships. Boats? Oh, you can say a boat.

Nobody in Canada actually says a boot. If anything, that would be something like Newfoundland Maritime, you know, Jimmy from Digby kind of stuff. But no, nobody says a boat, a boot. We say a boat. And the closest that Americans can hear is a boat. So, you're saying there's something wrong with our ears. Boats and a boats. Well, it's like Bjarnestra. Oh, man.

It can't be pronounced. I'm never going to live that one down. And I deserve not to. I had to go online to find out how to pronounce it. And they do this epiglottis flip that no humans can actually do. All right. Well, I think that will cover it for Christopher's question period. Andre,

do you have any last thoughts you'd like to leave us with? I'm so glad we had this time together, just to have a laugh or sing a song. Seems we just got started, and before you know it, comes the time we have to say, so long.

Wow, that's cool. That's the first time before you know it comes the time we have to say so long. Wow. That's cool. That's the first time anybody's ever sung a song on the show besides the cat. Meow. All right, Andre. Thank you. It has been wonderful talking to you.

Just chop it to bits. Nope. Nope. My guest has been Andre Chichak, a systems developer at CBF Systems Inc. I am so happy he was on the show. And as always, I want to say thank you to Christopher White for co-hosting and producing. If you ever want to have a podcast that sounds really, really good, my suggestion is to find a producer who loves you.

I'm sure that'll be easy. And thank you for listening. And to those of you who, like Andre, have written encouraging emails, a really big special thank you. If you'd like to say hello, hit the contact link on embedded.fm or email us show at Embedded.fm. I do have a few stickers.

I will be in Pasadena at the Big Hackaday Party on May 8th and 9th, 10th, that weekend. And we've listed my other conference fund, so I won't bother. I think that instead of a final thought relating to the Great White North and Geddy Lee, I will be going with a fact, one you may not know about nitric oxide, NO. It was proclaimed Molecule of the Year in 1992.