Embedded - 211: 4 weeks, 3 days

Episode Date: August 17, 2017

Dennis Jackson spoke with us about making the career shift from software to embedded. Dennis buys James Grenning’s Test Driven Development in Embedded C for his new hires and often recommends Elecia’s Making Embedded Systems. His tip that everyone should know was “Learn make!” and he has a reference for that: Why Use Make. He suggested Joel Spolsky’s reading lists from Joel On Software, even the ones that don’t obviously apply. Additional suggested-reading articles: 30 Pitfalls for Real Time Systems (part 1 and part 2) Rules for defensive C programming Why are you still using C What every computer scientist should know about floating point arithmetic The Power of Ten -- 10 Rules for Writing  Safety Critical Code . In his previous appearance on Embedded (#25: Don’t Be Clever), we talked about code complexity and measuring cyclomatic complexity. At that time he wanted a tool to monitor the code’s status. He has since found one: pmccabe. Dennis currently works at Element Science.

Welcome to Embedded. I'm Alicia White alongside Christopher White. We get asked quite a bit about how to get into embedded systems and how to get a job in embedded systems. I'm pleased to have Dennis Jackson on the show to discuss it with us. Before we get started, though, the dogs are doing well. Thank you for asking. The thing we didn't get to talk to Alan about was the Hackaday Supercon in Pasadena in November.

He'll be there. He may be speaking. And you can be there, too. Tickets are on sale now. Hi, Dennis. Welcome back. Hey, Chris. Hey, Alicia.

Thanks for having me back on. It's funny for me to have to ask you to tell us about yourself because I feel like I know you pretty well, but that's kind of not fair to the people listening. So could you tell us about yourself? I can, and things have changed a little bit since last time I was on. But I'm currently the director of firmware at Element Science, a medical device company here in San Francisco. One of the few in San Francisco, which is pretty awesome. For those paying attention from the last episode, my math was a little off, but I've been doing this for two decades now.

And I enjoy every aspect of it. But I've been working in medical devices with a little stent in drones for a few years, for all those 20 years. And my goals are always to make sure the software going out is really good, really solid, and make the world a little bit better every time a release goes out or every time a new product goes out. I like to stay focused on medical devices because of the safety-critical aspect of it and also just the helping of people. It's just the thing that I started out. I got lucky and I got started in medical devices and stuck with it.

And you were on show number 94, Don't Be Clever, which was a great show that starts out with a lot of discussion about drones and airware. And it was before the FAA had gotten that certification process. Which is now null and void. Which is now null and void. But we spent a while discussing what it might be, only to find out it was never going to be. Yeah. So I'm looking forward to talking more about the code side of things.

Before that, though, we have lightning round and we have a lot of lightning round questions. So we're going to need you to be very fast. All right. And I've never done lightning round, so I'm actually a little nervous. You listen to the show. You know it's coming. I know. Okay.

Favorite, we were just talking about this, favorite movie or book which you encountered for the first time in the last year? Well, there's been a bad year for reading. Favorite movie. Any fiction. Any fiction. You know, I actually read Handmaid's Tale. And I know a lot of people didn't like it, but I thought it was a really interesting view of the way things can go south really quick in a society so i think that's going to be it cool favorite tool but not software i want a physical tool hammer is a really nice tool

especially for fixing hardware best perk you've ever had at a job? You know, just flexibility. Being able to take the time to do things with family or, you know, I need some time off to take care of something and I'll make it up another time. If you woke up tomorrow as an animal, what animal would it be? Well, funny thing. I was thinking there might be an animal question i know these aren't fast uh rhino it's a good one preferred voltage three point uh no uh yeah 3.3 favorite airplane oh the sr71. Although the Piper Arrow is pretty nice. I've flown those before. If someone gave you $10,000, but you had to spend it today, what would you spend it on? I'd probably buy a vacation.

Okay, technical tip you think everyone should know. Learn make. That's a good one. Chris, you had one more? Favorite office arrangement. Not open. We knew he was going to say that. Okay, so we got in a discussion not too long ago about people wanting to get into embedded systems. And once they're kind of there,

how do they get a job in embedded systems if all they have on their resume is computer science or software in their past? And I want to talk more about that with you because I know it's something that you come across pretty often. Yeah, it happens quite a bit, actually. Either current co-workers, former co-workers, people who have been trying to submit resumes to where I'm working,

or friends of friends who say, hey, I hear this is what you do, and how do I get into that? So what do you suggest? What's the first pithy, easy thing they can try? So it's been a huge change in the world uh in software development especially embedded development in the last 20 years like being able to access some of this stuff um you know raspberry pies um discovery boards all these things just didn't really exist 20 years ago um you can get one of those today and start writing something. Try to make something

that you want to see. Usually what I say is, is there something in the real world that you want to see interacting with? Do you just want to turn lights on? Do you want to measure temperature in your house? Yeah, you can go out and buy a thermometer. You can go out and buy a thermometer you can go out and buy um you know a light but this is something if you want to if you want to try it and you want to write some code for it this is something you can get you started with something very simple we totally agree of course if you want to get into embedded systems the first step is to try it and these systems are super cheap. I haven't used the Raspberry Pi, but I've used discovery boards and you can find them for $20. That's an easy, cheap way to get started. The compilers are free. The tools are free. There's a lot of resources out there. I think the Pi Zero

is $5. So yeah, cost is not really an issue anymore as long as you have a computer, which is probably the higher cost. Yeah. So the discovery boards, one of the hard things with them is that it's a big opaque system. I mean, we talk about Arduino and I love Arduino, but the discovery boards are a little closer to what you might use in a professional environment. And yet people get pretty stuck on them. What do you suggest for compiler and just development environment? So for those ones, the vendors have tools that you can start with.

So with the discovery boards, you can use ARM, you can use arms Kyle compiler and that comes, there's a free version of it. It's it's a, it's a decent compiler and decent IDE. It comes with sample code. So you can turn the lights on. They have accelerometers on there. So you can do some basic things with accelerometer. They have buttons so you can do basic interaction like that. And they have samples of all this. you get that board you get you download the compiler and you can at the beginning just compile and run it make sure your your computer's set up make sure your compiler's set up

the connection to the board you have power to the board they're usually powered over the usb so once you have that on you know you're going to be able to talk to it. And then once you do that, then you've got the start of it. And their sample code is good enough to give you an idea of what you should be doing. Now, you're not going to turn that into production code, but it's good enough to start out and start learning how to play with the board. I would add Embed to that. Embed is an online compiler. You don't have to install much, maybe some drivers. And many of the discovery boards are embed enabled. Yeah, I've heard about that.

I haven't tried it. It's kind of nice because in an Arduino-like fashion, they have huge numbers of libraries and example code, so you don't have to write everything from scratch. It's also very C++-oriented, so it's not as much just straight C that you get in the tutorials from the Discovery Board. Yeah, and the embed boards are more capable than Arduino. They have networking, a lot of them on it,

so you can do IoT things if you like. IoT. Why are you making a face oh because the arduino boards are so big now that the default arduino so you need to get a shield and stuff arduino unos are kind of brain dead but that covers so much now and the embeds cover the same things okay maybe it's funny it funny because I, I know these things exist, but I haven't, I haven't used them. Cause typically I'm working on a project that already,

that, you know, we're using, usually an arm core somewhere in there. And, and we have, we have the, you know,

you know, industry tools on there. And usually there's an E on the team or someone else on the team. It says, Hey, we want to try this new sensor out on ball. By, I got an Arduino and wrote something to try it out. I'm like, oh, you did that yesterday. We talked about the sensor yesterday and you already wrote it.

I'm like, that's pretty awesome. Arduino gives you a lot of speed. That initial prototyping is really useful. But then we go to discovery and other things that make it faster and cheaper. Faster to run, not faster to produce. Our new college grad in CS has a day job doing software that he doesn't really like and he wants to do embedded software. He got a project or she got a project and has decided to blink some lights cool I mean that's that's a tutorial there are lots of examples on how to do that out there what next I mean they

can't put that on their resume and say hire hire me. Yeah. So this is an interesting story. A couple months ago, I was hiring from where I am now and I had a resume come in and the resume looked really good. Definitely junior level, but really good. And I talked to him on the phone, did a phone screen and answered all my phone screen questions really well. Like this, this is, this is a really good potential candidate. So let's bring them in, brought them in. And as we started talking, uh, I was the first interview of the day and the whole schedule lined up and we started talking and just, I don't know. I don't remember. I unfortunately don't remember exactly what I asked, but I asked something and the way he responded, I thought, hmm, that's not in line with everything else I heard. So I said, let's, let's just write a little bit stuff up on the board. And we started doing a little bit on the board and within, within 30

seconds, I knew, okay, this guy has a potential, but he's not going to be the guy I'm going to hire. And I said, hey, hang on a second here. I just got to stop you. And you have a lot of potential, but this isn't going to work out. But we were going to talk for a while here. So I went out and told the rest of the interview team, this isn't happening. We stopped the interview. And I went back in and talked to him for another hour and a half. And I just basically broke down, here's where you are, here's where you need to go. And he, he was in the fortunate position of being at a company that was doing embedded work. He just wasn't on that team. He was, he was helping test and he would help, you know, find bugs. But as, as we're talking, I said, well, you know, you guys have a version out. You're probably going to have a new version coming out. And he said, yeah, you know, we're actually, the processor we're using is coming end of life. We're looking at new ones. You know, we've heard a lot about ARM. And so we're thinking about going to that. And I said, this is a perfect opportunity for you. Go buy Discovery Board, you know, get the company to pay for it. It's 20 bucks, but worst case you buy it yourself.

And you already have a list of the sensors that you're using. You can get samples of those. You probably have them in house. You can talk to the EE and say, Hey, can I get one of those? And you know, I don't want it on the board, but I want to be able to hook up some leads to it and hook it up to this board. And that is the, you can take that, get the sample program and write, you know, either look at sample drivers or start writing a driver that talks to it. Get the data sheet. And you can use that as your starting point for developing yourself. Because now you're looking at new hardware. You're doing the embedded work that you want to be doing that you already have some experience in. You just don't have enough experience with the writing of the software part

of it. And you can use that as your starting point. So to generalize that, it's another thing you can do is say, hey, you know, is there a project at work that is using embedded software that you want to get into? Or, you know, if you're not at a company that has embedded software, but that's the role you want to switch into, is get some hardware. Get a couple sensors. Like we said, you get the Arduino, you get any other parts like that, and you can download drivers and make them work right away.

And that's fantastic for rapid prototyping. But when you go to that stm part which you will more likely be using on a production system you're not necessarily going to have the driver for you know the current limiting chip or the current reading chip or the accelerometer or you know the temperature sensor or any other sensors on there but get something that you can you can play with and use and look at that data sheet and write a very simple driver for it. Everyone says, well, you're working in bed. I got to worry about power usage. I have to worry about size limits. I have to worry about putting the processor to sleep. It's like, you know what?

That board's going to be sitting on your desk. It's not going into a product. It's not going to be running off a coin cell battery that has to last six to 12 months or longer. It's going to be plugged into your computer running whatever speed that you set that thing up for. Just run it at max speed, run it at one of the low speeds. It doesn't really matter, but get it running. Get that driver written and make sure you can read it. You have the data sheet there, and usually there's a, you know, if it's I2C, if it's SPI, even serial, there'll be usually an identification byte. You ask if it's, especially in I2C and SPI,

you send a command and it sends back to you, like, this is my ID. Get that command working first. Then you know you're talking to that piece of hardware um you know but if you're using the accelerometer get back x y and z and now you can pick that board up and you can start moving it around you write a little something that um you write something on your in your processor that that it continually reads that and you know do you have printf do you have you know are you using debug an led an led yeah some way of getting output back on to say hey the thing is moving and it's reading now the other thing is oh i have to worry about interrupts i have to worry about all this stuff just pull it that's you know that's a very very simple way to get

started do the simplest thing possible. Then you can graduate into those other methods to make it more efficient. And more likely, the sample programs are going to have examples of interrupts and DMA transfers. That's intermediate and advanced level. Just get it talking. So I would say, just to add to that, if you're going to be playing with SPI and I2C, you probably want to have some sort of logic analyzer.

Because even the simplest devices that you think are just going to work, sometimes they don't for some reason, or your driver isn't quite right for some reason. Or you wired it wrong. Right. And if you're just doing it blind, it's impossible to figure out what's wrong. That should have been my favorite tool, Aselia. That thing is amazing. Okay, so we've relaxed the constraints. We've decided we are not going to try to build the most power efficient, most RAM efficient, most cycle efficient system.

We're just going to build a system. And we're going to use the example code as much as we can. And we're going to learn to use these tools because, I mean, we all can use oscilloscopes and logic analyzers because we've been doing this for, since the dinosaurs roamed, or since 8051s were in the wild. No, they're still in the wild. And yet, those are all skills. It's like you're making an embedded system and you're not doing all these things that people say,

oh, embedded systems are hard. What you're doing is you're learning the whole thing. Cross-compiling is something that you don't know natively, and using a debugger on a system that you aren't running on, it's different. So, yes, I agree with your approach that you don't worry about everything at the start. You just try to make it work. And at that point, you will realize there are things around you that you're working on. Well, nobody learns to drive on a racetrack.

No, you didn't. Might have been easier. Yeah, I know. And that's exactly the right thing is you're going to read, especially if this is where you want to go, you're going to read about all these different methods to keep things tight and efficient. And that's not where you're going for when you're learning.

You're learning, you're going for, can I compile it? Can I load it on the board? Can I turn the lights on? You get those down and then everything else, you'll start approaching all those other things like, oh, you know, I can time this and say, oh, it takes a really long time to run this and it should be a lot faster.

How do I make it faster? Then you start reading, oh, this is how I do interrupts. Oh, this is how I reduce the amount of code that goes on it to initialize or to read those values or do those computations. And the reason I know that printfs take a long time is because it has bitten me. And the way that people will learn how to do these interrupts and callbacks and changing how they do printf is because at some point you'll realize, oh, that's the problem.

Yeah. But how long does it take? Okay, so I'm in a software company. I have found a hardware project. Maybe we don't do embedded. Maybe I've decided to do some physical monitoring of our servers or something that will get me a project, some project I can do as a 10% project at my work. And I've done that. But that's actually a really good one. I'd forgotten that one is, uh, so at, when I was at Airware, we had a DevOps guy who wanted to do

some embedded stuff and what he did, I'm pretty sure he used an Arduino. I'm not 100% sure. But he wrote this little device. It must have been, because I know he was using, I'm pretty sure he was using embed. But he wrote this little device that sat on our network that got messages from our continuous integration server when builds finished, and it would monitor those and then keep track of it over time, create a little summary, and then post to Slack.

Now, if that is not an integration of a lot of technologies, I don't know what is. It was doing network stuff. It was network. It was processing internally. Now, it wasn't using... Oh, and actually, it did blink light so we knew like the status of things um so it was just it was amazing little thing and it just it was a little board probably like two inches long and it just set plugged in next to his desk and we would interact with it via slack and just you know we'd

ask it questions like basic questions of like, what's the build status? And it would return back a list of, you know, pass and fails. It was, it was a cool, it was one of the coolest little embedded projects I'd seen. Well, that's a good idea too, because that's normally something somebody would just do on server somewhere. Yeah. There's no reason you can't do any kind of software on an embedded device. If you have to find an excuse for a project, take some agent that you've got already running that is doing testing or monitoring and put it on some little device.

Yeah, yeah. And the embed, I said it had networking, but it does have networking, but it also has an RTOS, which makes a lot of that stuff easier. Embed is like a class of boards. Yeah. It's kind of like saying ARM.

Well, but it's also the framework and the library they provide for it. Yes, and the compiler. So it's sort of like saying Arduino is not just one board. Right. But yeah. Okay, how long do you have to do these projects before you can get a job?

Well, let me ask that a different way. It's sort of been rattling around my brain because I haven't done interviewing in recent years, at least of new people, like junior people. So I haven't encountered people coming in with, oh, well, I don't have embedded work experience, but here's some projects I've done. Have you seen that, Dennis?

And what do you think about it and how do you evaluate that? I guess that goes to her question of how long does it take? In a way, asking more, what are you looking, how much are you looking for in personal projects to decide, okay, this person can do something. I have because especially at Airware, we were hiring interns. And now it's a little bit different than hiring a junior developer fresh out of school looking for a full-time job. It's just a little bit different.

But the thing that surprised me was how many projects people had on their resume. And I was just thinking back when I was in college, I wasn't going for, um, you know, embedded software. I was actually a physics, physics major and, uh, just happened to end this industry. But, um, when I look at these resumes, I'm like, Whoa, uh, that's a lot of like, they've done a lot of, they've done, they have a lot of experience experience. I mean, classroom experience, but like they have running projects so that's that's pretty awesome um for someone like just coming out of school like if you've if that's where you want to go if you're in school and that's where you want to go like make sure your classes are are helping you create these things you know create projects um or join their robotics team. Yeah. Or do projects to do things outside of class that are,

that require some kind of embedded work or do, and do personal projects. Like, like we started the whole conversation with, I want to make, I want to make something that does this thing. And you don't have, here's the other thing too,

is like, you don't have to think about it in terms of, I want to sell this as a product someday, so it has to be perfect. And there isn't a product on the market that already does this thing. Just do something simple. But coming into it, it really depends on the companies that you're looking at. an early stage startup that has five to, you know, five to 20 people, they're not, I'm sorry, but they're not looking for someone who's looking for, looking to gain all that experience on the job because they want someone who can help turn things around quickly. Now, as in, if, if, if

they're in that stage and they're looking for interns, like that could be a thing. But if you think, I really have a hard time believing I started, I was going to say, Hey, you've never done embedded work. You've done some projects. We're going to hire you to make our first product. Um, that's, that's going to be a hard sell. Um, the bigger companies like those are going to be easier to get into and are a good way to get a lot of experience quickly. Um, my job, I had never done embedded development. It was about 130 people at the company, but I had done a lot of software and, you know, I got, I got lucky that there's a lot of luck in interviewing, but I got lucky that, you know, the hiring, the folks hiring

saw a lot of potential. And I remember that first week on the job and they're like okay here's the here's the you know we're working on this pump fluid pump and here's like here's your compiler here's your here's how you talk to it and and I remember the first time I just I typed something in and I saw I saw I heard things clicking and moving I'm like okay this is what I'm doing the rest of my life actually I want to stop you there for a second. Sorry, if you can hold that train of thought. Chris, how did you get from, well, you had a math degree, but you went into software, you went into, well,

you went to Cisco and routers, but for a long time, you didn't consider that embedded. Because I didn't know the word. I don't think anybody was using that word. Yeah, they were using firmware. then cisco i mean cisco by most people's definition would be embedded i was cross compiling i was downloading images to a device it was not a computer we were on some weird you know mips processor and with some purpose you know uh home-built operating system with yeah so it was embedded and i actually you know thinking back learned some of

the things we're talking about remote debugging and cross compiling and i got exposed to those through cisco and it didn't even seem weird to me because that was just oh i thought this is the way people do things um you know i'd done software development of of course, on normal computers before, but it didn't seem that different. The first time I encountered moving things, like real embedded where you're doing SPY and I2C and things like that was two jobs later. And that was a medical device company. And that was, I'd already, that was i'd already you know watched your career so i knew about all of these things so it wasn't it's not exactly fair i wasn't really just dumped into it and shocked by a lot for me i went to hp and did servers um they made big servers

and wanted monitoring and i kept getting lower and lower in the hardware. I wrote drivers for all the different operating systems, you know, OS2, Windows NT. OS2, that's awesome. You wrote an OS2 driver? I did. I've written several. That should be your entire resume. Yeah, I would hire you right on that. And so I kept writing drivers and getting lower and lower and then blinking lights on the hardware.

And then when I went to go look for a new job, they were like, why are you applying for software? You should be doing embedded or firmware was what they were saying. And I said, I don't know what you mean. And they said, do you really have a signals background? Can you do PID loops? And I said, oh yeah, sure. And they said, well, we'll teach you everything in between. And then I actually got to do the cross compiling.

Before that, I didn't know about remote debugging and cross compiling. So yeah, I went from software to embedded in large part because of luck, because somebody said, oh, you have all the skills, so let's just put them together. Yeah. And seeing output over a serial port and that's your only interface to think. I mean, there are all these little bits of, this is different that you have to pick up. It's hard. We're telling people to do projects but none of us did we just kind of lucked into it i that kind of did maybe yeah i mean you're gonna there's gonna be a lot like when you're learning this you're i mean when you're in when

you're in the real world you're gonna be doing all kinds of crazy things like you're gonna it's gonna be like the circuitous past but like you got to get started somewhere um true and maybe maybe you get a job at a company that does have embedded stuff, and that's what you start working on. So my first job was at Decker Research, and they were an embedded company. The products all had processors somewhere inside of them. Then I went to St. Jude Medical, and I went there to work on the firmware team. But we had other people that were not embedded. They didn't work on firmware. They did not work on embedded software.

They worked on application software. But they had said, hey, I wasn't involved at this time. But whoever was in charge at that time, they said, hey, I really want to get involved in the embedded side. And I remember there'd be conversations about, oh, so-and-so, he's only ever, he or she, he is actually a man and a woman. They have only written application software for the desktop. They're not going to be able to do it. And you just show them like, hey, you don't have, you know, a gigabyte of memory to use.

You don't have, you know, this infinite hard drive space to save stuff to. You have, actually the device I think had 256K of memory. Like you have this amount. And they just said, oh, okay, these are the new constraints. And, you know, they learned some techniques and things to look out for. And they became like super valuable members of the team it's just it's so you touch on a couple of topics in that part of the conversation i have almost never used the word firmware like i'm a firmware engineer except for where i am now because that's just the way they've divided it up i've always

said embedded software and the reason I say it that way is, to me, like, people have asked me, like, what's the difference between firmware and software? I'm like, just a couple letters. It used to have meaning. Yeah, I think it's lost it. Yeah. Yeah. I mean, if you think about like, okay, you know, I'm writing the control, the software that controls this motor that lives on a pick that lives in something. Okay. Maybe you can call it firmware, but you know, at the end of the day, it's taking inputs, taking output, it's doing something, it's taking inputs, doing something and outputting something. You know what software does? It takes inputs. It does something with them and does

outputs. It's all software. So to me, there's no distinguished, I don't distinguish between them. The distinguishing thing is the constraints. Do you have lots of memory? Do you have little memory? Do you have power requirements? Do you not have power requirements? But it's still software at the end of the day. I personally just don't like distinguishing between them

because I think it does a disservice because now it's like, oh, I write firmware. I don't have to follow those rules for software development. I'm like, what? You have to follow those rules. You know why? Because your software, your firmware, is running on something that has to work.

So, I mean, rules are good naming, good code, unit testing, documentation. Like, what does this thing do? Why does it do it the way it does? Now, you don't have to write pages and pages of documentation to say this motor controller takes an input and computes this commutation and then outputs a PWM signal to something. I don't need pages. One page is fine, but you have to write down what it does so when you leave that

job, when you get sick and you're out and someone else has to do something with it, or when you go on to another project and someone else takes over, or you come back six or 12 months later to fix something, you understand what the heck is going on there. So that to me is why I don't distinguish between them. Because at the end of the day, you still have to write. You have to be, you have to,

you're a software person, even if it's, you're flipping bits on something at the lowest level, you're still writing software. I do think you're getting at something there that I haven't really realized is that, that naming of roles has, has some power because people do say,

Oh, you know, I'm an embedded software engineer or I'm a firmware engineer, and they have some understanding of what that means, and that might be colored by their decade of career, and it might not apply really anymore because if you're releasing a quarterly update

that firmware updates your device, it's hard to call that firmware anymore. Yeah. Right? And it's hard to say this isn't an application even. And the techniques that you go through to secure something that can be updated frequently are far different than if you're burning a ROM, right, that you're never going to touch again. Yeah.

And I think people need to realize that we're all kind of moving to a space where we're converging with normal software development. And all those techniques that came from even web development now apply to us, except Agile. Except Agile, I concur. It used to be more Wild West.

It used to be more just make it work. Once it works as well as it's ever going to work, we burn a million ROMs and are done. And we drop the code and we go on. Because whatever we're going to build next is going to be on a different processor, is going to be totally different. There's no reuse or continuity.

Testing is external to the code. And that's not true anymore we write software we we write software primarily for humans and it the fact that the computer the processor does what we want is sort of a byproduct we should be more aware that the humans are the people we're writing for yeah yeah i think a lot of this went out the window as soon as the first radio module was put on a device for more update i mean it means that our firmware it means that our software it means well it means it can be worse than we want it to be the first time we ship it but it also means it can be it's much more insecure so that those are two strikes depends

on what you're working on depends onends on what you're working on. Right, right, right. So you touched on another thing there. It was a lot of times just get it working. And that's still the norm. It's like, okay, we have all these pieces. We need to make them work. So let's pull something together quickly and make sure everything works together.

Now, the scariest thing is having a working demo. Because once you have a working demo, everyone's like, oh, we're going to ship this in like two weeks. We just got to put the packaging around it. And we're like, oh, no, we just got it working. We just got to take this down to South America. Never mind. Yeah. Yeah.

We got it working once. Yeah, we got it working once. Yeah. We have it working once. Yeah. That's, that's the big risk. But that, and that's where it comes in on, on the team and not just, not just the lead, but as the team who says, we, we aren't, you know, we can show that it works here in the lab. Like we can put one on someone's desk and they can use it. But here's all the other work that has to happen to make sure that can be done not just one time, but 10, 100, 1,000, a million times. And it works reliably.

Like secure firmware update. Yeah. Okay, back to the question of how long do I have to do projects before I start actually getting a job? Which is not a real question because how long doesn't make any sense. It's actually four weeks and three days. Oh, I see. I didn't know. No, I mean, again, it depends on the type of company you're targeting and you can get your foot in the door. So for me, especially when I was talking to interns

or if I'm looking for a junior developer, I want to see, have you done something? They haven't shipped anything, but have they done something and been able to demonstrate it to someone else and it works? If you have something you can bring to an interview, something um you know they they haven't shipped anything but have they done something and vanilla that demonstrated someone else and it works you know if you have something you can bring to an interview that'd be that's really cool that is such a great way to control an interview because now you are going to be talking about something you know to control it from the from the interviewee

perspective oh yes like i mean if you you're holding this thing of course they're going to ask you about it and now instead of asking they're going to ask you about it. And now instead of asking you about their product, which you don't know anything about, they're asking you about your thing, which you can talk endlessly about. Yes. But getting, you're right, it's a pretty open-ended answer. So when I'm looking at these developers, I'm looking at like, so I always open my interviews, like we're going to do some, we're going to write some code on the law at some point. And I'm not looking to see if you can put semicolons in the right place or your parentheses are balanced and all that. Like, I'm not like the compiler will tell you if you've done that.

Like, if you do that, I'm like, hey, I'm noticing that. But I'm not going to throw, you know, this guy is not going to, or this person's not going to work out because they didn't put the semicolons in the right place. Like that's just stupid. But I'm looking like,

how do you, what do you, how do you think about the problem? Like I, you know, if I asked you to write a, a UART driver to, to,

you know, talk to another device, like how would you approach that? Like, what are the things that you have to do? And this is actually a really good question, a really good thing to think about if you want to get into bed. How do I talk to a couple different devices? What do I have to do in order to talk to you?

Oh, I have to be able to open it, I have to be able to close it, I have to be able to set baud rates and bit rates, parity. If I'm talking I squared C, how do I address those? Is there, you know, just address, is there command response? If it's spy, you know, how do I address different things? How do I write to spy? How do I read from spy? Like, there's addresses that you use that all spy devices use.

Now, you don't have to know those, like, oh, it's this address, and this is exactly how I do it. But having that, we can have a conversation. But I'm also looking at just general software development. That's why I'm not thinking, oh, you're an embedded person, you don't have to follow the rules of asking questions, asking about a spec. What does it have to do? What kind of data rates do I need to be able to handle? How often is this going to be called?

Do I have to put it in a route? Do I just, you know, do I just put it in a, do I just pull it whenever I need that information? Those are the kind of questions I'm going to ask. And just if anyone ever comes to interview with me, I'm going to leave holes in the questions. I'm going to leave, it's going to be vague. I want you to ask those questions. That is an interview thing that I don't think I understood

right away. That the question may actually be impossible as stated. And what they are trying to find is, can you ask the questions? Or are you just going to bash yourself against those rocks until you realize you can't and then you're going to leave angry because our jobs are about asking the questions. You're describing my last four weeks. It's just addition is traction, man. No, no, there's other stuff. There's other ways. Okay, so you said I squared C and spy and U are and do they need to know all of these terms

and if so if they've only done a couple of projects how do they know these things they're not easy i don't think you have to know them you don't have to know them inside and out so that was another thing when you guys were we were talking about first jobs um i i was thinking back over my career when you guys had the past people on the episode. Unfortunately, I missed it. But I was thinking about when is the first time I really, like, truly wrote like a low-level driver? I was like, oh, I had to be back at DECA, my first job. And I was thinking, I was like, no, no, I think it was at Airwear. I got hired on Airwear.

That was not that long ago. I know. It's hired on Airware. That was not that long ago. I know. It's just a funny thing. So, like, I've been doing embedded work for 20 years, but the first time I really wrote some low-level stuff, I think, was at Airware. That was, like, 16, 17 years into my career. But I was writing embedded applications.

I was writing, you know, I was working on control algorithms. I was working on, you know, I knew I had memory restraint, memory restraints. I had power restraints. I had all of this stuff, but I was like, well, I, I, you know, I blinked some lights early on. I definitely wrote some stuff that like turned displays on and wrote some stuff to display, but the display drive was already there. Like someone else on the team had already written it i was like oh i was just putting stuff on the screen i guess i did respond to some button clicks oh but someone

else wrote the thing that listened to the button tell me the button was clicked or not i'm like wow yeah it was pretty far into my career before i wrote some real like quote unquote true embedded yeah i think i'm trying to think it must have been 10 or 11 years into my career maybe nine somewhere between 9 and 11 years before i did that too and it was much earlier for me yeah that was because i actually went to embedded yeah so i guess this is throwing a little kink and all the stuff we've been talking about we're all liars all this stuff yeah we're all liars um yeah It definitely gives you a leg up, especially like for me,

if I'm, if I'm hiring someone to do embedded, I just want, I want some, some exposure to that. But really, if you are, if,

if you can understand and write software and a whiteboard question is a very, it's a very limited scope. It's very, it a very limited scope. It's very time constrained. It's what we do in the industry. It's not maybe the best thing to do. But take-home coding tests, I've done those once, and I actually hated it. I think that was really annoying. But it's a way to show some knowledge. And really, my interviews are a lot more

conversational. I'm not going to throw you up on the wall and say, okay, write this thing. I definitely have done that. And I know I did that a few times at Airwear. But I like to have a conversation about like, how do you approach this how do you do that um like one of my questions is always unfortunately the answer almost is always the same thing is like do you do unit testing and it's almost always no um get some experience with that there's a lot of great frameworks out there you know you don't even have to do unit testing of embedded code just get some experience with it if you walk into an interview with me and you say yeah i use i do unit testing of embedded code just get some experience with it if you walk into an interview with me and you say yeah i use i do unit testing and this is my framework i'm like you already went up 10 notches on my scale um because that means you're concerned about the quality of the code

not just does it work like does it work when i write the you know i write it and throw it on the on the processor like you're concerned about writing code that can be tested and that is going to be solid that I know is like, we're going to write that and we're probably not going to touch it ever again. And you know, I don't ask in interviews about iSquared C or Spy or UART. I hope that they have used them, but I don't care if they've written a driver for them. Yeah.

I want to know more that you can understand how a system isn't all spaghetti, that you can make boxes. I'm really impressed by people who can make block diagrams of their system because they understand that these are separate pieces of code that potentially have reuse that can be tested on their own. I care a lot about testing. I don't always, you know, unit test is a word pair that is particular meaning. I'm okay with understanding more about how do you test it? How do you make it testable without saying unit testing? But yeah, we want you to know what all of these little protocols are,

but you don't have to have implemented every single one of them. If you've implemented one and can talk about the difference and talk about how logic analyzers work and why it was important to know that in order to debug a spy driver, that is awesome.

More, I want you to be able to say oh i had an accelerometer i had an led i used a little bit of code to make it so that the led color changed based on the accelerometer and this is why i did it and this is how i kept them separate and this is how i changed the accelerometer to something else i mean it's the separation and the testing that are a bigger deal. So, I mean, tying that all back together is when we talked before and when I tell people things they can read to become better developers, I do have some embedded software articles, like read these. But one of my favorite ones is Joel Spolsky's lists. And they have nothing to do with embedded software. They are software in general.

If you're writing web servers, web applications, embedded applications, things that sit on desktops, these are articles. You read those and apply what you can. Almost all of it applies. You know why? Because firmware is software. That leads me to a question of what are we assuming the minimum experience of the person is who is trying to get into embedded? What background are we saying is,

okay, you need this background before you even attempt to play with embedded and attempt to get an embedded job? Because there's plenty of people, I think, who listen to our show who maybe don't have a software engineering background who play around with Arduino and are learning this other stuff. I mean, most people I've worked with have not had computer science

or software engineering backgrounds. They've been mathematicians, they've been physicists, they've been electrical engineers. They took some software classes in school. I mean, they've done those, and've had, you know, basically had a knack for it. Like, oh, I like writing software. And that's where, I mean, that's how I ended up in it. When I was in college, people were like, oh, you must be a computer science person. I'm like, no, I'm physics. But I just, I'm always in the lab writing software because that's kind of what

I like do. But I'm writing software for physics stuff. because that's kind of what I do. But I'm writing software for physics stuff, so it's just the way it worked out. I mean, if you want, I mean, so software is a craft. It's not just like you are going to, if you like doing it and you want to do it, you need to do it. You can't just say, wake up one day and It's like, I'm going to be a software engineer. I want to be a software person and just say, okay, now I'm going to go get that job. Like you have to be practicing it. Um, you have to do it at home, you know, or in your current job. Um, you just, you have to get that experience of working with a computer. You know, everyone, anyone who doesn't write software thinks,

oh, I want the computer to do this one little thing. It should be really simple for you, right? And the fact that you're laughing is because you know that's never the case. It's never the case. There's always a million little details that you have to get just right. I mean, the compiler is brutal. If you don't tell it exactly what you want to do, it won't do it. So, you know, having that experience.

So, you know. And shifting between the detail-oriented, where do I put my semicolon and how do I talk to a register on a peripheral to the high level of how do I keep my code separated and clean and testable and architected well? And there's this, you have to slide back and forth in mental shifts that orders of magnitude difference. Yeah. Yeah.

You mentioned resources. And I agree, Joel on software.com, he's got reading lists on that front page that are just fantastic um it's funny when i go back and read them how many things i must have read 15 years ago and internalized so much that now he it feels like he's plagiarizing from from my head uh so i agree with that you have other books you suggest uh other articles other books um, everyone knows I'm a big fan of unit testing and test-driven development. There's James Grinning's book. That's geared towards embedded C, but there's a lot of other TDD books out there. Now, I always get asked this like,

oh, you're a real advocate for TDD. You must do that every time. I'm like, actually, I don't. And I mix it. It's kind of like the whole Agile thing. I'm not a fan of all the stuff with Agile, but I like some of the stuff from it. So I've taken the stuff I think is appropriate. Some other ones, actually, is your book, Making Embedded Systems. When someone asked me, I'm like, you don't have to read this thing cover to cover,

but read through it because these are the kind of things you need to know about because you have everything in there from the high level, like the block diagrams, like how's the system going to go together to low level debugging. So it's a good place to start. So other articles are things like, uh, if you're using C there's some, this article is probably, it's gotta be like 25, 30 years old now, uh, rules for different defensive C programming. And it's just, it's really, I think there's like

15 tips in there, 10, 15 tips. Um, it's just like naming, you know, putting units on your names. Don't use pound defines because that can lead to other problems. And I, I found out from a guy I used to work with at DECA 20 years ago, I was, I was gone from there a few years and he said, yeah, we found this problem in the in the in a released version of software and all came down to a pound defying being incorrect and it had gone through all the testing and it's gone through everything and it'd been on the field for a while but like i went back to that article and reread it i was like right there um some other things like i know a lot of the embedded world uses c and c is a great language it's also it's very powerful and it's it it'll let you it'll let you cut your own foot off um but the article is

why are you still using c and it's an it's advocating for using c plus plus and um i've got a mix back and forth between c and c plus plus um current project's all in C. But it's funny because I've been there seven months now, and as I'm looking at it, I'm like, oh, you guys should just use C++. Because the code is written to try and do that. All the things that C++

gives you for free, they're trying to do in C. 20 years ago, the arguments were, well, C++ has all this extra overhead, like you have all these things you can do. And you know what? C++ has gotten a lot bigger. You make spaghetti code, says the person in a bowl of spaghetti. Yeah, yeah, yeah. It's like, you don't have to use all those features. You can use a subset

of it, but you know, take a look at it. I do remember a story from way, way back that when I started at DECA, they were using C++ and they had an interview candidate come in and say, you absolutely cannot use C++ in embedded software. That guy didn't make it very far in the interview process. The robot escorted him out. Yeah then um there's another there's another set of articles that were in i mean i think ee times way back in the day um 30 pitfalls for real-time systems development and that's another one is just you know i i i have this set of articles i keep them i i have them on i have pdfs of them and whenever someone starts I'm like read these or actually I usually do it when you know we've made them an offer and they've accepted and they haven't

started yet I'm like here's some stuff to start out especially and interns too I send them the articles like these are good things to read like I'm not going to quiz them on day one but you know it does come up in conversations and I can usually tell if they've read them or not um you know and in that time it's like it's like when you uh you get a new a new device like you want to just start playing with it and seeing how it works and you put the manual to the side but if someone were to hand you the manual before you ever got the device you would actually read it you might not read it cover to cover but you're going to look through it so i like to give these articles when they're excited about the job but they haven't come in and seen like, okay, now I have to make all this stuff work. So it's that little window of opportunity to sneak these things into them. And I usually give them the TD, I buy the TDD book

as well and send that, either send it to them or it's sitting on their desk when they start. But those are- Just the TDD book? Sorry. I have the making embedded systems as well. Maybe I should add it to my list. I will. I will.

Especially the more junior developers. None of these are new. None of these came out in the last six months. No. That goes back to software being software. Yes. Well, that's why my favorite tool is make um there's another there's a great article that someone someone had uh given me when i

uh just a few years ago that now i i pass around when people say what make make is like 40 years old i think it just turned 40 like this year maybe last year and i my favorite thing is like oh we can't use whatever tool is because it hasn't been updated in six months, 12 months. I'm like, you know, Make hasn't been updated in a long time. And it's based on a 40-year-old plan. But you know what? It works really, really well.

It's got some quirks. It's got some things that you got to watch out for. But it works really well and i i see all these other systems people are like oh we've recreated we've replaced make with this new system that sits on top of make that sits on top of make um either sits on top of make or recreates it poorly um because they're like oh it handles this one thing that that make doesn't do i'm like yeah but doesn't do the other 40 things that make does really, really well. You don't do them at all.

I'm like, oh, but that's not that important. And you see them every week, they have a new release that does another little feature. I'm like, yeah, I'm sticking with make. I'm old school like that. Sometimes you find tools that work and even if they're quirky, you keep using them because they're effective.

Oh, so another question you didn't ask in the lightning round was, I think you asked it recently, last week or recently, was IDE versus command line. And I use both. I use the IDE to debug. I use the IDE to do simple, like whatever project I'm working on to do the build right then. Cause I'm, I'm just, I'm in like, I'm in the groove, but every project I work on can build from the command line. Um, it's a make target and I, you know, you say make, uh, either make and it's the all talk target is the first one or, uh, a special, a special

package. The reason you do this is because, um, when you go to release this, you don't want to say, Hey, we're doing a release today. Uh, you know, is Joe here is Sue here. Like she's, she's up for the release. You don't release from your desktop. You release from your continuous integration server. Um, this is is another software in general best practice is having a computer that is not a development machine that can do all your builds. And your IDE is not going to, your build machine is not going to open your IDE

and click build and save off the packages. All that stuff has to be scripted. It all has to be driven from a command line. That thing sits in a closet and you know it either runs every day every night every branch every commit you know you can set this you can set that whatever you want mine runs on every commit by the way commit to to the main repo that thing, when you go to do the release, all you do is you kick off the release branch, you update, you know, the version number, any other little pieces of documentation that you want to have on there, you push it to the server and, you know, depending on how long your bill takes five minutes later, you have your release

candidate. Now you should go off and do your formal test. Oh, you know what? All your unit tests ran on the release server as well. You know all your unit tests pass. If you are really far along, you've run basic smoke tests on hardware from your integration server. There's probably still some stuff that you have to do manually. Someone on your team, someone in QA, they pull that release binary, they run the tests and they give the thumbs up, thumbs down. You merge the release back in, you tag it. Hey, look it, your release is done. But you know where that started from? Being able to build from the command line. Being able to build the entire project from the command line. That's actually one of the things from

Joel's list as well. Can you build in one step? And it's amazing how many people don't do that. They say, oh, we're going to release. Go pull down the latest branch or the latest whatever develop is or master, however you do your release process. Change the version number and hit compile. Okay, here's the thing that I'm going to copy into the network drive.

I'm like, okay, did you know your tools updated? You updated your tools and now you're on a different tool set. Now you just broke something for everyone else. Oh, you didn't have whatever, you know, the special version of some key or whatever you had to have in place and you didn't do that. Now that version is completely different than all the ones we built before there's so many things that will go wrong

if you're doing it from your desktop but if you have your integration server doing it, it's going to be the same every single time it's just, that's what you should be aiming on, now that has nothing to do with getting into embedded software that's further down the line but just do that and you should be aiming on. Now, that has nothing to do with getting into embedded software, but that's further down the line.

But just do that, and you'll be happy. Being able to say that you understand those things is useful. As a software engineer, you may have an advantage with that. You may be able to say, I know how to do these things. I may not know how to write an I2C driver, but I can make a test system that will actually work so that you're no longer dependent on one computer that may go up in flames at any time yeah yeah and that's you know if you come you know you come in to me and say hey i want to

be embedded i have a little bit experience but i also do all i've seen all this other stuff i'm like oh you've used those. Okay, let's start talking. Yeah, your software knowledge is not wasted. Yes. How do you balance being confident in both the theory, like all of the Spolsky stuff and all of the TDD stuff, and the tactical information like processors and specific peripherals. It seems like it's so much knowledge to get all at once.

What do you look for for them? Yeah, that's a great question. I'm always thinking of myself like, okay, I know this and I know that and I don't know anything. Yeah, I feel that way a lot. I think the key is to realize I feel that way a lot. I think the key is to realize you're part of a team. And this is my little kumbaya moment. But you're part of a team.

You don't have to know everything. Especially as I've been on the management side of things. Like my current job, I have the compiler. I have hardware. I can do some of it but unfortunately i don't i don't touch the hardware all that often as much anymore um i'm doing a lot of other things and especially setting up build stuff for making the teamwork more efficient but you know i rely on the fact that i have this team, and this person knows a lot more about the language, the intricacies of the language. And this person knows a lot more about processors. This person knows about sensors.

It's not just on the firmware embedded team. It's also the EEs. If you have the current product, that's what you have. You've got to make it work. But when you start talking about that next generation one, you don't have to know everything. Everyone's going to bring something to the table. I keep working on teams where I'm the only person.

That's the flip side. Then you do have to know everything. Yeah, then you do have to know everything. Or you know people who know other things, and you reach to them. Like, I've been in the industry long enough that I have friends on both coasts. I have people I've worked with. I have medical, non-medical. of airware i was like hey uh we need i need some i need some of these you know i need uh i was looking at static analysis tools and like you know embedded software like that's not the first thing on your list is i need to know about embedded and static analysis tools but i reached out to my i reached out to old co-workers and colleagues and friends and said what are you guys using and just took a took a sample and i mean i poked around the web and seen what people are using.

But being able to talk to people is huge. And it wasn't like hours-long conversations, throw an email or phone call, say, hey, you got five minutes. You're going to build those resources up over your career, make use of them. And how do you convince management? am management i know i know no but there's always you haven't always been there's always a layer above you there is a layer above you um that's and that's uh that's the hard one that that you know people talk about writing

software is really hard like convincing people is even Um, people aren't compilers. They're not, you know, they're not a piece of hardware. You just, if you, if you get it just right, they're going to, they're going to go your way. A lot of it is demonstration. Um, you know, if you, if you need to lay out, you know, thousands and thousands of dollars, like you're not going to just go get that tool and start using it, but maybe you can, you know, maybe you can demonstrate some of this, like, like a CI server, a continuous integration server. If you go to, if you go, like, if you're, if you don't currently have that, if you go to someone and say, Hey, you know, I need a server,

I need a computer, uh, it needs to live on the network. It has to have these kinds of resources and I need to have this software. Like that's going to be a hard sell. needs to live on the network, it has to have these kind of resources, and I need to have this software. That's going to be a hard sell, depending on the company you're at. But you know what you can do? You can find, hey, is there a computer that's just an older computer that no one's using anymore, an older laptop? Can I have it? Can I use it for a few weeks? Is there a lab machine that I can use that is just sitting there gathering cobwebs? You don't need the most powerful latest machine. You just need something that runs.

There are free CI servers out there. There are free tools. Jenkins is a very popular one. A lot of people use that. You're going to have to learn. I'm in the business. I'm of people use that. You're going to have to learn. I'm in the business. I'm learning it right now. I've got it set up, but there's so many things about it that I just don't know. But get that set up. Have it sitting on your desk. Especially if you're a boss or your team lead is like, we got to get this thing out and you don't have time to work on these other things.

Lunch hour is a great time. Grab your food, eat it quick and get back. After hours is- And baby steps. Yeah, baby steps. You're not going to get it in the first step. But get it set up. Can it read from your repository?

You might have to create some, depending on where things are, you might have to create some, depending on where things are, you might have to create an account for it. But until you get that all rolled out, use your account. If it's on a remote, if you're using GitHub or Bitbucket or any remote code repository.

This isn't your production level. I want to prove that this works and get it set up to run once a night. And you can come in in the morning and say, look, it did this or it runs at lunchtime so you can see the results right away. So a lot of times, that's one example, but a lot of times it's just setting up

an example of why you want to go somewhere. Now, if you're using embedded tool chains that have really expensive compilers, you're not going to be able to do that. But maybe you can run it on your machine. Now you are working on your machine, so you don't want to distract, you don't want that running while you're doing things,

but you can set it up to run, like if you don't take your machine home, have it run at seven o'clock in the morning or six o'clock in the morning before you come in. Or have it run at the lunch hour so you can sit there and watch it and make sure everything is working. That's for like a CI server. For other tools, I've seen this before too. It's like a static analysis tool.

On Linux, you have Lint. Again, it's an old school tool, but it gives you a lot of information about your software. And maybe you can't go to your project lead and say, we now must use Lint or whatever static analysis tool you want to use. That's the only way we can do really, really good software. Run it on your code. Run it on the stuff that

you're working on, you know, make it, make it part of your, part of your, part of your development process. Um, and, you know, in your code reviews, hopefully you're doing code reviews. Um, you know, people start saying, huh, you know, you're, you're not, you know, QA is not coming back for saying, hey, we found these problems or your tests are always passing. You can start demonstrating the code is getting better. Or you can just look at the code and say, look, as you're reviewing other people's code or as you're making modifications to other parts of the code base, you can be making these improvements. And when people say, hey, why did you add this check in here? as you're making modifications to other parts of the code base, you can be making these improvements. And when people say, hey, why did you add this check in here? Why did you change this? I was like, well, that's a common error that gets hidden by this other thing.

And how did you find that? How did you know about that? Well, I used this tool called Lint, or I used this other thing. Clang. Clang or Cpp check that's a free tool same thing um you know i use this tool i just i've been running it over the code and identifying you know potential problem errors and like not they're not 100 guaranteed to be problems but these are things that could cause a problem down the line um or yeah i mean if you

have a gcc based, or you have a project that can be built in clang, run it with clang and see if it's, you know, their, their compilers is really good at spitting out a lot of other, a lot of other warnings about what could go wrong. Like your tools are, use your tools. They're there to help you. Yeah. So, I mean, and that gets back to, like, you can do things within your own little sphere. And then maybe, you know, the person sitting next to you says, hey, I've noticed you've been doing that. Like, can you show me it?

Like, oh, yeah. And, like, I have it built in here. Like, now they start using it. It's a little bit of a guerrilla warfare, but sometimes that's what you got to do. Especially if you don't have a lead that's like hey you know if the lead's like we just got to get this out like maybe you can help change it from the bottom up and that does work but going in and putting your foot down and saying

we should do this and you have 25 000 errors in your code it's not it doesn't work and you bash your head against the wall and it's no fun for anybody. Yeah. And yet if you can start being the person that everybody goes to because your code works. Yeah. And your code works because you use a tool that tells you, you use a tool, you use unit testing and suddenly you're the person everybody wants to go to. It takes a while. It does.

It's baby steps and it's a lot of learning the tools. Yep. Because you don't necessarily want all of those warnings. That's actually a good one. You need to start going. You need to start doing it. And yes, it's hard to start. Another really, really simple one is if you don't have warnings as errors turned on in your build, start eliminating those errors and then turn that on.

And now other people on your team are going to turn on your build, commit it to the repo. Talk to people. Say, hey, I've eliminated all the warnings. One, they should notice it. If they're not noticing the error, the warning count go down. Start wondering about your coworkers. Yeah. But, I mean, once you get it down, if you can eliminate all those, talk to the lead of the project and say, hey, I took some time.

I'm delivering. Make sure you're delivering the things you've promised to deliver on. But say I took some time and I've eliminated the warnings. Can we turn this on so it's now an error. So we don't get any more. And I'll just say, you know, if we, the warning, my favorite thing is leaving warnings on and people say, Oh, it's just a warning. Don't worry about it. A warning is a compiler telling you something, you know, you're doing something a little weird. I'm not sure that's really what you want to do. And maybe it is. And if it is their compiler flags that you can go in and say, ignore it for this one piece of code, because we're doing something really, really clever. Don't do clever things and that you'll

be better. But you can turn it off for very specific cases. That's the better thing to do. And then just say, oh, just ignore the warnings. Because those warnings are going to come back and bite you at the worst possible time. You're going to be doing a demo in front of your CEO, in front of the company, in front of a customer, or that thing's going to be shipping and it's in the field. That's when that warning is going to break. So we have this Slack channel that is private. And Dennis is on it, and a couple other people, Andre and Svek, and some other experienced engineers who are all nominally supposed to be writing for the Embedded FM blog.

But I'm noticing some deficiencies there. Anyway, we have this channel, and we talk on it. And Dennis often mentions unit testing and continuous integration servers and things he's working on at work like that. And everyone on the Slack channel wants to work at Dennis's company. I don't think we place enough understanding that not only are these making better pieces of code, better software, better devices, it's making a better environment to work in. And I wish that the management and leads who don't support test-driven development, unit testing, better lint checkers, all of these things, I wish they understood that, yes, it's a little hard to get started,

but once you're there, it's so much better. And experienced people know that. That's the funny, that's actually really interesting because folks on my team here, when I came in, they were doing, you know, pull down it, you know, here's the branch, hit build in the compiler, here's a release. Like, there were a lot of things that were not in place

and you know you talk about how do you bring this into a place when I started at Airware I'm like I came on really hot and I definitely tweaked some noses when I did it and so I scaled back and then I started building things within like the stuff I was working. I did the whole process that we said before. I just kind of built it out from there. Like they already were doing unit testing, so that was great. But I just kind of introduced more things to it, and they had CI servers set up. So I learned a lot of amazing techniques from folks at Airware.

But eventually, all of a sudden, I was like, okay, now I had the team. I had more of the team. Now I'm in charge of all the embedded development. And so when I came to this company, one, I came in at that lead position, but I also knew there was an established team there. I can't you just can't come in and say you guys are doing this all wrong and they weren't doing it all wrong they were doing they they had functional code they had they had you know they had things working so you know the greatest thing at the end is like does the device work like if it works then you did something right

like the technique to get there may have been you know questionable at some points but um like they had they had a functional device now taking that to production like there's things you can do to improve but especially in those first in that first week or two first couple weeks just looking at how people were developing i'm like okay this could be better and here's what we're going to start you know we're going to start adding on how we do things. And getting that CI server in place so it was always building. There were no more broken develop. There was no more broken trunk.

We were using SVN originally when I first got there and we switched to Git. But people started noticing like, oh, I don't have to worry about such and such thing happening because, you know, it tells me there's issues coming up. We just unfortunately took a lot longer to get in. But that was my fault. But we got unit testing going and across the team, every single person said, wow, I just found a problem that I would not have found or would have been impossible to find on the hardware. But I wrote a unit test and it's it it failed and i was not expecting that and now i put the fix in and now it's better and or you know we have this code that has like low level driver stuff and application stuff kind of mixed together

and they're like it's going to be impossible unit test so i need to refactor it and pull out a driver level piece of it and an application piece of it. I'm like, all right, good, go for it. And they come back and say, okay, we've separated out and you go look at the code and it's like so much more

readable. And the whole team was like, this is really good. This is really like, everything is from, from my, you know, above me,

like people are like, still like, okay, where's, you know, software needs to go faster and faster. But I look at what we're doing and we're working so much faster than we were before when I started in the first couple months. We had someone recently start on the team, and they're like, wow, you guys work really fast here.

I'm like, we're still jogging. We haven't started running yet. We still have a ways to go, but we're getting there. So these tools and techniques and just looking at how things work and making them work better helps everyone. My favorite actually had this conversation just yesterday. For some reason, our build is recompiling everything every single time instead of doing partial builds. And I didn't notice it because I hadn't been using it. And so I said, yeah, it's been doing this for a little while now. I'm like, okay, when things like this happen,

we have to fix them right away. Because if you compile, if it takes three minutes to compile the entire project and you do that 10 times a day, that's half an hour of your day just sitting there looking at the compiler. But if it's doing partials and it takes three seconds, that's 30 seconds of your day. Which would you rather that's 30 seconds of your day which would you rather use 30 seconds to just what would you rather be spending that other 29 and a half

minutes like thinking about the code working on stuff or would you rather spend that time watching the compiler go compiling file compiling file compiling file so all those little things they add up really quick over the course of a day over the course of a week over the course of a project i feel like it's redundant now to ask you for a thought to leave us with know your tools where we are in the uh in the show okay so yeah know your tools or do you actually want no i mean i think that's it is like really understand your tools. It goes back to what I said earlier. Software is a craft. You're not going to go to a woodworker who's going to have a dull axe, a dull knife. They're going to keep them sharp because that's their trade.

It's the same with software. Your tools are your trade. You don't have to know every single option that they can do, every single thing they can do, but the ones that you need to use, make sure you understand how they work and use them. And if things start looking like, pay attention to how long things take. If things start taking later, what changed?

Start taking longer, what changed? But just really know your tools, understand them and keep them sharp. Thank you. That is good advice. And I think you've had lots of good advice through this show. So thank you very much for being with us, Dennis. Oh, it was a great, it's always a great fun talking to you guys. Our guest has been Dennis Jackson, Director of Firmware Engineering at Element Science. Dennis sounds familiar because he was on episode 94, Don't Be Clever, when he was at Airware making a drone operating system.

I would like to say thank you to Christopher for producing and for co-hosting. And of course, thank you for listening. Special thanks to Patreon supporters who let me send Dennis a microphone. I am looking for a new gig, ideally something mathy or robotics-ish. If you know someone needing a strange set of embedded systems architecture, prototyping, code implementation, and personal communication skills, please keep me in mind. I should get a job at some point. And a quote to leave you with this time from Voltaire. The perfect is the enemy

of the good. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.