Embedded - 270: Broccoli Is Good Too

Episode Date: December 14, 2018

James Grenning (@jwgrenning) joined us to talk about Test Driven Development, dealing with legacy code, and cleaning out very large pipes. James is the author of Test Driven Development for Embedded C. If you want to take his live online course, check out the remote delivered TDD classes on Wingman Software. His blog has many great articles including TDD How-to: Get Your Legacy C into a Test Harness and TDD Guided by ZOMBIES. Book: Working Effectively with Legacy Code by Michael Feathers James mentioned Given-When-Then, a testing design pattern (brief intro). Kent Beck also wrote about test && commit || revert style testing. James and Bob Martin present IoT implementation strategies in a series of videos on Clean Coders. James mentioned working with a Synapse Wireless radio.

Hello, and welcome to Embedded. I am Elysia White, here with Christopher White. We're happy to welcome back James Grenning. He's the author of Test-Driven Development for Embedded C. Hi, James. Great to talk to you again. Great to talk to you, too, as well. You've been on the show twice before, but sometimes we have new listeners. Could you tell us about yourself as though we'd never met you?

Let's see. Well, I started programming and I tried to avoid programming, actually. When I was in high school, it looked horrible. There's punch cards and typing, which in the 70s, guys didn't type. But in college, I ran into programming. And in my first job, somebody handed me an 8251 data sheet. I became an embedded systems engineer. Twenty years later, I ran into Kent Beck and Company and started learning about extreme programming and this thing called test-driven development.

And that's kind of led me down the path of figuring out what that means for embedded systems and writing a book and building a business around helping people learn that. Excellent. And your business name is Wingman Software, right? Yes. Okay. If there's a nice metaphor there, you know, when I'm working with a group in a training environment or in a coaching environment, I'm often looking over someone's shoulder. And then there's also a bit of history in the name, which is my dad was a World War II fighter pilot.

And there's some aviation art where he's featured as a wingman on D-Day in a P-38, which is on my logo, you can see. So we want to do lightning round where we ask you short questions and we want short answers. And if we're behaving ourselves, we won't ask you how and why and for more details. Are you ready? Okay, I'm ready. Do you have any role models? Well, my father's one. Cool.

Do you have a preferred listening music or whatnot when you're programming? I generally don't have music on it in the background, and then it would probably be Steely Dan and things like that that come up on Pandora when you say you like Steely Dan. All right. Preferred way to learn new technical things, reading, videos, trying it out without reading the docs. Do I have to choose from that list?

No, you can. Experiment experimenting and phone a friend okay do you have a favorite animal donkey donkey that's a new one all right uh tip that you think everyone should know it's easier to keep the system working than to fix it after you break it. And last one, preferred programming language. Oh, gosh. It depends on what I'm doing. So it would have to be CC++ Ruby and Python. I like that you say that all in one. Yeah.

Okay. So test-driven development. Can you give us a short elevator pitch for that? Sure. I kind of have a joking name for the way that most people program in the world. It's called debug later programming. You write some code and then you go figure out what you did wrong. And you get this idea because there's a long feedback loop you actually get this idea that you're good at programming because you do get stuff to work and you don't really realize the rate at which you make mistakes so a test-driven development is is about putting a feedback loop on your work so you're working in small verifiable steps using this thing called cause and effect and every time you take a little step you find

out if you made a mistake and your mistake is the last the thing you did in the last minute or two so it's easy to find and so you cut out debug time that's the first benefit and then there's a whole bunch of other things that happen after that which we can talk about so that's yeah that's and i'm just expecting you to go on with all of the other benefits because there are so many. And part of it is the debug time and part of it is the figuring out

what the code is supposed to do. That's always been something that I've had trouble explaining to people that we sit down and we type, just type, type, type, and don't think about what it is supposed to do. What is this? What is the end effect?

You might be thinking about the solution. And so I like to quote Kent Beck when he has, he's got lots of interesting little things to say, but there's a tweet he gave a couple years ago, which was, TDD gives you a really good excuse to think about the problem before you think about the solution. And I see this all the time in my training classes is where I give people the problem and they immediately want to dive in because they've got parts of the solution in their brain and they want to get them typed out without really thinking about what would be a reasonable order to solve the problem.

They're just thinking, well, here's what I thought of, let's put that down. It pours out of your brain. Yeah, it's like somebody says, I have this list that I need ordered, and everybody jumps to Quicksort instead of asking, well, what does the list contain? And how do we determine what ordered means and so test driven development would say well let's make the list and small and try it and yeah yeah what if you had an empty list yeah how would it be to sort that that would be pretty easy so is this just another way of saying you should do designs and isn't that is tdd in some ways a backdoor way to force people to think about design?

Just kind of, I don't, you know, TDD won't make someone good at design. Although if you're, if you're aware of the signs and you're aware of good design, TDD can help you achieve good and better designs. That's what I think. And that's kind of how it works for me. And it certainly might be a better way for people who are code focused to to do that because they're actually writing code it's just test code right yeah you know so we're when you get start talking about td you start focusing on that the micro cycle of write a test watch it fail make it pass

write a test watch it fail make it make it pass, write a test, watch it fail, make it pass. Refactor is built in there too. What happened before that? You know, before you sat and did that, you've got to think about what is this thing? What is the architecture of this thing? What are the different pieces that I'm going to need? Okay, which piece should we start with? How much of that piece should we build?

Which test cases will we need? And then you start to pick, you to pick a path to take you from I don't have anything to I have something. Why is it test-driven design instead of test-driven implementation? Test-first programming. Never mind. He's right. The original name, just history, when I saw it, it was called test-first programming. And because you were programming and you were writing tests to drive, to pull the code out of your brain and, you know, make sure it does what you wanted it to do. And then it evolved into tests. I think as people did it, it evolved into, well, it's more than just programming it's really it helps you think

about design it helps you consider um testability it helps you consider a lot of things you know right kind of an in front in your face and that's what sort of distinguishes it from just unit tests right i mean everywhere i've worked we've done some unit tests, but they're usually developed maybe concurrently, maybe after the fact, and then they're stuck into the continuous integration system and forgotten about until they break. And then you throw them out. Yeah, yeah. So, this isn't just, hey, you should write unit tests. This is more of a whole programming philosophy, right? Yeah, it is a technique and I suppose a philosophy.

And then there's something interesting you know the unit tests that are being written that go into your ci system and then later get thrown out because when you change something they're too hard to fix um i often get with clients that have started to do unit testing before you know i'm teaching them what tdd is and what they have are like everything tests they'll have a test that you know basically does 10 different things in one test and you know if one of those 10 things breaks and that test is broken and now you've got to go fix it it's hard to fix so what do we try to do ins? We try to design little scenarios that describe one path through the code. So I use, in my training class, circular buffer.

And so one of the things, if I put something into the circular buffer, and if I took it out, did I get back the thing I put in? There's simple scenarios like that that drive the work. And so you wrote your book about seven years ago. Has anything changed recently in TDD? There's a couple of other things coming that I need to read about, but it's really kind of interesting. I think back, you know, when I started with TDD, it was new, you know, 1999. Not many people were doing it. And it's largely the same

that it was back in 1999. I think people have refined it. The way of talking about different things around it has evolved. But basically what you're doing is very similar to the, well, it's basically identical to what I saw Kent doing then there's, you know, hundreds of test harnesses. And, you know, there's a lot of, and then people have learned how to write tests. You know, what would be a decent structure of a test? There's a vocabulary around that. You might have heard given, when, then vocabulary, which is around behavior-driven development. Given some situation, when something happens, then something should be true, arrange, act, assert.

So there's a lot of evolution to help people understand what it is, but it is pretty much the same. Now, Kenda's talking about some new thing, which I think kind of, I haven't looked at it yet, but it's something like test, commit, or revert. Test and commit or revert. So it's like a play on the Unix command line. So if your test passes, the second part of the and will happen, which you'd commit. Or if it doesn't, you revert and try again. And basically this idea is if you're working in small steps,

you didn't invest a lot. Debugging is such a waste that you work in small steps so you don't have to debug. So I need to read about that. So I haven't read about it, so I might have it totally wrong, but given kind of what I know about him and his approach, I'm guessing what that is. But I've been seeing that on Twitter, and it's on my reading list. And you used CPPU test in your book.

Yeah. I know you've been doing some development recently. Is that what you used for your development? I've been using, I know you're Python. I don't know if you're Pythonic, but I know you work in Python. Um, and I use, uh, I've been on the project I've been working on. A lot of the work I've been doing is in Python. And I've been using, uh, the built-in unit test, uh, framework in Python.

What's it called? I think it's called cleverly unit test. Ooh, sneaky. But I do use CPPU Test. And in full disclosure, I'm one of the authors of it. And it became an open source project maybe about 10 years ago. And a friend of mine in Singapore does most of the work to keep it an open source project. And I don't really do much with it anymore.

Not much in the way of contributions, but it detects memory leaks. And it's the only one I know that does it out of the box. It also follows the no news is good news approach to tests, which kind of a, you know, Google test is fine. It looks a lot like CPPU test. Tests are structured similarly. But when you run a test, you get five lines of text for every test case. Yeah.

And that means that if your first test out of 5,000 fail, you get to scroll back 5,000 lines to find out what went wrong. And CPPU test will just give you 500 or 1,000 periods and a report in your face about what's wrong. changes if you were going to publish an updated version of your book what would you change uh you know i don't know i probably should go back and read it and find out um it's been a while since i've read it and i read it about a million times back then but of course now i read and go who wrote this i'll get quoted every now and that's like really i said that um uh there's actually what i probably i probably wouldn't do anything drastic to that book. I'd probably think about a second book, which is, yeah, but what about my legacy code? And, uh, that's a great title too.

Um, so I'll have to remember that if it's a good title. Okay. So, yeah, but what about my legacy code would be, that's a very short book. You hire a new team that looks at your legacy code and says, this is crap, and then they start over. Well, that's what they want to do, but you can't afford it as a company. Yeah, you can't afford that. And you might have looked at Michael Feather's book on legacy code, and one of his very insightful statements was, if you were to redesign your code, the only document you have that's detailed enough to allow you to redesign your code the only document you have

that's detailed enough to allow you to redesign your code is the code you have yes and it's like oh hmm maybe we should take a different approach then and that's where you know refactoring of legacy code is really the the the path you might want to try a few kind people have that have attended my training have sent me stories about what happened to their code base over, you know, a few years of doing this kind of stuff. And it was interesting to see some of these stories about how their code base totally changed from something that was in deep trouble to something that

people really liked and respected. But it is a long-term change. You're not going to give a class at a company and the next day all of their code is going to be testable. No, no, because the way I go to school on Michael Feathers, the approach to dealing with existing code is, if a feature is taking you to a certain area of the code, that's where you're going to do a little cleanup. And, you know, the idea is, you know, the Boy Scout rule, we're going to leave the campsite cleaner than we found it. So if we go in there, we don't have to

solve all the problems, but we're going to make it a little bit better. So instead of every time you touch the code, it gets a little bit worse. Every time you touch the code, it gets a little bit better. And as you learn what better means, and a lot of people don't know what better means, but as you learn and start to structure your code to be better, then over time, things improve. So one of the things you mentioned earlier was, well, I guess Chris mentioned continuous test servers.

That's one of the things that seems to have changed to me in the last 10 years, is that that's common now. Before, you ran the tests sometimes, or you ran only the tests related to your files because it took too long. But now, we commit and Jenkins does its thing. Right. Do you use a lot of automated testing? I do a lot of automated testing i do a lot of automated testing i'm typically in my uh stuff i'm the only person working on it and i probably should go through the stages of getting my stuff in jenkins i used to have a pretty involved jenkins server because i needed it I deployed my training course to Visual Studio version 6, 2008, 2010, 2012, GCC, various versions, Mac Clang, and all my code had to work in all these environments because I had no idea what my clients were going to throw in my face when I show up there.

So I had a pretty involved CI system. Every time I made a change, I would find out if I was compatible with all those different environments. It was a virtual machine sitting under my desk with Windows running in one virtual machine and Linux in another, and then an old Mac running stuff on another machine. And that really saved me from the compatibility because the best time to fix the problem is as soon as you find, as soon as you know what happened.

But now when I deliver my training, I've taken that problem away because I provide my server. I use this thing called Cyber Dojo, which is a cloud server. It basically is a simple development environment in it based on GCC. And so on day one of a training class, I don't have to worry about it anymore. So I don't use CI for my training materials. This product I'm working on, if there was more than just me working on this part, I'd probably do that. At some point, I'll probably turn this over to somebody else.

But I haven't had a need for it in my normal and my side project work. After all that, can I ask you to go back and define Jenkins? I know I brought it up, but somebody should. Okay, so Jenkins is a piece of software that runs on your computer or in the cloud. And what is it? You give it credentials for your source code repository. And so it can be watching your source code repository and it can say, oh, that changed. I think I'm going to go replicate the build that, I'm going to go run the build that was checked in

and make sure it works. Not only, you've heard of works on my machine, W-O-M-M. Oh, yes. So this is part of Jenkins is to help stamp out works on my machine w-o-m-m yes um you know so this is part of jenkins is to help stamp out works on my machine because you have to really be explicit about what your build environment is jenkins is what's known as a continuous integration server and a lot of people don't use it for continuous integration continuous integration is one of the extreme programming practices so jenkins you could say is a one of the extreme programming practices. So Jenkins, you could

say, is a descendant of extreme programming, even though people that don't do extreme programming use it. And so continuous integration would say, whenever your code is working, check it in. How do you know if your code works? It passes its tests, and it's of the quality that we're, you know, that we want.

So anyway, that's, is that enough about Jenkins? Yeah, yeah, I think so. Chris, do you have anything to add? No, so anyway that's uh is that enough about jenkins yeah yeah i think so chris do you have anything no no that's all people need to know until they decide to use it well yeah then you go use it and you had to go figure it out yeah you know so we're not gonna do that yeah but you know i having done it you know five years ago or so um i spent a day or two on it and it was well worth it for the benefits I got. And I've had clients that I say, well, you should really set it up. And you go, I don't think I need it. It's like, okay, but I'll do it anyway. And it's like a week later,

it's like, oh, gee, thanks. That was really helpful to solve these problems and that, you know, so a lot of these things, it's hard to know if they're going to be helpful to you until you start to do them. Yeah. So many of these things, it's hard to know if they're going to be helpful to you until you start to do them. Yeah, so many of these things. Because I feel like I'm constantly being told I should try this or that or rust is the best thing ever. And then, you know, if I do spend a couple of days on it, that's pretty expensive, actually. And so, I want to know what's going to be useful. And yet I can't always know ahead of time until everybody's telling me, you know, Jenkins is the way to go if you want continuous integration and test-driven development is an excellent way to program.

Yeah, so it's hard to know. Yep. Which are the good ideas? I guess you have to find people you trust and then see what they have to say and see if you can feel the same kind of benefits that they might feel. You know, what problem are you trying to solve really is an interesting question to ask, too. Because as engineers, oh, try this. It'll be helpful. Well, how?

What problem is it going to solve? know why should i change and if i listen to them i change my ide every week well i would just say which you know definitely there which problem you're trying to solve and for instance if it was i want decent refactoring in c and c plus plus and xyz actually has it and you could demonstrate that. It's like, okay, then I might want to switch to that. So back to the automated testing and continuous integration. Do you find that there's a tradeoff between being able to commit easily whenever it works and testing everything, even if the tests take hours to run?

Well, there's this idea. It's a really complicated idea. There's fast tests and slow tests. That wasn't that complicated, really. The fast tests you would like to run every time you make any change. And, you know, for instance, CPPU tests, if we were making a feature change to CPPU test, it has a thousand tests.

And they run, I mean, aside from a clean build takes a couple minutes, but an incremental build takes a few seconds. And so every little change, you can run all those thousand tests and make sure you didn't have any unintended consequences. If I chose instead to just test one class, there might be unintended consequences that maybe one of those other tests might detect. So if I could determine that sooner, that would be better. Now that said, then there might be other tests that take time. So think of an integration test or system test where maybe you want to have a real network connection in it or a real database or some sensor that has a time lag or whatever it might be. Okay, now that test is going to take longer and, you know, you can't run them instantaneously. So you got to have that as a consideration. You know, something like Jenkins, you can set, at least the last time I used it, you can set its check interval.

And so what that would mean is if, let's just say, you know it takes an hour to run your full suite of tests, you might give it a one-hour check interval or a two-hour check interval. The sooner you know you broke something, the easier time you're going to have to know what it was you did to break it. And, you know, so you want that to be as short as you can do. Now, if I have a system that's got an eight-hour test cycle, well, I might say, why? And maybe there's some systems that need to have an eight-hour test cycle. But most likely, there's something wrong with the way you've decided to test things. I was working with a medical robot company out in the East Coast somewhere.

And it was really interesting. The bottom part of their system was real-time CNC++. And then the top side where the intelligence was was C-sharp run by kind of the management of the machine. And there was a handoff between two of these robotic machines where samples of something would have to go from one machine to the next. And the person who wrote that code, they had to sit there and wait like 40 minutes for the machine to cycle up to get to the point where it would actually try to send its first sample across this little conveyor belt.

And, you know, if they get the if statement backwards, if they forgot something, if there's a no pointer, you know, whatever it is, they get to go do that. They get to relive that half hour sitting there over and over and over again. You know, their hands were tied. They didn't think there was anything they could do. But it's all about the design of your system. Is it designed to be tested? You could test that logic independent of that machine. So it was a really fun consulting job

to help them see what they could have done differently. But that also suggests that while you might strive to do a good design for your system, if you want it to be testable, you also have to incorporate that into the design. You have to design the test infrastructure and the way you've architected your system to be appropriate to be well-tested. Because if you do a terrible job at it, like you're saying, you end up with a suite of

hours and hours of of tests that you can't really pick apart because any one part of the system might have broken any other part you know if you get back to uh you know benefits of tdd's now so in the beginning when there is no code uh it's very easy to write tests for it because there isn't anything there. So if you let tests influence your design, so for instance, imagine the happy path of a design and then all the error variant cases that you need later. The happy path is pretty easy to test, and it was buried down inside of a big domain object or a big you know object um and now they're now we have to deal with all the air conditions and it's buried

in these private areas of this design it's like we can't get to it it's like well why is it private because no one needs access to it okay but that doesn't mean you don't need access to it to test it. And, you know, so it influences, you know, the separation of responsibilities in the system. You know, so it's kind of vague to talk about it without a real example. But a lot of this happens when people do a big design up front and then they get down to the details and they find out, well, are complicated but it's buried inside of this thing we didn't want to reveal this interface and you know so but now you know there's another universe down there in the fractal nature of software you know we're going to layer it or um you know structure the insides in a good way so they can be tested you Your car is not welded together. They don't do that with Teslas, do they?

No. They're not welded together. They're bolted together so you can take them apart and look at the different pieces even though you're not supposed to. You've got a steering wheel with a gas pedal. Yeah. Or accelerator, I'm sorry. It is important that it is testable in units, but it needn't be testable in units by the consumer. Right. Right. Absolutely.

So you mentioned a side project. And I want to talk to you about that because it is an instance where you have hardware and you are are back to doing embedded systems development with the tdd mindset and how's it going what are you working on my brother has a business which is he's got a lot of interesting ideas about how to uh automate some low-tech um industry stuff where you're measuring water pressure. And you need to know the flow rate of water. And so one of the things we started talking about a couple of years ago is how we might automate this. And the idea would be there's a hose nozzle that's flowing water, and it could have a little—right now there's a guy out there with a gauge and a clipboard looking at the gauge that's jiggling as the water goes through and making his best guess about what that first tick mark means between these two numbered things and writes that down in a checklist.

So anyway, that's a little bit about the—and then this information is kept as records for the long-term health of the equipment that's being monitored. So what we're looking at is a way to measure maybe a dozen pressure points, And then getting that information back into the building, into its basement, where the guy is running this, you know, the source of the water for this test, controlling the flow. And then there's certain key measurement points we have to take. transposing digits and stuff and using maybe a thousand gallons of water a minute while they run the test okay now you guys live in california so that'd be really popular out there here in chicago we can do it you know as long as you know we got a big giant lake here filled with water so um but the technology is really interesting it's's a very, you know, so you're asking me how does TDD impact this. And so we're looking at, we already had a product that used a certain ADC. And this ADC communicated with a little microcontroller through SPI.

And so we had that, and we had a high a high resolution gauge. My brother's company has that high resolution gauge with a little computer and a display, right? So here's an electronic version of that gauge with the jiggling pointer, except there's no jiggling pointer in this because it gives you a digital readout. So now how do we put that technology into a bigger system? Okay, so we started looking at vendors of, you know, what you consider IoT technology. And through Twitter, I found some guy that I taught TDD years ago said, hey, you might want to look at our stuff because we made this easy. And so I said, okay, we'll look at it.

So if I describe the layers in the system at the bottom there's a adc with spy talking to a micro microcontroller and now that micro controller is going to need to talk to a little um iot radio mesh network forming radio and there'll be a bunch of those and they will talk to a a little Linux box about the size of a pack of cigarettes. And then that little Linux box will make a Wi-Fi network, which the operator of the system will have a tablet that they connect to that Wi-Fi and monitor what's going on in the system. And so I just rattled through a bunch of layers there and pretty much the only thing i knew about when i was getting started with this helping my brother is the part i wasn't going to work on uh the embedded c underneath where the radio was going to be because he already had built that um so how does the mesh

radio network work okay well i don't want to have to learn that. Is there something good out there? And we came across this company. They're called Synapse, and they have basically a Python infrastructure that allows these little radios to communicate to a little Linux box. And a little Linux box could form a Wi-Fi network and communicate to a tablet. And so there's, at the lowest level, we've got this micropython that has to talk to an ADC, and then we have to get those measurements either in their raw hex form or somehow converted to PSI through the radio network up to the little Linux box, and then that has to get formed into something that this tablet application could look at, either a custom Android app or a web server. These were lots of alternatives that we had.

I think I told you about my Zombies article. I don't think you told us on air, but yes, I read it. Yeah, okay. And so it's kind of interesting because, well, Zombies is about procrastinating on the hard part. There might be some mystery about how something might work, and you get really simple test cases going first. Zombies stands for zero, one, many, boundaries, interfaces, exceptional situations, and keep it simple. TDD guided by Zombies is the name of the article. So basically, you know, if you have nothing like a circular buffer, the article is about circular buffer.

And using that as an example, you know, if I had a brand new circular buffer, it would be empty. And, you know, it's really easy to get it to pass that test because you just write is empty and you hard code return true. And now that test works and now you move on. You grow the behavior slowly and invite your listeners to go read that and maybe chat with me about it on Twitter or something. Now, that is procrastination, deferring some of the harder decisions, you know, in circular buffer, it doesn't get interesting until you're wrapping to a full buffer or wrapping away from an empty, you know, whatever, you know, there's boundary conditions in there that are interesting that you've got to integrate a bunch of little ideas before you get to that point. So, weighting is actually helpful in solving that problem.

Now, with all the technology risk I just I just mentioned there waiting in that is suicide. We might not know that, you know, if we chose to use synapse, like, would it do what we needed to do? So the first experiment I ran was, you know, before I invested any programming in synapse was, can I get messages? We needed one message a second from from a dozen sensors. So I thought, well, why don't we just make, you know, I'll buy five of them. We'll have them each chat at 10 messages a second. And let's make sure that none of them get lost as I, you know, walk further and further and further away from the little Linux box. You know, at what point does it break? And, you know, can we go through brick walls and stuff like that um so we qualified that with

an experiment um nicely enough the company um i said well could you give me a reference can you give me an engineer that could uh you know i could pay to write me a little script to do this and they said oh i'll just do it for you over the weekend um so he gave me the script and it proved out that their technology could do the job and it it was like 20 lines of code in the radio and maybe 50 lines of code in the Linux box. I'm thinking, hmm, this is pretty cool. But we proved that little bit of technology out. Now, the next question about that technology was, could it talk to our ADC? Because they claim to have SPI in their little radio.

And actually, Bob Martin and I did a little series where we were on video at CleanCoders, cleancoders.com. And we did several Bob and James hanging out programming together. We got his oscilloscope out and that sort of thing. And we were going through this activity of getting this little radio to talk to the ADC. It was really interesting what we uncovered. We were noticing some errors in some of the readings. And it's like, what's going on?

But we ignored them for a long time. And then eventually it's like, no, let's go find out what that problem is now. We worked through by watching the oscilloscope and checking the bits it's like oh that looks like a 8 000 80 722 hex is in this uh this spy reading and every now and then it gets clobbered with uh seven ones you know for really unluckily how often does that happen then we go figure out we've misprogrammed the device of course um and didn't realize that the device was we weren't using the device quite right but um discovering that became possible to do okay well now let's see how do i get my tablet to talk to the little linux box and so we talk about a number of architectural things to do

there should we have a custom app uh in the Android or should we try to do a browser app? And the intriguing thing about a browser app is then we only have one thing we have to change code in to deploy new features. And so we went down that road. Actually, I did experiment with a custom Android app for a while. And then we ended up with going down the path towards the uh the browser at least for now maybe later it will become custom but right now it isn't i probably talked enough there the steps you took the qualifying the radios the looking at things and discovering errors that are errors in your configuration. These are all steps that sound not TDD. Ah.

Well, you know, people in my training class used to say this to me. How can I possibly write a test if I don't know what the code is supposed to do? And I would look at them kind of funny and say, how can you possibly be writing the code if you don't know what the code is supposed to do? And I would look at them kind of funny and say, how can you possibly be writing the code if you don't know what the code is supposed to do? And sometimes that would elicit a laugh, and other times it would be like, you just don't get it. So, in the world, and in nature,

and in programming, the boundaries are where the interesting things are going on, and where the mystery is boundaries are where the interesting things are going on and where the mystery is. So if I look at this problem, and kind of my strategy for solving this problem has been to understand the boundary, and that means probing it, writing some code, seeing what it does, trying to break it, getting a Wi-Fi connection and then dropping the Wi-Fi connection and seeing which area you get from the socket,

experimenting with it so I know what this does. I could go and read a book on socket programming, or I could play with the socket. And what I really need to do is a little bit of both. Get some example code together, experiment, probe, find out what it's doing, and then so that I can think of this as discovery, discovering what the code needs to do. For instance, at the ADC, we get back a 24-bit binary number.

And the MicroPython as three bytes. As it turns out, we thought, oh, we should convert that to a PSI while we're down here, and then send up just a PSI number. Well, as it turns out, we can't send a PSI because this radio wasn't capable of doing that math if we wanted to stay in the Python environment. So the best thing we could do would get a, I don't know if it's a 16-bit number or whatever it was, but we were kind of limited in this MicroPython world to what was possible. So if we had specced out the interface that said, thou shalt produce a PSI in decimal

coming out of this radio, that would have been really hard because, well, we would have had, maybe not really hard, but we would have had to do some extended precision math, reinvent extended precision math in this radio. And why do that? We could just send that hex value up to Linux and deal with it in a very natural way there. So we discover what the capabilities are. And then from that, we minimize the dependency on the thing that's out of your control.

So for instance, this radio infrastructure is out of my control. They might go out of business next week. I have no idea. Or we might decide they're too expensive for our product, or maybe a number of reasons we might not want to continue. Maybe 2.4 gigahertz isn't going to work if we have to go through basement floors to get to, you know, maybe we have to have 900 megahertz and we have to change vendors. So part of this is discover how this thing works and then make it small and put an interface on it.

So I don't really recognize what that thing is. And now I'm in this area of purely a matter of software. So I have an interface. I can say, get me an ADC reading. And I can have another interface that says, send this to the hub. And in between is the formatting of the message to what I want it to be. And that's fully testable. So at that point, that thing is fully testable. Those other things, I make them work in the environment, and then I don't touch them again.

Well, and if I do touch them again, then I take on the cost of retesting them manually. And, you know, maybe it would, in some other environment, make sense to have automation for that. But in my environment, it's kind of the same strategy I use for my website. Certain areas have automated tests andios I don't have control over, so how do I test them? And it's nice to hear you say, yeah, there's sometimes, you test the parts you can, and the parts you can't, you try to loosely couple those. So that you can remove them later or look at the boundaries more closely. But also giving time to, just like you don't dive in and write the code immediately without understanding,

you don't dive in and write the tests immediately without understanding. You spend some time doing experimentation and consequence-free kind of experimentation that isn't going to necessarily be written code to gain some understanding so that you have something that you can design and put together in a sensible way yeah right right that's tough to explain to your boss sometimes well you know hopefully they hired a professional and you can just tell them you're right have that conversation with them um and it you know this i mentioned the script that the uh vendor provided me um and i also cause and effect is this thing that I've been, it's been something that's been not bothering me,

but I've been appreciating more and more over the last five years. So I had a thing that worked. On this other thing, the tip I gave you earlier, which is it's easier to keep a system working than to fix it after you break it. So the vendor gave me the script that worked. And so my strategy for turning it into what I needed was to keep his script and to keep adjusting it carefully, slowly, until it was feeding back the messages that I wanted as hard-coded strings

to see what's involved with that to get the messages to look the way that I wanted and throwing out more and more of his noise code that actually wasn't needed. And every line of code is there for a purpose. And I know what the purpose is because I know what happens when I take it out. And so this discovery thing heavily relies on things I learned from doing TDD, cause and effect. I write a test. It won't compile. What's the compiler error that I get? Oh, that means that I need to add the signature in the H file.

Add the signature in the H file. Oh, if I did that right, I'll get a linker error. Okay, now go add the code into the C file. And okay, now if I get that, I should get a test. Cause and effect every step of the way where you're doing these small verifiable steps. Number of interesting well so that impacted the way i approached this problem even though i wasn't automating the test everything i was doing was from a test perspective cause and effect perspective that was on the list of topics when we were talking about doing this show, and I had, you know, why is that important? But now that you say it, the opposite of that is write 900 lines of code and then wonder why nothing works.

Yeah. much you can't untangle the cause and effect because you have 20 causes and 30 effects and so that's right if you just keep doing it slow take steps don't take leaps that's that's really what it's about and um this might be a little bit of a tangent but um you know getting a program to work is a challenge and it's hard and uh i call it the aptitude test if you can get a program to work is a challenge and it's hard and i call it the aptitude test if you can get a program to work and i've my brother's company's had a few of these programs written for him and he's asked me to do stuff to him afterwards i came up with this because the apps did work but that meant the programmer passed the aptitude test they were able to get the past the the program to work although it wasn't really in any kind of good shape to be able to maintain in a long-term way.

So there's more to being a programmer than being able to pass the aptitude test, although important to be able to pass it. And if you've gone to Stack Overflow and grabbed a chunk of code and said, oh, good, it works. It works for now. Yeah. Well, so I, that's part of one of my ways to, to learn how stuff works,

but then I don't just leave it that way. It's like, I start to go find out, well, let's see, why does it work? You know, if I,

if I only use the first line of code, what would I get? And what if, and then what does the second line do? So there's this cause and effect thing that I learned by doing TDD that pretty much permeates, actually permeates to my home improvement projects too. It's like, you know, if I'm making, putting in a light switch, it's like, okay, now maybe I should turn the breaker back on and see if it works before I go screw it into the wall.

Because I don't want to have to take it out of the wall again. And, you know, little things like that. Did the thing I do, does it really work? Yeah. I mean, I totally agree. That's a life thing. Yeah. I see that a lot myself.

And I sometimes see the, I did 97 things and I don't know why. So I see both sides. And the cause and effect is a lot easier to debug. On this list of topics for this show, you had another one that had me a little puzzled. It said having friends. And I'm generally and cautiously in favor of that but i wonder if you have something in mind so um one of the appealing things to me about using this radio system was that it was written in python and i'd heard i should learn python and i never had and

so i thought okay um let's let's learn Python with this. And I happen to have a friend who's totally Pythonic. And so when I run into something, I can call him and he's really kind. He answers the phone even after multiple times I've called him and he'll help me with stuff. And I ran into another guy at a conference who knows JavaScript and UI stuff. And he said, oh, that sounds like a cool thing you're working on. I would pair with you on that. And so he helped me make the UI. Another thing I don't know anything about is JavaScript and websites. And so having these friends and then, you know, the vendor was a friend by providing me some working code.

It's easier to keep a system working than to fix it after you break it. They gave me a little bit of working code, and you can do interesting things with something that works. You can experiment and see if you can keep it working. Are there other ways of doing it and that sort of thing? And so having friends, that's, so my friend, my friend, Tim Oettinger, he's my, my Python go-to. And then Rob Richardson, he's actually from a different universe than me because he's from Windows world. And, but, you know, he's been really helpful and generous to, you know, give me some of his time to help me work through some of this stuff. If I had to go read all the docs, I would have just given up because that would have, you know.

Working examples is what works well for me. Maybe something else works for other people, but having something that works and then being able to do variations of that. Yeah. I am one of the people who tends to read almost the entire manual, but that's because I like to read. And then do you integrate it all and know what to do? I mean, it depends. Not the Cortex-M4 manual, because that was a million pages long. But some of the other processors i can internalize well enough or i can at least understand where the important bits are where

is this part that tells me how to set up a gpio for spy interface in the spy section or in the gpio section or in the interrupt section. It's spread across all of them. Usually, yeah. But just reading through, at least paging through and seeing how they've structured their document helps me a lot. Yeah. But yeah, I like data sheets. It's weird, I know, but I'm happy to read data sheets and manuals.

And then I forget to read the errata like everyone. I might read them, but second. It's like if there's this, so I get the vendor to give me a script and it works. And now to understand it, I'm going to go back and look at the reference things. I know when I was first getting into test-driven development in the early 2000s and trying to do some simple thing in Java, like use a log. And so you can go to the Java doc

and find pages and pages of stuff on how to use a log and then go try and instantiate one. It's not going to go very well and then you know spend a day trying to figure it out why couldn't they just have a little bit of example code that is here's a very simple log here's how to get started but no read you know 14 pages of java doc and integrate that and know what it means and then you know then just write the log correctly the first time now i'm sorry that didn't work that way for me anyway. That is where Stack Overflow becomes helpful.

You get the initial and then you can read the doc. Yeah, right. So on the order of things that are different among people who agree on many things. Wow, I have no idea where that sentence was going. But you want to have different approaches. things wow i have no idea where that sentence was going but uh you know i'm looking forward to it different approaches i heard recently from a friend who was watching a jack gansell uh presentation that jack is recommending people do code inspection before testing uh group code inspection before testing that might be out of context it might totally be out of

context it was like and and many of the other things that were reported to me about that were clearly lies um but that one i kind of believe jack's really into code inspection what does testing mean in that case this is the out of context part yeah i mean the what was reported to me was any testing like you barely compile and then you you send it out barely compile all the time all right let's see so that i so i don't have to be totally at odds with my friend jack um test-driven development is misnamed. It's not about testing. Okay, you happen to create a lot of tests, but test-driven development isn't testing.

Testing is something you do at the end. Test-driven development is something different. So maybe Jack and I could come to a truce in something like this. I'm in favor of, I love it when I get to pair program. So now that's not a formal code inspection, but it's two brains are better than one solving a problem. My blind spots are not your blind spots and vice versa. And, you know, so interesting. I haven't seen that.

I haven't looked for that from Jack. Well, I think I agree with you that it's probably not tests like test-driven development. It's tests like integration tests. But where do you see code inspection falling in your process? I would, so before people take my training class, they answer a survey. And if anybody's interested to look at it, you can find it on my website. Underneath resources, you can see answers to feedback, and you can see what your colleagues are doing. And people say they do code inspections.

One guy will say, yes, we do them all the time. The next guy in the same company will say, we do them, but they're really crummy. The next day will say, it says that we do them in our process document, but we don't really do them. Yeah. It's the same company in the same team. Yeah. It's all guys that work right with each other, and they're kind of looking at each other as I read through these. Things that come up in the course of people learning td they're worried about uh what if i can't think of the test what if uh what if there's this problem what if there's that problem um well

what do you do right now what if you wrote the wrong test okay why did you write the wrong test well um because i had the wrong idea of what the code was supposed to do well um this is you have this problem whether you write tests or not, if you have the wrong idea of what the code is supposed to do. What do you do about that now? They'll say, well, we do code reviews. It's like, okay, keep doing them. So for me, I think socializing what's in code is important from a lot of different perspectives. One, you're going to learn so much from the people you're working with, and your people you're going to work with are going to learn from you as well. I'm presuming people want to learn.

And, you know, so I would say code review is this thing to make sure that everybody knows what the code is. Part of this continuous integration thing, all these things are connected, so sorry about my rambling. In extreme programming, I don't own the spy bus module, and you don't own the socket module, Alicia. And Chris, you don't own the user interface. We work on features that go across the whole system, right? And so we learn what the whole system does,

and we move around the system, okay? And you try to make the code look like it was written by one person. So there's a number of these kind of lofty goals there to help improve the quality. So, you know, sorry about the long answer, but I think there's a place for reviews, certainly, making sure, you know, it depends on, it's all context. How big is your team

um you know that sort of thing so i would i would like real-time reviews and then spreading code around we're we're finding at motorola when we're doing this stuff in uh 99 2000 that uh code reviews changed a lot we stopped reviewing the production code and only reviewed the tests and the interfaces to the things we were building. We thought because we were pair programming and everybody on the team was seeing it through rotation, that the code was going to meet the coding standards, etc. So we assured that we were finding nothing in our code reviews um and then the interesting things were the interfaces and the behaviors that we had defined in the test and that was more significant for us to review so it kind of changed that i think um tangentially you mentioned people saying well i don't i don't know what did you say i don't know how to write the test now, or I don't really know how to do this particular thing.

It kind of suggests to me that there's, with all of these kind of development philosophies, there's a feeling when I approach them that there's an expectation of perfection. Like, well, if I do TDD, I've got to do it for everything, and I've got to do it right, and I have to continue doing it for everything forever. And it's still better if you kind of don't go all the way, just as long as you do something. It's kind of, you know, the perfect being the enemy of the good. Yeah. I think stops people from pursuing things a lot of times, and they just fall back to, well, I'll just write the code and whatever happens, happens.

Because I don't want to try to do this new thing and then not do it 100% like everybody says I should do it. Yeah. I think, you know, if everybody says you should do it 100%, then somebody's not thinking. The places where you do TDD, this is an interesting thing, the places where you do TDD, you will do it 100%. New code that you're test driving and you've isolated the responsibilities, you'll get 100% test coverage and that code will be test driven. So if you look at my code, the places where I test drive, it's all test driven. And then the places that I don't are minimal and usually straight line, low complexity, maybe no if statements at all in the code, or maybe, you know, maybe one level of if statements at all, something that would be easy to check. You know, so there's a judgment and you can go read stories about people

trying to learn TDD and it's like, yeah, I thought I'd try TDD and well, we used it on the UI and it was horrible and it doesn't work. So I'm going to give up on TDD. It's like, well, yeah, I mean, go read about what people say to do about UIs. They say, don't bother. Go do something else.

Model view controller. Go test the model or go test the controller. Don't bother. You know, go look at the view, you know. So, it's, I think, a very pragmatic kind of, I would like to consider it a very pragmatic thing. This all leads into my next question, which is, I hear that you have a pretty good course. I've heard that. I read it on this show once. Pretty, pretty good.

So, I did, in fact, in some previous show, mention James' course was pretty good. And for me, that's a pretty good compliment. It's a good compliment. It's a great compliment. That's a circular definition of pretty good. Yeah. Okay. Pretty and good.

Okay. It's a course I enjoyed attending, and I learned a lot, and it did change my programming philosophies. And you should all take it. And now you can, this pretty good course. I think you should rename the course, James. The Pretty Good Testing Course. Yeah, Pretty Good Test-Driven Development Course. Yeah.

Because, you know, it's not a testing course. Right. But now you have a live via the web course and Cyber Dojo and people can learn in a group and learn on code. And maybe I should let you tell me what it's about. Yeah, okay. So, it's basically my two-day training course that I spread over three days because being on a webinar for a day is horrible for everybody involved. And I go through three cycles of explain a problem and a potential solution, demo a little bit of the solution, give people an exercise, hands-on exercise that they do

in my CyberDojo server. I can monitor every change they make while they're doing it. After the exercise, we do a debrief session where we find out what people's reactions to it were and that sort of thing. I do that three times over three days, taking people from the initial point of what is TDD and how is it different and why would we do it? And, you know, what are the steps? And then we get past that and we get into a more complex example. And then finally on the third example, we're working on a device driver. It's pretty much the same thing I do when I'm on site. But given the, you know, the technology that's available, I think I just scheduled my 18th or 19th one of these.

I've started doing them about three or four years ago. And I had really kind of no business doing them when I first started. It was kind of a mess. But now I've evolved my infrastructure, my website, and my delivery to really support it. And so that's kind of what it's about. And who should attend? Is this people who are advanced in their careers or new in their careers or not even in embedded careers yet?

Just ask yourself how much time you spent chasing bugs the last week. And if that number is greater than five minutes, the course is probably for you. And if it's not, you're very suspect. If it's not, I don't believe you, or you're not programming. Right. Okay, so I just,

you know, so anybody writing code is probably doing some debugging. But no, more seriously, if I were to think, you know, so I started to do this live via the web because my normal thing would go and work with 15 or so people on site for the better part of a week. And that means you need to have 15 or so people.

And that means it's a bigger investment for you. It's a bigger investment for me. And I would have these, you know, a couple of guys in Europe that would say, oh, I'd like to take your class, but it's just two guys. Or, you know, we've got a bunch of engineers, but we don't know that we want to try and do this.

You know, we're not sure. So to pay you to do a full week class is like, maybe we don't want to do that. So I had a number of these things hanging out there. And so I thought, well, I could maybe help them out, which I like to do. And it would be a way for me to not travel as much. And it turned out, even though it seemed to go horrible, the first time I did it, people kind of liked it. And then I started to, each time I delivered, I evolved my website to support the training class a little bit more.

For instance, in the beginning, I'd have to email a link out to all the attendees about where the exercise was, which meant, oh, you know, I didn't get that email. And it was very disruptive. And now I have a webpage where, as the course progresses, the exercise links appear, and it's easy to know. After the first one, you know what to do after that. And it's a lot more seamless of a mechanism. I use the same mechanism now when I'm live, when I'm at a client site, just because it just works better. But when you go to a client site, mean the web the web version is a couple days when you go to a client site that's like the first couple days and then the next couple days you do different things you use their code what else do you get what else do you change when you're

on site with a client yeah if we're on site then uh typically, you know, the day after I leave, before I started doing this, what I call legacy code workshop, the day after I leave, people didn't know what to do. And the take up rate was not so great, I don't think. It's hard for me to know for sure. You know, some people do it and I hear about them and then the other people that give up, I don't get to hear about them. And then one of my clients said, you know, we're going to bring our leaders from, you know, around the world to our office in Texas. And we want you to stay with us, do the training and then stay with us while we work in our code. It's like, sure, I'd be happy to. The first problem we run into, the person who ran into it, it's like a header file that was tied to a certain DSP,

I think a TIDSP, where, you know, if you tried to test that code off target, you would say, you can't do this, you know. Well, we solved that problem. And, you know, so we worked through in the legacy code workshop, we put their code up there, we kind of now it's evolved into a mob programming activity where pretty much the whole group is working together, solving one problem at a time to go through one example. And it might take us a couple hours to get the first bit of code that

was never designed to be test tested, or never designed to be compiled off the target environment to get it into the test environment on your whatever it might be, a PC or a Mac or something or a Linux box. After we do that once as a group, then people typically break off into smaller groups where they work together and solve these problems. And I kind of go around and see what people are getting stuck on. And I do have one of my, as I was doing this in the early days,

after every trip I would write another article about the crazy thing we ran into this time and what to do about it. And so there's an article on my website called How to Get your legacy c into a test harness and in it are a bunch of uh one of the steps is don't panic yeah and uh and then when you hit the don't panic you're supposed to go see if you can recognize your problem in this list of problems and then if it is you click through and you see a detailed example of what to do about it um so we do we're doing that and typically an interesting thing for me in this activity is that people really start to get it.

And then some of them are ready the next day to keep going. And that's the guys that were, no, this is a terrible idea on Monday. They're like coaching everybody on the last day. That's got to be pretty rewarding to see. It's fun. It's fun to see people getting it and seeing that it might be helpful to them. They still have more work to do to prove it to themselves. I can't prove to anybody that this is a good idea, but I try to give people an opportunity to prove it to themselves.

That's pretty much the approach. People might think I'm trying to convince them, but I tell them, at least in these words, it's your opportunity to convince yourself. I think we have kept you long enough for a weekend day, which is hopefully as pretty there as it is here. Do you have any thoughts you'd like to leave us with? I've got one little saying I kind of like that I heard long ago. It was actually on one of the marketing people's wall in a company I used to work at in the 80s.

And it said, it's easier to act your way into thinking differently than to think your way into acting differently. Yeah, that's neat. There's a lot of goodness in pretending you're good at something. You know, it's just, you go through the motions, and then you might get to appreciate, you know, how I relate it to, you know, kind of things we've been talking about here is, you got to try it to find out if you like it. You got to try it to find out if it would help you. And a lot of things it's not going to hurt to try. Yeah. Yeah.

The broccoli is good, too. You know, I even didn't think so when I was a little kid. Our guest has been James Grunning, author of Test-Driven Development for Embedded C and founder of Wingman Software. That's wingman-sw.com. You can find him at JW Grenning on Twitter. Thanks for being with us, James.

Thank you, Alicia. Thank you, Chris. Thank you all for listening, and thank you to Christopher for producing and co-hosting. You can always contact us at show at embedded.fm or hit the contact link on embedded.fm. And now I have a quote to leave you with from Charles M. Schultz. Sometimes I lie awake at night and I ask, is life a multiple choice test or is it a true false test? Then a voice comes to me out of the dark and says,

we hate to tell you this, but life is a thousand word essay. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.