Two's Complement - Special Guest: James Grenning

Episode Date: January 13, 2021

Our first guest! We speak with James Grenning about his work (re)building embedded systems using Test Driven Development. Then we ask James about his involvement with the creation of the Agile Manifesto in Feburary of 2001, and find out how 'Agile' has changed over the last 20 years.

Hi, I'm Matt Godbolt. And I'm Ben Rady. And this is Two's Compliment, a programming podcast. Today, we're delighted to have our very first guest. James Grenning joins us. Hi, James. Hi, Matt. How are you doing? Very well, thanks. Hi, Ben.

How's it going? Long time. Yeah. So, James, do you want to give yourself a little introduction? Sure. Yeah, I'll try to make it short, but in high school, I had a friend that was programming, and it looked like a terrible thing. It involved punch cards and rooms with no windows. So I tried to stay away from that. And then in college, I bumped into it as part of engineering school and found out it was actually fun. And then, well, here I am, you know, 42 years later, still enjoying it.

Have been lucky to run into a lot of good people along the way. Well, like Ben there. We were working together about almost two decades ago. I think maybe not quite. Gosh. Not quite. Ran into Bob Martin early in my career. We worked together at the same company.

So I did some work at a big company for about 15 years and then ended up consulting. So I like to say I've been unemployed since 1996. And then somehow Bob said, let's go to Snowbird. And snowbird and i said oh yeah i'd love to go skiing and then there was that agile manifesto thing so i got involved in that totally by accident we talked about that later and uh my background is embedded systems and i learned c back before you know they had all the bugs out of the compiler and in embedded systems he is used a lot. When did they get the last bug out? No one told me. Yeah.

Well, not all. I mean, the bugs are still winning, but at any rate. So now what I do mostly is try to help people learn how to do test-driven development in embedded systems. And I wrote a book about that and that's my business. And that's kind of why you were an obvious choice for us when Ben And that's, I wrote a book about that. And that's my business. And that's kind of why you are an obvious choice for us when Ben suggested, hey, I know a guy, Ben and I have been talking about testing over the last few episodes, and in particular, testing in C and C++ and why it's different from everything else. Why? I mean, maybe it isn't as

different as I think it is. But certainly certainly in my experience compared to some other languages where it's very easy to install mocking um write tests around the edges of things all those kinds of things where c++ you tend to have to design your system a bit differently which we've discussed to death but i would be interested in your experiences but sure the first question i have really is what makes something an embedded system how do you define embedded good question by the way for me so that my market is as big as possible but it involves a lot so is my cell phone an embedded system yes i mean certainly there are embedded aspects to your phone and anything that you kind of look at well actually phone is kind of hard to distinguish from a computer these days

or a supercomputer already 10 years ago, right? Yeah, sure. But if you looked at like the early cell phones and such, they of course had computers in them. And you would have certainly thought that was an embedded system because you recognize it as a phone, not as a computer. And so for my definition is kind of colloquial, I suppose. If you don't recognize the thing as a computer it's an embedded system got it now that means it could be quite big it could be you know the a phone a camera all kinds of things have computers buried in them cars millions of lines

of code in cars it's like really scary but yeah so a car could be an embedded system and in fact yeah some of us have cars that are very much explicitly you know self-drivable things which is terrifying but but i was trying to explain to my my kids earlier actually i said about what embedded system was and i was pointing it like the oven and there's a little clock display and i said that's an embedded system and then that's right you know and then i was sort of working my way up and that's what made me think is a cell phone an embedded system i know you know the modem in it is a has a baseband firmware and is doing some crazy rf manipulation stuff and that definitely fits my mental model

of what an embedded system is but the phone itself with all the the arm cpu and doing the the computer things it still shares some of the aspects with what i think of as a traditional embedded system that is it's it doesn't have a keyboard i can type on i'm not developing on it and i can't just write my tests on my my computer and have them run on it in the same way necessarily that they do on the actual device itself which i think is maybe what makes embedded testing different from other testing or am i backing up the wrong tree there um i'm not sure a tree you barked up but uh you i was off like thinking about can i ssh into my phone it's like you know it is a little linux box after all but uh you know so but you know so are you were you asking me like is uh

testing of embedded systems different than testing anything else right i suppose that's yeah that's that's the question i have uh well yeah of, yeah, of course it's different, but, uh, it's also the same. And, uh, you know, so I came back to, uh, well, we were working on a fairly large embedded system at Xerox in the late nineties. It was one of, uh, object mentor, Bob Martin's company's clients. And it would have filled up the office I'm in. And it was taller than me and longer than my arm span. So over six feet tall and over six feet wide. And then you could chain them together. Wow. That was an embedded system. I'm used to being smaller than, sorry, the embedded system being smaller than me. That's

another thing in my sort of mental checklist of, is it embedded? But you didn't look at it as, it was a printer, right? You didn't look at it as a computer. And it has all the problems of an embedded system. So what's the problem with an embedded system? One of the problems is that you can't just fire up the compiler on it, for instance. Like my laptop computer or my Mac, I can run the compiler on that. Maybe you can run the compiler on your Android phone. I'm not sure. Probably could. It probably does when it's updating and stuff.

I don't know. So typically people doing embedded systems, they have cross-compilation involved. So, you know, they're working on one system, you know, a Mac or a Linux box or a PC, usually with specialized tools that really aren't C anymore in some cases because the silicon vendors have done you a favor and extended the language. Yeah. There's a Mac quote there for that. Yeah, there's an air quote there. You know, is it a handcuff to their silicon when you start to use their special features? Like

they'll start to make addresses. I'm sorry. They'll make registers in the device appear like global variables. And now the compiler does something special. And then they'll annotate things, say, well, this is not just a plain old function. It's an interrupt function. Put a special keyword on it. That's like, it it kind of breaks stuff but there's ways to get around all that and i suppose that's really where that's my peaks my interest because that's that's the kind of thing that is harder to test i would imagine is if you have got as you say like uh some apparent global variable that has to be a global variable because really it's a system register on your embedded system that controls, I don't know, the LEDs that are lighting up

or something like this. Exactly. Yeah, right. How does one go about testing that? So let's pick a, say, serial port. That's a great example because it's got to be driven by higher level bits of code, but ultimately you're going to be sending stuff

to a shift register or doing some magical thing. Maybe there's an interrupt associated with it. What kind of things are you thinking about when you're designing for test or just testing a piece of code that talks to a serial port? Well, it's been about 40 years since I did that, by the way, but that doesn't matter. I still remember. Okay. You know, so if you're at low level code doing one thing, you know, its job is to wait until something happens on the serial port, and it has nothing to do until then.

So I'm going to make it as simple as possible, no interrupt for starters. The software might be sitting there saying, is there a character? Is there a character? Is there a character? Is there a character? And then maybe every now and then it would go and change the LED so that you knew that something was happening. Every thousand cycles through there or something, maybe it changes the LED so that you knew that something was happening, right? You know, every thousand cycles through there or something, maybe it changes the LED. I don't know what the right cycle, what the right duty rate for that would be today. Yeah, but then once it said there is a character there, then you would go read it and then you would do something with it.

You know, so, you know, a single purpose driver for a UART or something, let's just say it's waiting for a line of text. So it would grab that character and would say, are you a character turn? Are you an enter button? And you'd say no, and you'd put it into the queue. You put it into the circular buffer, whatever it is. Are you one?

And until all that happens, and as soon as that happens, then something else would take over, right? So that would be, you might build a little thing that knew how to take characters from a device and put them into a FIFO so that something else could take it out, right? You know, so separation of concerns, you know, lots of people doing embedded systems are, well, I got to first, I often tell a joke and hopefully I won't alienate anybody that might be listening that will be offended by this joke. But what's the definition of embedded software? I don't know. What's the definition of embedded software? Software written

by double E's. I see why you said that was contentious. Yeah. Okay. In good humor, please. I know. Just kidding, everybody. Hey, I studied double E2. So don't worry. Right. But so you were talking about separation of concerns. And so presumably the sort of the take home there is that, first of all, to make something testable, one has to separate one's concerns. And then you were kind of alluding to something perhaps negatively, which maybe negatively which maybe things that maybe by default that's not what happens when people write well yeah so again not to sound like uh you know there's a lot to know in this world and to uh get an app to work

is amazing okay uh i like to say being able to get an app to work means you passed the aptitude test dad jokes the dad jokes are coming out spelled app-titude now it's amazing to be able to do it at all now the next thing is

can you do it such that someone else could actually look at your work and know what you did or could you even look at it yourself two weeks later and know what you did i keep wondering who keeps breaking into my ruby on rails website and messing it up but i kind of know who that is you know so i'm the only one with the keys so it's got to be me right um it's that previous you darn him yes so knowing uh discovering really in emergent design, right. Discovering where the boundaries in a system should be, you know, so in the beginning, I, I see all kinds of fun stuff when I work with my clients. Um, about 12 years ago, this happened with one of my clients.

They said, well, you know, after your training class, you're going to stay with us for two days. And you're going to show, you know, we're going to use it on some of our code while you're still here. It's like, okay, that sounds like a good idea. I usually tried to sell that. And I kind of gave up because nobody wanted to, you know, buy extra consulting time. So I just took the class. And this client flew people in from around the world to participate in the workshop. And then we ended with a couple, a day and a half in their code.

And, well, we ran into some crazy stuff in their code. So it worked well. So I started going for this company to numerous places around the world. And one place I was at, they brought their code to the workshop and it was one file with main in it in about 5 000 lines wow and it worked and it's doing this critical job and but there was absolutely no separation of concerns as an employee that would make my heart sink but as a consultant does that make you think, hooray, I can come in here, I can save you from yourselves?

Well, it means that I can probably keep working as long as I feel like it, pretty sure. As long as it continues to be fun. And it is great fun. Now, that also opened my eyes because I thought I would show people TDD and toy problems in a clean room environment, right? A green field. And they would be able to integrate that into the way they think and then be able to take it to work. And then they started showing me their actual code and kind of some of the problems because I didn't actually run into those compilers that

are helping you, more air quotes here right right and uh you know one guy said oh does this mean i don't have to test my code he was looking for a an out it's like no we don't give up that easy the pre-processor is an amazing swiss army knife it's a it's a yes and like a swiss army knife you can cut yourself quite badly with it as well but obviously sometimes it's the only thing you've got when you're up against the as you say vendor strange extensions and you're like well i can replace studio.h with my own one maybe and then we can record what the printf was doing or whatever heinous trickery i've i've been around i've seen a few of the things you can do yeah but I'm interested what other

you know that that's obviously we can talk about that in a minute specifically but uh you know you're you're saying that there are some tricks in the bag that maybe are not obvious that it takes somebody who's had the experience like yourself of knowing how to unpick that 5000 line file into testable components and the kind of techniques that you can use. And the fact that it is kind of worth it as well at the end of it all, which I think when you don't know those techniques and when you're not really sure that it's going to be beneficial, it's really hard to convince yourself to do the work or to, moreover, like Ben, you would often say at this point or to convince the people

that pay you that it's worth doing the work yeah yeah yeah certainly something we were talking about uh in another episode is you know when is the right moment to sort of point out the elephant in the room and say hey you know every time we make a change to this code we introduce two bugs so every time we fix a bug we get two more more bugs. So if we keep doing this, eventually we're going to have an infinite number of bugs. And if we don't change what we're doing, then that's not going to work out well for anybody. And so, you know, I don't know about you, James, but certainly when I was consulting and sort of for some period after that, when I would be mentoring or talking to people about testing, the most common question I would get is, you know, Hey, I have this legacy system.

I have this old system. I have the 5,000 line main method. How do I do the things that you're talking about? And, you know, my unfortunate response is that's probably like the hardest thing that you can do with testing, right. Is, is retrofitting that existing system. But there's always a place to start. You know, I never want to give people the impression that you shouldn't even try, right? There's always a way to start pulling things apart. There's always a way to start making more testable areas of your code. Oh, yeah, there is, especially with the Swiss Army knife in your pocket. Yeah, yeah.

And sometimes you've got to pull a few dirty tricks to get there, right? Like if that's what it takes, then that's what it takes. But I would love to hear some of your experiences sort of rescuing some of those code bases. Like what tricks were you able to pull to get some testable pieces of code out? Now, tricks is not the right term. No, that's techniques. I usually call them engineering techniques. There you go.

I mean hacks. Right. And then I have a whole webpage dedicated to this, by the way. And it's how do you get your legacy C into a test harness is the name of it. It's on my website. And it's because of that trip around the world I did with this one customer, every time I went, I saw something different. And I would write an article about it. And that way, the next time I ran into it, I could say to the person in the

workshop that has it, it's like, oh, you've got this problem. Go read this article. And then I could manage a group of 20 people if some of them are off reading articles, right? Yeah. So there's some things there. And, yeah, jokingly, I call them, now we're going to learn an engineering technique. And then I mean hack to get past this problem. And don't think I think this is a good idea for you to do and leave it in your code. Because you did this hack, it means you need to go and fix something later. But let's just see if we can get your code under test first.

Yeah. This is the mindset would be don't change the production code. Can I do something external in a build or, you know, the make file or one of the Swiss Army knife techniques is to put a include file on the command line and shove it into the beginning of the file. Right. And so if I've got one of these deeply embedded processors

that has changed the language and introduced like interrupt as a keyword and the regular C compiler says, interrupt? What the hell is that? Okay, game over. You can put a forced include in that turns interrupt into nothing. And now that line of code isn't the problem.

There'll be another one that'll be a problem in a little while, something else. Now that line that thinks it's a register, well, I can put in the forced include file. We'll start to collect all my little hacks, right? I start to put those in. And, you know, if it wanted something, capital I-N, kind of looks like a macro, it's actually a register on a certain device, I can turn in into a integer.

Got it. Okay. Now later I get a linker error. Right. Yeah. And this is to have like some kind of uh to be able to build this on a non-embedded compiler is that the sort of that's right oh yeah yeah yeah because yeah because if some of these deeply embedded things you're not going to fit a test harness in there you maybe you can get assert and printf maybe but the first step is to make sure you can run it on your own computer as part of your

regular testing and this is one of the techniques that you could employ to bring embedded code into a world where you can build it on your windows machine your linux machine or whatever yeah right that's right yep and uh yeah just teasing the code off of its environment uh if you could if people actually thought about portability which of course nobody does because what they're thinking about is we've got to get this product to work. Now one of the things with hardware is it changes so fast. You might have to redo an embedded system because the part that you depend on in your product became obsolete.

Gosh. Well, now we never wanted to change the product, but TI decided to stop selling this part, so now we never wanted to change the product, but TI decided to stop selling this part. So now we got to redesign the product. Or no, we could save 10 cents if we redesign the product. Okay, then you go and spend a half a million dollars of non-recurring engineering to do that. It might be worth it. But that almost points at a better design makes it easier both to test which would you know moving away from

modular separation of concerns all the things that we've sort of talked about but if for example i have to change i don't know my audio output device of my little embedded system because it's 10 cents cheaper to use a different dsp or a different whatever i'm going to wave my hands a lot here and you're going to give me a lot of yeah but if that is encapsulated as the audio subsystem in a nice way that makes it first of all easy to test without actually having an audio subsystem yeah it also means it easier perhaps to just switch it out and say well i can write the test for my new audio system and drop it in and we're good to go have you observed that is that a is that a thing that can happen yeah there's a there's a name for this it's called hardware abstraction layer okay um it's an old idea which gets ignored a lot and then i was listening to your last two

podcasts and you guys were talking about virtual functions are too expensive to use sometimes so you can just imagine the kind of conversations you get in where, oh, we need to add a function call here. It's like, I can't have a function call there. And gosh, you know, so I mean, a regular old direct function call, I can't have a direct, this needs to be a monolith of 5000 lines with go to's. I'm sorry, I don't have a stack. It's like, is this true this true yes this can be true oh my gosh i mean i live in a luxury land of of mostly modern compilers when you start talking about the vendors that you're and so i make all these these grand claims in in presentations i do about

the amazing things that compilers can do if you give them the right information so sometimes that kind of thing becomes a moot because oh no the inliner can see through 12 levels of indirection and you know that that function call goes away and it's just you know register access plus plus or whatever but in the kind of compilers you're dealing with that may not be true you may be talking about a 15 year old compiler with tons of magic extra keywords that aren't properly supported by any kind of optimization within the compiler so you have to think about these in a very different way. That's a really interesting thought.

Gosh, that's terrifying. It was kind of funny. I went to Llewellyn Falco's approval testing course a couple of months ago. Oh, yeah. And on the last day, I asked him, would you like me to show you one of the things I use for getting legacy code into the building?

And they wanted me to show it. So we stayed after a day. And then he had this revelation about five minutes. And he goes, oh, man, I've been so lucky. All my code actually builds right away. You know, he's working. And it's like, I mean, one of the givens for your world is that sometimes a code won't even build. And it's like, that's right.

Wow. I've started automating the legacy code build process. And I started toying with this a while ago. So my typical engagement with a client is I'll go and do a training class that lasts two or three days. Now I've kind of shortened it to two because people are overwhelmed at that point after two days anyway. Their brains are like kind of ready to explode with things that they'd never even considered before.

And then we spend the next day or two in their code. And, you know, so the first two days are in this ideal world where, you know, it's new code and we can see what the future could be, right? So you're talking about how do we help people motivate to start to change. And you're familiar with Cyber Dojo? I think I am. It's sort of like a web-based typing code. Kind of like your compiler thing that you're working on.

Is that? Right. Compiler Explorer. I mean. Similar sort of goals, I i think in terms of like they allow you well i didn't used to be able to run code but i think code cyberdojo is a thing which lets you actually write and run the code and that's kind of the focus is that that's right yeah yeah so i use it for my training it's really revolutionized my uh my business oh cool and uh

so what that means is that people when they're in cyberjo, it's a virtual environment. I have a cloud server that runs it, and I've customized it with my own code and that sort of thing. So I start people off on simple problems, and they find out how hard it is to get simple code to work, even if there's nothing else going on. So we build a FIFO, and what's that like? And can people actually incrementally grow a FIFO from a series of's that like and you know can people actually incrementally grow a FIFO from a series of test cases what we're trying to do you know starts out empty then you put something it's not empty you can know what the sequence is and you're talking about a software FIFO here just a pure boring data structure FIFO not like an actual hardware FIFO you're just saying no software you know the interrupt routine happens you pull the character off off the

device you put it in the FIFO something else consumes it later got, you know, the interrupt routine happens. You pull the character off the device. You put it in a FIFO. Something else consumes it later. Got it. And as you say, you're opening people's eyes to just the sheer number of things that can go wrong. If you start TDD, designing a FIFO from nothing. From nothing. And then working up.

And by that point, you've also extracted all of the embedded specific issues. And you're just saying, let's learn about testing or how one could write this code, first of all, without thinking about that. Gosh. Yeah. It might have been called from an error routine. Right. Which brings its own fun. Separation of concerns.

Yeah. Yeah. And so they get to see over three cycles of I show that, present a problem, demo a little bit, have them do an exercise and a debrief. We do that three times. They get to see, you know, how hard it is to get code to work at all. And even when you're making little tiny changes. Yeah. Right.

And then we go and then it's like, okay, but what about our code? Now what do we do? Right. And guess what the first problem is? Everybody's got these builds that take hours or, you know. I think you guys were talking about build time last time, weren't you? We were. We were.

Yeah, I listened to that. It will not be the last time we talk about build time. Eight seconds, I think, was your guys' number. And you were talking about this rule of eights or something, right? Yeah, yeah. Eights, 80, and yeah, okay. Yeah, so I asked people beforehand to complete a questionnaire about how long their build time is. And when they tell me that they don't have time for TDD, I say, hmm, well, you're – I'm spoiling it if any of your listeners are going to come to my class.

Oh, you know, so you don't have time for TDD, but you don't mind waiting a half hour for a build? Yeah, yeah. You got a lot of time for TDD. Yeah. You got lots and lots of time. What your problem is, you're building everything at once. You have a production build. You don't have any test builds.

Yeah. You think a build is something that's permanent. A build is something you just need. You might need just for today. So we start to do that. And actually, then the problem is I've never built this code in GCC or in Visual Studio, whatever they're choosing to use for their thing. I was just going to ask, how many people do you run into that have never tried to build their code for anything but the target platform? Does that happen? You can look at my survey on my website and see.

I think I actually cut it off to only the first 100 entries, but just because I haven't implemented the next button yet. And the database is getting kind of big. It was kind of bogging my system down a little bit. And I'm cheap, so I didn't feel like buying a more expensive web server. But Amazon, the next price up would have cost me $10. Never mind, I should stop talking about my cheapness.

You're just efficient. He's just engineering efficiency. Most people that are doing embedded systems, very few are actually testing off target. Now, I'm starting to see more of it. And oftentimes, it's the person whose idea it was to come to my course and to bring his colleagues with them.

Right. And so it's more common to only build for the target system. Right. And I think back to my first 20 years of development. That's what we did. Yeah. We might have run the compiler on it, that's what we did. Yeah. We might have run the compiler on it, but we never tried to run it. We tried to run little pieces of it.

Then as soon as we had the target, we never ran it anywhere but the target after that. And part of the story is the targets have problems. They're built by humans too, and they've got bugs. And so I kind of want to be able to do finger pointing more accurately. It's like, well, my code I know does this. The idea that people haven't used multiple compilers on their code is always useful. I mean, as a developer who's mainly in the server space, if you just point Clang and Clang Tidy, static analysis and gcc and all the different various variants of gcc different versions you always learn something new and you turn all the warnings on and you look at there's just so much information that the compilers can give you about your own code

um and if you've only ever pointed one and especially if you've only done like you say like a release build because um very often i I understand it, these embedded systems don't have the same amount of memory or storage as you would want if you were to do an unoptimized build. Or even I've had people arguing for not using some abstractions in C++ for embedded systems quite reasonably because if you don't build them in the full level of optimization then those those abstractions don't go away the compiler isn't able to optimize them out and then you miss like timing windows for hardware access and stuff like that so it does seem like a very different world for the target device in release build versus having the opportunity to build it with uh like the memory sanitizers that we now have access to on development machines you know valgrind and it's like and

all of those other things that come with it and it seems like a no-brainer but obviously without knowing the techniques that you were talking about you know without like force including some compatibility header it seems insurmountable to take your 5000 line code that every third line is using some register keyword in a very unsuspicious way and and get it building on your windows machine and running it how much easier in your experience especially with embedded systems if you start out you have the luxury of a greenfield project and you start out with enough people you know part of the team that's building it where they sort of, you know, they buy into this philosophy of, okay, we're going to write tests and we're going to have, you know, we're going to target

multiple platforms and we're going to, you know, set up the environment so that we can build and run these tests quickly. How much of an improvement is that over trying to retrofit it later in your experience. I was lucky in my younger days to be in a lot of Greenfield things because nobody knew how to program computers then. Sorry, I'm laughing. You're making yourself sound very, very old, which you're not. So please carry on. Well, I'm older than like 95% of all the programmers probably at this point, right?

People that are practicing. That might kind of always be true though. Isn't there like a curve? Yeah, probably. Yeah. I think Bob Martin was saying that every five years it doubles. Yeah, right. Exactly.

Yeah. Yeah. So pretty much almost everybody has less than five years experience. Right. Right. Yeah. Yep. So, okay.

What was I talking about? We're talking about whether – starting Greenfield versus retrofitting later and the relative costs there. All right. Yeah, okay. So, well, most of the world doesn't have that luxury because they're having to deal with somebody else's thing that they already did. Or they're moving it to a new platform because the hardware keeps changing. Yeah. And so it's really common to run into – you know, one of the questions I'll ask in a course is, okay,

who's got 5-year-old code? You see the hands go up. Who's got 10-year-old code? Who's got 20-year-old code? Who's got 30-year-old? If you're at a telecom company, it's like, we've got 50-year-old code. It could be, yeah. Probably C, you're not going to have any C older than 50 years, but you're going to have 30 and 40-year-old C code, right? It's possible. For sure. Right now, right?

So a lot of people live in that. Now, for sure, it's easier starting from a green field. And it also helps if you know what to do, because a lot of people don't really know what to do in that situation because they don't get it very often. Right. You know, it's like, oh, well, you know, when you first write tests, when you have no code, the tests are kind of stupid and trivial and testing ridiculous stuff. Okay. But it doesn't take long for the code to go beyond that, right? You guys familiar with the zombie acronym?

No, I've never heard that one. No, what is zombie? I've got an article about it. TDD Guided by Zombies. It's on my blog. And zombie stands for zero, one, many. Okay. Okay, it's a two-dimensional acronym or maybe a multi-dimensional acronym zero one many is a linear part and then the other

side of the matrix are boundaries interfaces exceptional situations and keep it simple and uh basically starting off in something in tdd is find the zero test first so it it metaphorically it's good with something like a collection because, well, what if the collection is empty? What can you do? Right, right. You can check to see if it's empty. You can make sure it's not full.

You know, if I'm looking at, let's use circular buffer because that's what's in the article. And then you're kind of out of stuff to do at that point. And then what can we do next? Well, you can put something in. Oh, then it's not empty anymore. And it's probably not full and it's probably not full because it's probably not a one element thing. So you kind of grow your way.

And while you're doing this, you're paying attention to different things. Interfaces. In the beginning test, you're really trying to get your interfaces right so that they're convenient to use. And it's trivial to get them to pass because you don't really need much code to say,

return true, it's empty, you know. And most people go, oh, that's such a worthless test. Your code is wrong too. So why are you so happy right now? Yeah, but that test saves you later, right? Yeah. When you add the code that makes it possibly not return true, but you don't. That's right.

Yeah. So I'll hear from people after this first experience. They'll say, well, it's just against my principles to write code that I know is wrong. It's like, okay, but you don't mind writing code that you don't know is wrong. You write code that's wrong all the time because we all do. You're writing code that's wrong and you're okay not knowing about that? Okay. Yeah, yeah, yeah.

But the – yeah, it's certainly easier with that i did this other conference talk which uh what is it called it's a tracer bullets and uh zombies or something like that and uh the idea with the tracer bullets you know it's a pragmatic programmer idea i didn't i was using their term uh but you go start looking for the unknowns. And, you know, it said, you know, oh, we're going to build this. I was working on a project for my brother, IoT radios. And, you know, I didn't know anything about IoT radios. And pretty much there's a long list of all the stuff I didn't know anything about. And he's asking me to help him. So it's like, sure, this will be fun. What you might do is you might go work on the business part of the problem because you understand that and you'd be comfortable with that, right? Oh, when we take a reading,

we convert it to PSI and then we can do these formulas in PSI. We know what those already were, so we could have worked on that. Except all the risk in this project is all the things we don't know. And so with the tracer bullet project is all the things we don't know. And so with the tracer bullet approach is you go start looking for those unknowns and try to build the walking skeleton of, you know, can I get a pressure reading from next to the fire hydrant onto the tablet for the guy through all the layers, you know, which involved IoT devices. And, well, it's about nine layers by the time you're done of stuff I didn't know about. When I first started, I only thought there were five layers of things I didn't know about. And then by the time we finished, there were nine.

Yeah, yeah. And we could prove the concept then, though. Now, where I'm getting to with this, and what's interesting is that all those little boundaries, the code that's in between is just like hard, you know, it's like a hard-coded, well, you know, none of the error checking is really there. It's just about can I transport something from the bottom all the way to the top. Right. And the little things you learn along the way.

And now after you get, once you've proven you can do that, pretty much the high-risk stuff is gone. And now you can say, well, we get a reading once every second. And from 10 sensors, we get readings once every second. You've got to collect those together and present them in a JSON package to the web browser. Now, once you've proven you can do anything, then you start to do the stuff which is pure software. Right. And so you're shooting the tracer bullets to find out all the stuff you don't know and getting something to work on this edge where you're not going to bother writing tests for that because you need an integration test, not a unit test. And then you try to minimize that thing that's outside your control and maximize the thing that is pure software. Right. You can play the zombies game. Right, right, right, right.

Okay. You know, so. Yeah. Right. So if I only had one, if I had no sensors reporting, what would I report to the web browser? Nothing. If I had one sensor, what would I report?

If I got two, you know, what would I, you know, so you can start to grow the stuff in between. And. You know, as much as we talk about, you know, engineering techniques, aka hacks for testing embedded systems in strange and interesting ways, it's amazing how many, so many of those techniques are so portable from one programming language to another, from one library to another, from one environment to another. You know, once you start learning the sort of basic philosophy of, okay, I'm going to test this with null values, with zero values, with empty lists.

And yes, the test will initially look very dumb. But they're going to lay a foundation that I can then build on as I add more complicated things. You know, techniques like you're talking about, like the tracer bullet. You know, sometimes people call that the steel thread where you cut through all the different layers of the system, defining sort of simple versions of the API as you go. And because those are testable, you're confident that you can change later. It's like, oh, we have to support this and this and this. Yeah, but don't worry about that yet. With the tests in place, we can change our mind.

We don't have to get the perfect design up front because we know we're confident that we can change it later and not break anything. And it's those techniques, like some of the techniques that you learn are just not portable. Like I can tell you terrible things that I've done to test asynchronous Python in the last six months that pretty much are not useful in any other situation unless you're using asynchronous Python and that's it. But there are lots of these techniques that you learn that can be applied in lots of different environments. And you sort of get a lot of bang for your buck when you're sort of learning those techniques. Yeah. And you remember all those years ago when we were together in Texas?

Mm-hmm. And I wanted to do something with a socket. I'd never done it before. And so I was talking to you and you said, oh, it's easy. Just do this, this, and this. Do you remember that? You know, I don't remember that. But I'm really glad that whatever it was that I said was good.

Yeah. Well, so you helped me. So we were trying to adapt fitness to this thing at Sabre. Oh, yeah. Sabre Airlines. I remember. Right? And I was trying to listen in, trying to intercept the traffic and record it. And I'd never done anything with a socket before. So now people ask me, well, if I want to test drive code with a socket, what do I do? It's like you wrap the socket up in some way.

Yeah, yeah, yeah. And you integration test that, and then you build yourself an abstraction layer over a socket. What are you trying to do with your socket? I'm trying to send messages from one thing to another. Okay, your application shouldn't be about sockets. Your application should be about getting something from something that delivered you one and right

the thing underneath it happens to be a socket yeah absolutely so you helped me down that knowing what to do when there's something that's outside your control is hide it I saw Matt's making the I thought he was trying to get us

in the conversation I love this manual heart emoji there I saw you, Matt's making the, I thought he was trying to get us to end the conversation. He was kind of loving it. I love this. I think it's just wonderful to hear. Manual heart emoji there. Yes. I mean, for me, it's just interesting to hear like stories of early Ben.

So I'm tapping you up here to learn about my friend here through your interaction, you know, two decades ago. No, Ben was great. He knew way more Java than I did. So that was good. Yeah, that was back in He knew way more Java than I did. So that was good. Yeah, that was back in the Java days, man. No, I mean, it's, and you know, we were talking at the that hide, yeah, okay, this is a socket and we made that for testing. But, you know, maybe it could be a file and you wouldn't even know the difference, right?

Or like, you know, a basic thing is like standard in, standard out, right? Like I'm reading from a socket. Could I also read from standard in? Yeah, if you design those abstractions to be minimal and simple, you sometimes get that for free. Yeah. And that's, you know, one of the great benefits of testing that, again, I think cuts across all different environments, all different languages. It just helps you no matter where you're working. Right.

You know, you run into people that say, oh, no, test-driven development couldn't possibly work for us because we're special. Yeah. Because we do this. It's like, okay, you're special. Yes, you are. But you know what? because we're special. Yeah. Because we do this. It's like, okay, you're special. Yes, you are. But you know what? It doesn't matter. Okay.

So the problem-solving techniques, you know, the problem in an embedded system is you depend on hardware. The problem with a web-based application is you depend on the database. Right? What do we do? We break the dependency, isolate the, right? Okay. So the ideas are portable. So you got lots to learn from different people. Matt, you mentioned memory checking. So CPP you test the, uh, the open source, uh, test harness that we use a lot for embedded

systems built on C++. We built in memory leak detection into it. Nice. And it's extremely important because everything leaks pretty much if you're not paying attention. Right. And I mean, especially if you're dealing with older C compilers that don't have some of the new and nice things that come with C++, which have closed a lot of the doors. Yeah. Then you've got to know how to work those too.

And someone might not want to pay for them right i mean from uh because you're talking about constrained environments right you were i know in one of your earlier episodes you're saying well we we can't afford the virtual function okay i remember the first time that happened now your compiler explorer would have been really useful to me back right that's kind of why it exists is exactly because of those type of things so yeah is that why it exists okay because here i am answering those questions of like can we do this thing i don't know if we can afford that let's have a look at what it does yeah but yes you're at motorola yeah i'm at motorola 20 years ago and uh i'm proposing that we put an interface on

something with virtual functions and the lead engineer says, no, we can't use virtual functions here. It's like, why not? Because our document says we can't. And well, I'm exaggerating a little bit. And I said, well, why does your document say you can't? Because they're too big. Oh, okay. And I would often did an experiment and I'll use a little error. So the virtual function dispatch was about this big and the direct function was about this big but guess what the alternative was a switch case which was this big now yeah right for the purposes of the listeners the switch case was

considerably larger than by either of the other two things that James gestured it as it took two switch elements to exceed the virtual function dispatch gosh so that's really interesting you can't just look at the micro you two switch elements to exceed the virtual function dispatch. Gosh. So that's really interesting. You can't just look at the micro.

You've got to look at the big picture. Absolutely. Now, the last thing I want to just mention, because if there's anybody in this kind of world trying to drag existing code into a test harness, and I wouldn't mind if anybody wanted to contribute to it. It's a thing I call Gen X fakes. And I mean, no disrespect for Gen Xers. Okay.

It's Gen dash X fakes. And X fakes stands for exploding fakes. Oh, yeah. Okay. That sounds painful. I mean, you're going to... Well, exploding fake. So I'll make this fast because when you're in an embedded world,

we never got over to the what about legacy code part. So first, what would be possible, right, is kind of how I start my course out. And then we have to deal with the real world after that. And your mileage may vary. But pulling code to get under test. First off, you say include something. You can't find the include file. Okay. Then you got to go find out where the include file is and put it in your test case. And then now it can recognize your code. And then of course, your include file won't build because

there's other include files missing. So you go in this loop for a long time. I call this crash to pass. And it's up on my blog too. Eventually you get to linker errors. And when you get to linker errors, this whole process is a very incremental way to drag code into no batch mode, just one error at a time. When you get to linker errors, you get like a hundred linker errors, right? And I'm only trying to test like one little function out of this thousand line file and I want to test a hundred lines of it, let's say. And if I got, I'm just going to pick some numbers, 20 unresolved external references that are unique, I really probably only need two of them to be resolved into real fakes, spies or fakes or mocks or something but the other ones i've got to get the linker to

be happy so i need something for them too so gen x fakes will look at the linker output and produce a bunch of little test stubs and if you happen to execute them it will fail the it'll exit the test ah got it so this is a way of of essentially as you say taking the the things that you know that you called that you think you don't need and generating essentially just stubs that say uh we're done yes as you say explode they abort they print print out and they fail the test and it's like no you you need to either stub this out or fake it out or mock it out or you need to remove the dependency on calling that function. And this allows you to link a subset of

your... You build and link against a tiny subset of your giant program in order to test just that one small element of the program. That's amazing. Yeah, so if you had a thousand line or a 10,000 line file, and I've seen 100,000 line files in my

legacy adventures i know you know to drag that under test is you know it's going to have so many linkers people are going to they're going to keep wanting to give up right and so uh now gen x fakes i've added to it a little script because the there's a series of things that you do a lot of. Like, what include path do I need to add to my test build? So if it gets the error that says can't find include file, it goes and runs a find command on your directory structure and then produces a little line that you could add to your make file. It's like, here, put this in your make file.

Right. And now it can find that file. And, you know, there's a series of things like that. Like a toolkit for making it as straightforward as possible to get to a point when you don't have a sane build system like you know i'm used to having these big sprawling projects that have been set up by me to be sensible and like subcontained and like everything can just include itself and whatever yeah but if you're not in that luxurious world where you just put in the header file you're like no this is a great way of taking just one small subset and building out a fast test. So you're talking about –

Yeah, it starts to get there. And then you have people – well, okay, so that's a start. And then three years later, somebody will send me an email. It's like, oh, it's really great. We did the stuff you talked about, and now our code base is completely different and really great. Yeah, that makes me feel good you get to that point for sure. Well because of three years of not, they weren't trying to do that

but as they went to different places they followed the Boy Scout rule. Yeah. Which is make some improvement. Just make it a little better. Make it a little bit better and if you choose always to just take another shortcut then it's always going to get worse but if more times than not you choose to make it a little bit better then eventually it's going to get worse. But if more times than not you choose to make a little bit better,

then eventually it's going to get better. Yep. Really, that's the only way to sort of revive those legacy systems, at least in my experience, is you just got to change your philosophy, which can be hard for other reasons, but it takes no real technical work. Change your philosophy and then just apply that philosophy going forward, doing the things, accomplishing the goals that you would try to accomplish otherwise, right? Like we have products that we need to get out. We need features we need to add. We need bugs we

need to fix. Yeah, do all those same things. Just apply a different technique and eventually you'll find yourself in a very different place, which is, you know, it's a little bit like boiling a frog. You don't notice that it happens, but sort of looking back on it, you go, wow, we really made some serious changes here. It's kind of amazing. So speaking of looking back, I wanted to ask you about the sort of the upcoming 20-year anniversary of the signing of the Agile Manifesto. For me, you know, and we've kind of talked about this on earlier episodes of the show, the agile movement, XP in particular, for me was a welcome change from other processes that I was using when I first got out of school, right? Like I worked at this place that was using the Rational Unified process and had these sort of

very, you know, formal mechanisms for defining requirements and writing code. And we never did any sort of automated testing. It was all kind of manual testing. We had manual testers that would verify our code. And for me, Agilent and Axby in particular was a movement toward more engineering-focused tasks. Because at the end of the day, software is written by programmers. Well, sometimes it's written by double E's, but usually software is written by programmers. And so orienting the project, orienting the team, orienting all these things around what the programmers are doing was, at the time, a novel idea, but, you know, to me, made a lot of sense. So I just kind of wanted to

ask you, you know, over that timeframe, you know, did Agile turn out to be the thing that you expected? What was different about it? What sort of like, you know, looking back at the world as you saw 20 years ago, what would you have predicted that has actually happened today? Nothing. Nothing? Well, Wow. So, okay, so that's a short answer. Because what was the expectation at the time? The expectation was, who cares? You know, will anyone care?

We had no idea that anybody would care about those four bullet points. And by the way, there was no signing of anything. So there was the writing of those four bullet points and then a debating of the 12 practices that backed it up over email afterwards, which I kind of watched. I was really learning from these guys at this time. As a 20-year person having done stuff, helping my former employer learn waterfall and trying to-oriented programming, you know, evolutionary type stuff was starting to be talked about. And I think late 90s when the words extreme programming were uttered, it's like, what does that mean? Because extreme sports were big then. It's like, oh, we'll have to wear elbow pads. Climb up a mountain with a laptop. Yeah. Right. Because extreme sports were big then. It's like, oh, we'll have to wear elbow pads.

Climb up a mountain with a laptop. Ridiculous jokes. Yep. Right. And, you know, so we were at Object Mentor, Bob Martin's company. We were learning from Kent and Ward and Ron and Martin Fowler about how to do these things. And so we're about two years, two years into it at, uh, snowbird. And,

um, you know, if somebody asks me to go skiing and snowbird, what am I going to say? Okay. Right. I'm going to say yes. All right.

And so, yeah. Right. Okay. So, uh, that was, and then you're going to go hang out with these guys that you respect and talk about software development.

So that's what the meeting was. A couple of things that I forgot, but a friend of mine who I did a lot of consulting with out there at the time, John Johanson, a few years later, he reminded me, he said, you know, after that meeting, you said that you brought the note cards. This was something us XP people would do. I was part of the XP extreme programming contingent at that meeting, you said that you brought the note cards. This was something us XP people would do. I was part of the XP extreme programming contingent at that meeting. And, you know, these all could be false memories by now, having talked about this stuff. It made them a little more interesting each time you talk about it. But so my friend John said, yeah, you told me like right after that meeting that you tossed the note cards out. And one of the techniques that we were using in those days, well, actually in the 80s as part of total quality management at the company Bob Martin and I worked together at total

quality management was about structured problem solving and communicating in a way that was non-threatening and that sort of thing. And so we would do brainstorming on note cards, you know, and you'd say, well, what do we, why, you know, what do we want out of, what's the problem with software development or what do we want? And so we started writing on note cards and then seeing if there was any agreement, right? And so then it probably moved quickly to the to the whiteboard. And it turned out to be, you know, well, having a plan isn't bad, but there's something, you know, plans change. So we we should have a plan, but we shouldn't not.

So, you know, the whole something over the other thing that that started happening. Yeah. That particular phrasing in the Antigone Manifesto of like, these are okay, but this is better, right? Like that's just – And most readers of that now, if they're kind of against the idea of agile, only read the other side and say, well, of course, they're against plan. So, you know, get rid of that. Yeah, yeah. Right? Right.

But it was really about, you know, trying to see how we could collaborate better. Now, like you, Ben, I was in there because I thought this was a better process. I was really kind of into the process-oriented thing. I'm an engineer, so I'm thinking about problem solving and effective ways to problem solve. And, you know, so I'm there thinking that. and effective ways to problem solve. And, you know, so I'm there thinking that. And Alistair goes, oh, the process, yeah, it's important, but it's more important the people you're with. And, yeah, that first bullet point was about people.

And it was just kind of like, I came here to talk about extreme programming. They're talking about people playing nice together. And, you know. There's so many software engineering problems turned out to be people problems. It's taken me 20 years to realize. Yeah, that's a Weinberg thing, right? He said, there are no technical problems, there are only people problems.

And I think you become more and more aware of that the more you program with people. Right. Yeah, so that was kind of interesting. And then the fact that it became popular, like a friend of mine, you remember Lowell, of course, Ben, right? From Object Mentor. He was a business manager. Yeah. looking at starting my own company because I wanted to get embedded people doing TDD and stuff. I couldn't get them to come to Object Mentor. And Lowell said to me, James, you're part of the Agile Manifesto.

That's like instant credibility. It's like, what? Why? I just wanted to go skiing. Yeah. It's like, well, you were there though. It's like, okay. Right place, right time.

And it's turned out to be a really good thing for the resume. And, you know, so right after starting to learn about extreme programming, I was working with a company doing embedded stuff and I had 20 years of embedded and so I just started experimenting with them and it turned out it was a good idea there.

Fantastic. Now, as far as how Agile has done in hindsight, I was pretty pumped about how it was doing 10 years ago. Because I thought there was a lot of improvement happening. And now I'm kind of disenchanted because I think we're back. I think we're in the dogma following rather than problem solving. Right. Right. think we're in the dogma following rather than problem solving right right and uh you know scrum is wildly popular yeah if you go to scrum gathering there are no engineers there right which which i mean except for me they invite you to go and talk and i complain about there not being any engineers it's like where are your engineers? It's like, well, they don't like Scrum.

I mean, they don't say that. Well, it's true. If you go online and you read about what engineers think of Agile, they think it's horrible because, here's my rant, is it's not focused on the engineering skills. It's not what the world has done. And if you remember, did you ever go to one of the extreme programming immersions back in the old days? No, I didn't have the travel budget back then. I didn't get to go.

Okay. So the way we, the first extreme programming immersion was, well, filled with, you know, people that already knew it and loved it. But we tried, not everybody, but I think we tried to start from the top down, management practices down to engineering practice. And it was, if everybody didn't already want to be there, it would have been a disaster. It didn't work. And then we started teaching the engineering practices first. Yeah, yeah.

And, you know, by Thursday when you introduce stories and planning, it's like, oh, big deal. It's like, right now we know, now we know what a little piece of work is. Yeah. And if you start the other way, and this is one of my gripes about agile is it starts the other way, the hard way, and it alienates the engineering people that have been learning how to do stuff, you know, by planning and, you know, not writing code until it's too late, right? Instead of starting sooner and learning sooner. Yeah. For me, the most amazing thing about XP in particular was that the realization that if you follow this set of engineering practices, which are things like test server development, pair programming to a certain extent, continuous integration.

There's a whole bunch of other stuff that you don't have to do, like project manager stuff that you don't have to do. And that is extremely threatening if your job is project manager, right? That's not what you want to hear, right? You want to hear the opposite of that. And so that's, I think, part of the problem is that the reason that Agile has kind of gotten away from that being, you know, not just, you know, to say it's rooted in engineering practices or should be rooted in engineering practices is almost underselling it. It is like 95% engineering practices. And once you have that foundation in place, you only need to add another little 5% to make the whole thing work, right?

Like it doesn't take that much. It just takes some note cards and a couple of conversations and an understanding of what you're trying to build. And you can do amazing things without having to add a whole bunch of cost. You could probably even be successful with JIRA at that point. I mean, let's not take this too far. I'm talking out of the side of my mouth i don't know anything about jira but i do hear people complain about it people complain about whatever tracking and management system you put in place so yeah yeah it's true i still have to try to explain

to people it's like the whole idea behind a story card is that it's a promise to have a conversation like it's about the conversation and we're going to talk about this we don't need to we don't need to have like this huge jira system for capturing all the requirements putting all the things and figuring everything out that's not what this that's not what a story is right the story is we're going to talk about but i don't know i did now ranting. Thank you for not calling it a user story. Yeah. Because it's really a story. That word is like it's unfortunate.

Yeah. There's a lot. I think there's a lot of that. I would call it nuance, but it's really the most important part. But a lot of that stuff that kind of got lost over the years, which is a shame. But, you know, some of us still know what it is. That's good. Some of us still know how it's supposed to work. I was out with these guys, but I'm out to dinner after a conference with a business guy,

and we were talking about Agile. I said, yeah, people are only doing the easy half of Agile. Mm-hmm. Yeah, so they're only doing the easy half of Agile. And then he said to me, well, what's the hard half of Agile. Mm-hmm. Yeah, so I said, they're only doing the easy half of Agile. And then he said to me, well, what's the hard half? I said, the technical practice is the hard half. He goes, no, that's not the hard half.

The hard half is the people half. And it's like I hit myself in the head. It's like, oh, damn, there's three halves. You got the planning half, you got the people half, and you've got the technical half it's like oh yeah okay we're gonna need a bigger boat that's yeah well that sounds like an absolutely perfect place to to stop the conversation yeah obviously keep on for many hours oh man a lot of interesting things to talk about here. But before we go, James, do you want to let

our listeners know how they can find you online? Like you mentioned your blog a few times and any other way of contacting you. Okay, so my website is wingman-sw.com wingmansoftware and I wrote a book Test Driven Development for Embedded C

if you search for that, you'll find me at jwgrenning on Twitter. Those are ways to find me. If you search for that, you'll find me at JWGrenning on Twitter. Those are ways to find me. Fantastic. Great. Well, thank you so much for coming on and talking to us. This has been so interesting

and enlightening and certainly lovely to hear other people struggling with different types of testing in C and then obviously to hear about the Agile stuff too is just mind-blowing for me. So thank you so much for being with us.

It was great to talk to two guys at Get It and I think I really liked your first two episodes of your podcast. Hopefully it's just a few. I like the little game theme. That's cool. One player, two player.

One day someone will figure out how to actually get to the podcast from the main page, but I'm not holding my breath. Cool. You've been listening to Two's Compliment, a programming podcast by Ben Rady and Matt Godbolt. Find the show transcript and notes

at twoscompliment.org. Contact us on Twitter at twoscp. Theme music by Inverse Phase, inversephase.com.