Two's Complement - Episode 0 - The Origin Story

Episode Date: December 24, 2020

Matt and Ben talk about how their careers were on the same path in the late 90's, but then diverged at a critical juncture. Then they talk about automated testing. Ben gets out his soapbox. Matt is a kind, patient soul.

Hey, Ben. Hi, Matt. We should do a podcast, right? Oh, that's a fine idea. You and I have chatted a lot about loads of interesting things, right? But I think, wouldn't it be interesting if we shared these views with the rest of the world? We could do a lot worse than that idea, for sure. Yeah.

All right. Let's give it a go. I dig it. what do you want to talk about good question well you know i was thinking the other day you and i started out in essentially the same spot and then something happened and then we arced around we went on two completely different career trajectories and now we've landed up working together and yeah do you want to talk a little bit about that i mean i started out in the games industry so age 16 17 i was making games at home i was hacking around i went to university that was a bit of a mistake really most of the time i spent in the computer

lab doing basically video games yeah and that was fun it was a lot of fun that was the thing you did right and then i got a job doing that yeah see that's where our paths diverged decade doing that but yeah you did something else right so why don't you tell us talk about what happened to you well yeah so when i was a kid i very much wanted to be, uh, wanted to make video games. And I, you know, had been programming in some degree in high school and really, really, really enjoyed it. And I got, I was a little bit into graphics, but more into AI. I very much wanted to build, you know, the bots that live in video games. And I spent most of my college career you know i got a computer science degree but i spent a lot of time sort of playing around with not only that but also some graphics libraries

like opengl and direct 3d at the time mostly because i knew i had to have some understanding of that to get a reasonable job in the industry but really right i just wanted to right i just wanted to make the ais but um when i graduated i got an offer from a company in houston and i at the same time had uh proposed to my now wife and we had agreed that whatever town or city we both got job offers in that's where we're moving and this company was in houston and she had gotten an offer from an oil company in houston being a chemical engineer and so we were moving to houston and we had you know planned our wedding and had gotten an apartment and all this stuff and this company calls me up and said hey ben we're moving to austin can you move to austin

and i'm like no no i cannot move to austin they're like well we're moving austin so if you want a job you can come to austin so shucks fast forward 20 years and i have never worked in the games industry oh no um i've worked in a lot of other industries but not that one uh in the mid late 90s when i was in the games industry i mean you didn't really miss much i don't think i mean it was a fabulous experience like loads of stuff i learned and you know crawling further and further further down the tech stack towards the the hardware and trying to get more explosions on the screens trying to get more triangles drawn trying to get all the cool special effects done that was what i was doing

and you know eking out the performance from these you know 200 megahertz machines as they were back then was super super fun but working 18 hour days for six or seven days a week for many months was not fun i'm very glad to be out of it from that respect i believe it's gotten a bit better from talking to friends of mine who are still still doing it but you know it's it's definitely an industry where it's cool to be in it so like sort of hollywood there's a certain amount of cachet from doing it and unfortunately they the the industry knows that and will happily milk the people coming in for all their worth to get that but and in a lot of ways i mean again certainly in the late 90s when i was there and there were a lot of people like me who weren't trained computer scientists or software engineers or any of that thing we were just you know bedroom hackers and so if you have an entire industry

founded on people just making it up as they were going along then there's a different tenor to the way software is engineered and in particular things like testing was just not something i didn't learn to do until like 15 20 years later when I worked at Google for a bit. And they were like, yeah, we need to write tests for these things. I'm like, yeah, well, we kind of had a couple of asserts in the code somewhere we would run. And if it passed through the asserts, then it was good enough to ship. And testing to us was you hand the code, a build of the code to somebody with a VHS recorder and a PlayStation controller and said, you know, can you find any bugs in this? That was testing for us. Yeah. But, you know, you've kind of built

a career out of, I wouldn't say career. One of the facets of your career is talking to people about testing. And certainly I've learned a ton from you about how tests, how testing should be. And in fact, kind of one of the things that got into my head about really talking to you about this podcast and getting us online was the something you said the other day when you said, if you're doing testing right, you should be able to go faster. And I think that was a huge mind leveler for me, realizing that that is is actually true and it's not just lip service that you do yeah absolutely what do you mean when you say that yeah so that's i mean there's a lot there's actually a lot of things to unpack in that statement um it you know testing what do you what do i mean by testing and faster is both of those are are important things to figure out. But all of the words are up for grabs. Pretty much.

It's a what is is is kind of situation with this. But yeah, just to explain a little bit. So the kind of testing that I'm talking about are what some people call unit tests, what some people call micro tests. I have sometimes just called them fast tests. They are tests that do not hit the file system, do not talk to the network, do not talk to a database, do not do much of anything except exercise a very small piece of code.

And these tests run extremely fast, hundreds of tests per second, depending on the language that you're working in. Just to sort of concrete, you've done these in languages such as Python, JavaScript, what? Python, Java, JavaScript, Ruby, Clojure, a little bit of C and C++ now and again, although I don't purport to be an expert in that. When I was working in Rust a little bit, I tried to write tests in Rust. That was a very interesting experience. We should talk about that sometime. But these kinds of tests serve a whole lot of purposes at once, and that's sort of where they get their magic, is they help you in an obvious way test correctness. Right. Give you confidence that your code is correct. They help you design your code as you're going.

Mm-hmm. And they serve as a sort of executable documentation for the people that will come after you when, you know, a lot of times the people that will come after you are just you six months later. Right. Or even the next day. Or the next day after you slept, right? What of times the people that will come after you are just you six months later right or even the next day or the next day what was i thinking yeah yeah yeah exactly so these are like literally hundreds of tests that you can run in very short order i press a button and it does this or is there a what what other ways i mean i the way i generally work is i want to make sure that the tests run on every change to the code that I make. So every time I hit save, it's a little bit like a micro commit, right? Got it.

Like I'm saving these changes. I expect some result in the test. It might be for the test to fail in a particular way. It might be for them to pass. It's usually not more than, you know, a few seconds, maybe a minute in between saves, but it's very rapid. And every time I hit save i want my test to run maybe there's a compilation step in there first depending on the language again but i hit save the test run i have a result and you literally got that like up in another window

like yes adjacent to your editor saying hey yeah i noticed you changed something right i ran the test and here's here's a bunch of green or here's some red and you were expecting the red because you literally wrote the failing test or whatever. That kind of thing. And like you call it a micro-commit. I think that's a really interesting way of phrasing it. It's like, I have done some typing. I have done some rumination.

I've moved some things around. I've made some decisions about what my coach would look like. The very act of saving is kind of my, I'm leaning back in my chair for a half second just to think about what I'm doing next. Where am I? Is that a fair assessment? Yeah. And you need a fair amount of real estate, screen real estate for this, because if the tests are green, like they're green, that's fine, whatever. If they fail, you really want to make sure that they fail in the way that you expected them to. And even more than that,

before you hit that save button, you really want to have in your mind what the failure is going to be based on your current understanding of the state of the code and the problem and everything else. Because if it doesn't meet those expectations, you might be doing something wrong. Right. Like the test failing doesn't mean that you're doing something wrong. The test failing means that you're gradually adding behavior to the code and changing it in ways you may be anticipating a failure right it doesn't happen and then you're like i'm sure i had a test for this but i've just commented out a key piece

of the code and clearly i'm not testing the opposite case or i'm right or i'm not testing the thing that i thought i i mean one of the the Yeah. Like, the classic mistake that people both make and anticipate when they're learning these kinds of techniques is, well, you know, if you don't trust my code to be right, why would you trust my tests to be right, right? Right, of course. And the problem is, a lot of times this stuff is taught in very static ways. It's like a blog post or like a pull request that somebody sends somebody and you see the code and you see the tests and seeing it in that static

form, you do kind of have a little bit of a question about like, how do I know that this is right? The real way that you get that confidence that the code is properly tested is by assembling both the code and the tests in the right order, such that you see that failure that tells you like, oh, this test is absolutely testing that I'm writing to this file because I can see this error that says that the directory doesn't exist. And I haven't created the directory because I'm not going to create the directory because I'm going to mock that out later or whatever. But I can see that failure that really gives me very strong confidence that this code is doing what i think it's doing and you won't ever see that if you just look at the resulting code and test together right you have to see the process of

how it was created to really think it's like a living breathing process where perhaps those who haven't seen it done this way like myself included really i really, I go to GitHub, I look at the code, I sort of page up and down a bit. And then maybe I'll look for a test and page up and down in that as well. But that's not really capturing what you're talking about. Yeah. This sort of living, breathing, interactively. So when you say we go faster when we have these tests, what is your thought about what faster means? So faster often, not always, but often means literally just the fastest possible route to production working code.

I am taking it as an assumption, and I think this is mostly true, but maybe my perspective is off on this. So I'm going to put this premise out there sure that my that i am thinking about this correctly my premise is that these days people don't think of code being done until it's actually running somewhere right like it's in the hands of a user or it's in a at least a testing environment if not a production environment right like you know gone are the days where we write a bunch of code and we sort of throw it over the wall. You know, kind of like you were saying with the PlayStation controller. Yeah, right. Hand them over and say, oh, I'm done now.

You know, like, ah, compiled, that must be good enough. And I'm going to throw it over to the wall of the testers. And then the testers are going to maybe find some bugs, but they won't because I'm perfect. And those days are gone, right? I think. So the question is, how do we get to that point where people can use the code that we write to do their jobs and make their lives better and accomplish whatever it is that they're trying to accomplish? You want to try to get to that state as quickly as possible.

So when I say faster, I mean that state. Faster from the point you're making the change to the point that it's usefully impacting somebody, be it a user or the downstream person who is going to take your code and move on to the next step or or it's deployed and it's actually running in a in some environment somewhere be it the actual live environment or or whatever that yeah that makes sense to me so when i interpreted that original statement you made about testing making you go faster the thing that i've internalized about writing tests is going faster because i have the confidence to make changes that otherwise i might either avoid doing or i would spend a long time picking

around the edges before i actually committed to because i know that if i make a change and i break something the test will catch it right that state is a really interesting state to be in and it's hard if you're trying to like graft if you're trying to preach to somebody who hasn't already got a decent amount of tests in their their system or if they have got tests but they like take two and a half hours to run then the safety belt that you're putting on with that doesn't really ring true right you're like yeah yeah i go faster because you know hey i make a change and then i hit the button and then i get a green and i know i'm good to go i check in i don't have to do anything else that's that's my my process and so for me that definitely has

been liberating and in particular some of the things we were talking about over the weekend one of the projects that i was working on which has got nothing to do with anything but like is it's a text editor for a dead language right and the fact that the syntax highlighting i could write tests for the syntax highlighter meant that i can take a bug report that a user's given me write a single test that reproduces it make the obvious change to me and know that i haven't broken anything else that was previously there whereas before i was playing whack-a-mole right with the well okay i load it up and i click around a bit and so that aspect is how i think about going faster i could knock out like a dozen bug reports in a couple of hours knowing that i haven't broken anything know that i'm not going

to be called back and said oh you know that thing that stopped working the other thing has stopped working right right and so that liberates me and i you know obviously that was in a like javascript which is traditionally more associated with these things but i could also have this in my day job with some of the c++ code that i mean and i feel that a lot of c++ engineers are missing a trick by not having a similar setup to this well and and i mean my experience is working in c++ are limited but i can tell you that one of the things that definitely can happen is you sort of get this broken windows effect with the compilation slowing you down, right? Yes. Where if you get into a situation where your build takes a minute, right? 60 seconds.

Mm-hmm. You might say, well, that's a pretty fast build. Well, but the problem is that there's a huge difference in the way that you interact with something. A friend of mine once told me about this thing called the rule of eights, where if you think about how you interact with a computer or any sort of digital system in, you know, 800 milliseconds versus 8 seconds versus 80 seconds versus 800 seconds, 800 milliseconds for most things is, know almost instantaneous i like how you

qualified that most things but yes for human interaction purposes yes short of all right the gap between me pressing the fire button and my my my guy on screen shooting is maybe that that's different but yeah for most things 800 milliseconds is as near damn it instantaneous yeah yeah yeah it's certainly interactive enough to where you're not going to lose your train of thought 800 milliseconds is you just keep on streaming the flow keeps flowing right eight seconds there's a there's like a there's like a beat there's a pause and you're sort of waiting in anticipation for like maybe something's gonna what am i gonna see right yep you might in a moment get distracted 80 seconds oh you're definitely getting distracted

you're gonna go check your email tabbing, 80 seconds, I'm all tabbing. Yes, you're all tabbing over to something. The train of thought is gone. And then 800 seconds, that's just right out. It's like, fine. We're in fast job territory there. Yeah, exactly. So thinking about the interaction in those terms, if your build gets up to 60 seconds, you've lost the flow, right?

You're no longer in that territory. And so not only do you have the burden of maintaining a fast test suite, but you also potentially, depending on your language, have the burden of maintaining a fast build. But I argue that those are investments that are absolutely well worth making

on a number of different fronts, not the least of which is it creates a development workflow that allows you to just keep thinking, right? You never stop. You never get distracted. You never lose your train of thought. You just keep the flow going.

And exactly. Flow, I think, is the key word there. And I think that's probably why you and I bonded so well over the way that I structured the projects, the C++ projects that we worked on together when you came and worked together. Because I've always felt this, the need for long builds for most things is gone now, I think. At least for debug and for essentially a fast build, which lets you work on a relatively small area of your code base and have that kind of turnaround time which i love it to be 800 milliseconds but i think with the best will in the world it's going to be closer to the eight seconds yeah it is the eight seconds

is okay right you can deal with eight seconds yeah right and a certain with a certain amount of mental pipelining and understanding that like okay i'm saving and running the test now and then i'm going to be paging up and down looking at the code while I kind of make that last stage commit, whatever, those kinds of things. But that's very, very different from the usual 30, 40, 50 seconds to do something, which is a big deal. And I agree completely that that flow state. And I think that it's such an enabler. It changes the way that you write and develop your software. Absolutely.

In the way that TDD sort of does because it makes you be your own client to start with. And if you can get the very human pleasure of writing a dozen lines of code at most and saying, okay, I know I have a test that's using my API in a way that doesn't exist yet. So I'm going to build it and it's going to say, no, there's an error. And you're going to go, okay,

well, I'm going to go and stub that out. Okay, I'm going to run it now. Oh, now the error is the test fails and we're talking another four seconds from the point which you click the button to get in that and then you start working on well let's make it pass and then oftentimes that can be a few minutes worth of work and then you get a green and you're like hey i've got the reward the endorphin kicks in i did something important i i went forward in in towards the goal and I got my little reward and I can move on with my life. I can even commit that.

That's useful. I have a useful change that makes sense. Is this just you and I bonding over the fact that we both think that and typing the perfect code out, is a valid way of developing too. Well, that's a great point. And I think it actually speaks to something that you said earlier, which is you're trying to trying to get confidence right that's what the tests were providing you in that instance but there are lots of different ways to get confidence you know i've i've i had a talk at one point i don't know if you saw this but i talking about confidence in software engineering and saying how software engineering is essentially a faith-based activity because there's this sort of magical state. As much as we like to think that

it's completely objective and it's all ones and zeros and everything is a meritocracy and all these other things that we tell ourselves about how software engineering works, the way it actually works is it just comes down to belief. Because you have to have confidence to deploy a piece of software. I'm going to take this, I'm going to push it into production. Millions of people are going to potentially use it. How do I know that that's okay? It comes from an internal sense that is often based on experience, but it comes from an internal sense of whether or not you feel confident in that code to do what you expect it to do.

There are lots of different ways to get that confidence. Automated tests are one way. Manual tests are one way. Manual tests are another way. Observability and tracing is another way. Just sitting down and thinking about the problem for 30 minutes is another way. Honestly, if I'm doing anything multithreaded, that's the way I do it. I sit down and I think about it really hard.

For a really long time. Maybe I shouldn't do multithreading. Yes. It's like, yeah. At first I ask, why are we using threads? That's the first question I ask. That's the first question anyone should ask. Can we not use threads? Is there any way we can avoid this?

Yes, exactly. But that's sometimes what it is. So it's not that there's one particular. And when I say, you know, if you're doing tests, it should make you go faster, that's true. That's not the only way to develop software for sure. It's not even necessarily at the individual level the right thing to do. As I was just saying, multi-threaded code, I don't rely on that. They're testing a different aspect, perhaps, of your code. You're sort of throwing the chaos monkey at it. Right. Sorry, carry on. Well, no.

To that point, it's like testing makes you go faster in the aggregate. It's there as a mechanism for gaining confidence. It is a mechanism that I think people can use quite often. It's maybe an 80% solution for how do for gaining confidence. It is a mechanism that I think people can use quite often. It's maybe an 80% solution for like, how do I gain confidence?

There are definitely people in the world that are able to do that without writing tests. They can literally just think of the code they want to write and bang it out. But I'm definitely not one of those people. And I have not found a lot of those people in the industry. And so the problem then becomes you have to start thinking about the dynamics of working within a team. And what does the team need? What does the team expect? For example, the team has to at least in some way be committed to building this suite of tests.

Otherwise, you're not going to have the experience that you had working on this syntax highlighter because if if the if the existing behavior that you're afraid of breaking isn't tested or isn't tested enough you don't really have that confidence after you run the test yeah the belief system being destroyed slightly by like i the way that i test it now is back to my old ways my old tricks are like when i run it and i kind of like page up and down the logs or i poke on the click buttons in the ui as fast as i can to try and break it and that's not a good feeling that's not right it's not a sort of intersubjective thing it's not like i can check it and say like this atomically is a sealed okay change and anyone can check it out and be the computer checks it for them right and right we've we've worked with well i've worked with definitely with with folks who

are very definitely to sit back and think a lot about it. And on a project at a previous company, I definitely had that experience of there was the lead developer who is absolutely awesome. One of the cleverest people I know, very thin on the ground when it comes to tests, because he didn't make the mistakes. He just wrote the code right. And that was amazing. And that worked perfectly well. But it didn't scale when I was added to the team. Because I'm not that person. I'm definitely not that person. But also, I didn't understand the code that he had written.

And I couldn't add to it without asking him 100 times, does this work? What about that? Oh, no, that won't work. That'll break because of something. I don't know. I just have to keep asking you. And that's not a great experience necessarily for someone coming onto the team. As it happened, we eventually ended up with an expansive enough test suite that we were

both happy and then we could add more people onto the team. And it was, it was a joyous moment, right? But that isn't always the case. Yeah, absolutely. And it is, uh, and I don't know if your experiences on that team would, would mirror this, but I've certainly found that unfortunately, writing these kinds of tests, these sort of fast-running tests, it's a little bit of an all-or-nothing thing, right? Because doing it some of the time still leaves you in a situation where after you've made a change, you don't have confidence that it works, and you have to go test it manually anyway. Or find some other mechanism for testing it, whatever mechanism. and there are lots of them, but find some other one. The tests don't give you the confidence that you need to move forward. And so it's not that they have no value, but they have a tenth of the value that they would have if you could maintain that confidence.

And the thing about it, one of the things that makes it hard is that confidence has to be shared among everyone on the team. Everyone has to believe in the test and everyone has to maintain them to a level that is extremely hard to quantify. I mean, you can try to do things with code coverage and all this other nonsense, but yeah, that's not going to tell you anything but you it's really hard to quantify it so you have to sort of have this like shared sort of communal knowledge of like what does it mean to write a good test and do i believe that these tests will tell me right and as soon as you lose it as soon as you lose it it's just it's just really hard to get back so yeah it's hard i mean

and you know this is another kind of point we were talking about this earlier this week i think the reason why a lot of this stuff falls down, we're talking about all these different techniques. They're hard, right? It's hard to learn how to do this. I think the mindset that a lot of people have in the industry, and this is especially true of technical managers, is that writing these kinds of tests is a choice. It's a choice that individuals can make to write tests or not.

Well, to me, that's pretty ridiculous. It's like, all right, well, you Python programmer with two years of experience, I'm going to put you on this legacy C++ code base. And if you don't perform to the level that I expect, well, then you made some bad choices and I'm going to fire you. It's like, no, you don't have the skills. You don't have the skills. It's not a choice. You don't know how to do it. You don't know how to do it in a way that isn't... I'm going to spend the next six months learning how to mock out interactions with a socket library to make sure that I've handled all the different disconnect situations in a realistic way. That's a skill. You have to learn how to do it. So as an industry, if we're not providing the resources and the time to teach people these skills that they don't

necessarily have it is totally unrealistic of us to expect for them to do it so you have to take that into account uh and if you don't you won't get good tests or what you will get is a smattering of tests you'll get the test all the tests that were easy to write will be written that was you you're taking the the words out of my mouth here which is that that's typically what one does i mean you know designing software to be testable observable all the things that we we we we cover is is a skill in itself and i think that's another problem like so i've sort of been sort of mulling this thought through in my head about whether you know how how would we advise somebody coming in who had a project that wasn't tested like we just described that didn't have a build like i would like to have if it was a native project or it had like some some some first

step that took a long while every single time you know like there's the web packs of these worlds these days is the new build time maybe not even lined up uh or organized in to be testable in the first place there's a lot of things that need to be lined up to be in our lovely world and you and i have been in a luxurious position recently of starting projects pretty much afresh and so we can start out the way that you and i like these things and you know you incrementally add on and then it's the tax doesn't seem so high but i wonder if for anyone listening thinking well this sounds lovely i wish i was in this world where my tests took under eight seconds to build and run and gave me feedback you know and then one of the things you sort of said is like if

if you don't have the 100 confidence or some thresholding level like so let's say 90 confidence measured in whatever system of confidence right you can have across the team then it's not as valuable as like an 80 confidence is like much less valuable than a 90 confidence because now everyone's a bit you know fuddish about well i made a change and i'm not really sure i guess i'm am i going to push it am i going to run it in the test environment forever what am i going to do like there's this sort of thresholding you've said so how can one get from a project such as i described more towards uh our mindset or your, I should say, your sort of thought?

Well, the problem is this is literally the hardest thing to do with testing is take a legacy code base. And I'm using Michael Feather's definition of the word legacy here, which is a code base without tests. Take a legacy code base and put tests around it. That's worth underlying, actually. Michael Feathers suggests that the definition of legacy is an untested code base, which I think is interesting in itself. I just love that.

Yeah, I think it's a great definition. I think it's particularly interesting when you sort of get flexible on the definition of tested, right? Because it's like you can have code where it's like i don't know that this code has ever been run we've all found how do you do with that right lines of code you're like looking at how did this ever work and then you're like well it never it never worked it maybe never it was never called because if it ever was called you'd be in trouble yes yes exactly so so i so i love that But yeah, it is an unfortunate truth that it is the hardest thing to do is to take a legacy code base and write tests around it. Because the code was probably almost certainly not designed to be tested in this way. And you have no real automated tools for checking to make sure that you didn't break it. So you have to change it and you don't have a way to, a mechanism to make sure that those changes are okay. And I've dealt with a lot of code bases like this. I'm not every project that I've ever been on has been a Greenfield project. That would be a charmed life, wouldn't it? But the, the ways that I've dealt with this,

and I also worked as a consultant for a while and, and, you know, we, we worked on these kinds of code bases. Like that's what you do as a consultant. But also in, in, you know, my, my sort of non-consultant career, I don't know what the adjective is for that. Whatever, the normal career, I guess, in different industries and all over the place. And I think there are some things that you can do to sort of deal with this. The first is decomposition right so try to find the sort of seams in the software where you can start to pull things apart in a safe way and then that's the the key there is is safe you've got to find something small enough that you have confidence extracting the three lines of code in the buried in some horrible giant class and saying i'm gonna make this its own little thing and i'm

gonna use it and i can't test that i haven't broken it by extracting those three lines. Yep, yep, yep. Yeah, and I mean, there's, you know, sort of like very coarse-grained kind of replay tests that you can sometimes pull out of legacy systems where it's like, okay, component A talks to component B over some, you know, protocol. We're going to just tap that protocol and play in the input and record the output, and then we're going to play that back. And we can all sit in a room and agree that if we get the same output, we're pretty confident we haven't broken anything.

Yeah, like state dumping, and before and after I run the whole program and I redirect the output to a file and I run a sed script to pull out all the timestamps, and then I say, did anything change apart from the timestamps? You're like, yep, okay, we're good. And yeah, I know that there actually are testing frameworks that I've seen that people use that have this kind of characteristic about them.

And in fact, have a workflow around a sort of almost a temporary crutch test system where you can write a test where you're like, okay, the diff comes out. And then you can click a button to say this diff is okay this time and it kind of blesses it there and then or else it fails which is only an interim but it allows you to sort of leg into writing tests the way that you just described where you put you say i know that extracting those three lines of code putting it into its own class that i can now go off and test in my own wonderful world and in fact in the idealized world if you can get to this wonderful point where in a compiled language, you're literally only compiling that file and its test and linking them against each other

and then running the test and going, hurrah, look, it takes 300 milliseconds, 800 milliseconds, sorry, 800 is the magic number, right? You could do that and that's feasible, but you can use this golden system to give you a little bit of confidence that even extracting that hasn't broken things.

Right, right, right. Yeah, and like you said, those tests should be very temporary, right? Like we don't want to live with those for a long time. They're just there to get us to the point. But, you know, another problem that people will ask, and this is sort of almost an orthogonal question of, okay, well, there's all the technical hurdles of, you know, changing a system to be tested in this way. What about the organizational hurdles? What about the, my boss is going to let me spend six months redesigning a system so that we can add tests or whatever? Right. Right. And that's a very hard argument to win. And I think that some pretty

reasonable arguments can be made in some situations that it like, as much as it, as it pains me as a developer, sometimes the right answer isn't not to make these changes just to sort of limp along. You will suffer the pain of doing that, but you know, it's hard to argue against dollars. Sometimes the, the opportunity that I've seen for this to happen and really be a positive

impact for everybody is when people start talking about a rewrite, when people start saying like, this system is old and crusty and every and we have the critical mess problem every time we fix one bug we create two more bugs so i don't know how to come back from that right um when people start talking about a rewrite you can start offering an alternative now it's hard to not be tempted by the siren song of the rewrite. You know, oh, I get the green field all over again and I get to redo it.

But trust me when I tell you, the rewrite is not going to be nearly as fun as you think it is, right? There are some situations in which it makes sense, but more often than not, when people start talking about a rewrite, the actual best thing from an economic perspective is to start applying some of these techniques, right? Like let's start looking at the systems that are

changing recently, right? Like these old legacy systems, some parts have probably stabilized and you know what? Those parts are never going to have automated tests. Just let it go, man. Like you're just never going to change it. But there's probably the reason you're talking about the system in this meeting or wherever you're at is because you're trying to change it, right? So where are you trying to change it? Where are you trying to make those changes? If you can localize your efforts to those parts that are changing frequently now and find ways to decompose them safely and start to get tests around them, you can wind up with a system that has a safe area where the changes are generally happening

and then an unsafe wilderness and yeah when you go into the unsafe wilderness the dragon it's gonna be rough man you know bring the shotgun it's gonna be a hard time but you can spend most of your days and most of your time in that safe area where you can make changes with confidence you can work in short cycles you can do all the things that we're talking about and you can live like that for a pretty long time so that makes a lot you have to find the right moment to sort of bring up the the approach because otherwise the people who are cutting your paychecks are going to be like really six months of nothing that's what we're doing that's an awesome observation that yeah there are economics at play here and the area under

the graph sometimes doesn't warrant the time out but But like you say, if you can cut off that part of the giant monster that's causing the problems and make it a nice place to live, then that's a sensible way to start. So I think that probably answers the question. Yeah, I think that's a great place for us to stop. I mean, there's a lot of us more to talk about. Testing is a huge, huge area. I'd love to talk more about how to make c++ programs

testable um there are some really interesting challenges there that i'd love to pick your brains on at some point because you know there are some interesting uh usually if you're picking c++ you're picking it for a very specific reason right than anything else and so that brings with its own challenges yeah there's definitely more meat on the bone i mean we should maybe we should even try and talk to people like michael feathers and see if he wouldn't want mind talking to us sometimes so i've got some ideas yeah we should we should speak to some people but this has been a lot of fun yeah i guess we'll do this again yeah absolutely you've been listening to two's compliment a programming podcast by ben rady and matt godbolt find the show transcript and notes at twoscompliment.org

contact us on twitter at two cp that's at t w o s c p Theme music by Inverse Phase