The Changelog: Software Development, Open Source - Building actually maintainable software ♻️ (Interview)

Episode Date: September 2, 2022

This week we're sharing the most popular episode of Go Time from last year — Go Time #196. We believe this episode was the most popular because it's all about building actually maintainable software and what goes into that. Kris Brandow is joined by Johnny Boursiquot, Ian Lopshire, and Sam Boyer. There's lots of hot takes, disagreements, and unpopular opinions. This is part two of a three part mini-series led by Kris on maintenance. Make sure you check out Go Time #195 and Go Time #202 to continue the series.

This week on The Change Law, we're sharing the most popular episode of Go Time from last year. This is episode 196, and we believe it's the most popular episode because it's all about building actually maintainable software and what goes into that. Chris Brando is joined by Johnny Borsico, Ian Lopshire, and Sam Boyer. There is lots of hot takes, disagreements, and, of course, unpopular opinions. This is part two of a three-part miniseries led by Chris on maintenance. Make sure you check out Go Time 195 and 202 and continue the series. And guess what? Go Time records live, so join in on the conversation on Tuesdays at 3 p.m.

U.S. Eastern. Hang in the Go Time FM channel on Gopher Slack. And watch and listen live at youtube.com slash changelog. We'll see you next time. Fastly.com and our friends at FlyLegit run your app and your database closer to users all over the world. It's like a CDM of your entire application. Try it free at fly.io. This episode is brought to you by InfluxData, the creators of InfluxDB. InfluxDB is the open

source time series platform where developers build IoT, analytics, and cloud applications. And I'm here with Paul Dix, founder and CTO of InfluxData. Paul, all the open source software that InfluxData creates is either MIT licensed or Apache 2 licensed. These are very permissive licenses. Why are you all for permissive licensing? The thing is, we like permissive licenses because we want you all for permissive licensing? The thing is, we like permissive licenses because we want people to do whatever they want. Because of these three reasons, freedom, evolution, and impact. Freedom means being able to create a business and create your

livelihood off of this code, regardless of what you want to do with it. You can modify it, look at it, do whatever. Evolution means you can create a derivative project and rename it, put it out there in the world either as an open source project under a permissive license or you can relicense it under a copyleft license or you can create a business off of that. And then the last bit is impact. We believe more people benefit from open source when that code is permissively licensed, despite the changes that the other infrastructure vendors are making. Influx remains permissively licensed open source with commercial software. Well said. Thank you, Paul. That truly summarizes the spirit of open source.

So if you want the option to have freedom, the option to have evolution and impact, use InfluxDB for your time series application needs. Check it out and start for free, of course, at at influxdata.com slash changelog again influxdata.com slash changelog let's do it. It's go time. Welcome to Go Time. This week, we're going to be doing part two of our multi- it comes to software, the maintenance and the long-term prospects of it. focused on building actually maintainable software and what goes into that. And this week, I am joined by Sam Boyer. Hi, Sam. How are you doing?

I'm lovely. How are you? Great. And to give you a little introduction of Sam, Sam is a principal engineer at Grafana Labs, where he just switched teams to be responsible for Grafana's Go backend. The team is nominally in charge of around 250,000 lines of code. Yeah, that's a huge amount of code right there. Sam thinks a lot about code evolution and quality, usually under the umbrella of package management,

an area he's been working in for the better part of a decade, as I'm sure many of our listeners are well aware of. And I'm also joined today by Ian Lopshire. How are you today, Ian? I'm doing great. Yeah, and to give you guys an introduction of Ian, Ian is a senior engineer at TimeHop, where he's responsible for keeping TimeHop's go-back-end in working order.

TimeHop integrates with multiple social media platforms to surface millions of user photos and posts each day. It's like your own little day in history. Yeah, I remember TimeHop from back in the day. You guys are still going little day in history. Yeah, I remember time hop from back in the day. And you know, still got you guys are still going strong. It seems we are. And I'm also joined by my fellow host, Johnny Bersico. How are you doing today, Johnny? Not too bad. Not too bad. I bring hot takes. So yeah, I'm ready for some good Johnny hot takes. I thought of a couple unpopular opinions. But if you have extra despair, maybe I'll just lean on you for those, Johnny.

Plenty to go around. Excellent. There's going to be a lot of unpopular opinions in this episode, I feel. All right. So with that, let's get into talking about maintenance and how to build maintainable software. And I want to start with thinking about how do we build build maintainable software. And I want to start with thinking about, you know, how do we build actually maintainable code for new codebases, right? Because there's, you know, some maintenance you do for old codebases, but then you have, you know, new codebases.

And we always start with these great ideas when we make, like, a new one. But rarely are we thinking about, you know, what's it like when that codebase actually goes to production? What are the steps to get toward it? So I guess we can start with, let's start with you, Johnny. What are the things that we should be thinking about when we build a new code base and we're aiming to get it to production for more of like, I guess, a maintenance standpoint? Well, that's the thing. You don't know if whatever it is that you're working on is going to be around for the long haul, right? So we have this assumption that all the pieces of code that we write sort of is worth sort of getting all

production ready and everything else, like all the time. And that's really not true. So as you say, like a lot of times you start out with basically doing some prototyping, some R&D. It is an unfortunate fact that a lot of times, right, due to business pressures or whatever, timelines, sometimes most of the time manufactured, you know, timeframes for things that the stuff you have ends up going into production. And you're like, man, like we really didn't do all the due diligence necessary or the all the prep, all the operationalization, all the production readiness stuff that should have gone into this, right?

It was just a prototype. Now, you know, management wants it and, you know, to be deployed and whatever it is. So you're playing catch up now, right? But in the ideal scenario, right? You figure out basically what is it that I'm building, that I'm tasked with building, right? And making sure that everybody understands this is the scope of this work. This is really, it's meant to show you something, right? Maybe you're trying to determine product market fit. Is this thing real? Does it have legs? Whatever the case may be, but with the intention of actually making it ready for production. These are very separate steps, right? When you're exploring and when you're making something production ready, these are

very, very different things, right? Again, it's unfortunate that a lot of times, you know, the play, the toy ends up going to production. But yeah, you kind of have to basically ask yourself, start out by saying, hey, do I have an agreement with whoever's asked me to build this piece of software? Do I have an agreement on where this actually needs to go? Right. Is it a toy that I throw away at the end or is this it? Right. Because there's going to be very different approaches to these things. Yeah. Yeah. I think that gives me a lot to think about. You just started unwinding.

I told you, man. I bring the heat. I've got opinions about this stuff. That's the start of the diverging path though, right? We can all imagine the best of intentions for making things maintainable over the long term, but those pressures exist at all times. And certainly, I don't disagree at all. That initial stage,

you don't know if the thing that you're making is trash or not. And the only way to figure out if it's trash is get it to the point where it actually runs and see if it's not trash. And the faster you get to that point, the faster you figure out whether it's actually worth putting the effort in. So my only point is to say, no, I agree. But I think that the point there is that the diverging tensions between like a sort of high-quality or maintainable code base, and I have a thing to talk about

whether we think those are different things later, but I think those paths start diverging right from the beginning. Is it high-quality, maintainable code base, or does it do the job? And resources are finite, time is finite. And I think it's a good point, those pressures are there right from the beginning.

Ian, do you have anything you want to say? Yeah, it's kind of along those same lines. I think an important piece of the beginning there is, does this piece of software actually, is this something that we can actually solve with software? I've been in the situation where, you know, you've got to build a piece of software to automate something or do something, and it turns out the edge cases and the error cases are just too numerous, and it's more of a headache to fix those than it is to actually just manually do the work. So I think at the very beginning,

you got to kind of start with an actual solvable problem. Okay. But let's say that we've already figured out that, you know, we have some code, right? We wrote some code, we prototyped, we're like, okay, this idea is solid and we can go forward with it. I guess like at that point,

what should we be doing or should we be thinking about to make that code base more maintainable? Or are we saying that perhaps we shouldn't be focused on maintenance in these earlier times and we should be trying to focus on it later? I guess, how do we start to strike that balance there?

Because I know a lot of us go into companies and whether it's moving from monolith to microservices or just, you know, you have microservices and you're starting a new one. And there's always, as Johnny and Sammy said, there's this hope. There's just like green field, this whole new path you can go. And there's all these different directions. And there always is that trade-off of time, as you also brought up, and constraints. But at some point we have to say, there's obviously a trade-off at some point where if you don't maintain, then you're going to have to wind up throwing this whole thing out.

And then all of that time you put into it is now gone. And you have to start all over again and pay those costs all over again. So is there a point or is there, I guess, some signal or some way of knowing at what point you should start focusing a little bit more on maintenance and stop trying to maybe optimize for those time or whatever it is, product features, whatnot. I think we need a definition from what maintainable is at that stage, right? Because I think if I'm speaking from a developer, like the person writing the code standpoint, my idea of maintenance is perhaps different from, say, an operator's viewpoint of maintenance, right? And again, there's a subtle difference. Well, in a lot of cases, not so subtle between sort of operating, right, a piece of software

that you and other teammates have written, right? And making that easy to operate, easy to maintain from an operational standpoint. And then there's the aspect of, okay, I'm working on a problem domain. I don't know everything there is to know. I don't know what business is going to throw at me next. I need to structure my code, right? And perhaps follow some best practices and design patterns, whatever the case may be, right?

To be able to extend the software easily, right? So different kind of views on maintenance and maintainability of long-term. And interestingly enough, different companies companies depending at their stage of sort of technology sort of maturity right or engineering discipline and maturity right are going to have they're going to be on different sort of positions on that on sort of spectrum right so i think really you can't look at sort of maintainability in a vacuum on its own you kind of have to say well for us what does maintainability Right. And that's going to vary from team to team.

I feel like Sam has something he wants to say about that. I'm not sure I can. Well, the problem is actually, I think I hung on to something you said earlier there, Johnny, about the difference between operating and maintaining. Certainly there, there are differences, but I don't know. I've had a few thoughts swirling around in my head about what it means to do maintenance. Because I do agree, like we need a definition for what that actually is, right? My sense is that there are two fundamental ways in which we can think about maintenance.

One is fixing bugs and the other is adding features. And those two things have tensions involved, right? But what I was thinking about when you were talking was how, I think there actually is a really important commonality between operational characteristics and development time characteristics. And to me, that's failure locality. It's the idea that I want the computers to tell me as closely as possible how the thing is failing so that I can fix it. And that is true whether I am writing tests, because to me, a good test is something

where when it fails, I know right where to look, I know right what to fix. And that's not something that I need to rebuild a ton of context for somebody else can come in and do right. Similarly, when something is failing in one of its operational characteristics i want to know as closely as possible like where to look where to go so i think there's a there's a there's a common principle there in terms of the way that we should be approaching making and that's mostly on the bug fixing side right mostly i think on the bug fixing side but maybe where the paths meet in the woods of the two approaches of the bug fixing and the feature adding is the extent to which, you know, your tests and your telemetry and whatever systems you have set up for consuming your operational information are able to tell you when the thing that you were adding over here broke some stuff over there right and quickly guys do the same thing because i mean ultimately what we're talking

about here is like does the software continue to be correct or not and how can you tell so but i apologize though because like i said that it kind of took off like a thing you said at the beginning and i feel like i missed the tail end of it which is why i was thinking about biting my tongue, but then Chris called on me, so I'm sorry. Ian? Ian's got some thoughts too. He lives deep in thought. Yeah, so this idea of locality, I think is important for maintainability as well.

I feel like if we minimize the amount of unrelated changes that have to happen to make a change, a desirable change that is maintainable, right? So because maintainability has all these facets, right? Perhaps we can sort of come at it from the other way around. What would we call unmaintainable software, right? How do we define that? I think that's something we perhaps all might agree on as these are the sets of things, right, in practice that make a piece of software unmaintainable

and i'm sure over the course of our individual careers we've probably seen a few right i mean you can start with anything you can live for right especially in go where there isn't a lot of disagreement about like what should go into linters that's a lot more common in other languages but i mean if you can live for it then, then yeah, like, you know, put basic docs on your functions and your exported members. You know, maybe don't have insanely short variable names for literally everything that you do.

There's the minimum bar, right? And I feel like we can almost just, you know, put a checkmark on the list and say, okay, if you can lint for it, like, maybe just, you know, do that from the start. And actually, I would loop back to the early question, right? Even in a new code base, like, throw the linters in okay, if you can lint for it, maybe just do that from the start. And actually, I would loop back to the early question. Even in a new code base, throw the linters in as soon as you can. Maybe you can ignore and just write some dumb one-liners for your function docs, but don't make things harder for yourself. Just start from that

and get yourself a nice little foundational baseline going of the basics. But I used words to identify the easy parts so somebody else can talk about the other things that might be harder to agree on about what makes unmaintainable code. Because I think it's a great question. I think coming at this from a negative angle, it's a good way to do this. Ian, do you have ideas on what makes unmaintainable code? I mean, I do. I think fundamentally untestable code is unmaintainable. If you can't know if it's correct, you can't make changes to it.

So some things that make things fundamentally untestable, like heavy use of globals, that sort of thing, I think is probably the biggest thing that sticks out to me. I feel like there's something you said earlier, Ian, that I would classify as making software unmaintainable, which is like, if we don't know what we're building, I feel like that fundamentally makes it very difficult for us to write software that we can maintain. I feel like if you don't know the scope, right? If you haven't sat down and written a scope and written a design, then the resulting software, it might do what you as an individual

thought the software should do, but that might be slightly different from what other individuals thought the software would do. And I think that's one of those longer-term maintainability problems. If people have different concepts of what a specific code base or a specific package or a specific function even is supposed to be doing, then when multiple people work on it over time, it kind of atrophies and it kind of falls apart. I think we've all seen these functions that live in code bases that have just been hijacked to do something completely different from what the original author intended. And you're like, how did that happen? And I feel like that's upstream a scoping problem. That's

upstream, like we didn't properly define what this thing was supposed to do. And I feel like that fits in that category of like gnarly things that Sam was just like, you did the easy stuff. This is a harder thing. Well, no, I agree. But I want to ask a maybe annoying question there, which is, so Ian said code is fundamentally untestable, right? How is code that's untestable different from code where the intent is not clear?

It's not. intent is not clear. It's not. The behavior is not defined. What if you're testing the wrong thing? Right. Right. But you have tests, right? You test that.

Right. The assumptions you made, though wrong, pass your test. Right. I feel like there's that cyclical testing that people tend to get into with unit testing as well, where they test at the the wrong level. And it's like, well, that thing is tested, you've tested it, it does the thing that you thought it was supposed to do. But like, the thing you thought you were supposed to do is not the thing that you actually wanted to do. And I think that's the difference there as well. Like you as an individual, if you write the function, and then you write the test, then obviously, I mean, I hope that the tests that you wrote have now

confirmed that that function does what you want. But if, you know, you, Sam, and I have two completely different ideas about what this function is supposed to be doing, we can both write tests and those tests can both pass. But that doesn't solve that, you know, initial problem of like scoping, like this function itself is still not well defined. So I do think that they are, they are divergent paths or maybe like one encapsulates the other, right? Like if you have untestable code, then you've most definitely probably scoped it wrong.

But scoping it wrong doesn't necessarily mean that it's untestable. Yeah, no, that certainly is the case. I agree. I do think there's a difference between these things, but I think that it's worth asking that question because having a clear sense of what it is that this code is supposed to do, the boundaries within which it's supposed to

exist, is astonishingly important to actually trying to maintain a code, especially as a codebase gets larger. Should this function go in this package? Should it be a new package? Why? What's the logic by which we are grouping these things? Is there some like broader theme that we can use to decide that this is how we actually organize our code? This is where we ought to look for something like these. As your code base grows larger, you can't just kind of, you know,

oomph your way through like finding things inside of it, like having larger patterns for why code gets grouped in different ways, having larger structural patterns, whether those are something formally defined by type-checkable interface contracts, less formally defined in terms of naming convention patterns, or really informally defined, but still very important in terms of general patterns and responsibilities, and we're going to put a collection of packages under a single tree that are service-y shaped or something like that. As your codebase gets

larger, and take note of that intro bit where I'm currently thinking about and mostly learning a 250,000 line codebase at the moment, having patterns and structures like this, I think do an enormous amount to orienting the maintainer who inevitably, given is the first step towards being able to figure out what should be tested, which is the next step on the path to figuring out whether the thing does what it's supposed to do in the first place.

So you don't wake up one day and you have an unmaintainable code base, right? No. You're going with the same terminology that we established. So you gradually get there. So we've already sort of created, or at least identified the nuance between correct code and testable code. The two are not necessarily the same thing, right?

But I think to me, you start to gradually get towards an unmaintainable code as you start to sort of let your technical debt, which is, that's not a bad word. That's not a dirty set of words, right? Technical debt is absolutely, I think personally, I think that's necessary, right? And when you're evolving software, as long as you pay it back. If you don't have debt, you haven't done anything great.

Exactly. Exactly. Yeah. Yeah. Literally you need technical debt is part of the currency, right? That you have, right? To trade for things, you know, basically to pay an upfront cost for a certain feature, right? And then to come back and actually fix the things that really make it maintainable in the long term. So when you don't address your technical debt, you start to creep towards that maintainability, to the point where it's like, oh, man, I'm looking at a code base,

which has been around for a couple of years, and there's three different ways to do the same thing with a slightly different parameterization. This one accepts an empty interface here because somebody wanted to make it super flexible, but they didn't understand enough about the problem domain. Now you end up having to create another one with more specific. You can see those like sprinkles of just different people

trying to solve the same problem in different ways and not basically saying, okay, we've done enough sort of damage, right? We have an understanding of what it is that this thing's supposed to do now. Can we just take a minute, take a step back, take all the different ways we would do the same thing, refactor, right, for maintainability, right? As opposed to somebody coming in next week and says, oh, now I need a slightly different version of this thing. Now you have four ways of doing the same thing, right?

So you start that march towards increasingly unmaintainable software. But is any software truly unmaintainable? That would mean that you can never do anything else to it unless you're basically on a code freeze. That's it. It's done. You're never touching it again. Like, as long as software is delivering value

for the business, you have to maintain it, right? So if you don't do the things you're supposed to be doing, right, towards making something maintainable for the long term, you're going to increasingly creep towards that increasingly unmaintainable state. This episode is brought to you by our friends at Square. Develop on the platform that sellers trust. Support Square sellers by building apps for today's business needs. As a Square app partner, you can reach millions of business owners searching for trusted software solutions. As a Square solutions partner, you can get hired by sellers on the Square platform, find new clients, and build apps that meet their needs. Square loves developers.

They work hard to enable you to launch fast with their developer tools. You get a full sandbox environment, an interactive API Explorer, live event monitoring, backend SDKs for PHP, Ruby, Java,.NET, Python, and Node. You get secure payment SDKs for iOS, Android, React Native, and Flutter. You get it all.

Learn more and get started at changelog.com slash square. Again, changelog.com slash square. Have you guys heard the rant about the word performant and how it's a made up word that doesn't mean anything? Like, is the software performant? What does that mean? What does that mean? How fast is performant? So I ask it here because I think maintainable is the same kind of problem, right? The point is we're on a sliding scale here and whether or not you would call something maintainable or unmaintainable, I mean, realistically, that's a question of how bad your day has been. But if you're being a little bit more high-minded, it's like, what is the appetite of the organization that I am in for allocating a bunch of time for being able to make changes

to this thing? So it is fundamentally contextual to the environment that you are operating in the same way that is the code performance is actually a question about what the appetite of the organization is for accepting latency along this particular path.

There is no objective standard here for it, which is why, as you say, Johnny, it's this risk of the creek towards you can continually add things that might make it more unmaintainable, but maybe would never necessarily reach there or exogenous factors might change, like how many things are relying on that code that will suddenly have it be in an maintainable state, whereas before it seemed fine. And I think you have a good call out there because I think the business is also on the hook for ensuring that there is enough space, time, resources for keeping the software maintainable. Because if you don't, and as I'm sure we've all either, you know, experienced or heard, like if you're never making room for improving your code base, not adding new features, not, not fixing bugs alone, but really improving the code base to make it easier to work in, eventually your, your shipment of features is going to come to a crawl

and everybody's like scratching their heads wondering why does it take like three four sprints of two weeks of pop to add just this one feature everybody's ends up scratching their heads asking that same question and the answer is always the same thing well we keep wanting to uh go back and fix these other things but we never get the time you know you know there's always a demand for ship this next thing. This is important, buy this, whatever it is, right? Basically, the business is not caring about what it takes, what behind the scenes, the things that they can't see, right? We're the engineers. We need to make the case for the time,

the resources, the space, right? To improve the code base. If we don't do that, the business is not going to do that for us automatically. To them, it's like, hey, can I get this? And you deliver it. Here's what's next. Can I get this? Because they have pressures from customers, from stakeholders,

right? As long as you keep giving them stuff, they're going to keep asking for more stuff, right? If you don't fight for the space and time to make your code base maintainable, right? Easier to keep adding things to, that's how you get in trouble. We had a discussion recently. We a we have a thing for fun called gardening week which i had not heard referred to this way before it's my first time hearing the term but you know after we do a release we have like a gardening week basically and there was a discussion about like well should we have a gardening week like is it a bad thing that we have gardening week wouldn't it be great if we didn't like need to have a gardening week and

my sense is that there are three. I'm going to try to pre-count the number of universes and then get it wrong. So I'm going to say there's three universes and then get it wrong. There is the universe where you don't have a gardening week, but you need one. You don't have a gardening week and you don't need one.

And you have a gardening week and you need one. I don't think the like have it and not need it is a super realistic one. But there you go. There's my there's my missed count. There's three, but maybe there's four. Anyway, point is, like, I had this initial reaction to seeing the existence of gardening week and, and seeing, oh, like, come on, can't we kind of do that as we go along? And then I realized, no, like, actually, I would so much rather be in a world where there is two words, gardening week, that has an understood meaning and an understood reason why it's valuable to the business and why it's valuable to the people involved, and that we have that time and that space

allocated in a sort of structured way because it is way better than being in a world where you need a gardening week and don't have it. I would love to be in a world where we don't need one at all. I'm not sure that's ever realistic, but having phrases like this, I think help to maybe make it less of a fight all the time to have to advocate for this. But if you don't have one and you need one, you need to advocate

because yeah, otherwise it's not gonna, you're just gonna keep on trying to push that Sisyphean rock up the hill and struggling and the business will only see things slowing down and not really understand why. And I guess I have a sort of question off of that, but I want to preface it with something.

The code bases I've worked on that have been unmaintainable, or it feels like unmaintainable, where it's like, this thing should take a week, and instead it's taking four months to do. This is miserable. I hate everything. Right. Whenever I've wound up in one of those situations, it's always like, it's not like one big thing. That's the problem. It's always like thousands of little tiny things. And then you look at those in isolation and everybody's like, well, that's a little thing.

So is it really worth it to go and fix it? There's all this other stuff we need to do. Like, it's always like, we know that it's death by a thousand paper cuts, but we never want to like, stop any one of those paper cuts from happening. So the question I have based on that is like, is gardening week enough? Or should we actually be pushing further and saying, we want like a gardening team? Because I think there's this, you know, myth that exists in our industry that like, people wouldn't like working on said team. It'd there's this myth that exists in our industry that people wouldn't like working on said team. It'd be like this miserable thing where you're like,

that's the team of people that doesn't get to do the fun stuff of building features and building new products and doing all of that. I have some friends that are like this. No, just give them code base. Let them go and just clean up some certain parts of code bases.

Scratch that itch. Like the garbage men. Like every city, imagine what our cities would be like if there were no garbage people, right? Not garbage people, but like trash collectors. That's a better...

I mean, it'd be good to have a city that has... I didn't hear it till you said it. I mean, it would be a wonderful world if we didn't have garbage people. Imagine a world where we didn't have trash collectors, right? Like, our streets would be disgusting, our cities would be awful, but, like, there's no one there

that's saying, like, and there's some people that are trash collectors, and they love their life. Like, they are so happy with, like, what their job is and how they live their life. And I are so happy with like what their job is and how they live their life. And I think there's like a significant portion of software engineers that want to do that sort of thing that are like, let me like take this part of the code base.

It has that function that has like 15 parameters. And that's just like, I'm going to think about it and refactor it and just make it better. So the next time someone comes through, it's like not as bad to be in that space. But I'm wondering if that's something that we should be pushing for or if there's like another version of that. Or is it just like, well, let's just start with gardening weeks and then we can figure out what we should be doing after we have this at like most of the organizations. I don't think that's something that needs to be optioned personally.

Right. If I'm running a team, if I'm an engineering manager, unless the team is gelling so well that I don't need to formalize the process, I'm just making it a formalized process. I'm just making it just like going on call. There's a rotation schedule. You go in that squad. Other people call it health squad. We can call it gardeners if you want. But you do your time in there.

And I say that, I don't want it to sound like it's a punishment or a chore. I think every engineer needs to understand what it's like to work on greenfield projects. And they also need to understand how you maintain existing software that's been around for a while, that is making money for the business and paying your salary. You need to understand how that software works. And because when we need to change it, maintain it, right? Add features, fix security holes, whatever it is, right? Everybody should be somewhat well-versed, right? In that software. And obviously different people, depending on tenure and seniority, whatever, they're going to have a much better

time at sort of holding the whole problem domain in their heads, depending on how large your code base is and all these other factors. But at least everybody's working towards a shared common understanding of the software so that we can all keep this thing alive that's paying our salaries. So I think this is something that every engineer should feel responsible for basically contributing to the health of a piece of code base, or however many you have in case of microservices or whatever. I would question, though, if that's actually a good idea. Because here's where I'm coming.

I always pull analogies from other things. I gave a talk at GopherCon where I basically talked about how we're similar to the publishing industry. And when I hear everybody needs to do a rotation on this team to help clean up the code base, I hear everybody needs to become a copy editor. And I don't like that idea. I feel like this is a higher form of engineering in a way than even just product or feature engineering. I feel like feature or product engineering, that's like, here's the

requirements. It's being done well. Here's the requirements. Here's the scope. Go make a design. It feels like this more structured thing. Whereas when you're trying to do gardening or doing maintenance of a code base, I know that's still kind of a nebulous word right now. But when you're trying to do this, it's like trying to pull out value when you don't necessarily have that level of structure, especially within organizations, to make that happen. And I think the thing I worry about is making bad trade-offs when it comes to trying to guard in your code bases. Because just like we have to make trade-offs when it comes to product features, we have to make trade-offs when it comes to guarding. There's a

thousand paper cuts that are happening here. We have to decide, you know, which ones are at a part of your body that's just annoying and which ones are like slicing an artery that is going to make you bleed out. Gosh, Chris. Your analogies, my friend. These analogies are escalating, right? Up and away. But I feel like that's a very difficult thing to figure out and to determine.

And I feel like there are people that are really good at that prototyping, that hackathon-style engineering. I feel like there's people that are really good at this more maintenance mindset engineering. And just like I don't want to put maintenance people into hackathon-style stuff because they burn out, they are miserable,

I don't want to take people that would rather just be doing product features or really rather be like give me a ticket i'll do that ticket and then i move on in a situation where now it's just like here's a code base go make it better or go figure out the things that we need to do to make it better so yeah i think that's where i fall on that i disagree with you on that and because disagreement nice so this idea of like greenfield development right like i think in a lot of ways it's an easier process you know you're you're starting from new you don't have a lot of things to consider but you can only do it well if you have had the experience of having to go back and change things so if all you do is build

greenfield stuff you're're going to leave a trail of debt behind you and never realize it. So if you have this distinction between a maintenance engineer and a greenfield engineer, I think you're going to kind of end up with bad software. It's not necessarily saying that you have to be one or the other. It's more so saying that like, we shouldn't make everybody like, that's why I don't like it being a rotation, right? That's the thing I was more objecting to than anything else. It's like, if people don't want to do this, that seems like an option that we can have in the same way that I think if people don't want to do product engineering, for example.

We're not like, every engineer that works at a company must do product engineering. So I don't think that every engineer should have to do maintenance work. I think they should have to be aware of the maintenance work that goes on. Just like in observability, we can't just like, or with SREs, we can't just be like, the SREs will just take care of all of our reliability things. The engineers still have to care about this. But I think it's important to make the distinction of like who winds up working on this like the most and who develops like the ethos of it. Right. So, yeah, I mean, but I don't disagree with you either. So I think it is important to get people seeing the repercussions of what they build.

Right. We can't have a world where there are just like, I mean, that's kind of the world we live in right now, where there's just a bunch of people running around creating stuff. And then they're never around to pay for, you know, the repercussions of it, whether that's because, you know, it's kind of built into the organization, or because, you know, it's a startup or whatever. And, oh, well, we built this thing. And now we've made an exit. And now the next group of people can deal with all of our terrible decisions. You know, whatever, whatever form it takes of like, cut and run, we don't want people to be doing that sort of thing. So I think it is important for people to understand maintenance, which is also why I guess I'm trying to like, raise maintenance engineering to a higher level, right? It's like,

you know, maintenance engineering, in some ways probably should be above product engineering, because you can't do product engineering without maintenance engineering, but you can definitely do maintenance engineering without product engineering because there's some code bases that have been around for decades that's just like, no, your job is to keep this going. We're not adding features to it. We're not doing anything new with it, but we have to keep this thing going. I don't know. I would point back to, at least on the maintenance engineering is a higher form than product engineering. I'm not sure I can compare know. I would point back to, at least on the maintenance engineering is a higher form than product engineering. I'm not sure I can compare them. I would go back to John's earlier

point about technical debt. And you borrow money to start a company. You take on debt for a good reason. And it's because you're trying to make something on the outside. But I would say the debt you have to take on needs to be good debt, right? There's good debt and there's bad debt. Well, we can push the analogy too far. But feel like this like this does come up in our code bases though because there's some code bases you're like this technical debt like this is acceptable for the trade-off we got and there's other technical debt you're like why this was not debt we need to take on or this was debt we were never going to be able to pay down. I think that's worthy of making sure we understand the debt that we're taking on, which I feel like is what maintenance engineers would understand a little bit better than

people that aren't as focused on what the repercussions of different types of technical debt we might take on. This episode is brought to you by Honeycomb. Find your most perplexing application issues. Honeycomb is a fast analysis tool that reveals the truth about every aspect of your application in production. Find out how users experience your code in complex and unpredictable environments. Find patterns and outliers across billions of rows of data, and definitively solve your problems. And we use Honeycomb here at Change.

That's why we welcome the opportunity to add them as one of our infrastructure partners. In particular, we use Honeycomb to track down CDN issues recently, which we talked about at length on the Kaizen edition of the Ship It podcast. So check that out. Here's the thing. Teams who don't use Honeycomb are forced to find the needle in the haystack. They scroll through endless dashboards playing whack-a-mole. They deal with alert floods, trying to guess which one matters.

And they go from tool to tool to tool playing sleuth, trying to figure out how all the puzzle pieces fit together. It's this context switching and tool sprawl that are slowly killing teams' effectiveness and ultimately hindering their business. With Honeycomb, you get a fast, unified, and clear understanding of the one thing driving your business, production. With Honeycomb, you guess less and you know more. Join the swarm and try Honeycomb free today at honeycomb.io slash changelog. Again, honeycomb.io. And no more worrying about access controls. Start shipping internal apps to move your business forward in minutes with basically zero uptime, reliability, or maintenance burden on your team. Some of the best teams out there trust Retool, Brex, Coinbase, Plaid, DoorDash, LegalGenius, Amazon, Allbirds, Peloton, and so many more. The developers at these teams trust Retool as their platform to build their internal tools, and that means you can too. It's free to try, so head to retool.com slash changelog. Again, retool.com slash changelog. is there a difference between maintainable code and good code in the way that we are talking about it good doesn't tell me anything though maybe okay i mean actually very good code that is obfuscated code and you know

it's it's designed to be short and terse and and it's good by that definition right so i always feel like we have to sort of provide the criteria sure well no but that's why i'm asking the question because when i was thinking about maintainable code i found myself sliding into thinking about good code and And the distinction between these, I think it's interesting where it is or isn't different. Like one thing that jumps out to me clearly is how I have internalized that belief that the best code is code that is easy to delete. That is the best code. It's not easy to extend. It's not super abstract and great nope you can delete that that's what makes so the point there being certainly i and i agree like i do have some of my own internal definitions

of good that i think have started to skew towards uh maintenance but i asked the question here because if we're having a discussion about like what is the relative value of maintenance versus like pushing forward right then it seems like part of the thing that we're asking is what our values are in terms of what actually makes code good versus not. So I asked the question to put a spotlight on, I guess, another definitional issue that seems to be at play behind some of this. I feel like they're different because I feel like there's code that I've seen and code that I've dealt with where I'm like, this is not good for whatever reason. Maybe it's like a Go code base that's written in Guava, right? It's Java, but just happens to have Go syntax. Wait, that's a thing?

Yeah. Wow, you never come across an I iterable? You've never come across some factory factories in your Go code base? Oh, okay, wait. I thought this is like an actual thing in your Go code? Oh, okay. Wait, I thought this is like an actual thing. Like, no, no, no. Like a Java person wrote Go. Like I was about to Google like Guava Go.

No, no, no, no. Something. Okay. Okay. All right. You probably heard of Gooby as well and Guyton. Okay. Yes, yes, yes.

We're back in the domain that I'm familiar with. Slightly less terrifying. So I've worked at some of these codebases before where I'm like, this is bad and this is gross and I do not like this. And it makes me as a gopher very mad that someone has done this. But I wouldn't say that those codebases

were unmaintainable or like weren't maintainable. It's like you're using interfaces everywhere. I hate you for it. But like there's a logic to them. They're arranged in a way that makes sense. And you can see like, OK, this is where we would add things. If we want to get rid of this is where we delete that. Right.

So for that reason, it seems like, yeah, we can have bad code that is still maintainable code. So I do think they are separate things. I do think in that case, like, you know, the reason we're saying it's bad code is because you're writing it in a language, but not embracing the language you're writing it in. And I think, you know, to your point, you could say good has to encapsulate both writing the language for the language and also making it maintainable, in which case then you can definitionally make it

so that yes, good code and maintainable code are the same thing, but I don't think you necessarily have to do that. I think they can be distinct. So, I mean, under that, nevermind. It's not even worth doing that. I'm going to like, let's go look at like, you know, a Go implementation of Paxos

and see if that's like super maintainable by someone, right? Is it good code? Is it maintainable code? I think that the it's complicated, but it's personal. There is certainly a degree of subjectivity to it, but that's actually, that's part of the thing too, right? Like it is personal, it is subjective. So is a given code base

maintainable in the hands of one team, but you swap out different people or a different team dynamic and it's not maintainable anymore the hands of one team, but you swap out different people or a different team dynamic and it's not maintainable anymore? I think if you're using Go, it's a bit more so than it would be otherwise. And I'm speaking like completely from my own experience here

because Go was sort of designed with that in mind, right? It was designed with the ability, I mean, our lenders, right? They all follow a similar approach, our lenders, right? They all follow a similar approach. Our GoFont, right? To remove everybody's pet peeves around formatting. I want my braces on this line.

I want mine on my, these kinds of things, right? So our entire sort of ecosystem prides itself on the ability of anyone finding a Go code base. You might not understand the problem domain that you're reading about but you could read the go code and the code itself will be readable to you right as somebody who's completely new to that code base right and since you're going through a two hundred thousand line line code base right now you can attest to that probably right so to me that's the thing like the technology that we use can help in the maintainability of software. So if we add that sort of lens to it, I think the technology plays a huge role in that as well, not just the people.

I think in that case, too, we have to maybe level up what maintainability means as well. Because it's like, okay, well, if we want to be able to move this code base between teams, say, like if we have, say, microservices, and those microservices might be handed off to different teams or reorg to make things make more sense. I think that's where you have to start building more, I guess, documentation in this case, or just processes and practices into your organization that allows that code to be moved between.

And that inability for a code base to move from one team to another team becomes a problem of maintenance, right? It's like, okay, now this code base that might be maintainable for one team is now classified as unmaintainable because it can't be maintained by two teams or three teams or however many teams

that you want it to be maintained by. And then that's a thing you have to go back in and resolve. And I wonder if that helps us frame maintenance as well to help us answer this various question that exists right now of what is maintenance? Maybe it is this thing of a sense of the team, so it's a subjective thing, and a comparator over time. So it's like, all right, well, this code base is currently maintainable

because we can do something that we weren't able to do before now we can do it now and we continue to sustain the ability to do that into the future if we add a new thing to our definition of maintenance then something that was maintainable becomes unmaintainable and we have to like bring it back to home of maintenance kind of it reminds me a lot of simplicity, right? Like what is simple? And the famous Rich Hickey talk of simple made easy and all of that. It's like, it's this very difficult concept

to like pin down, grab down, and you know it when you go into a code base, but you know a simple code base when you're in it and you're working in it. And it's hard to figure out when you've lost that simplicity, but you always sense that you've lost it. I feel like maintenance is the same thing. It's like, you know when you're in a it. And it's hard to figure out when you've lost that simplicity, but you always sense that you've lost it. I feel like maintenance is the same thing. It's like, you know when you're in a maintainable code base.

It has a certain feeling. And then you know when you've lost that. You know when something has diverged, and it's like, okay, this no longer feels like a maintainable code base anymore. There's something wrong with the way that we can properly maintain it. I feel like this is all subjective, though. I mean, this is all very much our own experiences. And I think naturally as an engineer grows from, you know, junior, intermediate, you know, senior, super duper senior, whatever other titles we throw out these days.

Staff. Staff and, you know, whatever. I prefer superist senior senior thank you very much right it's all i mean you learn you get that gut feeling that chris is talking about right he's like yes this is based on my experience but you know based on what i've been through this feels good and when you start to lose that grip on and when you can no longer hold in your in your head all the different strands that you've had to pull to understand one single feature in a code base once you've lost that then then you kind of, ah, you know, this could be better, right?

But is there a more scientific method, right? Like I remember back in my Ruby days, we relied quite a bit on things like code climate and whatnot to measure things like, you know, cyclomatic complexity and sort of repetition, you know, all these kinds of things like, you know, some heuristics and trying to figure out, okay, based on a common set of agreed upon don't do's in this case, the Ruby community, right? These are the things you should avoid doing your code base. So you get that feedback almost immediately, right? You open up a PR and then boom, you've got, you've got some feedback from a machine, right? Not from another human, from a machine telling you, Hey, we ran some lanterns and this is what we found.

RuboCop yelling at you. I mean, I remember I'd be fighting RuboCop every day on every NPR because I'd be like, oh, okay, fine. I have to go and fix that, right? Okay, fine. But you're paying that cost and you have a machine helping you, right, to identify these things and all in the hope that you will not get to a point where you've got so many of these the accumulation of these minor paper cuts right i recall them that the code basically becomes sort

of every time you're in there it's kind of feels yucky kind of feels that feeling that you're talking about chris like oh man we have too many you know little pinpricks here kind of thing is there such a thing and go up obviously we have our linters you know we can have that sort of immediate feedback mechanism but do we all agree on patterns and best practices and things the stuff that we we sort of usually thought it would idiomatic go is that our common set of patterns is that as scientific as we get i feel like for go we could probably get at least part of the way there like i once again i'm always thinking about writing because I'm a writer. I'm kind of thinking of manuals of style and how

when you have a manual style, it has a lot of very strong opinions about this is how you do things. Exactly. For those wondering what we're laughing about, Sam just pulled up a book on style. Yeah, Strunk and White. It's a classic. The book on style yeah strunken white it's a classic the book on

style in fact yes the book yep the book indeed chicago manual sales sitting on my bookshelf right over there oh it's a fight it's a fight yeah writer's version of tabs versus spaces yeah yep yeah exactly it's like the the famous you know, you have closing quotation marks, does the comma go inside or outside of that? Right? Like, there are these things that are like, I don't think there is an objective answer, but we still need an answer. So I think like, if we as a community, or if we as a group of people can craft something like that, then I think the answer to your question is like, yes, I feel like there is a more scientific approach we can take to things. But I think for us, especially, you know, if we take kind of wider angle of it, not just go, I don't think so.

I think a lot about like, you know, once again, writing. It's like, what's good in a romance novel or like one of those trashy, you know, airport romance novels versus what's good in the New York Times is going to be very different. And neither is wrong. And you probably wouldn't want to label either one as wrong.

I feel like code is the same way. What is good in Go is not the same as what is good in Java. And I think even within Go codebases, what's good in some Go codebases is probably not good in others. Like I think of the use of the unsafe package or the use of the sync package. Like some teams and some organizations,

that's a good decision. That's a good thing to use. It's like you have the engineers with the experience. Other teams probably shouldn't be doing that. Like someone's going to like blow off their foot with that. Man, gruesome analogies today. Seriously. I mean, and also I don't know about you,

but I start every main package with a compare and swap because that's just how I roll. Why use mutexes when you can use atomics? Like, come on. It's all the cool kids are doing. I think it's like

one of those things that like, if you define enough components of it, though, you can get toward a more objective thing. But I think it will always have hefty, hefty amounts of subjectivity that you need to kind of abide by. And I think it will always have hefty, hefty amounts of subjectivity that you need

to kind of abide by. And I think we as a Go community, I think this is like a thing that we need to do, is we need to start writing these things down more, right? You know, when you look at the Chicago Manual style or the AP style book, they didn't just appear as thousand page, well, the AP manual style

is shorter, but they didn't just appear as these like huge books the important parts they appeared over time okay I have problems with shrunken white but we are not style the good parts 85 the elements of pages what is this thousand page business you're talking about

continue there are some antiquated things in that but anyway they didn't just spring out of like out of nothing they were developed by like smaller style sheets for specific books over time and then compounded over time so it's like i think we as a community need to start just doing that action of like more people writing manuals and there's some out there right you go i think uber has one for go i think facebook has one for go you know look you find them but they're all very, very, very short, not even 85 pages, right? These are like,

you know, five pages. And I think that's what makes it so hard to understand what maintainability is at the end of the day. Because, you know, once again, to go back to the paper cuts, those small paper cuts are those small little decisions that aren't being aligned over time. It's the equivalent of like not making a decision about whether the comma goes inside or outside the quotes. And then it's like different for every paragraph in the whole book. And it's like, pick a way to do it and then stick with it. It's one of the reasons we love GoFumped. It's like brackets go here, like these things go in these places and that's where they live. And now we don't have to think about, we don't have to care about these things. That was a very, very long way of answering your

question. I hope I actually like captured some of it. Can I, can I try to sum that up into a pseudoscientific, still subjective, but something that has more numbery bits in it? Go for it. I want to loop back on the correctness bit that we were talking about earlier. I mean, everything you were talking about just now are essentially correctness criteria. Not formal verification correctness, but like it's correct if it passes linting and not if it doesn't. I would offer that the maintainability of a code base,

starting just accepting the premise, this is subjective because I don't see any way around it, is the ratio. You can understand the maintainability of a code base by looking at the ratio of time spent researching what is correct versus making the thing correct. Whether you're talking about trying to fix a bug. And so, you know, failure locality, right? Like, how long does it take to figure out why the thing is failing?

How long does it take to figure out the rule you should apply in deciding how to fix the thing versus actually fixing the thing. That ratio, which yes, will vary from person to person based on the length of their experience, they're familiar with the code base, but again, I don't see a way around it. I think that may be the kind of core of what we're driving at here. I feel like that sums up what I took like 10 minutes to say. Okay, so speaking about Go, because we're kind of getting toward the end of the episode here,

and I feel like we've been very light on the Go content in this episode. So it's Go time. We've got to talk about it. I don't know. I think it's been implicit in there. I think it very much applies. Well, thank you for saving me there.

But is there anything that I guess any of you would give as, you know, what makes Go a good language for building maintainable code bases? What things do we have that are like, yes, this is why I like and enjoy writing maintainable code for, you know,

knowing that we haven't quite defined maintainability that well, but this is what I like that Go has that makes it so I can write maintainable code., you know, knowing that we haven't quite defined maintainability that well, but this is what I like that Go has that makes it so I can write maintainable code. And then what are some things, if there are any, that make Go a bad language for said maintainability? If you each want to answer that in turn. I already gave my reasoning for that.

The fact that I can drop new person in a Go code base, and even though they're still learning about the problem domain, it's not Go to have a problem. They're not fighting the language. It's not Go to have a problem understanding. It's, you know, what is this type doing? Where is it used? What business problem is it solving, right? So that for me, I've never experienced a language that gives me that sort of room, right, with my engineering team to be able to say, hey, go into this code.

I know you've never worked in this code base, but this is what it's supposed to do, right? I have a bug fix or I have a feature, you know, drop in there, see what you can do. And then relatively speaking to other languages, they come back much more quickly with the fix or the feature or whatever it is, because they didn't have to fight the language, right? They easily understood what they were reading. Once they understood the problem domain, they were able to execute and get the job done. To me, there's nothing like Go that does this, in my experience.

Ian? I'll mirror that. The simplicity of the language just adds to the maintainability. But on top of that, I think the AirZus values really adds to this. Being able to explicitly see air paths and not worrying about exceptions and trying to trace these all the way back up really adds to like glance ability and

therefore maintainability wait so you don't use panic defer recover everywhere in your code all over the place all the time that's crazy i think that's what we all did sam's best practice. I got to go rewrite a lot of code right now, guys. That's not good. Planning-driven development. Sam? Yeah. I mean, I'll go with the definition I just offered, right? Like, I think that I will mirror these same statements and translate them into the

ability to quickly, with very little research time and mostly fixed time, hone in on the basic answers to the questions about structured code. I remember running face first into a very large Ruby code base and being gobsmacked about my inability to figure out what felt like basic questions. I had been really just focusing on Go code for a while and came back and was just like, wow, I can't even like, I can't even just look up symbol names

and find them in places. There has to be special pseudo-static analysis. The manner in which the structures and variance rules, whatever the language, make it possible to, even over large code bases, have clear answers to questions about basic things like, what are all the instances of this, you know, references to this type, or to this interface, or perhaps even find all the implementations of it a relatively difficult thing to do,

and yet so very doable. Like, it goes very innumerable. It's very analyzable. And that means that most of my cognitive effort is spent on dealing with the higher order abstractions that people have tried to create because all the boring questions are quickly answerable. And so I can get right to the heart of the matter. All right. And then final question. What would you add, if you have anything,

to make Go more, let's not make Go itself more maintainable, but make the code that we write more maintainable, but make the code that we write more maintainable. Generics. Hell no. No, no, no. I want generics, but just for me. Everybody else can screw themselves, right?

Because then I know what mine do, and I don't have to deal with any of your garbage, and we're fine, right? I'm fine. That's what's important here. Yeah. Oh, I would add Rust. That's what I would add. I would really like to have compile time. Unpopular.

There you go. I would really like to have compile time guarantees about shared access to global immutable state. Forget this go test race garbage. Come on. Static or nothing.

Is this a helpful answer? This isn't a helpful answer. I'm sorry. Well, we avoided dependency management, so it's okay. We did! We did! We did! Scooted right by it. Well done. Okay. Nice. It's time for unpopular opinions

i actually think she'd probably leave Ian, you're up first. Unpopular opinion. I really don't have one this time. I cannot think of anything. Oh, man. It's okay. Sam has an extra four for you. Oh, crap.

Let me hear some other ones, and I'll... Let me know if something pops in my mind. Get the wheels turning. Sam, any unpopular opinions? I totally had like two at the beginning. And then we were talking about all this interesting stuff. Don't, well, I was going to say don't use gRPC, but I can't actually back that one up.

I don't think that's unpopular. You don't have to back it up. You can just throw it out there. Oh, I can just throw it out there? Jeez, God, that's liberating. Okay, don't use grpc let the masses interpret all right uh don't use grpc streams unpopular there we go okay that's nuanced yeah i feel like that's not unpopular though like grpc right that's the

problem is that all of my opinions are right and popular and so how am i there's the unpopular opinion there it is there we go there we go there we go yeah and yet i also think that that one might be a relatively common one even if not necessarily unpopular in our industry we've done enough meta dancing i feel like johnny gets to go and then maybe i'll have something more real in a minute maybe johnny i know you have many non-popular opinions so yeah let me let me formalize it into a natural and popular opinion i don't think you should have separate teams feature teams health squ, folks who only work on greenfield stuff or whatever. And I don't believe in sort of because what if I was hired and I was put on the bug squash team and then I want to work on some feature stuff?

Like, why am I never going to get a chance to do that? Right? No, I think teams should be loose in terms of their memberships. And people can just if you want to have formalized rotation to put people in different teams, and that goes for on-call as well. I firmly believe that if you're on a team writing software that goes into production and it needs to be operated, right? I think you need to be on the hook for when something goes wrong, you're on the pager,

you get called as a responder, right? I think that perhaps that is another opinion, but I think you need to be part of that rotation as well. It all basically falls under this umbrella that as an engineer, you need to be exposed through basically to all the layers, right, of the stack as it pertains to running the piece of software that helps a business make money. I think you need to understand each,

maybe even spend some time in support, be at the front lines of customer requests and bug filings or whatever it is, right? So play a role in each layer of that. And trust me, this is not a punishment. This is going to make you exponentially better engineer if you understand the different vectors, right, of things that are coming at your piece of software

that you're writing, that your teams are responsible for. That's going to give you superpowers as an engineer. That's all I can say. I think I agree with that. Not the on-call stuff, because I don't wake up for anything when I'm asleep. So the thing's just going to be broken

until I wake up in the morning. So if that's okay, then sure. But I feel like maybe on-call should be for... You wouldn't do well on my team. This is also why I purposely avoid roles that have on-call components to them because I know this about myself. So I,

it's good to know thyself. Yes. Man who understands his constraints. I mean, respect, right? Like I like high level stuff. I like being at 30,000 feet.

I can come down to the ground at some point, but like, and on calls usually not at 30,000 feet. That's what the birds. You're a little bit lower. But no, I think you're right, though. I mean, I've been doing a bunch of security engineering-related work at work. And I'm like, more people need to understand security, not to implement it themselves.

Please don't go roll your own crypto. But I think from actually getting in and understanding how public key infrastructure, how certs work, how public-private key pairs work, how cryptography in general works, enough people don't get exposed to that because security is in the specialized area.

I think reliability is the same sort of thing where it's like, there's the SRE team. They tackle stuff. So I am in general agreement with you. I think software engineers need to be taking more on as far as like what their remit is. It shouldn't just be, go build some products, go write some code.

Chris is there. I agree, except when it applies to me. I can agree with something in general and know that it wouldn't work that well for me. I mean, like I could do on-call just during the day hours. Or more so, I just will write software that doesn't crash at night. Ooh, okay, okay. Or doesn't crash at all.

There's the, see, that's the thing. That's the trick. Or only crashes. Crash a lot of software. Which is a real thing. That's a great thing. Ian, I have another. I do have one.

If you, if you want another minute to think otherwise, go, you can go ahead. Do not use semantic versioning for any versioning system that you create, unless you can define what backwards compatibility means clearly and precisely. Yep.

Skating into dependency management. Just the, yeah, that is a little sub tweet there, but yeah, Precisely. Mm-hmm. Yeah. Skating into dependency management. Just the way. Yeah. Yeah. Yeah. Yeah. That is. Yeah.

A little subtweet there, but yeah. Yeah. Sneaking in. Yeah. You know. Snuck that one in. Just. Yeah.

Yeah. I got you. I mean, oh, I don't know if that's going to be. It'll be popular among some people and very unpopular among other people. So. The conditionality makes it very interesting. If I've had previous conversations with you about it, you know that there's a deep troll

in that statement too, but I'm not going to unroll it here. I was going to say, depending on how there's going to be a whole swath of new Go developers who have no idea of the backstory for all this, there's going to be a bunch of people who are Go developers who have no idea of the backstory for All Discipline. There's going to be a bunch of people who are like,

yeah, yeah, I get what you mean. We get what you're putting down. I think it's a statement that stands on its own, independent of any history. Right, yeah. Yeah, there's a lot there to that one. I think it hooks into the episode. We started out just talking about maintenance,

and we're like, maybe we should define what maintenance is. So if there's a specific thing that's at the crux, I feel like when it comes to semantic versioning, the thing at the crux of it is this idea of backwards compatibility because that's what all of the digits in it are about. It's like, how

much have you maintained backwards compatibility? You don't define what that means. You kind of have a problem. You wind up with that function that is like means different things, the different people. And then it has 14 parameters. And they still don't describe the entire possible space. Because thus far, we've been talking mostly about maintenance as though it's something which is kind of confined to a single team. Which kind of isn't true when you've got a blast radius

that is as large as your dependerspace. This is why this is a series and not an episode. But yeah, I've never, ever run into a code base that I've maintained that has a 14 parameter function that only has three lines of code. Never. I've never felt that situation happen. Not even one time? Never have I ever. The thing I've never felt that situation happen. Not even one time?

Never have I ever. The thing about it is in that situation too, that code is like, this is actually correct for the state of the code base right now, and I'm mad about it.

Yeah, just one 14 parameter function calling another 14 parameter function. Well, I would hope it's at least like a 14 calling a 13, so you're doing like partial function application

all the way down. You're just peeling parameters off like one at a time. Does it not do that? Now I'm picturing a code pyramid. It was an unpleasant situation when I had to refactor a lot of code to get rid of that. I looked at it and I'm like,

uh, this looks wrong, but it's not wrong. All right, merge it. Yeah, it was a bad situation. Anyway, Ian, unpopular opinion. Got anything? It doesn't have to be related to Go or code. It is not related, but I do have something.

Okay. I do not think variables belong in paths of URLs, especially in APIs. I think we have query parameters that are built for this. So all these APIs that are slash one

to get something, I think that was a misstep. You're old school, huh? When I started, REST was already a thing. I think it was just a bad thing, you know, like... Clean URLs.

Great idea for WordPress. Bad idea for APIs. That was what I went to I went to Drupal days for sure I feel like Roy Fielding would be very happy with you right now I think he has a couple of rants about that about just

your URLs are opaque stop putting stuff in them that you need to parse there's other parts of the protocol anyway yes all good unpopular opinions. Or so we think. They end up being popular anyways. I'm not really sure how many of these are going to wind up being popular.

It's like an actual, I'm wondering what the poll results will say. We do go and we poll all of these on Twitter. So it should be good. I mean, my Sember one really, like that one gets unpopular when you start suggesting alternatives. It sounds fine when you're just like, the world's terrible.

Don't do that thing. And everyone agrees. And then you like try to suggest an alternative. And everybody's like, go f*** yourself. And so it's, it depends on how far you walk it out, I guess. I'm going to fall over.

We haven't had a bleep on the show in a while, so this is going to be a good one. Sorry, sorry. I should have inquired at the beginning. Oh, I mean, Peter got his one swear, so I feel like you also can get a couple swears in. Okay. All right. It's okay. We're a mostly family-friendly show. Okay. All right. Okay. We're,

we're a mostly family friendly show. Gotcha. Noted. I'll, I'll remember for next time. Yeah. Thank you, Ian and Sam for joining us today.

And thank you, Johnny, for being my cohost through this. And thank you to all the listeners out there for, uh, enduring through another episode about maintenance. There are more to come. Yes, indeed, there are more to come.

In fact, there's two more episodes to enjoy on this maintenance series from Chris. Again, episode 195 and episode 202. The links for those shows are in the show notes. Jared and I hope you enjoyed this awesome rebroadcast of GoTime 196 as much as we did. Building actually maintainable software is very hard, as you can tell from all the disagreements, all the hot takes, and those unpopular opinions. A big thanks to Chris Brando for kicking off this series, Johnny Borsico for being an awesome panelist, Ian Lopshire and Sam Boyer for being awesome guests, and you for listening to this show. Thank you so much.

If you enjoyed it, tell a friend. That helps us reach more people with our podcasts. Thank you once again to our friends and partners at Fastly and Fly.io. Also, big thanks to the Beatmaster in residence, Great Master Cylinder. Those beats are banging. If you haven't yet subscribed to GoTime at GoTime.fm, this show records live on Tuesdays at 3 p.m. U.S. Eastern. Hang and go for slack in the GoTime.fm channel or watch and listen live at YouTube.com slash changelog.

That's it. This show's done. Thanks again for tuning in. We'll see you on Monday. Thank you. Game on.