The Changelog: Software Development, Open Source - Good tech debt (Interview)

Episode Date: February 6, 2020

Jon Thornton (Engineering Manager at Squarespace) joined the show to talk about tech debt by way of his post to the Squarespace engineering blog titled "3 Kinds of Good Tech Debt". We talked through the concept of "good tech debt," how to leverage it, how to manage it, who's in charge of it, how it's similar to ways we leverage financial debt, and how Squarespace uses tech debt to drive product development.

Bandwidth for Changelog is provided by Fastly. Learn more at Fastly.com. We move fast and fix things here at Changelog because of Rollbar. Check them out at Rollbar.com. And we're hosted on Linode cloud servers. Head to Linode.com slash Changelog. Do not underestimate the power of the independent open cloud for developers. Yes, I'm talking about Linode.

Linode is our cloud of choice and it's the home of Ch for developers. Yes, I'm talking about Linode. Linode is our cloud of choice and it's the home of changelog.com. What we love most about Linode is their independence and their commitment to open cloud. Open cloud means being unencumbered by outside investment and maximizing value for the community,

not shareholders. And that's exactly what Linode represents. No vendor lock-in, open at every layer. If you want to learn more, head to linode.com slash open. Again, linode.com slash open. What's up? Welcome back, everyone. You are listening to the ChangeLog, a podcast featuring the hackers, the leaders, and the innovators in the world of software development. I'm Adam Stachowiak, Editor-in-Chief here at Change Log. On today's show, we're talking with John Thornton, Engineering Manager at Squarespace. We're talking about three types of good tech debt.

And last October, we featured a post from John on this various subject in our news feed and newsletter. By the way, head to changelog.com slash submit. If you've ever written something your fellow devs would find interesting, share it with us. We'd love to feature it for you. We talked through the concept of good tech debt, how to leverage it, how to manage it, who's in charge of it, how it's similar to ways we use financial debt in reaching our goals, and how Squarespace uses tech debt to drive product development. Back in October, we logged a post from John Thornton, three kinds of good tech debt. And those words don't often go together, good tech debt.

It's often like this dirty word, this thing that you're trying to avoid. Yet, John, here you are writing about this. Tell us about these, the tech debt that is good. How is it good? Yeah, thanks for having me, Adam. You know, the title was definitely designed to provoke a little bit because I think tech debt often gets thrown around and used to describe almost anything an engineer doesn't like about their code base. And really what I'm talking about in the post is more about not spending time fixing the things that don't matter. So really good tech debt is the stuff,

the bugs that you don't have to fix, the race conditions that you don't have to fix, you know, the things that you don't need to make perfect so that you can focus your time on the things that do have to be perfect. So in the post, you draw an analogy to debt. Well, I guess the analogy is already there, right? But in finance, you mentioned that debt can be problematic, but it's not automatically

a negative connotation. If you're in debt to the point that you can't get out from under it, maybe you have student loans and there's no way of declaring bankruptcy in this country. And you have crippling debt. That's really a problem. But there's also smart use of debt. And we do it all the time. It's just part of the way that we live life.

You point out a home mortgage. So draw that analog for folks. Of course, we know there's a debt analog. But you say with a home mortgage, you're not crying about it. You're happy about it. Right, right. Or a car loan, things like that. Things that you're trying to use some of this debt to get something you need now.

It's very intentional. Exactly, exactly. You're using debt as a tool. And it's so that you can get something you need now and you can pay for it later. And sometimes that can be a pretty reasonable thing to do with like a mortgage on a house or a car loan, or it can be a pretty dangerous thing to do if you, you know, spend a bunch of high interest credit cards and go into debt on that. That's debt that's going to compound very quickly and get out of control. So a big part of using debt safely is doing it in a way that you'll be able to keep it under control. So I was thinking about that analogy to a home

loan. And it's interesting, even the difference in debt between a home loan and a car loan, because a car loan is a depreciating asset, right? You drive it off the lot, it's worth less than before. You buy a house and assuming no market crashes, the general flow of the markets is upward. And the real estate market tends to flow upward because, well, they aren't making any more real estate. Although there are a few countries who are building islands and stuff. But aside from that, land is a constrained asset. And so your value on that home goes up over time. You have a secured asset with a home loan. You have a secured asset with a car loan as well. But it's often the car isn't worth as much as the loan itself. So I'm just curious, when it comes to software,

and we look at these different kind of loan types, these different kind of debts, you have kind of smart debts, dumb debts, secured debts maybe, like ones where there's an asset behind it. I'm curious how far the analog moves and if it just drops off right there, or could you have like a mortgage-style tech debt

where there's some sort of thing that's a value in trading for the value. So I'm not sure this is a perfect analogy, but I think knowledge is sort of the really valuable asset that you can get with tech debt, where you can use tech debt to get a feature or a user interface or something that you've built out into the world in front of users and get it into use so that you can start learning about how that thing you just built solves the problem or doesn't solve the problem. And depending on the complexity of the thing, there might be a lot of stuff to learn

over time. And so your knowledge is going to keep growing. You're never going to lose that knowledge. I guess an analogy around like a car or something that depreciates is maybe when you're trying to rush shipping something that maybe you're not going to learn from, but getting it out, you know, a couple months sooner maybe brings in more revenue for the company or allows you to turn off a different system a little bit sooner, save some money on expenses. So in that case, like you're taking a shortcut for a very defined benefit, but it's not one that's going to stick around forever. This is a perspective, though, simply from maybe a developer's perspective. And the reason why I say that is that I talked to this fellow named Travis Kimmel, who recently had his company acquired for many millions of dollars from Pluralsight. The company was called GitPrime.

And he came on Founders Talk and kind of opened up this sort of idea of tech debt and doing it with intentionality. And Jared, you mentioned sort of this, you know, this secure debt and that kind of thing. Well, the startup behind the tech debt sometimes is this, you know, the asset, so to speak, right? So something he had said on Founders Talk was basically that, you know that all startups take on technical debt and often, hopefully often, do it very intentionally. And if you can think of that as a parallel to financial debt like we have been doing here, it makes a lot of sense because 90% of startups fail. And so he said, so imagine someone could give you a loan that there's a 90% chance that you would never have to pay it back. So imagine you're building a startup and it's good to build this debt because, hey, you might get to exit three years later for billions, maybe millions, who knows.

And the debt doesn't even matter at that point because – and maybe I'm saying it doesn't matter potentially with air quotes because you've got more money to solve those problems and maybe potentially more people too. It matters who's buying that company, right? Yeah, well, yeah, sure. Sure, and it matters to the developers who are going to be addressing that tech debt. But if your company's getting bought for a billion dollars, I could put up with fixing a lot of tech debt for that. Right. It's easy to hire that technical. They pay it off. You pay off the debt. I think that's his point too, is just this different perspective,

like you're offering here in your post, is this different perspective on technical debt. And if you do it with intention and with specificity, like you do with financial debt, or like that's a great analogy for it, then there's a lot of gain. So look at it differently. Rather than saying it's a bad thing, how can we change this bad thing into something that we can leverage? Yeah, and the place where I've found it most useful

is for rearranging the order in which I'll build things. So one example I talk about in the post is I was working on building email campaigns at Squarespace, which is their email marketing platform. And when we started building it, the very first place we started was the email editor itself, the place where you built and styled your email. And that was because Squarespace is already well known for its website building tools. And so we knew we needed to get the editor experience like really, really good. But in order to like fully test that, you know, we wanted to get our co-workers using it to send real emails. And we didn't want to take all of the time to build like a real email

delivery infrastructure. Like, you know, we're now, we have a backend system that's capable of sending billions of emails, but it took us a very long time to build that. And we didn't want to have to wait just to get some feedback on the email editor. literally just a for loop that took a list of email addresses, looped through them, and one by one sent an email to each individual person, which was definitely good enough for testing in our 600-person company at the time. Obviously, it would have fallen over the minute a customer tried to use it to send a mailing list, but we weren't sending to real mailing lists. That wasn't the point. Exactly. So, you know, we were very deliberate when we built this janky for loop solution. You know, we spent about a week on

it knowing that it was going to be throwaway work. And another benefit we realized is because we were thinking about it as throwaway work, it changed the way we built it. We were a little bit cleaner with our interfaces because we knew that the code that interacted with this, we wanted to keep the editing code around, but we wanted to be able to throw away all the delivery code. So it kind of forced us to keep things very decoupled. That's interesting, maybe unintuitive to me, because I would expect if you're thinking

I'm throwing this away anyways, it would make you write it with more abandon, less care. But I guess what you mean at the interface is the places where that prototype or that throwaway system actually attached into your existing software, where the modules connect. You had to be more serious about that aspect of it because you knew you were going to be swapping it. Is that what you're saying? Yeah, exactly. Exactly. So everything behind the interface could be pretty bad. Terrible. We still did code reviews for it, but we all kind of laughed and looking forward to deleting this code soon. Right. Yeah. You know, so we didn't, we didn't spend time going back and forth trying to make the code inside the implementation perfect, which was another benefit is we didn't spend a lot of time talking about code that we knew was going to be deleted soon anyway.

Which is interesting to look at it from a habits perspective or a goal-setting perspective. The reward to you and the team was, can't wait to delete this code. And so deleting the code would have been a milestone. It would have been success. It would have been an outcome goal where you hit some sort of target and the team was, can't wait to delete this code, right? And so deleting the code would have been a milestone. It would have been success, right? It would have been an outcome goal where you hit some sort of target and it's like the day we get to delete this code is the day we hit our mission. We achieved our goal. We're at the next level, right?

And that's a good thing. That helps everybody sort of like stay motivated. Yeah, that's right. And it also compartmentalizes what's built and what's not built, you know. And so it was really helpful to have areas of the code base where it was known that this is scaffolding code. This is going to get deleted later. Don't worry about, you know, fixing small bugs and things in here.

But that also meant that there were other areas of the code base that were declared, you know, quote-unquote done or places where we would be polishing and perfecting things. And then I think ties in a little bit with, like, the broken windows philosophy in that, you know, when you have defects scattered across your code base, it can be a little bit difficult to prioritize which things you're fixing. And by being very deliberate about which parts of your code base, it can be a little bit difficult to prioritize which things you're fixing. And by being very deliberate about which parts of our code base were finished and ready for polishing versus which ones were still, you know, not even building the real version on

yet or which ones were in active development, really help us focus our efforts and also kind of avoid development regressions too, because, you know, we were putting in tests as we went along with this, but test coverage would obviously come in when we were building the real version. We wouldn't spend time testing the scaffolding. And just by separating each part of the system into components and being deliberate about like what stage of the development it was at, we were able to put the right effort into the right piece at the right time. What about team size? What's your team size? Does it get difficult to manage or determine or declare

what tech debt might be in a feature set? Does it happen in a meeting, demarcated, and code somehow with code comments? How do you even determine that and how do you communicate what is and is not tech debt to the team? That would tend to happen through the team's planning meetings. So we had a pretty typical two-week agile-ish sprint process. And so every two weeks we'd have a new set of goals for the team. And if we were going to take on a big project such as building the system that would send emails to mailing lists, before we got into working on the actual code implementation of that, one or two members of the team would go and write a design document that kind of laid out in broad strokes

the components of the system, how they're going to interact together, and it would sort of be the blueprint for how they're going to interact together, and it would sort of be the blueprint for how we were going to code this thing. And so there was always a bit of a lead time before we were splitting up tasks for everybody to work on. There would be a couple, one or two engineers doing research on this thing, and they would be talking to individual engineers to collect ideas and get feedback and so there was always like great awareness within the team of sort of like where the active development was which parts were finished which parts were throwaway work for later

and what was the total time span of the of this particular project are Are we talking a month, six months? It was about 15 months of development work from the first line of code until we launched it to external users. What about until the point where you said, okay, it's time to go back and pay down the debt? The prototype or the idea validation that you're referring to, was that a much shorter time span? Yeah, I'd say the email sending prototype,

that lasted maybe four months. And some of the other, you know, like we had some scaffolding for analytics and it turned out that that actually worked a lot better than we thought it was going to. So that one stuck around for 18 months. So you shipped it? You know, we only recently was going to. So that one stuck around for 18 months. So you shipped it. You know, we only recently addressed that one.

So that's the other nice thing is that you can build these things and plan to throw them away. And sometimes things work out better than you thought they would. Yeah. And your seemingly not-so-great code ends up performing pretty well. That is a nice scenario where you're like, this is a prototype, but well, it turns out it's pretty good, so let's ship it. Yep.

Can't complain about that. So if we were to categorize, I think the hard part is deciding when it's smart and when it's dumb. And you mentioned, or maybe misguided is a better term, you mentioned that it's all about intentionality. Like if you're intentional with it, then that's what's important. And I think that's true, but you can also be intentional, but miss the mark, right? You can misfire and make a bad call. And so there's got to be some heuristics and decision-making like times when taking on debt is smart and times

with paying down debt is smart. And it sounds like you've identified at least one which is kind of idea validation or maybe it's the idea of it's not even idea, it was that it allowed you to build a thing before you could have otherwise if you had gone in the other direction. So are there other ways that you can say in this circumstance tech debt is

wise and we can kind of use those as waypoints when we're making decisions? Well I think one thing that helps is even when you're building your tech debt implementation of it, you probably should have some idea of what the quote unquote real implementation is going to look like. So that as you're building the scaffolding version, you have some sense of what's going to go in place of it. And you're not just leaving it as a complete question that you're not sure you're going to be able to figure out later. You know, another thing to keep in mind is,

is the tech deck going to grow? Typically, the places where this works best is where you can take all the bad stuff and just hide it behind an interface and not look at it until you're going to replace that component. You know, if you're building something and some of the more tech-daddy aspects of what you've built are leaking out into other parts of the system that you're not planning to replace later, that's a big sign that you're doing something that's going to compound over time. And it's actually going to be a lot more work if you leave it in there. So isolation is important in that regard. Yeah, yeah. I mean, a lot of it comes back to being able to put this stuff behind an interface, choosing the right

interface, you know, so I definitely don't think building things with tech debt is easier than building things without tech debt. It's more a matter of just kind of like focusing your efforts in specific places. And again, the analogy of buying a house comes back here because why do you take out a loan to buy a house? Because, well, you could afford a house over 30 years, but you want to live in that house because, well, you could afford a house over 30 years, but you want to live in

that house today, right? And so you're willing to pay more for the advantage of living in that house today. And in the case of these features, you're wanting that feature now, you could build it very slowly and have it 30 years from now, right? But the market opportunity is today. So you take on some debt in order to afford yourself that luxury or that advantage. Yeah, and I think the house analogy can be extended even further because sometimes you can end up building things that you didn't actually need. And those things can have maintenance or operation costs that are much bigger than a different implementation might have been.

And you can almost compare that to buying too much house, more house than you can afford. And then you're stuck with all of this maintenance overhead. A lot of the reasons too is this data gathering process, like you're doing it to learn. And often the data that you can sort of gather from this debt informs your future the same way that a house will provide you shelter for the time being. It's giving you an opportunity to sort of level up other areas of your life while you take on this portion of debt with intention. How often do you think about internal tooling? I'm talking about the back office apps, the tool the customer service team uses to access your databases, the S3 uploader you built last year for the marketing team, that quick Firebase admin panel that lets you monitor key KPIs, and maybe even the tool that your

data science team had together so they could provide custom ad spend insights literally every line of business relies upon internal tooling but if I'm being honest I don't know many engineers out there who enjoy building internal tools let alone getting them excited about maintaining or even supporting them and this is where retail comes in companies like DoorDash, Brex, Plaid, and even Amazon, they use Retool to build internal tooling super fast. The idea is that almost all internal tools look the same. They're made of tables, dropdowns, buttons, text inputs,

and Retool gives you a point, click, drag and drop interface that makes it super simple to build these types of interfaces in hours, not days. Retool connects to any database or API for example to pull data from Postgres just write a sequel query and drag and drop a table onto the canvas and if you want to search across those fields add a search input bar and update your query save it share it it's too easy retool is built by engineers, explicitly for engineers. And for those concerned about data security, Retool can even be set up on premise

in about 15 minutes using Docker, Kubernetes, or Heroku. Learn more and try it free at retool.com slash changelog. Again, retool.com slash changelog. I would imagine that not all tech debt is equal or the same, but given your examples here, John, that the different kind of debt you've taken on for the features that we've talked about, when do you decide to start planning to pay that down? And maybe give me some scenarios that's not just this scenario, if you can imagine other times you've had, but what are some telltale signs for when you should start to pay down that debt? Yeah. You know, I think one sign is when you find yourself spending more time kind of maintaining the debt than it would take to fix it, that's a pretty

obvious case where you'd be better off just paying that debt down. But I guess it depends a lot on your requirements. And so, you know, one example is you can hard code things as a way of using tech debt where, you know, rather than building a system to manage adding and removing items from a list and uis to interact with that stuff you can just have a list of things in your code that can work really well as long as you don't need to make updates any faster than it takes to normally make deploys to your code But if you find yourself having to make urgent updates to this list pretty frequently, you're going to end up having a lot of fire drills.

And that can be another sign that the functionality of the tech debt solution you built just isn't matching the requirements you have anymore. So I think it comes down to paying attention to your requirements, paying attention to where your team is spending your time. It seems to me that a lot of people get themselves into untenable circumstances because they are always going to pay down their debts tomorrow.

The day of reckoning never actually comes because systemically, culturally, it's always pushing forward and the debt is ostensibly being managed but it's actually just being pushed underneath the covers and eventually the market's going to crash or whatever. That seems like that happens a lot. That's when the big rewrite begins. A big rewrite, a second system, or call the contractors.

There's entire firms. I used to do rescue projects quite a bit when I was freelancing. So there's a lot of people that are going to, the consultants are going to come in and help you save this system because you never actually pulled the trigger on paying down that debt. You just kept accumulating and accumulating. That's very normal because in business, I mean, forward is the way businesses move. And in the competitive marketplace, it's hard to slow down and to do things the right way

and to actually pay your debts down. And so maybe you can talk to Squarespace's culture around this and how you guys manage internally as a leader of a team or inside the teams. Do you allocate time for refactoring and paying down? Talk about culturally how it works. Yeah, I mean, that's a tricky problem, And you're right, I think a universal one, because customers aren't asking you to fix tech debt, customers are asking for features.

Right. You know, and that's what the business wants to be working on. So there's a couple techniques we use at Squarespace to manage this. First is that product management needs to be a partner in understanding what sort of tech debt we have. And product management needs to understand what the benefits of addressing tech debt are so that it's not just engineering versus product management competing for priorities. You know, in some ways we've done that is when we've used tech debt intentionally to accelerate getting a certain feature out there, we'll talk a lot about the tech debt. We'll almost like brag about how bad and how simple

the scaffolding component is just to make sure that all of the stakeholders for the project understand that we took on debt here and we're going to have to pay that debt back later. So we put a lot of effort into getting that into the planning process when the debt is intentional. But obviously not all debt is going to be intentional, so that doesn't work for everything. So another tactic we have on some of the teams I work with at Squarespace are we'll designate one day out of the sprint to work on tech debt. And I think Google is famous for their 20% time. I guess it's 10% time. It's one day out of a 10-day sprint. But we tend to focus that day on developers fixing the things

that bother them about the projects they're working on. So that's the day where you would work on making the build faster. And, you know, making the build faster is going to make everything faster because every time you make a change, you have to run a build. Or if there is some process that's manually operated by the engineers, such as updating a hard-coded list or something, that's the day that somebody who's bothered by that could work on fixing it. And a key aspect of this day is that the product manager, the team tech lead,

aren't the ones dictating what gets worked on. It's each individual engineer saying, okay, this is the thing that bothers me about the development experience of this project. This is what I'm going to fix today. You know, we have some rules around that to keep things from going off the rails. So things that get worked on in that day ideally should get completed in a day. We don't want to, like, pick off a three-month data migration without talking about it with the team. But if somebody wants to tighten up the Lint rules

or improve our UI testing suite, we want to have as little friction as possible for that. And we found that one day out of the sprint has been the right ratio for addressing that stuff. And really what it does is it helps us avoid the prioritization challenge of, you know, how do you compare the benefit of making your UI tests 10% faster versus, you know,

getting half of a user-facing feature built? And it's really difficult to prioritize those things when you're planning a sprint. So by just kind of setting aside some time, we just sidestep that prioritization process entirely. Once you've had a chance to do some of this stuff, how do you communicate those, I guess, upgrades or debt paydowns back to the team?

How do you update documentation? How do you sort of say, hey, we've got this Lint is now in a better situation or faster or has these new opportunities? How do you communicate those things? I guess through your processes so much like you said agile-ish before. So I'm assuming you kind of mean agile, but maybe it's agile-ish. I don't know. Well, I just mean that I think everybody kind of has their own flavor of Agile that works for them.

Right. For the team I work with, each they've set up, the team uses Slack to communicate, and they've set up their code review tools so that anytime somebody puts a PR for the project, that gets posted to Slack. And so there's generally a lot of visibility on what sorts of changes are going into the code base. You know, and honestly, if it's something like making the build faster or adding a lint rule, either you're not going to notice those things because they don't impact you. Or, you know, maybe one day you'll say, huh,

that build finished in six minutes instead of seven. Or, you know, you'll be coding and that new lint rule will find you when you interact with it. So we actually haven't had to be too intentional about communicating the little improvements. Because those have tended to be the things that get talked about naturally. I think the tech debt that's been the most challenging to address has been the unintentional tech debt. And there it's been kind of a challenge of learning how to articulate the benefits of paying down that debt and then figuring out how to come up with

a way to prioritize those benefits versus spending time building features or new products or other things like that. Yeah, it's an unintentional tech debt if we go back to the whole financial model. It's like accidentally taking out a loan or accidentally spending too much money. What's the analog to that? I think it's kind of like buying something with your credit card and saying, oh yeah, yeah, I'm going to pay that bill at the end of the month.

And then the end of the month comes around and it's like, oh, but actually I want to buy this other thing too. And then you carry a balance, and then that balance, the interest starts compounding. It's even worse in software, though, because you almost can't just accidentally buy things. You can just be so off or make such a bad decision that you

didn't decide to like go buy that outfit but you bought it because you didn't realize that this was just a really bad architecture that you designed and now you have to pay it down even though you didn't want to buy it like you were just trying to do the right thing but you did the wrong thing yeah so uh it can definitely bite you this is why why I'm, if you remember, if anybody listened to the Back to the Agile Basics episode we did last year, this is why I'm really down on estimations because I feel like it's like, do you want me to lie to you in big picture or granular? You want granular lies or vague lies?

That's kind of the way I look at it. And that makes me say this next thing very, well, let me just throw this out there. So I've been thinking as we talk, the nice thing about financial debt is that every month your credit card or your bank whoever holds your mortgage sends you an update and says this is how much you owe right and you can quantify that and you can look at it and you can say okay i'm doing poorly or i'm doing really well yeah or hey i got some extra cash i'm gonna pay some principal off of that loan. I'm wondering if it's worth, I think it would be valuable

to have some sort of an analog to that in software. Or a way to quantify the current software stack's debt. Where do we stand? Maybe some code quality things could help along that way. I feel like we move into the place of estimations. You have to go ask the developer, how much tech debt do you have? And they have to give you like a point system and it could be ridiculous. But I think there would be value, at least from a high level strategic place of saying like, no, we can't actually afford to continue to take on debt

because here's where we stand. We're at, we already owe a million dollars. We can't take out another loan right now. Right. Curious what you think about that your debt to ratio your your what your balance to your debt what you you have a certain limit i suppose your ratio to limit ratio is off yeah and i mean the point would be like visit adding that visibility that currently john you and your team like you said you just kind of like have to continually orally like report it like by the way remember this is not right a real thing we forget too we forget often and that's what i'm thinking like john how do you kind of like make a log i guess kind of what jared is saying almost a ledger i wasn't referring to like an actual kind of like a bitcoin ledger you know that kind of thing was just more like how do you maintain and know this debt over time

so that you can not forget it once you've shipped the feature and the rewards have been paid out and everybody's deleted the code that was yucky or nasty and you've moved on and you're onto the new feature set that the customers are demanding how do you keep this log of debt to eventually squash or just goes there looming i think it becomes part of your your project documentation you know so most projects are going to have, ideally, a readme that tells you how to build the project and run the application in your laptop and things like that. And hopefully you've got some comments in your code.

And I think it can work in either places. But what's key is that you're writing down somewhere in a place that a developer is going to find it when they're looking around the code base you're writing down what you expect the limitations of what you've built are you know so as an example of that with the email analytics that that we built for the first pass we knew what we were building wasn't scalable. And so we deliberately ran some load tests to understand when this debt was going to come due, you know, and we've been kind of monitoring how close we are to those limits all along. And so that's made it pretty easy because

having these numbers to watch kind of, you can just set an alert or an alarm or something like that. and then the computer is going to come and remind you to do it. Another way to do it is just with a code comment on the interface between the debt part and the non-debt part, kind of explaining the rationale behind what was built behind that interface and explaining what was intended to come after it. But as for keeping track of the unintentional tech debt, that feels like a much tougher one. I feel like if I could come up with a tool or a system to solve that,

I'd probably go start a company for it. Yeah, exactly. So one example you put in the post, which is a form of taking on tech debt, is not fixing all the edge cases. So basically you're staying mostly on the happy path. Like we're just going to address the happy path right now, we realize there's these 10 different things that could go wrong. And production grade software would handle all those cases. But today, we're just doing the happy path. And so a form of

documentation on that kind of debt is documenting all the things that you're not doing, right? Like here's the 10 ways that this could go wrong. I'm not going to code them up right now, but I'm at least thinking about it with enough intention that I'm intentionally skipping these 10 things. And it's worth me putting a number list right here, one through 10. Maybe there's more I'm not thinking of, but here are the edge cases that we're explicitly not solving today. At least then when you return to that area where someone says, go add a feature,

and then you realize, oh no, this is in production. This is only happy path code, right? Who put this in production? You can go back and say, well, before we add that feature, I have to do these nine other things that we skipped. Yeah, yeah. And you can also build your code in a way that it will tell you when it deviates from the happy path. And that way you can even sidestep the question of, you know, is this actually going to happen in production or not? Yeah. Ship it and let production tell you if it'll happen or not.

Yeah. Then you'll have error reports. Exactly. User tooling. Which can turn into tickets, which can turn into stories, right? Granted, if that's something that really does impact user experience, you might not want to take that approach. Or if it's something that might annoy your users,

you're going to need to be ready to respond to it really fast. So I think it's important to understand what the effect on the user would be if any of those unhappy path things happen. You bring up a good point with QA support. What are the different departments, I suppose, is probably an easier word to say. You mentioned project management before, product management, engineering.

What are the different stakeholders that care about tech debt that I guess need to collaborate and communicate through? As you just mentioned with, if you ship this and you get some errors and it diminishes the user experience to a point that users get upset, QA might want to know that,

hey, just let us give some slack on this particular thing or you might want to give support a heads up, like, hey, expect some emails about this and see me if you get anybody. Yeah, definitely. Customer operations or customer support is a huge stakeholder for that. Even to the level of we will build things in order to reduce

volume of incoming support requests. Because we have a large, really well-trained support team here at Squarespace, but we don't want to build things that put a bigger burden on them. So if we have a feature that we think is going to result in support requests, we'll probably end up changing that. So your section on hard coding and the topic you brought up earlier about hard coding made me think of something that I was doing just the other day. Kind of a style of coding that I've taken up in the small

that I like to share and get your thoughts on. So you work in a large company with large teams, of course, and we're a small company with a team of, well, you're looking at us. You're looking at our team. And so we carry many hats, and there's tons of indie devs in this area, whether you're freelance or startup or on your own, where a lot of the decision-making process isn't even, can we move faster? It is for us, it's like, what can we actually do?

Because I have these other things on my plate and software development is a part of what I do. It's not like there's a team that's right in our platform. It's like we carve out time. And so prioritization and what's important is key. something I've been doing the last few weeks which I think has been fruitful is thinking about things in terms of minimal viable feature set which is the same exact concept as an MVP only just in the small so I ask myself, here's a large goal we have

what's the minimum viable thing that I can do today to push that forward and be usable today or maybe tomorrow or maybe on Wednesday whatever sometimes week right yeah like what's viable what can I do and this requires a lot of sidestepping like this is not the full feature this is a step on that path and so it kind of fits in with my baby steps philosophy but it requires a lot of hard coding and so for an example of that is transcript notifications, which is something that I'm building right now. As we told you, John, each show gets transcribed.

Those are in Markdown and they're put on GitHub and they're synced into our website and they show up on the show page. Well, the transcripts don't come out until the show comes out. So it's actually, in the case of the change log, it's usually one to two days later. In the case of the change log, it's usually one to two days later. In the case of our other shows, it could be three, five days later.

That's just the way that our workflow works. We're not planning on changing that. We don't want to hold back the episodes so that the transcript is ready. What would be nice is knowing when the transcript is ready. We have features like email me when founders talk ships an episode. That's a feature that exists inside our platform. And it would be nice to have a button right there for a show that doesn't have a transcript.

You'll go looking for the transcript because a lot of people would rather read than listen, or they don't have the time, to have a button that says, yeah, let me know when the transcript is ready, right? So that's the feature that we're building, email notifications around transcripts. But that requires different moving parts, right? So that's the feature that we're building, email notifications around transcripts. But that requires different moving parts, right?

The actual knowledge inside the system of when the transcript has been attached to the episode, the ability to email people, the ability for them to subscribe to those email notifications, configure them, unsubscribe, et cetera, et cetera. There's probably five things. What's a minimum viable feature like what

can i build today that would be useful today and just to like move that along not trying to get all the way there but how can we move it forward and that i don't know if that's a technical debt but um it requires some hard coding some things so So, for example, on that particular feature, I thought, well, if I can just do it for myself, then that'll be forward. Like, that means I have to have the ability to know that the transcript's been uploaded. I have to have the ability to email at least one person, right? But I can sidestep the UI for the notification management.

I can sidestep the unsubscribe routines, all that kind of stuff. Oh, man, yeah. You could work so much tech debt into shipping that feature because, you know, it sounds like there needs to be a component that tells your application when the transcript is ready, you need some way to collect a recipient address from the user, you need some way to send the email. Right, exactly. You know, so I could see building something that,

you know, version one just has your system find out when the transcript is ready, and maybe it just outputs a log line to say that it happened. You know, maybe the first version only sends emails to you because your address is hard-coded in there, and that saved you from having to build the user interface to collect the recipient address. You know, as you're describing it, it all sounds like very reasonable, you know, MVP-driven development.

So I don't think, you know, good technical debt is that radical of an approach. It's just kind of putting labels to things that I think a lot of people are already doing in their software practice. Absolutely. I should say, I just got an email. The transcript for becoming an accidental founder is ready. So I mean, right now in our code base, what I did was what you said I did. I skipped step one. So combine those two. I had enough time to build the thing that notices, right, that the transcript has been added. And then hard code, like you can go check, it's open source. It says email,

and the string is jared at change.com. It's just going to email me when there's a new transcript. But it's already in a for loop, so I can go ahead and get to the next step pretty easily, add multiple email addresses. So I'm looking forward a little bit,

but hey, it's a feature that I've always wanted to know when the transcript's ready, and now I do. So baby steps. I guess you can learn the value of the feature, too, to you individually. Right. Like, maybe I don't care.

I'm like, ah, another one of these? Why are we building this? Right. Or you can see what the lag is even. It could be, like you mentioned before, John, some of this debt intentionality is sort of data collection, being scientific about it. Like, do we need this feature? How useful would this feature be? What is the user experience criteria around this as it plays out? So in this case, you know, a precursor to this might've been, you know, maybe just some smoke

and mirrors fake UI to say, do people actually care when they go to where the transcript typically is in the UI, find that it's not there, get upset. If we could show them some UI and say, tell me, and then essentially let that be this sort of like collection of interested parties, opportunity, essentially, you know, that might've been a precursor. Sure. I can see some benefit in it, but I'm just talking through hypothetical of like how you can get, you know, essentially gather data to determine, you know, if you should, and then if you should, how is it actually working out? Yeah. I've, I've heard that. So it, it sounded if you should, how is it actually working out? Yeah, I've heard that. So it sounded like you were sort of flipping it around and saying, what if we built the UI first?

And I've heard that referred to as the Wizard of Oz pattern where there's a man behind the curtain. Right, no tension. Well, I heard it as smoke and mirrors. That's what the Wizard of Oz is. You go. Well, I heard it as smoke and mirrors. Mm-hmm. Yeah, you know? Like a smoke feature. That's what the Wizard of Oz is. I know. You go in the room, he's got smoke and mirrors.

Smoke and mirrors, that's right. I love it. It all comes together. Well, you know, it always just kind of makes me laugh that behind that feature is a person sending emails to the people that signed up. It's not a computer. Oh, I love it. And it's sort of the lean startup methodology

as well, right? Like build a landing page, prove the idea, and then fake it until you make it. And not in the way of like you're lying to people. Like you're literally

faking the feature because you're doing it manually. And then we slap on names like it's a concierge sign-up process. Like, oh, that sounds nice. I think I'll have that. It's very hand-holdy.

Yes. I love it. Yeah. Start with real intelligence before the artificial intelligence. Exactly. Hey, what's up? Adam Stachowiak here. I got a question for you. Have you heard of our newest podcast yet called Brain Science? I'm not going to be offended if you haven't.

It's okay. But here's how you find out more about the show. Go to changelog.com slash brain science. We have 10 episodes on the podcast feed. So have fun, go back and listen to all 10 and subscribe to get more. I actually get to co-host this podcast with a doctor. That's what makes this podcast legitimate. If it was just me, it would not be as cool, but I get to team up with Muriel Reese. She is a practicing clinical psychologist, and it's so much fun digging into deep subjects around being human. Here's a preview of episode number 10. We're talking about shame. We haven't talked about this as it's relevant to creativity. And if you can see that when we are trying to navigate shame, this sense of

inadequacy, do you think you're going to be more apt to be creative or less apt to be creative? I would guess less apt because I'm trying to focus on fight, flight, or freeze in those moments, and I've got no time to be creative. I've got to be the most necessary Adam possible to get through, right, rather than be creative. Yeah, so let me tell you the dynamic. Adam, I need you to be remarkably creative so you can come up with the best, most user-friendly way for this to work, except you suck, you didn't do it enough,

and you need to do better. Don't ever tell me that again, Mariel. That is not nice. But I can understand how in that kind of moment. So if you're leading teams out there, don't lead with shame, okay? Well, it's really recognizing the way that you have to, right? If you can see, shift your mind into seeing this through a way of management, like I need to manage how I interface with other people, especially around creative endeavors. Yeah. That I need to be deliberate about identifying what they're doing well and even saying like create clarity, like what you want them to approve upon. All right. To keep listening, head to changelog.com slash brain science slash 10.

And that will take you to the episode titled Shame on You, where we examine the hustle of not enough, how shame relates to imposter syndrome, our fight, flight, or freeze lizard brain response to threats, and so much more. Again, changelog.com slash brainscience slash 10 or search for Brain Science in your favorite podcast app and subscribe. We'd love to have you as a listener. So whether the technical debt was intentional or accidental, eventually you've got to pay it off. And it sounds like y'all at Squarespace had some unintentional debt

that you have been paying down lately. You want to tell us that story? Yeah, that's right. So like many successful businesses, the software that powers the business is a monolith. We were talking earlier about when you're building something, you got to find out that people care before you really invest a ton in that thing you're building. And so often it doesn't make sense to start with a perfectly scalable,

you know, microservice distributed system architecture from day one. More likely you're starting with a monolithic application and, you know, eventually that gets extremely popular and then you have to scale out your system. And so that's where Squarespace is at. And so we're in the process of breaking up our monolith into separate services. We've been working on that for a couple of years. And one easy trap to fall into when you're doing that is, you know, a lot of thought will go into breaking off pieces of business logic into a separate service. But it can be easy to forget that there's also data that needs to be moved. And if your project ends up going slower than

expected and you're getting pressure from other stakeholders to switch to working on other things, you might end up building a microservice that's separate from your monolith, but leaving your data in the main database, which is a thing that happened at Squarespace. And the effect of that was what we've ended up calling the distributed monolith, where, you know, we have separate code bases for different services, but some of these services share the same database. And that database ends up becoming an unintentional coupling point for the services so that you've now broken up your system into five or ten separate services,

but in order to actually run any of those services, you need to run the other nine too. And you haven't actually made your development faster you've just made it more complicated and so a team i work with recently inherited one of these services from a different team and due to a somewhat we we sort of made lemonade out of lemons where uh the the team's product manager had had moved on to doing other things and the team was without a product manager for a little bit and then had kind of a caretaker product manager.

And we said, huh, well, this is actually a good time to take on some technical work. And what was even better was that the service that needed this data migration was going to be a key foundational aspect for future projects that product management was really excited about. And so we were able to kind of like sell this data migration, this very technical project, to product management and the rest of the business stakeholders by saying, yeah, we're going to make the development experience better.

But what we're really doing is we're laying a strong foundation for this big project that we're all excited about starting in six months. And it sort of, it solved a couple problems. The big one for the engineers is that it fully decoupled this service from our monolith so that it really was a microservice. And you really could just run it on its own. Or if you weren't actually doing development on that service at that time, our Squarespace's local dev setup is designed to just fall back to staging for any of the services that you're not doing development on. So for many teams that built things that

depended on this service, their local development setup got a lot better. And another benefit was that we were able to adjust the data model as we were doing this migration. And that's where we were building the functionality that product management was really excited about for the next project. And kind of the last key piece of making this project a success is we developed a system for tracking progress. Basically, we were migrating data access objects pretty methodically. We were migrating API calls pretty methodically. So there was a pretty easy-to-interpret burndown of the whole project. And that made it really easy to communicate progress, made it really easy to

get trust from other parts of the business. And we found that when we had that trust, it was really easy to get things done because you didn't have to spend a lot of time explaining why you were doing this or what the outcomes were going to be. Get buy-in, basically. So if you have unintentional debt looming over you, the question you might want to ask is, who would care if this debt was paid down? Yeah. Is that kind of what you did? And how can you bundle it in an attractive way to sell it, right? Because there's some key factors you mentioned.

They had buy-in early on. You had their trust, so you didn't have to keep going back to them explaining what you're doing and why you're doing it. You could freely solve this technical debt in ways you felt were necessary to lay this foundation. So getting by it was – Oh, it was huge. Yeah. What's interesting there though is the leadership part of that is the ability to see that. You're going to be at this Lead Dev Conference later this year giving a talk of similar name to this podcast slash your original article we're kind of referencing here but how do you help engineering managers bake that into who they are teams bake that into they are to see that cross that that

visibility and other teams to to kind of see the future of where product management is going how do you how do you do that just pay attention be a better employee be a better manager be a better you know individual contributor how do you get that kind of insight? How do you get that personally? Gosh. Pay attention? Making lots of mistakes and trying to learn from them. Okay.

You know, there was something that I heard last year, and I'm blanking now on where I heard it, but it was the idea that instead of talking about paying down technical debt, instead reframe it as making technical investments. So instead of talking about making the bad thing go away, you're talking about what are the good things that are going to happen after you've done this technical work. And I found talking about it that way, it's a lot easier to get product management or the other parts of the business

to understand why it is you want to do this or having to come up with the explanation of why this is a technical investment can maybe sometimes tell you, the developer, that you're pursuing the wrong problem. If it's hard to explain concretely what the benefit is of paying down tech debt

aside from the bad stuff goes away, maybe you don't need to pay down that tech debt. That's a really good point. What kind of environment do you pitch this to? Do you call it one-on-one? Do you do it in all hands? Do you do it in your Agile-less workflows that are say, your stand-ups, if that's what you're doing?

When's a good time to broach the subject? Usually when – so at Squarespace, we'll do our planning on a quarterly cycle. Every four times a year, we'll get together and we'll write down what each team is planning to do for the next three months. And we're not just coming up with that on the spot, obviously. We're thinking about it on an ongoing basis. But these quarterly roadmaps are usually a good way to understand where all of the teams are trying to get to. And so they're a really good guide for ways that you can kind of hitch your project on someone else's existing project, which is usually the easiest way to get something greenlit is find something else that's already greenlit and say, hey, my thing is going to help with that thing.

Like a parasite or a tagalong. I was trying to find a good word. Parasite's obviously not a good word for that. Like a leech. Tag along is a better example for that. But it's definitely parasitic, but with good intention, not malintention. Symbiotic, maybe. Okay, symbiotic.

Symbiotic relationship. Yeah, everybody loves a good symbiotic. You know, I think because the benefits can go both ways in that. In that, you know, ideally your tech debt project is doing something that does meaningfully move this other project forward, but you're also getting buy-in for fixing the technical issue that you want to fix. Yeah. That's good advice in the large. I have a little bit of advice for this problem in the small, working with individual clients as I have for many years, people ask, how do I get my clients to pay or to buy into testing and refactoring and these things that we know are good practices but aren't working on a new feature? And my answer to always that has been, well, I don't tell them. And it's not that I'm trying to hide it. It's like, nope, this is

how I write software. I write tests, I refactor as I go and that's just part of what you get when you hire me it's not like it's a line item on my estimates or my bids it's just part of how you write software if they aren't interested in that style of software then they can go and hire somebody else

to write it for the fifth time for less and accumulate debt total cost of ownership is an important aspect of that too. That's interesting. Cool, John. When is this talk you're giving and where is it? Tell us about it.

Yeah, so I'm going to be speaking at the Lead Developer Conference in New York City. And the conference is April 7th and 8th. And I think I'm going to be speaking on the 7th. And it's a great conference. I've been as an attendee before. I've never spoken

there but anybody in a technical leadership role or people who aspire to be a technical leader, it's a great conference. Two day conference in New York City in a decent month. What's the weather like in April in New York City? Is it beautiful? Is it amazing? You never know what's going to happen.

You can get any of the four seasons in April in New York. Take your chances. Sounds like a really good conference. Bring shorts and a parka. There you go. Well, Suze Hinton will be there, Jared. Nice. Brian Louse will be there. These are two of the

several people on the homepage that I know. And now you, John, you'll be there. So that's awesome. NewYork2020.thedeveloper.com is the URL. We'll add that to the show notes and deep link to your topic. John, thank you so much for sharing your wisdom here. More importantly, thank you for writing it down. I think it's an important process to future educating future Johns, future Jared, future Adams, or the listening audience. Write it down. I think it's an important process to future educating future Johns, future

Jareds, future Adams, or the listening audience is like, write it down. So I definitely want to encourage you to write more of your wisdom down, John, and we'll be glad to feature it and or have you back on the show and talk about it. So thank you. Awesome. All right. Well, Adam, Jared, thanks so much for having me.

All right. Thank you for tuning in to The Change Log. If you aren't subscribed to our weekly newsletter, you're missing out on what's moving and shaking in software and why it's important. Hate email newsletters? Fun fact. KillTheNewsletter.com was created by someone just like you who wanted ChangeLog Weekly so bad

they wrote a program to subscribe on their behalf. And of course, it's 100% free. Fight your FOMO at ChangeLog.com slash weekly. When we need music, we summon the Beat Freak, Breakmaster Cylinder. Our sponsors are awesome. Support them, they support us. We've got Fastly on bandwidth,

Linode on hosting, and Rollbar on bugs. Thanks again for listening. We'll talk to you next time. Thank you. Субтитры добавил DimaTorzok