Two's Complement - Source Control and Conway's Law

Episode Date: May 12, 2021

Matt and Ben compare monorepo vs multi-repo layouts, explain Conways Law, talk about what a 'team' is, and what Visual Source Safe isn't. Ben defines how big a service should be. Matt recalls a brief interlude with Clojure.

I'm Matt Godbolt. And I'm Ben Rady. And this is Two's Compliment, a programming podcast. Hey, Ben. Hi, Matt. So, normally we have a plan when we do these podcasts. That is usually our plan, to have a plan. We have a plan to have a plan when we do these podcasts. That is usually our plan, to have a plan. We have a plan to have a plan. And in this time, we spent so long discussing

what it is we should talk about that I said, why don't we just hit record and go with it? Yep. And that's what we're doing. Right. You know, we're doing it live. So what are we going to talk about? It's a very good, very good question. We could go ahead for a second and talk about how we decide what we're going to talk about? It's a very good question. We could go ahead for a second and talk

about how we decide what we're going to talk about. That's a brilliant idea. Let's do that. Because you and I are both staring at a shared Google Doc right now where we have just as over the last few weeks, we've been gathering thoughts on interesting topics, often over

a cup of tea that you and I will have like on an uh an afternoon and then we're like we get 20 or 30 minutes into our coffee date and then go damn i really wish that we were recording this because this is gold this kind of conversation is exactly the kind of thing we want to talk about in the podcast and so we write it down yeah and then when it comes to the cold light of day we don't actually go oh that doesn't it's not as good doing it when it's staged right we had the really interesting conversation but we weren't recording and let's not try and pretend to make it something different yeah so that is the current process is a big google doc and us scrolling notes all over it yeah and we've got

wow what's that two whole pages of thoughts. And you'd highlighted something which says trunk-based development as potential discussion today. So why don't we just start talking about what you meant when you wrote trunk-based development and see if something comes out of that. Yeah, I think that's a great idea. Some of our experiences at the company that we're currently employed at and contrasting that with some of the experiences that I know I've had at other companies. I'm sure you've had at other companies. Right. Where there are lots of ways to skin a cat. Why are we so mean to cats? I keep using that phrase.

I don't know. We don't need to skin cats. How about zero ways? There are lots of ways to do things. Cats get this all the time, right? There's cat gut, right? If you're going to make strings for a musical instrument, you make it out of cat gut, which I don't believe has ever been related to cats,

but it sounds pretty horrible for the cat also. Maybe after you've skinned the cat. Oh, man. No, this is bad. Moving on. But yeah, there's lots of different ways to kind of do this, where this is organize your code, right? Right. Like, there's a lot of different ways.

And some of it can be kind of bike shedding, right? It's just sort of like, you know, Coke versus Pepsi, this or that decision that doesn't really have that much of an impact. Some of that can have very significant impacts on the success of your project or the success of your company, even. And sort of everywhere in between. I mean, you have that expression about Conway's Law. Yeah, yeah. I mean, you know, Conway's Law is a feature or a bug, kind of depending on how you think about it.

Do you want to just quickly, what is Conway's Law? Yeah, yeah, yeah. So Conway's Law is an observation that the structure of your software will match the structure of your organization, right? So if you have a software engineering department with three different teams and you tell them to build something, you're probably going to get a system in three parts, right? Right.

Seems obviously right without even evidence, right? Which is a dangerous thing in general, but we've observed that it is generally true. Yeah, I mean, it's an observation, right? It's sort of like looking up at the sky and saying, I see that the sky is blue, and that's the blue sky observation. Do you also see the blue sky observation? Ah, we intersubjectively share the blue sky observation. Well, that's amazing.

So it's something that people observe, right? Like when you build software in an organization, the structure of the software tends to match the structure of the organization, right? And the question is, is this a bad thing? I mean, I think of it as a thing, right? Whether it's good, bad, whatever labels you want to try to throw on it, this is a thing that happens. So you can think about it in whatever terms you want, but it's a thing that happens. You should either work with it or have it work against you. There's really only two things, right?

Either you go with it and say, okay, you said this before. It's either a feature or a bug, depending on which way you go with it and say okay i'm gonna you said this before it's either a feature or a bug depending on which way you look at it and you it probably makes sense to choose to see it as a feature and say how might i arrange my organization such that i get the code arrangement that i want or how do i make sure the code arrangement appropriately mirrors the organization i would like to build around it right right and if you have some preconceived notions about how your organization should be structured, or how your code should be structured, and those two things aren't in alignment, you're going to have a bad time, right? That makes sense. If you think, well, we should have four different teams, and you have one giant code base, it's going to be real hard to have four different actual teams, right? Like you might have an org chart with four groups on it, but that doesn't mean you have four teams.

So all of those things can have a pretty significant impact on your success. And so it's important to kind of think about them. And some different ways that I know I've seen of just sort of structuring, I'm just going to say chunks of code. You can call them projects. You could call them things. You could call them things. I don't know what you call them, actually. I almost always think of things

these days, I tend to think of things more in terms of repositories, just because of GitHub and open source and all that. That's just sort of my brain has been warped to that after years and years of GitHub. Remember, there was a time before GitHub.

Oh, man. It seems odd that you'd have to go and find some random CVS repository and pull from something or, God help you, SourceForge, which still exists. I don't know how, but there are still projects that are hosted on SourceForge, which just seems like a terrible waste of your time to visit. But, yeah, those were dark times, man, right? Oh, man.

I still can't roll up to a CVS and not think it's the pharmacy repository in CVS. to visit but yeah yeah that was a those were dark times man right man just tar balls occasion still can't roll up to a cbs and not think it's the pharmacy repository and cbs like that's just they even had the slash there every time oh gosh i'm just like that's the pharmacy repository and then there was there was there was perforce yeah um and then there was uh subversion of course which was kind of the and of course there was visual source safe which was none of those words i'm sorry that's just my favorite expression okay so moving off of source control because whatever we i don't think we've had source control has not been a topic that we've discussed yeah and i feel like these days it's just kind of like, use Git, right? There's not really much to debate about. You just accept it. And Git is the least worst thing. Yep.

But getting back to the environment you said, you were saying about repositories being like the unit that you think of. Right, right. Where I derailed you. Yeah. So it's like you can have a debate about monorepo versus multirepo, which is certainly a debate that I've been a part of from time to time. And, you you know do you never very rarely not been in that debate because you're either in one state or the other and there are always the few vocal folks who have very reasonable reasons to move to the other approach

and in my experience like it's either a tooling problem or an org problem or if your org is big enough you need a tool department to manage it you know very very large web search companies that i've worked at before have had entire departments devoted to making sure that their giant mono repo is manageable given the way that they would like to administrate it and then of course if you have worked at one of those places you think mono repos are the best because like look all these benefits you've got you know i can just put in code from some other part of the repository without really thinking about it it's really trivial it's really easy there's no build issues i just it's just all there it's all there i can

you know get grep across the whole code base how amazing and then you don't realize that that doesn't necessarily work when you move to like a four-person company and you don't have a giant team of people that are administrating the systems that make that feasible be it the build systems themselves be it the source control um code review stuff be it like code owners i mean obviously github is starting to pick up all of these things and we're starting to have those we're starting to have those kinds of features come but um but yeah it's it's a thing um and obviously then if you are bitten by that, you might decide it's just easier.

Hey, I don't need to jump through all these hoops. I just want to make a little repository in my style, in my way, away from the rest of the code base so that I haven't got thousands of other commits polluting my history, or I don't have to fight with someone for a continuous integration server slot or merge conflict, blah, blah, blah blah i've got my own little way so they're both stable places to be the problem is we live in this extraordinarily high dimensional space of trade-offs right that

is engineering yeah and any one set of of uh issues is only one possible solution in a set of equally valid solutions to the problem of, like, can we work effectively? Yeah. So what would you do? And I don't think I've ever asked you this. So let's discover that we suddenly have a big difference of opinion. Or not. We'll be fine. What would your natural tendency like if we were if you and i would start up a company and start developing with say like 15 developers just picking a sort of big enough to

be multiple teams but not so big as it's like an unwieldy setup you know we've got like three or four things to do yeah would you go for the mono repo or multi repo what's your sort of default thought so i think it entirely depends. I'm going to answer your question very specifically, but I'm going to start with what sounds like a wishy-washy answer. Okay. Okay. I think it entirely depends on the people involved and the problem that you're trying to solve. Because again, there's this deep relationship between the code and the organization, right? And you have to keep those things in sync or you're going to have pain. And so you need to

think about who are the people on my team who are the people in my in this group that want to work together right either because they share you know like a common technology like oh these are the c++ programmers and these are the javascript programmers or whatever i don't know or um you know these people have worked together before and they know that they work together well and that was half the reason we hired them because we know they all work together well. So, it would be kind of a shame to split them up, right? So, think of the – because the people parts of the organization, I feel like, are much

harder to change. I kind of have this rule that people don't change until they do, right? Which is to say, if your plans depend on someone changing, you need to get some new plans. Because they might change. You never know. People change. But if your success depends on this person doing something that they've never done before and have shown no inkling before to do,

you need to get some new plans. You make a very interesting observation. That is kind of the the good advice to give somebody who's planning on getting married to somebody else like if if your future happiness depends on changing that person to be someone that they aren't right now yeah there's probably not a good idea to get married to that person they're not going to change they might change they might they might change you have to understand people do change and that's yeah of course fine and many people improve and all that kind of stuff but like if that if you're

betting your future on it probably better not to and yeah i guess to an extent you know there are a lot of similarities between getting married to someone and hiring somebody right you're bringing someone in you're going to spend a bunch of time with them probably more time than you spend with your spouse i mean that's exactly right notwithstanding but more waking time with them you've got to work closely there's going to be conflict from time to time there's gonna be differences of opinion you need to be able to trust that you can get on with with with a sensible way of getting uh through those differences of opinion and um yeah and it's very expensive to get rid of them if it turns out that it wasn't a good hire,

which is not a good way to think of the thing, right? No one goes into either of those things expecting for the worst outcome. So that makes sense. You don't expect folks to change. So design, again, as if that's a feature, not a bug, right? Right. Hey, these people work well together. Let them work in the way that's most effective for them.

Exactly. Exactly. And in a lot of ways, this is just another kind of engineering. You're just solving for constraints, right? right the constraints of this company are these four people work really well together uh we can try to break that constraint just as you can try to force you know a uh type to be cast into something else and you can kind of squeeze it into the box and make sure all the bits align and you're fine is that really what you should be doing well probably not right yeah so i mean you can if you need to, but probably not. It's like you just need to solve for this just like you'd solve any other constraints. And my argument is that the people are the hardest and most expensive thing to change. And so, you should start with the people and then work back to the organization of the code, right? Right. of the code, right? So if you have 15 people that all worked at Google and all worked in the monorepo

and really liked it and they all sort of understand how it's supposed to work and two of them are willing to be the tools people that build all the tools to make that work, take a monorepo.

Sounds like it'll work great. If you have three people that worked really well together, five other people that hate those three people, and then the remainder of your people are just like, I don't really care, whatever. Make sure that you have at least two teams

and make sure that the boundaries between those code, that code is very well defined. Hate is very strong there. Let's just say don't work effectively together. Enjoy working with, right? Right, okay. I mean, I have worked at companies where there were people that just really, you know,

for various reasons, political or otherwise, just didn't like each other, but you go to war with the army that you've got. So, you know, that's how it is. And you've got to think about this stuff, right? If you try to argue these things, I think, entirely based on the technical

merits, you have to think about the technical merits because if you only look at those, I think that you're going to wind up in on the technical merits. You have to think about the technical merits because if you only look at those, I think that you're going to wind up in a really bad place. You have to consider more. That makes sense. I don't know if it was you I was talking with about this earlier in the week, but I was definitely talking to somebody about

a couple of people that I was working at a previous company where we'd reached a sort of nirvana state where we didn't need we weren't really big on pull requests and all that kind of stuff or code formal code review we'd obviously read each other's code as you went through it and sometimes just go through the log and just see what the other people have been doing but like we had reached this point where you it was almost difficult to tell who had written what code all of the code looked and fitted together beautifully

we'd kind of come to a gestalt mindset about how to approach a problem such that you could, if there was three different ways of solving a problem, you could probably, you could bet good money that in a vacuum, the four people that were working on this team would probably solve it in the same way. We're using almost identical naming conventions, algorithms, approaches. And that was a beautiful, that's almost like a flow state for a team, right? You're kind of interchangeable. It was wonderful.

And it meant that we never really had any conflict. We didn't think about it. We just got on with the job at hand. It was like a high point in my career to be working in a situation where I was just churning out code and there was nothing really stopping me and not even within the team.

The team felt like it was moving forward. And that could easily be a bubble. Now, in that particular environment that we were in, we were in our own bubble. We had our own build system. We had our own CI setup. We had our own deployment that was slightly different from the rest of the team because we felt strongly,

as two or three of us, that this was how it should work. And it overlapped like 80% with the rest of the team. It helped that we were in C++, everyone else was in Java, so that there was an obvious line that we could draw. But then we kind of widened the line a bit and said, no, we think it should be done this way, and so we are going to do it this way, and we'll make sure that we work with you, we interoperate with you.

Yeah, yeah. So I guess what I'm trying to do is citing that as both, I'm very lucky to have had that situation, but also that that is a sort of natural occurrence i think yeah at least it appeared natural maybe my my team lead was uh was actually maneuvering it that way because he knew that things were going to work better if it was that way but you know possible uh but you know it was our it sounds like we had stumbled on a nice low point or a high point in the multidimensional space where that worked really well for us.

And both the Conway's Law version of like, we were our little team in our own little world. Yeah. Yeah. I mean, it's funny because it's sort of like you guys had that sort of independence, right? Of being able to do things your own way because you were in C++. And sometimes that's a convenient excuse for creating that. But honestly, maybe you should do that on purpose, right?

Like if you have a group of people who are all C++ programmers, that doesn't mean that they should all share the same build system or share the same makefile or share the same project structure just because they're all in the same language or even using the same technology, right? Like, it's much more about what the people who are going to be tasked and responsible for solving the problem want and how they can work together effectively. I've said this thing for a while that a team is a group of people who succeed or fail together. And succeed is sort of an interesting word in that phrase because it can mean sort of different things.

And the definition there is a little fuzzy, a little bit on purpose. But success for most programmers looks something like being able to come into work every day and have the experience that you just talked about, right? Like, yes, there's financial success. And that is a necessary but not sufficient portion of being successful, I feel like, as a programmer. But real sort of, you know, Maslow's hierarchy of needs stuff, self-actualization, comes from coming into work, being able to work effectively, do your best work, do work that you can proud of,

that you're proud of, and solve a real problem. And if, you know And if that's what you're optimizing for, then I think you should be much more cognizant of what group of people can I assemble as a team that all see success in the same way, that all operate in the same way to create that success, that will not create conflict between two people because they have different definitions of success. Right? Interesting. Because then you don't have a team anymore. Right?

Yeah, that's a really, really good thought. Yeah, I hadn't really thought about it. That's a great characterization of what makes a team, right? Because you can define a team as these old people are working like notionally together. They have a tech lead and he's in charge of them. And they have a manager and she's in charge of that whatever you know that set up and then you know that's defining it using conway's law from the top down like hey your team is defined by the org structure yeah yeah but then you've got the team of like well this person they want to do everything as beautiful functional

programming and that's really their goal their goal is to turn the world into Lisp or the world into whatever. And we've all worked with people that have various bends and we've all been guilty of similar things, right? You've got and then other folks who are just like, no, it's got to be just as fast as possible or other people

that know I want to refactor it or it's going to be testable and obviously there's a sort of core of things that are just required to be successful. As you say, necessary, not necessary, sorry, sufficient, necessary, not sufficient or whatever the right way around is. But yeah, if you get a group of people

like in the way I described before, where we all looked at the code and we're all like, no, the code isn't done unless it looks like this, where this was how we all wrote the code by that point so that's that's an interesting one so can i ask a question about that though because the it sounds great it sounds amazing and it's something that i've been thinking a lot about recently and is there a problem that you end up with a group of folks who just think the same way is that is that a problem i mean obviously from a diversity point

of view diversity of viewpoint and all the other axes we measure you can you can get into a really bad situation if you just say no only people that think like me and work like me can work together yeah yeah i i so i have some very nuanced views on this i mean first of all from like a cultural diversity standpoint i think you need to be as diverse as possible right like never do you want to give up on those kinds of constraints ever like try to try to make your companies your teams as as culturally diverse as you possibly can because you'll just make you better problem solvers right people from different experiences will have different insights and you'll get different solutions that are more holistic while we sit here looking at each other, two 40-something white men, our backgrounds alone

have already shown us that measuring that one tiny dimension, seeing the world in two different ways is very, very valuable. And just to have more dimensions to that sounds, the more the merrier, right? The more viewpoints, the better. Absolutely. I personally think that it is completely fine and maybe even more functional to have when it comes to sort of like technical diversity, right? Like different ways of problem solving that are not necessarily cultural, but are technology-based, right? Like the classical functional programmer versus object-generating programmer, for example. Right, okay. Yeah, yeah, yeah, that level. Something like that, right?

I think it is very reasonable to have within a company a diverse culture, but within a team, basically a monoculture, right? I do not think that's harmful. And in fact- And this is a monotechnoculture. Monotechnoculture, right? Like all the functional programmers grouped together into a team, that's fine. You don't need to split it up so that there's two functional programmers and two object-oriented programmers on every team, right?

That's sort of like a little bit of a straw man, but that's kind of what I'm getting at. Yeah, I see what you're saying. Okay. saying okay yeah i i don't think it's harmful to say we're going to take all the closure blisp folks and put them on a team and give them the tools that they need and build something in closure right and then that's a monoculture from that sense on that very specific point i've had so the very first uh team i came into uh when i joined the company where you and i met was uh a desk where there were there was some closure and i ended up pair programming with somebody in closure and it opened my mind to the whole idea of functional programming and i was a very very dull boring procedure not procedural

what they call imperative programmer program like i mean as you know i've crawled my way up the stack from assembly so you know that's how i think about the world. It's one statement after another. And to see this very implicit, very much more abstract and different way of thinking about it, I found it extraordinarily irritating for the first couple of weeks because I'm like, I just want to do my job. Why am I pressing about with this? This seems so much harder. And then I had that moment of realization and my experiences opened up. And so I was exposed to something new and I learned a ton from that. And I resolved at that point to try and make sure that I look for those kinds of opportunities and take them every now and then. Obviously, you know, this is the exploration versus exploitation, right?

If you've got a good team that gels well together, sometimes you want to just all right you four folks you go off and you do your thing and that's great but every now and then you probably do want to kind of perturb it a little bit a little bit of simulated annealing like hey we've got this this uh this girl wants to come in and do this thing over here and you know she's her background is in is in uh you know i I don't know, genetics. And so maybe you can work with that and do something. There's something else coming in where it might shake it up a bit. And then everybody learns something. But I guess you have to be super, super careful that you don't put people in awkward positions, particularly if they're the person coming into an established team and they're the outlier, as it were.

Yeah, yeah. In my case, I was the outlier because I was just the weird C++ programmer being dumped into a Java enclosure department. But I learned a ton and I was glad of it. I mean, going back to the metaphor of this was sort of just a form of engineering, there are always trade-offs, right? And if you do some of the things

that I'm talking about here, there are trade-offs. And one of the trade-offs is that you will have less, fewer, you will have fewer opportunities to get the kind of exposure that you had working in Clojure. And my general theory on that is that you just need to be aware of that, and you need to solve it in a very specific way, which is you need to move people around slowly. If you move people around quickly, that will be very disruptive, right? But you need to have a career path for the people in your organization. This is something they're going to want. This is something you need to give them. So, you're like, you're not

just going to be on this one closure team for the rest of your time at this company. Otherwise, the time at this company is going to be pretty short, right? You're going to get sick of it, or you're going to want to grow, or you're going to want to do something else. Or to want to do something else uh or not right again people don't change until they do right maybe you have people that just want to stay there forever and that's fine right so you need to be able to handle both cases and so one of the things that's really important is understanding that within the team there is a little bit of a culture like there's there's usually a little bit of a of a spree to core it's this thing that you kind of talk about where you had these four people that you worked with and everybody just sort of like had mind melded, right?

And like everyone knew what to do for any particular situation, right? I would say that's a team that is probably ripe for either moving somebody out or moving somebody in, assuming that there is somebody that wants to do that, right? Because what you have is a very, it's almost like a sourdough starter, right? Like you have a very healthy, strong sourdough starter. That's a perfect analogy.

Now it's time to make some bread, right? Like you're going to pull somebody out, you're going to put them somewhere else, and you're going to see what happens. I mean, that team was two to start with. That was me and the team lead. And it did take us a while to find our our level right um i think i've alluded to to people before who just sit down think a lot and then type out the right answer that was my my lead he was very much the think about it a lot and and less on the actual test

because his code didn't ever really go wrong and And there was only one of him right when I joined. And so then I joined, I'm like, I don't know what I'm doing here. I need the safety net of more testing. And I think we found our level, right? And then we became more and more productive. And then we wanted to grow the team. And we had a fresh out of university person join us. And that was really interesting for us both because we got to bring someone up to speed who was a blank slate and that was a really exciting experience for me personally to actually sort of be able to help someone on their journey from pretty much i don't say nothing obviously if you've been to it you've got a degree in computer science you know a lot about it but like we also know that like you know it's like when you learn to drive a car you pass the test and then you learn how to drive yeah right there's

the second step of actually doing it for a bit um but as you say the sourdough starter i feel that we had got it into a nice situation where we could bring them along and that really really helped them um i believe anyway uh and then then when we added another person to the team it was just a natural way but that was not the kind of thing where we were throwing in the curveball that wasn't like the closure person came in and and we were learning back and forth that was definitely a very much a here's a blank template in the first case or here's an industry veteran who came in as like the fourth member of the team can we just make sure that they they fit in the role that we if it's three to three to one then it's a good chance that you're even those edge edges that you might have where you don't fit in exactly they get

bumped knocked off pretty quickly or or else the whole team adapts a little bit which i think you know is a natural process so yeah yeah so yeah sourdough starter i like that right i like that as an example for a decent little team that can can grow but yeah i mean i think that metaphor works well but i mean so so all of those things then inform the technology so if we're going back to the whole debate of like mono repo versus multi repo like you might have a situation where there's like a one to many mapping from teams to repositories right where like each team manages a set of repositories. You might have a one-to-one mapping where each team basically has what is effectively

a monorepo. And even though that team might be building lots of different services, maybe even unrelated services, they all work out of the same monorepo because the tooling is there and the flow is there. The utils. You know, everyone's grab bag you know yeah we just need the case intensive string comparison routine the same one that we copy paste into everybody or whatever it is yeah so so you can if you think of things that way then i think that the the technology choices become constrained in a good

way in that good way that you sort of have like know, you were saying this sort of multi-dimensional problem that you need to try to solve, you take a few of the dimensions out and all of a sudden the possible solutions become a lot more easy to reason about, right? Interesting, yeah. And so, you know, a lot of those things can kind of solve themselves in that way. A friend of mine once kind of mentioned, and I had an observation I thought was really insightful once. He said, teams are immutable. And what he meant by that is every time you add or remove someone from a team, you don't have a different team, you have a new

team. Because you have to kind of incorporate all of the needs and all of the abilities of that new person into the holistic team and reconsider all the things that you considered before and some of us have very concrete situations where someone joins a team and all of a sudden there's a four-hour meeting about why we do things a certain way right like well yes and some of us may have been the cause of those meetings yep exactly exactly oh dear um and so it's it's like it is both an opportunity a little bit of a burden perhaps but an opportunity to sort of re-evaluate why it is that you do things the way that you do and reassess them in the context of what is

essentially a new team and my point here is that the technology is sometimes the thing that changes after that right like you have a new team sometimes the result of having a new team is that the technology is sometimes the thing that changes after that, right? Like you have a new team. Sometimes the result of having a new team is that you need new technology. So it's sort of like, well, previously we used Make and now we're going to use CMake. Or previously we used CMake and now we're going to use Bazel. I don't know. Or whatever it might be, right? Or even on the subject of like the repo, monorepo stuff there,

a lot of, in my experience, of the arguments for and against those two things is, again, the tooling. We've talked about like having teams. If someone says, well, there's a tool that can make it easy to extract, publish a library, maybe we can now use Conda or Conan or VS Code or insert Maven.

Everyone loves that one, I heard. To extract out some code and then and but now you get an interesting sort of side effect of that is that now you've changed the way you develop and maybe that's a positive maybe that's a negative you know like we've all been in situations where versioning some chunk of code that's not it used to be inside your repo where you could make a change and then you know test how it works across a bigger code base in situ is a convenient thing to be able to do and so you're like well that's an argument for a mono repo right if i have to refactor and change the name of this particular uh interface

it just works like the refactoring tool takes a while but it can refactor across the entire code base fantastic right right but also we've also been in the situations where like well i really want to break that API, but I don't want to go through the pain of changing everyone who's using it right now. I'm going to do it in my repo, and then everyone else can pin to the oldest version that does what they need,

and I can move into a new version of that. And so there are trade-offs even along that dimension there. And so, yeah. Sorry, I've derailed you from you talking about the new member to a team being... Like creating a new team. I've derailed us completely. Yeah, creating a new team, yeah. Sorry, I've derailed you from you talking about the new member to a team being – Like creating a new team. I've derailed us completely. Yeah, creating a new team.

Yeah. And we've moved into something else, which I wanted to talk about actually anyway, which is these kinds of things, these kind of trade-offs. Yeah, yeah. Because what I just – I mean, unless you want to finish off on that point. No, no. My point was that that can change the technology of the team. And so, I think it's actually kind of a natural thing to say, okay, what are some of the technology changes that can come across from that? I don't want to jump the gun there.

No, no. I think you were maybe like 30 seconds ahead of me. It was perfect. Perfect. Well, so yeah, that's one thing. So that can be a feature that you can extract a library and then you like pull in version 1.0 of the library and all that kind of stuff. But that sort of begs the question or asks the question i should

say because people will complain if i use the word beg to beg a question when nothing ever is actually begging a question um is should you be at trunk all of the time yeah because when you have a mono repo you are kind of by definition you're at trunk um at least for those kinds of changes right if i do a uh a, then if somebody edited some library, I'm going to use air quotes for this library because it's just a subdirector of my monorepo, then I've got the change immediately, right? That's something I get straight away.

It's more of a, there's more decoupling. There's a sort of clutch plate between parts of the code if you use a sort of ci server to build and deploy a sort of compiled package and then you pull back down again into another project you've kind of got this uh separation which can be a feature sometimes yeah yeah but if you don't live at the trunk all of the time, if you allow yourself to float away, there's a danger that you can float too far. And then one day you're like saying, oh, yeah, yeah, that bug that you fixed in version 2.7, can you backport it to version 1.8? Because, you know, we never quite got around to it.

Now it's a huge big deal. Yeah. Those are painful. I mean, what kind of tradeoffs, other tradeoffs? What else am I missing in that? I'm sure there are tons of other things. And what are your thoughts? Yeah. I mean, there's a lot-offs, other trade-offs? What else am I missing in that? I'm sure there are tons of other things. And what are your thoughts? Yeah, I mean, there's a lot of trade-offs there for sure.

I tend not to use libraries as a form of dependency management to solve that problem if I can avoid it. Right. Or it's maybe a better thing to say that I use them very intentionally. I use them when the difficulty to refactor across that library is, again, sort of like a feature and not a bug. I want it to be difficult because of reasons. And the reasons might be I don't want inexperienced programmers mucking with something that is very core to the business or very proprietary or very difficult to understand or whatever it might be and so i'm going to encapsulate that in our library or i don't want non-programmers having to solve very trying to solve very difficult software engineering problems like multi-threading

or parallel computing so i'm going to build a library that solves that those problems on their behalf and if i've done a good job of that they will hopefully never have to reach right you just tuck away tuck it away yes yeah almost as a just a you know putting the panel over something saying you know like hey no like service warranty yeah exactly right right and i mean i don't want to say yeah that's that sounds uh sounds pretty bad because obviously anyone can look through the code anyway. You could go find the source of that code. But it's just an extra sort of insulation layer that you're using, again, structurally within an organization to say, like, you don't need to go past there. Obviously, there will be people who will.

You want to encourage the cross-team communication that will be necessary if changing this code requires going to somebody else and being like, hey, can you change this function for me? Be like, actually, I can't. And here's why. Right. Yeah. And have you considered maybe trying this instead? All those kinds of nice conversations that might never happen if you don't put those

sort of speed bumps. Now, again, they are kind of speed bumps, and I generally don't like it because it sort of creates a slowness. You just want to do that intentionally. You only want to do that on the block with the kids playing sign you don't want to do it literally on every street in the city right um so so i use it intentionally the other thing that i tend to do is i personally tend to prefer breaking things out not as libraries but services uh not microservices but i i had this phrase that i used a while ago called one second services to sort of like define this like approximate size of

a service and how much behavior it should have it's it's this is the first time i've said testing in this podcast it's one second worth of unit tests oh Oh my gosh. So about a thousand. We're about 30 minutes in and you said testing only now. That's impressive. Yeah, I know. And I'm not going to talk about it anymore. I didn't have that on my bingo square. I'm not going to win this one. I'm so mean to you.

I deserve it. I deserve every minute of it. But no, it's sort of like, if you have a thousand tests, that's about the size of a service that is nice to work with. It's not hundreds of thousands of lines of code. Obviously, the tests only take a second to run,

so they run super fast. It's not a microservice. You don't have this thing where it's like, okay, I'm bundling up my single lambda function that's eight lines of pad left. Pad left, right, I'm bundling up my single lambda function that's eight lines of... Pad left. You know, pad left, right, as a service that you call, right? That's insanity, in my opinion. But it's also not a giant, you know, monolith of a single service.

It's got, like, you know, a huge interface to it and a thousand different things and connects to, you know, a database and a redis queue and a kafka queue and a you know dynamo db instance and everything else in the world right but a second's worth of tests is just as good as anything of licking your finger and putting in the windows like yeah this is about the size that i mean i just know instinctively the kind of level of complexity that comes with that amount of code and obviously you said a thousand tests there because the other sort of aspect of this is that the test should be fast which we've talked about a lot so we won't cover again here not gonna talk about that um right so you prefer services and i mean all of these are apis they're

different ways of making apis right this is what i the layering between software bit like how you call into a library and you can't actually see the library because it's there's no code you just got like a header or a couple of like uh bits of uh frameworky stuff that you talk to or you make it a service and now you've kind of got some kind of rpc mechanism that again defines a very strong api and enforces it as well yeah i do so those things is a little different though because the library it's usually an all or nothing upgrade right you can't have four different versions of a library active in your application at the same time well not without pulling heroic tricks yeah right right yeah whereas like if you have an api right you can have backward compatible like if you're making a breaking change to your

api that's not strictly additive you can have both versions of that api live at the same time right oh interesting clients can even use both versions at the same time right like they don't have to make a choice of we're moving to version 4. They can be like, well, for this section of the code, we use version 3. In this section of the code, we use version 4. Very interesting. I'd never considered that. I mean, obviously, there's got to be a cost to writing a service that seamlessly supports some number of older versions of it.

Right. But I can see the benefit that comes from it too, right? Yeah, as you say, this allows a sort of more transitionary, that's not even a word, an easier way to transition between older and new code without it being the code, where the code is like, well, you're either calling Bob, the Bob function with two arguments,

or the Bob function that takes a some other completely different structure now obviously some languages will let you do some amount of overloading and no other trickery like that and of course you can have versioned bits of code and you can probably do that with libraries as well and kind of pull in bits of this library and like enable yeah there are various tricks to do it but it's very clear and clean in a microservice environment. Yeah, potentially. You have at least the opportunity to do that. stable APIs and the services that interact within a team have more flexible, dynamic APIs,

because maybe the team has a single monorepo with its 12 services in it. And all of the external facing APIs have, you know, hardworn integration tests, but all the internal APIs can all be deployed at once because it's all a big one monorepo. And if you want to change the signature between three services, that's fine because they all get deployed at the same time anyway. Then you can do a lot of cool stuff with that. And's just it's all about being cognizant of the boundaries between the people and how that reflects in the boundaries in the code got it so you talked about deploying there in terms of like deploying a whole bunch of services and obviously that fits together if you uh you have a like exactly as you describe a team that's that's got all

these microservices in one kind of gob that they can deploy but in general deployment especially when you have got tiered things like v1 v2 v3 v4 there's usually a lot of machinery to make that work there's some kind of dispatching system somewhere higher up that you can i will i want this microservice to appear here in some logical uh space how does one deal with that kind of stuff with with regards to you know experimental stuff maybe can i make do what what do you think like maybe i've got like some um i'm rewriting a server and i want to like deploy it uh somewhere should i be doing that in trunk should i be making a branch and deploying that in some other namespace thing how How do things like that work?

Talking about that, that seems tricky. I mean, I will say that I have never built a system. I mean, there are a lot of people that are going to be way more experts on this than me in terms of building systems with like hundreds or thousands of running services, you know, deployed on AWS with all different kinds of versions and stuff like that. And I'm not going to – if that's what you're doing, you should not listen to me because I've never done that before. And I have maybe even intentionally avoided finding myself in that kind of situation because there's a lot of stuff going on there. The systems that I have built where I think this model works well are systems where you have sort of dozens of services, all of which are large-ish.

They're these kind of like one-second-sized services, right, where it's not millions of lines of code. It's also not eight lines of code. But it has a non-trivial amount of responsibility. Right, right. And some subset of those services are dependent on each other in potentially breaking ways. And most of the time what I've done has been able to deploy the things that are hardened against each other as a unit, right? So it's like, if you've got three services that all need to change together,

then they all get deployed together. There's not even an option to deploy them one at a time. But for the things that do have sort of those harder interfaces, you can deploy them independently. And that's what you plan to do, right? So if you're going to change an API, you're like, all right, we're going to introduce the new API in service cluster A. And then we're going to update service cluster B to use the new API.

And then we're going to go back after that's done and everybody's happy and remove the old API from service cluster A, right? And you do that in sort of a three-step process right and that's kind of just the way that i have done it for these types this type of structure but you know i have just been recently doing a whole bunch of work off on my own little um branch of a big big code base is essentially a mono repo for all intents and purposes for in its own domain right there's a lot of different people and different ways of writing code they're all in that same repository it doesn't represent the whole company's code at all or even like the whole domain's code but if it has a mono repo feel to it and so i've been in the world of making changes on the side in my branch and then obviously suffering the pain of merging

in all the time to try and keep myself up to up to pace but there is another world in which i could have done all of this in in in trunk and i know like at again big web search engine company feature flags are the way forward for that and i and i've done i've done those things i've used feature flags for like turning stuff on and off but it can be very difficult to do that well especially when you're making structural changes yeah it's very hard to say well i redid the class hierarchy behind a feature flag you're like well now you've just got two entire copies of the code base right and somewhere you have to make that switch. And so I've been thinking about it.

I realize this is slightly tangential to the things that we've been talking about, but it sort of fits into workflows and how teams work. There is a world in which everybody lives in their own branch, and they kind of occasionally merge back into a master or a main trunk of development. And I guess just channeling what I've learned from you in this conversation, again, it's let the teams find the right way to do that. If it works well for having people maybe breaking off one or two at a time and going off and doing something and then bringing it back to the center,

then go with it. But, yeah. going off and doing something and then bringing it back to the center, then that then go with it. But yeah, I had a conversation today with two people that work in that very same repository that you're talking about, which shall remain unnamed. And we were talking about them doing some work in another repository. That is just the three of us. It's me and the two of them.

And I was telling them, you guys don't need to send me PRs. You can just put your work right into master, right? Because there is the benefit of the sooner you get it into master, the sooner I can start coding against it, right? If you've changed some method signature or some class hierarchy,

the sooner I see that change and integrate it into my own work, the less painful it's going to be for me. So I want you to do that as soon as possible. And if that means that you need to check in code that's not running yet because it's been disabled by what is effectively a feature flag, I would prefer that to you

sending me a PR and then doing all your work on a branch and all the other things that go along with that. There's a benefit to that. There's definitely a thing on the, like, I have a personal extreme, and we haven't got enough time left for me to go on and wax and lyrical

about how to do a good pull request, but certainly a way to do a bad pull request is to drop a 100-file PR on someone because you've been working away in isolation for six weeks. So we'll come back to that another time.

Yeah. But does that not encourage people to check in like slightly not ready code? Well, I think one way that you can balance that is with continuous deployment. So if your policy is code that's pushed to the trunk automatically gets deployed, assuming that it goes through

the continuous deployment pipeline, then that's sort of setting the standard for code that gets shared. I mean, I think it was Michael Feathers, I keep quoting him, code is a way we treat our co workers. It's just like, you know, not filling up the coffee machine, right? Or, you know, leaving your three week old lunch in the office refrigerator. And so for me, a reasonable approximation for is this code good enough to inflict on my coworkers is, is it good enough to go to production or is it good enough to get deployed basically? And so coupling those two things together to where whenever you push to the trunk, it's going to go to prod. So you better make sure that it's not

completely terrible, right? it's not perfect there are certainly ways in which you can write code that you know isn't going to negatively affect the production environment but will negatively affect your co-workers right but it's it's a it's a it's a sort of strong enough correlation where i personally i'm i'm a big fan of that now that it's more measurable yeah than just dissatisfaction of your colleagues, right? Which is harder to measure. Maybe once a year during the peer review that it might come up, but it might be a bit late. Right, right, right, right.

So yeah, I think that's a reasonable way to do it. And then, of course, that implies, given all the things I was saying earlier, the organization of your code is such that that is also the unit of deployment, right? Because otherwise, you can't do that, right? If you have to deploy a bunch of things together and it deploys automatically then practically what's probably going to happen is that they're all going to be in the same repository and now you have what is essentially a mono repo but just for those things right yeah otherwise you're building very complicated tooling to make all that work i was just to make that point yeah the tooling is is the other solution

to that problem. But it amounts to the same thing, isn't it? Like if you can carve off an area of the problem space and make it work in this way through being in its own repo, through tooling to isolate a unit of deployment, testing and deployment, and then have this kind of thought process where you say, check into the trunk, and then you don't need to do pull requests or anything

like that because the standard which you hold yourself to to say this will go to production as soon as I commit it kind of is the barrier now I can get behind that but I think

pull requests have other benefits which we can talk about in maybe a pull request related episode I think we should do. But I think pull requests have other benefits, which we can talk about in maybe a pull request related episode. I think we should do a whole topic on pull requests. On pull requests. Trade-offs. Code review or trade-offs. I'll add it to the list since we have a list of those things. You added to that list we were talking about.

Fabulous. There you go, everybody. This is how we do things. And I can actually, look, I can see his cursor moving around in the document now. He's not even lying, ladies and gentlemen. Oh, my God. Okay.

Well, you type that and I will, as I say, have very strong opinions and I have a number of open source projects and I get to be on the receiving end of a lot of reviews for those things and I've developed strong opinions on those. So we can talk about that some other time. Cool. All right, my friend. Well've we've definitely covered a whole bunch of stuff and uh we made it up as we went along which is kind of how we do it always right yeah it is there's no pretense here that there's a heavily scripted aspect to this uh this extremely high production value

yeah but one thing we've never been really good at is ending podcasts because, you know, we're just chatting and then they're like, oh, I guess we should finish now. So I guess we should finish now. And cut. You've been listening to Two's Compliment, a programming podcast by Ben Rady and Matt Godmolt.

Find the show transcript and notes at twoscompliment.org. Contact us on Twitter at twoscp. Theme music by Inverse Phase.