Soft Skills Engineering - Episode 235: Bus factors and toxic time bomb

Episode Date: November 9, 2020

In this episode, Dave and Jamison answer these questions: Questions I work as an IC in a team which owns 3 very different and large parts of the system. Our team is 4 experienced engineers and 1 intern. Historically each person was assigned to a single part and, as you might expect, we have a bus factor problem. With this layout we’re making as much progress as possible and it helps us to compete on the market but creates a dangerous situation if someone would decide to leave (spoiler: I will). What would you do if you were IC, team lead or a manager in such a team? We’re already exceeding headcount so it’s not an option. I am a developer with 1.5 years of experience, and was put on a greenfield project to rapidly develop a new application. We have a contractor that came onboard to help with the process. On the very first day of meeting this person I noticed their propensity to not allow anyone else to talk and interrupt. Fast forward several months and this person has really become a micromanager, they’re requesting the source files from our UI contractor, they got another person kicked off the project because they didn’t like the changes they were making interfering in their development process, they have constantly hoarded all the real dev work and work frequently until 9pm. I have voiced my concerns to the PM, mainly about the bus-factor, since layoffs are likely coming and this person likely won’t be converted. At this point I am just tuning out on this project. I do the scrap issues the contractor basically doesn’t want, but I am seeking learning opportunities elsewhere within the company and have nearly zero interest in the project which I see as a ticking time bomb. What would you recommend? I could potentially escalate the issue to the manager of our team but I basically see working with this individual as toxic and the PM as autopiloting to the finish line. Show Notes https://www.computerworld.com/article/2534312/the–640k–quote-won-t-go-away—-but-did-gates-really-say-it-.html - apparently the bill gates quote is apocryphal

it takes more than 640k of ram to be a great software engineer this is episode 235 of the soft skills engineering podcast i'm your host jameson dance i'm your host dave smith soft skills engineering is a weekly advice show where we answer the non-technical questions about the technical field of software development have you heard this quote before dave yeah 640k ought to be enough for anyone yeah apparently it's apocryphal it's attributed to bill gates as a way to like dunk on him as what a loser he was wrong but i guess he didn't say it sorry bill that is so the internet you know what i mean like the internet exists just to like debunk things and basically spoil everyone's fun dunks and blocked dunks if we could harness that energy but they're

they're always retroactive blocks though it's like 20 years later oh by the way you never he never said that yeah that's true it's like retractions yeah you know who said that albert einstein i love those abraham lincoln quotes on the internet yeah about the internet all right do you want to thank our patrons dave yes i do thank you to those folks that are contributing at the amount where they get a shout out each week they are oladapo fadi yik yarns fainton alexander microconfig.io nick hathaway travis sanders dennis bogdan of braden kane steven armand lee john grant vinlock the agile ventures charity nick canter and philip if you would like to join this illustrious crew go to softskills.audio and click support us on

patreon if you do that for a large amount we'll say your name every week if you do that for any amount greater than zero we will send you an invitation to our slack community where you can join a bunch of fun hilarious insightful and smart people and pretty darn nice people and chat about all things software and not software the whole gamut all things all things no qualifications we uh heard from our listeners about coping with distance during the pandemic we asked about this a couple episodes ago and here are some replies this is from someone named brandon our team did a zoom escape room and it was hilarious we hired a company that offered escape room style situations through zoom the one we did was stranger things styled with our team

solving puzzles via screen share and each of us finding clues through our browsers on the internet we actually had a lot of fun okay that sounds really fun like if you had told me just online escape room i would have said lame but if there's like clues hidden on various websites on the internet that's cool it almost sounds like a like a capture the flag type of thing or an advent of code type of thing yeah which would also be very cool this is another message we got from courtney ross my team has a weekly d distancing 30 minute chat that can be about anything sometimes we talk through the tough bugs we are working on or news or podcasts also we are going to have a virtual cooking class so we can do it together while safely in our homes. I love that. I love the

term de-distancing. That needs to be a new thing. Yeah, closing. I could use some de-distancing. I could use some closing. Yeah, I like that idea. I've seen that start to become a little bit more common with my company as they've remained remote after going remote for a little while, and they're sort of starting to talk more about how, not just about how hard it is vaguely in general terms but the negative effects of missing social interaction and and like explicitly scheduling things to work around it so that's kind of nice to see that that's spreading around yeah okay we got one more message from someone named tarje skarset okay i actually they're not named that i can guarantee that's not their name but that's how i pronounced it sorry

maybe not super clever but it really works my team sits on discord all day when we have different rooms and everyone can see who is sitting together at any given time We usually pair using VS Code's Live Share or just sharing our screens. I could imagine pair programming would be a good way, a good thing to do in this environment. And that's probably the first time I've ever said that. I have enjoyed pairing a lot more since becoming a remote worker than before. Yeah, interesting. I think it's easier to introvert while still pair programming.

Like I can not smell them if I want to. I can contribute technically, but not be exposed completely to the frightening energy of another human. It only comes from proximity. Exactly. Yeah, it's psychic energy through the ether. Anyways, on to our questions. You want to read our first one, Dave?

Sure. This comes from an anonymous listener who says, I work as an IC, that's individual contributor, in a team which owns three very different and large parts of the system our team is four experienced engineers and one intern historically each person was assigned to a single part and as you might expect we have a bus factor problem with this layout we're making as much progress as possible and it helps us to compete on the market but creates a dangerous situation if someone would decide to leave spoiler i will what would you do if you were ic team lead or a manager in such a team we're already exceeding headcount so that's not an option so bus factor problem but

according to my advanced simulations if you have five people on the team and three major projects you have a bus factor of 1.6666666 oh great which is higher than one so you're doing fine and according to my even more advanced calculations if one person leaves then it'll be down to one point three three three three three three three which is still higher than still good you have 30 redundancy baked in easy peasy i guess if you only count interns as if they count as zero people then you only have a bus factor of one once you leave but yeah that's that's a problem that's cruel if you count them not as humans do you want to explain what a bus factor is because you just did some really advanced mathematics and i'm not sure everyone followed yeah okay that's a good

point the the metaphor of the bus factor is how in trouble are we if someone how many people need to get hit by a bus for this project to be in trouble right it's kind of morbid when you think about it i guess but if you have a bunch of knowledge like just all enclosed in one person's brain and that person is incapacitated either by bus or more likely by quitting or being fired or i don't know switching projects or something then how much damage is that going to cause i just have this image of like a venture capital firm investing in a bus company so they can just drive it around and hit their competitors engineer that sounds a lot like organized crimes business model where they just murder people so they have fewer competitors right

there's some like tired joke about yeah what do you think bunch of venture capitalists organized optimized crime but right i'll leave that up to you all to oh man so so this team's operating pretty lean yeah or efficient oh heavily optimized the flip side of the coin yeah i was telling dave two stories that didn't make any sense so i'll skip to the point of those two stories which is that the more optimized something is the more fragile it is like you can have this highly tuned team where you have exactly the right number of people and everyone's stretched thin but just knows just enough to get stuff done and that's fine and and probably saves budget from headcount if if you're not i don't know

wasting people by having extra people that that aren't contributing as much as the dollar amount that it costs to add them but as soon as something disrupts that then you're going to pay a pretty big cost in the delicate machine being thrown out of whack well if it was less efficient you could theoretically you would be spending more money over the long term but you'd absorb these changes more easily right and i was just thinking like okay if you throw the delicate machine out of whack by losing someone now the question becomes do the costs that result from that exceed one more head count or whatever number of head yeah i would need and these are unknowable no one will ever know the answer to this. Like impossible, impossible to tell. But I will say that people

that work on teams generally feel like they have too much work and people that allocate resources or people to teams generally feel like they don't have more money to just give all the teams to hire more people. Or more specifically, they feel like asking, how can you do the same thing with fewer people? Exactly. Yeah. Like those incentives are, it's pretty consistent that people on the ground feel like we don't have enough people people deciding how many people to add feel like i don't have enough stock options so we're at an impasse here and good thing i make the decisions can you imagine if this was a democracy or something goodness me that would be very hard for me

oh i got a little sweaty all right good thing my tesla has air-cooled seats they don't by the way Oh, okay. Well, now I feel better about not owning one. Yeah. So I was just thinking, like, what happens to an airplane with zero redundancy? You know, the more I learn about airplanes, which is still very little, the more I realize, like, so many of the systems, there are two or three of them, you know, copies. And I'm like, surely this makes the airplane heavy, which means the airplane uses more fuel and costs more to build, but they don't crash very much. And it all depends on what dollar amount you assign to the value of human life, Dave. That's right. Easy.

The answer is we waste a lot of money. Yes, we do. Yeah, that's a good point. There are industries that are very good at making that trade-off of extra cost versus risk that it avoids. And software is just such a fuzzy field that I feel like most teams are not operating at that level of analysis. and I mean it costs a lot of money too it takes an enormous amount of time and money to study your designs and test them and simulate things and that gets in the way of your two-week sprint so it's it's tough to pull off in a lot of software environments yeah and not not to mention the soft most software teams don't need to be as safe as say an airplane yeah if my if my modal

background does not dismiss 150 people do not plummet to their death to a fiery death yeah imagine the pilot touching an ipad in the front of the plane oh no uh-oh another modal won't an update now oh no how will we land hey uh folks this is your pilot we're waiting for an ios update this is kind of a form of debt i think we talk about technical debt where teams have taken on bad practices or bugs or issues that they knew they were going to have to address eventually, but not now. And this is kind of like that. I think you're kind of running on borrowed time. And the risk when you take on technical debt, the risk is that the creditor will come calling

before you're ready. And in this case, it's kind of the same thing. The creditor can come calling and the creditor here kind of looks like a bus or a resignation letter or a job offer that you can't match because you're running so lean yeah you already converted all that money into bonus for your executives it's gone i mean teams like this are kind of defying gravity or defying reality right it's like they're kind of just seeing how long they can do this and get away with it but like you said it's really hard to know that you're doing this because there's not a number or like a thermometer sitting on the wall that says uh-oh danger zone you know yeah so it's hard to know but but they are they're defying gravity and and there's like this real cost to operate and build

the software and there are companies like in this case that are trying to run that at a lower cost yeah almost like they're being incentivized to to cut cost there's a a sneaky non-helpful answer which i will give you're welcome which is you could try and simplify the system so you have these three different and large parts of the system and this will probably take more work so if you're already running lean like you got to figure out a way to do this while keeping these up and running but right you could try to increase the leverage of those people so that i don't know maybe you combine some systems or you make them easier to operate or easier to understand or make changes to or something like something to make your team scale their efforts more to scale their

efforts more like hey be smarter and go faster all right yeah good talk you tried just being better at your job no but i mean there's presumably some amount of maintenance that goes into these three systems if you can make that smaller then you have more time to do other things or or more time to do stuff that you can't get to right now because you're just stuck maintaining them there's maybe some incidents that happen maybe you can reduce the rate of those i mean there's stuff you can do to simplify systems it's just yeah do you have the the resources to do that do you have enough time to do that so simplifying the system you're saying is a means to an end and the end is that you can spend more time understanding the neighboring

system that you don't currently know anything about? I think so. Yeah. Like, man, I can't think of a way to do this without just butchering the bus factor metaphor. All buses are not created equal. All, that's not it. Cause the bus is the thing where you leave the project. You don't need a whole person for every project. If the project is kind of self-sustaining, well-designed enough for it. Yeah. Yeah. Or easy enough to operate, then you might still be spread thin, but not as thin, I guess. It doesn't take as much of you on, on each project. That's a good point. And then There are also practices that you can do to mitigate the risk of the bus factor, which I'll just name one very basic one that I kind of thought was becoming a de facto standard, but that's code reviews. So if there's individual engineers who own certain parts of the system, they should still have to get their code changes signed off by neighboring engineers.

We don't have time. The bus is coming. We don't have time to do code reviews. Yeah, that's the danger with just do more things to dig yourself out of this hole is that often you only realize you're in this state when you're so overwhelmed that you can't keep up with just keeping things running. And to say, well, add all these practices or take this big refactoring project on or redesign project on in addition to all you're doing is hard to juggle. Absolutely. In fact, every idea I'm having, like, for example, design reviews with each other where you review each other's design, maybe architecture discussions to just deep dive into each other's architecture. every every idea i'm having is just going to take you away from the day-to-day work which you're saying yeah it all costs time yeah it's basically not an option but i'm saying you can either take that time now and slowly meter it out under non-pressure circumstances or you can wait for

the bus and then you're going to have to figure it out anyway very fast sometimes you get hit by the bus sometimes you get on the bus that's right yes so i mean the question is what would you do if you were an IC team leader manager for such a team? And to me, the answer is, there's not like one thing where you do it and then you go, all done, it's fixed. Instead, you have to bake systems into your day-to-day work so that knowledge sharing happens automatically.

And the way you do that, if you can't afford to actually have multiple people working on one system at the same time, then the way you do that is through peer reviews of designs, architecture, and code review. And I just really can't think of another way to do that other than spending your nights and weekends reading your peers code for fun.

I mean, you could think about maybe like relaxing your SLAs. So, no, I'm serious. Like, if you don't have time to maintain the same level of operations or feature development or something and also fix the problem, then you will never fix it. So, at some point, you have to say, well, just stop doing this other thing that seems really important but is less important than the future of these projects. and it'll cause this cost, but good thing we can exactly estimate the cost and benefit trade-off. Yeah, I mean, because our estimates are so good,

you can just write down an estimate of zero cost. That's true. And then tell the business, look, I'm going to give you something for nothing. And they'll be like, great, I love it. And in turn, we will give you something. I mean, yeah, have we answered the question? I think so.

I think there's one more aspect of it, which is how do you convince your team to buy into this idea when they're already super frazzled and barely able to keep up with the work. And I think you kind of subtly have to tell the team, look, we can't afford to run at this pace because we're running in a very high-risk scenario. But then you have to tell the business some way of quantifying that risk

in a language that they understand, which is usually dollars or delays. Dollars and delays speak very loudly. And if you say something like, hey, remember that customer we wanted to sign up and we were able to crank out the code in two months? Well, we can't afford to operate that way anymore because if this bus factor happens, we're going to go two months without being able to deliver anything.

And do you want to say that to your next potential client? I think the convincing the team part, I bet the team feels it, right? Like it's pretty common to feel overwhelmed and like you can't keep up and you wish stuff was better or easier or you knew more.

And so I think if you told the team, hey, we're going to invest more in making life less hectic and making these things easier to own, I think they'd be on your side with that. It's the business part that's trickier. that's an age-old problem of how do you sell like kind of engineering improvements to business exactly hey you know what i want i want you to pay me the same and i'm going to give you less

deal but i pinky promise that later on you will get more yeah or rather later on you will not have gotten even less i will prevent bad things i mean disaster avoidance is always harder to sell than disaster recovery exactly and that's where you are you got to stage a disaster that you narrowly avoid by right by doing this oh a false flag how about resigning but it's a fake resignation psych you're welcome that was me helping you boy wasn't that scary we should probably make some changes to our engineering practices they're like yeah starting with you all right now i think we've answered it shall we uh move on to our next question Yeah, I'll read the next one. This is from an anonymous listener. I'm a developer with one and a half years of experience and was put on a greenfield project to rapidly develop a new application. We have a contractor that came on board to help with the process. On the very first day of meeting this person, I noticed their propensity to not allow anyone else to talk and to interrupt. Fast forward several months and this person has really become a micromanager.

they're requesting the source files from our ui contractor they got another person kicked off the project because they didn't like the changes they were making interfering in their development process they have constantly hoarded all the real dev work and work frequently until 9 p.m i voiced my concerns to the pm mainly about the bus factor since laughs are likely coming and this person likely won't be converted i assume from contractor to full-time at this point i am just tuning out on the project i do the scrap issues the contractor basically doesn't want but I am seeking learning opportunities elsewhere within the company and have nearly zero interest in the project, which I see as a ticking time bomb. What would you recommend? I could potentially

escalate the issue to the manager of our team, but I basically see working with this individual as toxic and the PM as autopiloting to the finish line. Ooh, another bus mention. That's a transportation metaphor. We have autopilot too. Autopilot, we got airplanes and buses. that's a feature autopilot some people pay lots of money for that yeah getting it for free yeah thanks to this the contributions of your toxic contractor huh this is a hard one i think that the one point in here that makes it really complicated is the fact that this contractor is working crazy insane hours it's one thing if you have a team member who's you know working short hours delivering crap and not getting things done it's a totally different thing when you have

a contractor who's highly effective and works tons of hours because management likes one of these people you know yeah and if you're like well i want to try to get this person removed from the team so we can all work better that's a little harder to do it could also be it might not even be that they work lots of hours maybe they just work a shifted schedule oh it could be where they start work later and then they work later and that's fine if you're collaborating well but if you're not collaborating well then and it sounds like the contractor has more experience overall than this person with only one one and a half years of experience yeah so i can see that being tough if they just kind of forge ahead on their own and you don't really know what their plan is

and you can't kind of predict and work around their weird schedule and they're hard to communicate with in general that would be tough do you think this bus factor mention was a thinly veiled threat against the contractor's life sorry they took up your startup idea yeah hey can i borrow your bus for a minute thanks demonstrating the bus factor as a service as a platform oh man so in the question the the listener calls this person a micromanager right did i read that right yeah i don't really see the behaviors here as micromanagement necessarily yeah maybe that part's not described i can kind of see it where they they don't let anyone else talk they kind of dominate the direction and design they sort of like hoard

the issues and dole out specific ones i can see that coming across as micromanaging because if you try to do something bigger they might jump in and say well you have to do it this exact way that i would do it i don't know it feels a little connected but they're not your manager but if they're more experienced than you then you can still be micromanaged it turns out yeah yeah for sure we should make a word for someone who's not your manager but they micromanage you anyway micro bug micro annoy you micro irritant yeah what do you think is the worst case scenario here if you just kind of finish the project i mean you get sad that's a possibility yeah if layoffs are coming then it's

possible that the project will not go very well and you might be impacted while we're talking worst case that's a really good point like if layoffs come and your portfolio so to speak is just full of these kind of crappy little jobs side tasks that are unimportant that might put a bullseye on your back yeah that feels like the biggest risk yeah i agree i was thinking like before the layoff comment i was thinking well just kind of tough it out it's a contract yeah be done soon but and one of the benefits of a contractor is usually they're pretty transactional by definition you can just say hey your contract has ended or i mean sometimes they have fixed dates but often there's a possibility of changing it or something like that right it's much easier

to end a work relationship with a contractor than with a full-time employee yeah and and like he like the question asker said that that's probably going to happen here yeah and so maybe i don't know but the question is will you also get removed yeah i'm seeking learning opportunities elsewhere in the company and have nearly zero interest in the project which i see as a ticking time bomb so it sounds like they're already doing some work to kind of expand their network and show value elsewhere which makes sense to me and feels like a reasonable thing to do yeah that's quite an issue to the manager of our team yeah i think you should do that good idea you should tell your manager I was assuming the PM was the manager, but maybe that's someone else.

I think it's someone else. So you voiced the concerns to the PM, but that was maybe the wrong person to have the conversation with. Yeah. There is a reason that your PM is not super concerned by what you've raised. And it would be good to get that context from them and say, okay, I've raised some concerns. They don't seem to be concerns to you.

Can you explain why? And a lot of times people can. So I'll tell you, there may be some factors here going on that are outside of your understanding or your knowledge, just because it's outside the lens that you see the world through right now. And I'll tell you that a lot of times as a leader, people will come to you with concerns, but you can see the bigger picture and the concerns they have are actually not real concerns. And I think one of the things that you have to do to develop good judgment is understanding when your concerns, which feel very real, are actually not real concerns. They aren't as intense as your

feelings would would indicate that happens you know and i had once had a manager say that to me and i think i had maybe it was because i just finished a gripe session with him but he told me you know dave sometimes i feel really concerned about something and then i let some time go by and i realized it wasn't actually that much to be concerned about and i'm like yeah and then you know like four days later i'm like ah he was talking about me got it weird i don't know what has to do with what i just told you but uh yeah whatever i guess i'll file that away so wait a minute and and this is this is a very tricky thing for me to say because sometimes you have concerns and they are very real and everyone around you is trying to convince you that they're

not real yeah so that's also hard and that's all part of developing good judgment is experiencing enough of these situations to say which ones are real which ones are fake yeah so i could see the PM not being concerned if they feel like sure sucks for your job but the project seems to be moving okay you know like things are getting done and it broadly seems like it's gonna deliver what it needs to on time so maybe they're right or maybe they don't see the the cost of the contractor's technical decisions or information hoarding or something like that but yeah hopefully they're looking at it's not shocking to me that the PM is not thinking about like what it's like to work with a contractor because that's not really their job. It'd be cool if they cared

about that because that'd just be a nice human thing to do, but that's sort of your manager's job. And the PM is more about, is the project getting done? That's a very good point. It's kind of not their job. Also, I would say, the listener mentions 1.5 years of experience. I'm going to assume that was all in this one company. And I'll say at 1.5 years, life's too short to live in an environment like this. And there should be plenty of other opportunities for you. and i would look into those and we haven't said this in a while but this might be a good place to say quit your job to walk away like a cool person from the explosion slow-mo walk yeah okay i like it all right have we answered the question i think so good luck sounds painful i'm sorry

quit your job slow motion explosions don't hurt though that's i've verified that by watching a lot of movies they just don't they don't get you so you should be okay sometimes they might make your hair kind of flop around a little but that's it yeah okay what can people do if they want their own questions answered go to soft skills.audio and click ask a question where you can fill out our form thank you so much to everyone who has done that we really appreciate it you keep the show going and if you want to support the show financially go to that same website and click support us on patreon you can make a one dollar donation a million dollar donation either way you're going to get access to our slack community and we would love to see you there all right we'll

catch you next week