Soft Skills Engineering - Episode 353: Easter outage and unethical things

Episode Date: April 24, 2023

In this episode, Dave and Jamison answer these questions: I work for a startup with a distributed team. Recently one of our clients experienced a production outage. As a small startup, we do not have an on-call rotation, and teams usually resolve issues during business hours. However, during this particular incident, most of my colleagues were on annual leave due to an Easter break, leaving only 10 out of 70 engineers available to assist. Although none of these 10 engineers were part of the team responsible for the outage, I was familiar with their codebase and knew how to fix the problem. Additionally, I had admin access to our source control system which allowed me to merge the changes required to resolve the issue. This was the first time I had done this, but my changes were successful and the problem was resolved. Now that the break is over, the team responsible for the codebase is blaming me for breaking the process that requires each pull request to have at least one approval and for making changes to “their” codebase without their approval. They want to revoke admin access from everyone as a result. However, I disagree with their assessment. While it is true that I made changes to a codebase that was not directly under my responsibility, I was the only engineer available who could resolve the issue at the time. I believe that helping our clients should be the priority, even if it means bending the rules occasionally. Did I make a mistake by making changes to a codebase that was owned by another team without their approval? Should I have refrained from getting involved in the issue and adopted a “not my problem” attitude since the responsible team was not available? Thanks and I hope I’m not getting fired for helping a paying client! J Dot Dev asks, ‌ What’s the worst thing you’ve had to do as a software engineer with direction from your employer? Years ago at a webdev shop we had a client who didn’t want to pay for e-commerce set up. My boss’ solution was to implement a form that included name, address, and credit card information fields that we would read on form submission and then email all of that information to our client in plain text. “Is that really ok?” I asked my boss. “Why wouldn’t it be?” “Isn’t that insecure?” “Only if they have her password. Just make it work so we can be done with them.” To top it off, they also had me email the information to myself just in case the email didn’t go through to the client or in case they accidentally deleted it, so I’d have all of this information just hitting my inbox.

it takes more than piping curl to sh and hoping for the best to be a software engineer this is episode 353 of the soft skills engineering podcast i'm your host jameson dance i'm your host dave smith soft skills engineering is your weekly advice show about all the non-technical things that go into the technical field of software development like how to admit to your co-workers that you uh accidentally gave someone your your your aws authentication keys when you piped curl to sh trying to install some cool new tool yep i was thinking curl to sh is not bad but curl to pseudo sh is like asking for trouble i have seen some of them start to say like hey read this first before you do that which is which i'm sure you did right yes i do usually now i don't but boy

have i not a lot of times and always if i read them too there's always some like okay go download this thing and execute it i'm like i don't go read that thing yeah exactly exactly i can't be reading two things who has time for that that's not what this show's about nope um dave do you want to thank our patrons yes i do we have a one-time shout out for soundstack engineering hi sam and weekly shout outs for thecomputersciencebook.com kyle boss connie lee valentin at datafold Santa Hopar Noah Frazier-Logue Kent C. Dodds

Jenny Kim Owen Charlton Craig Motlin I Love Mavis The Stochastic Parrot Alice Jost At least we no longer have that awful name

Flocheni Cooley Philippification is common and Lanfair Twillquin Glick Go go go

Oh Schwindrodrob Junaiden Hutton, Ohio I see I looked it up Patreon.com.au We're hiring Ira Chan

Monkeyface Emoji Jonathan King Testingisdocumenting.org Oladapafadiyi Willangel Ragnar Nick Hathaway Travis braden canes nora trans rights codes john grant bartek takowski code of sale nick cantar philip

john basile if you would like to join this illustrious crew of great smelling and generous patrons go to soft skills.audio and click the support us on patreon button where you can pick any dollar amount and submit it and we will give you an invitation to our slack community and if you give us enough money we will say or try to pronounce almost anything even if it embarrasses us especially if it embarrasses us we'll try extra hard to pronounce it yeah so i feel like the name of that place in wales has changed i don't think this is actually the name because i thought it ended in go go go i think it got cut short here earned a rob oh okay it hits some that's only that's only like half the letters in that name well guess you'll have to sponsor

the show twice to get the other half of the i wonder if they hit a character limit so there is a limit to what we will say and it is imposed by some uh undoubtedly some relational database schema owned by patreon imposes a limit we'll go see about increasing that do you want to read our no no do you want me to read our first question that's exactly what i want you to do in this moment okay yeah i almost made a fatal error all right this is from an anonymous listener who says i work for a startup with a distributed team recently one of our clients experienced a production outage as a small startup we do not have an on-call rotation and teams usually resolve issues during business hours however during this particular incident most of my colleagues were on

annual leave due to the easter break leaving only 10 out of 70 engineers available to assist none of these 10 engineers were part of the team responsible for the outage however i was familiar with their code base and knew how to fix the problem additionally i had admin access to our source control system which allowed me to merge the changes required to resolve the issue this was the first time i had done this but my changes were successful and the problem was resolved now that the break is over the team responsible for the code base is blaming me for breaking the process that requires each pull request to have at least one approval and for making changes to quote their code base without their approval oh they want to revoke admin access for everyone as

a result however i disagree with their assessment while it is true i made these changes to a code base that was not not directly under my responsibility i was the only engineer available who could resolve the issue at the time wow i believe that helping our clients should be the priority even it means bending the rules occasionally did i make a mistake by making changes to a code base that was owned by another team without their approval should i have refrained from getting involved in the issue and adopted a not my problem attitude since the responsible team was not available thanks and i hope i'm not getting fired for helping a paying client yikes oh man crazy this it this almost feels like i don't know it kind of feels like someone is uh

trying to let me put it this way have you ever noticed that in the news cycle sometimes something really bad will happen but then something will happen shortly thereafter that just captures the news media's attention and everyone forgets about the first thing yeah this feels like that yeah let's not have a discussion about a retrospective about why we didn't leave anyone watching the store let's instead have a discussion about source code access yeah let's get down to the real problem here which is someone didn't get approval from us before fixing the problem yeah clearly clearly that was the uh the major issue here that needs our attention yeah this is i mean there's there's two levels to this one is like are you crazy no you are not crazy you are absolutely

right and you 100 did the right thing yeah and and they are 100 wrong for being upset at you about it that doesn't make them not upset though yeah and there is some like negotiate i don't know navigation you need to do because they're feeling defensive about this i think and and yeah uh being right will not necessarily make them not do this dumb thing that they want to do since when since when has being right ever really mattered yeah when it comes to doing dumb things but but just just as a sanity check like yes you did the thing you are supposed to do yeah and if i owned this company i would be singing your praises for not letting a process stand in the way of success yeah and also i'd be having words with the team of who left their system in such a

state that it broke over a holiday weekend and there was no one around to notice one other weird thing where a small startup do not have on-call rotations you said 70 engineers that feels like a not a small startup and certainly not small enough that like oh we just don't have on-call rotations yes i i would expect an engineering department of that size to have on-call rotations this is not small this is not a mega huge company but it is not by any means small in my book yeah is this like a european thing i don't i don't think the listener said where they're from but i wonder if this is a like when you're when you're out on vacation you're really out on vacation and and being on call while you're out is is not an option yeah like

a cultural norm there or something it might be i um i mean i'm latching on to one key word here which is easter holiday easter break that is not a united states phenomenon we don't usually have uh breaks or or holiday days for easter but in london i know they do and it is a weekend kind of thing like it's a multi-day break so i think so i think you're probably looking at london here look at us detectives putting this together with the clues yeah i think incident response is fascinating i like it i like thinking about it and talking about it and learning about it and i have ideas about what you could do here but again that doesn't address the problem of this team being defensive about someone touching their thing like they lost the right to be

defensive and grumpy about people not following their process when they let their thing break and left no way of fixing it for anybody without without breaking the process yeah if the process keeps your thing from your service from being fixed then you go around it then you have the wrong process and that should be what yeah discussed here but what do you do what if you're the person that fixed it and now you're sort of in trouble or they're trying to position you as being in trouble uh and then trying to pivot the discussion about how do we revoke access from all these all these uh rebels coming in here breaking our processes so many things here this team should have a post-mortem for why their system broke and maybe they are and that's just not talked about

but yeah like you said that should be the focus here why did our system break how can we learn and improve based on that and if that's happening and this is also kind of like a side thing then then that seems less egregious to me but i think it is if no one is saying hey let's have a post-mortem yeah you should say that even if it's yet more meddling with this other team this team needs to be someone has to yeah yeah exactly the first thing this team needs is a little meddling to make it a successful meddling where you actually get the outcome you want and you don't just end up being in trouble you might want to recruit some people with clout in your organization to get on your side bring the story to them explain to them what happened make sure they

know that you're clear that you broke the process but that you felt that the need trumped the process in the moment and so i'm sure they'll be on your side and you know if you find anyone reasonable in this organization who's not a member of the team whose process was breached and who neglected to keep someone on staff during the break go find someone who is in that position and get them on your side somehow like and maybe maybe there's someone in your organization who's in a position to demand a retrospective find them work with them and say hey i'm happy to help i was there i'm sure you'd like my input on the things i did so i can fill in the details on some of the things that happened while the rest of the team was away you know but always in a very supportive positive

like i want the best for the company and the best for our customers attitude and i think i think that's how you can kind of build up a little defense layer against the these weird attacks that are actually defenses i could see folks having genuine concerns about a lot of engineers having admin access to your source code or your version control system because i mean potentially you have kind of like checks and and i don't know stuff to make sure that code that is merged doesn't break some build process or something and often admin access lets you bypass all that stuff to say no just slam it into main and and so so it opens the the cicd process up to to working around it if admin access is common i think one thing you could propose is like a break glass solution

i often hear this talked about where it's it's not like every engineer or many engineers have admin access but there is a way to get admin access if you need it in emergencies and for this exact reason like no one is around to approve the pull request the the system prevents the pull request from being merged without an approval we need to do it anyway so we need some way to work around it and like if if they want to revoke your admin access i could see that feeling bad but really like what do you need it for if the thing you need it for is situations like this i feel like this break glass process to say there is a way for for senior engineers who will be around to get this when they need it might address the need to have a workaround without just telling

them like nope i'm gonna keep my admin access um i think their motivations are suspect here but yeah in general it's probably better to have fewer people have admin access to this system I mean, yes, it makes sense. It also depends on the level of checks that they have in their deployment process. You know, if they've got good unit test coverage and whatnot. I don't know, why wouldn't you allow other people to push? But maybe not. I mean, I've worked at companies that are very large, thousands of engineering teams, and they did have permissions that would prevent other users from actually merging, for example, pull requests into their repository that would then go into the deployment flow. So that's normal. another normal thing though is is there's an emergency path like you have your normal path and permissions and processes and tooling and stuff to make sure that things happen in regular predictable order but if stuff is broken you have to be able to do things yeah and and so it's pretty common to have some way to get elevated permissions on on uh like your cloud provider or yeah like like bust through the red tape because often that's useful so so i think what i'm suggesting

is i don't think you should admit any fault or guilt it sounds like they're trying to blame you to say you did the wrong thing you merged without our approval i would push back really hard on that and say your system would still be down and that's worse than merging without your approval so i did you a favor by by i saved your butts while you were all on vacation yes yeah like imagine if you would let it sit for four days broken yeah now what yeah then they'd probably be saying well you had admin access i mean we're okay with 95 uptime all right buddy yeah um everybody knows that websites go down for easter yeah in observance of the easter holiday our computers will not be working frankly i'm okay with having a admin or with having a uh restricted access to a team's

source control system like this provided that they also have a hundred percent on-call coverage for that system yeah you know if you're going to rely on other people to come in and fix your stuff when you're not around which also is a viable strategy you just better make sure those people have access to actually do stuff and not get stuck yeah so if you're gonna block out access to the only people who fix who are available to fix your system then you're going to have more failures yeah so to me it's like okay it's fine to do this but it has to be two prong you can do the admin restrictions fine but you also have to now set up a process whereby you have coverage on this system 24 7 yeah i think it also might be useful to go back to shared goals here because this does feel kind of

confrontational and like they're trying to maybe deflect blame or feeling guilty or defensive hopefully you have an underlying shared goal of wanting your systems to work so that your customers can use them and they're focused on this like intrusion into their code base because it offends some idea of ownership maybe, or makes them feel guilty that their stuff broke and someone else had to fix it. But if you focus on the shared desire to make systems work and say like, this is what I was doing. I was trying to make the system work. You were trying to make the system work as well. And let's kind of go back to that common point that might help diffuse some of the conflict or tension here to speak productively about it. Yeah, I agree. It would be very easy

to stoop to the level of the people that are accusing you of wrongdoing by accusing them correctly of the wrongdoing that they are doing. But that's not going to be the productive way out of this because that just turns into an endless cycle of ego preservation and prideful deflection.

Instead, let's focus on the common goal here, which I think they all share and they know they screwed up. And you can make sure their management knows they screw up through other means, but you don't have to make them admit it out loud. There's no value in that. I hope I'm not getting fired for helping a paying client.

That would suck. If that happens, then you're better off to find a new company anyway, because any company that would fire you for this, that's a really bad company to work for. Yeah. Well, have we answered the question? I think so. Good luck.

Tricky situation. Honestly, I would love to hear how this goes. If you could write in with a follow-up question later, I think it'd be very interesting to hear how this team reacts. This team that is obviously very touchy and willing to say and do some kind of silly things to defend their turf. i'd love to hear how they react when you bring it to their attention if you defend your turf that hard your turf better be immaculate it's not that immaculate cars up on cinder blocks and stuff

well i'm afraid i'm afraid that the the indication that they do have cars on cinder blocks is yeah the fact that their system went down when they weren't looking all right do you want to read our next question i do this comes from j.dev who asks what's the worst thing you've had to do as a software engineer with direction from your employer years ago at a web dev shop we had a client who didn't want to pay for e-commerce setup my boss's solution was to implement a form that included name address and credit card information fields that we would read on form submission and then email all of that information to our client in plain text is that really okay i asked my boss why shouldn't it be my boss said isn't that insecure i said only if they have her password

just make it work to top it off they also had me email the information to myself just in case the email didn't go through to the client or in case they accidentally deleted it so i'd have all of this information just hitting my inbox oh having worked somewhere that cared a lot about pci the standard for handling credit card data this just makes me cringe so hard you mean the pci spec doesn't say that email is a an acceptable means of transmission of credit card information i mean i didn't read it saying that so i guess maybe it does somewhere in the parts i haven't read yet it's like 500 pages or something but uh the part i did read said five hundred thousand dollars per violation so it's a lot of how much money do you think it costs to send an email

like i don't know it's probably like fractions of a cent right except for these emails which were potentially 500 grand a piece yeah exactly oh boy yeah this is insane but at the same time i didn't i there was a time in my career where i did not know this i'm trying to think if there was a time in my career where i didn't know not to send a credit card number in an email or at any point in my life. I think I sprung from the womb knowing this. That's something your parents instilled in you from a very young age.

In utero, they read me stories about engineers. They read you the PCI standard. Now, young David, always remember. I mean, I think I knew credit cards were a big deal because the places I worked always had some kind of vendor or library to handle this stuff. And I knew we didn't touch it because it was kind of scary. Yeah, you're like, we didn't want to. We didn't want to build that. Yeah. It's like building your own encryption layer. Yeah. Boy, do I know now. Yeah. Only if they have her password. What is that supposed to mean?

Like her email password. That's what I assume. Oh, only if they can access her email. Well, her email is password protected. Don't worry. No one will ever see these emails. except the owner of the emails yeah imagine okay imagine you think you're checking out somewhere and you type your credit card information into a form i feel like you'd be able to tell like wait this didn't do like email sent is like the the validation message you get when you click buy or something imagine that sinking feeling you'd have like oh email sent where don't worry we've emailed your payment information to our our payment processor her name is emily yes she will she will process your payment shortly

actually i'll bet you a lot of people just wouldn't think anything of that honestly but the developers would be like digging their fingernails into their desktop so hard they'd leave trenches but everyone else would be like oh good i'm glad i'm glad she's taking care of it that's nice i wish more websites would tell me the name of my credit card processor yeah so this was a web dev shop yeah which i imagine it it sounds like a probably a relatively low cost contract or engagement or whatever the client didn't want to pay for e-commerce setup maybe a thing was already signed and a budget was already agreed to and i assume the people who who did that business side of it didn't really have an understanding of how much

work it is to handle credit card data securely what would you do if you were this developer and someone was like please email the credit card information to to our clients i could give multiple answers to that question depending on how many years of experience i had at the time yeah as a young whippersnapper i would have refused on moral grounds i probably would have called my employer some rude names and then they would have asked someone else to do it i wonder if there's like a like a whistleblower fee the finders fee or something there probably is you're a portum get a fraction of that 500 grand yeah no it's like well what do you do in that case well i think what you do is you wait a few weeks for the emails to pile up in your inbox and

then you bulk forward all of them to whatever regulator regulation body enforces the rules around dci and then say i'm entitled to 10 of the fee i think as a as a more senior engineer I feel like it is our responsibility to inform our employers about the risks they are taking on with their technical decisions. I mean, just today I was in a meeting where I was informing our business about some risks. And so I feel like it's my duty to say, here's the likelihood of this risk coming to be. And if it comes to be, here will be the cost of the business. And here are the costs to mitigate this risk. You know, like as an engineer, that is a major part of your job.

And in this case, it's like, look, the risk is very high. But the business may choose to accept that risk. I mean, this would be an insane risk to accept. $500,000 per form submission is like, no one would accept that unless they just really wanted to give the government some money, I guess. Also, I mean, a big part of the PCI penalty stuff is you can lose your certification or like you're not PCI compliant anymore. Which means you're not. Which would not be a problem in this case because. You already weren't.

They're not. yeah like are you i mean i don't want to ask you to answer terrestrial law questions but are you are you technically not allowed to receive credit card information or request credit card information if you're not pci compliant i don't know it's a good question i don't know i'm pretty confident that the card vendors would somehow block you as a merchant if if they found out you were risking their customers uh yeah like i think there's some vendors somewhere in this chain that has to accept their credit card information. And that's probably the thing that has the certification

and they might lose it. So bad for them. Anyway, this is a really tough situation, but I would not build this. But there are kind of two ways to not build something. One way is you just refuse, stomp your feet, fold your arms and bite your lip and say no. But the other way is to clearly articulate to the business

why this would be a business destroying technical decision. and then stomp your feet fold your arms and bite your lip say no yeah and i think i would do the latter because like it's just under and there's no circumstance under which i would build this and uh i would want to make sure that the business also chooses not to have someone else build this by clearly communicating the risks honestly the fact that this is tied to money makes it easier than if it were something more ethical and less directly financial exactly because you can say this is unethical and also it has these huge right yeah business destroying financial consequences that are very directly linked to it not just like people might find out and then and shame you or

something revealed for for how evil we are but like right no it will cost all of our money right and then some for one of these emails exactly i mean i'm assuming this contract is is going to be like the thousands of dollars level of of revenue uh yeah if they were not willing to pay for an e-commerce site, I'm going to guess it was well under 500 grand. But yeah, you're right. And I think this makes it easy because there's teeth behind the wrong decision. Whereas if your employer asks you to, I don't know, take advantage of a loophole to not technically defraud, but still defraud a bunch of people, that one's harder to say no to because it's like, I don't know if there's regulation on this. I just don't like it ethically. Have you ever been in a situation like

that i mean not that i know of maybe maybe looking back i was and i just said oh seems fine sure thing one one sci-fi horror tracking system coming up i guess it's easy to always do right oh we want to screen our our customers for depression and kick the ones that are depressed out because they'll be less likely to leave us good reviews sounds good here it comes this is gonna have a major positive impact on our app store ratings yeah great sounds avoid all those sad people you know i don't think i can think of any examples yeah i can think of an example where someone told me i was asking them to do something unethical and i disagreed very very strongly i would love to hear

this yeah so this was at when i worked at a giant megatech co and we were building internal tooling for the engineering team of that giant megatech co okay and so everyone that uses this is an employee all right all the stuff in the tool is is like non-private data it's like configuration for for load balancing basically all right um like configuration ui for load balancing and i wanted us to add an impersonation feature to the the ui so we could say like log in pretend like you're you're this person and see what they see and take actions yeah um and and had designs that made it so like we would log the fact that you were being impersonated so you could tell someone wouldn't be able to kind of go destroy stuff and then say you did it and i i had someone

who who absolutely refused to build it because we were not asking our users for informed consent to impersonate them and but the users were employees of the company they're employees of the company yeah so they don't have that right yeah and then i just built it instead see and and that's because the other person didn't make a convincing enough case to you yeah and i still disagree with them i mean i i think if this were a publicly used tool or yeah if if if the people were not employees working with like configuration data that had nothing sensitive then i could see it but i just really disagreed strongly that there's this moral obligation to inform and request consent yeah boy i've got some i've got some

bad news about that employee's email and slack messages yeah oh boy uh it was super useful too it was very helpful it made our system way more robust i found all kinds of i thought you were gonna say it was super useful i was able to blame other people for mistakes all the time thanks to this impersonation feature that never happened but we did find a lot of broken stuff because we used it also, but we had different permissions and access and stuff. So we just didn't really see what the average user saw. Well, I have never been in this situation where I've been pushed to do something with engineering decisions that was highly unethical. The closest I've come is I had a customer, which is a big organization. One of my customers in that organization told me

to lie to another member of the same organization about whether I was going to be in the office on a certain day because they did not want that other team which they felt was kind of an internal competitor with them uh did not want that team to have access to me what holy cow they were very protective of their turf and then when that team approached me and said hey can we get some of your time this friday do you want to know what i said to them yes i did that's exactly what i said i just said yes you said yes yeah i just couldn't i couldn't lie about it i mean it was like and then i told my other customer later and they were just so pissed off they were like i told you not to give not to talk to them i was like sorry not gonna i didn't say this out loud but i'm like i'm

not gonna lie for you not because not because i felt like it was a huge moral quandary or really you know like a soul-destroying ethical decision but rather i just didn't want to lie about it plus there was nothing in it for me there was no money he didn't bribe me at all i'm like look you want me to do something like that you got to pay me a lot of money way more than zero i mean look at our patreon feed we'll say a lot of stuff for a little money yeah someone someone could very easily put the word no into the text field we would say it we'll say it for the right amount and they could also put right before it a different thing that's a question we'll say no to any question you put in i guess technically i'm not

sure how they get ordered though so maybe there's no way to guarantee that the question comes before the no yeah you just got to sponsor more times if 99 of the patreon supporters are your question then it's pretty likely that the no will be right after the question that's true it's for enough money all right well did we answer this one yeah we've answered the question i think i'm glad this was a long time ago and sounds like you you are older and wiser now yes and hopefully your boss is too or not in charge of stuff anymore yeah well what can people do they would like their own questions answered dave go to soft skills.audio and click the ask a question button thank you so much to everyone who has done that we really appreciate all your questions we love

them we read them we nurture them we water and feed them and we look forward to more of them we will catch you next week you