Soft Skills Engineering - Episode 64: Negative Peer Reviews and On Call

Episode Date: June 15, 2017

Jamison and Dave talk about these questions: How direct should I be in a peer review of a coworker who I really dislike? How do I convince developers to go on call?

It takes more than great code to be a great engineer. This is episode 64 of the Soft Skills Engineering Podcast. I am your host, Jameson Dance. I'm your host, Dave Smith. And this is the podcast where we answer your non-technical questions about technical fields like software development. Sorry, I was just doing some mental math because this is the sixth Power of Two episode.

It sure is. Do we just take the episode off then? Is this the end? Yeah, that's right. All right, and that was our show. to celebrate the sixth power of two it's going to be a lot more episodes until we get to the next power of two in fact it's going to be the exact same number we've done so far you're approaching hard skills yep i'm flirting with danger yeah uh you should flirt with soft

skills and the stuff you're gonna say i think flirting is a soft skill actually not one that i recommend using in an engineering context yeah i've been married for like 10 years so i don't i don't know what that means anymore okay flirting is doing the dishes when you're married that's yeah it's okay all right so uh i had a little bit i wanted to share with someone this last week i had a chance to pay someone a compliment at work and i realized that i haven't gone out of my way to do this in a while and i just wanted to offer a little reminder to our listeners that if you see someone do something good take a moment pull them aside and tell them that you were impressed and that you thought what they did was good and the effect of this in my experience has been

increased trust better working relationship and everyone just feels good so there you go little soft skill tip of the day thank you what an excellent tip dave oh thanks that makes me feel good to hear you say you always have the best excellent tips um we also have a a comment from a listener so this is from klaus and he says you answered my question about how to approach a salary raise talk just as a follow-up it worked more or less as expected with a double digit raise without much ceremony i appreciate you taking the time to give some valuable tips yeah good job um i'm going to say that's more down to you than to us but we will happily take 10 yeah yeah this is just a small commission yeah as an extra i would recommend the book fearless salary negotiation

for the situation um it's not the most thorough and not well written yet it covers more or less everything i needed a strong lukewarm endorsement i haven't heard that book i will check it out i haven't i haven't read any books about negotiation not thorough and not well written but don't worry it's good you drive a hard bargain but i accept i will buy this book uh thank you for the feedback klaus i'm glad that it worked out well for you yeah congratulations okay i think uh i think i'll read our first question yeah this comes from an anonymous listener uh and it is directed to the point it says how direct should i be in a peer review of a co-worker that i hate to work with can i just say please fire this guy he's a jerk thanks

if you do that i would love to hear what happens i haven't thought about that what if we pivoted the show instead of trying to give good advice what if we tried to give the most interesting advice and most interesting for us personally right like most entertaining how do you know i haven't already done this my my advice that you thought was good has actually been fake the whole time i'm just i just really suck at giving bad advice that's what it comes down to oh so i assume this is a 360 review a which is a review where peers give feedback on a peer not directly to the peer they usually give it to a manager and then the manager collects it and

then shares the hopefully anonymized feedback with the peer and then fires the peer and then fires them if enough people vote them off the island um yeah survivor style have you ever done this jameson i have not been part of anonymous 360 reviews at my last full-time job we did um they were like deep heart to heart sessions that i do not think would work well with every team but but it was very like honest and open in person every quarter we'd sit down we would go around the whole room and everyone would give feedback uh on on the person whose turn it was and it had you had to have constructive and and positive feedback um and at first it was like i think sometimes you you tie your shoes a little too slow it was like fake made up constructive feedback

because everyone was scared of offending people but after a while it got what would appear from the outside to be uh intensely honest but it was it was great just because of the trust the team had in each other but it wasn't filtered through another person it was just direct face to face face to face that's really hard to do yeah yeah i i don't know that i will see that work again in my lifetime but it was it was interesting to be a part of i think you're a jerk and you should be fired it the the people that did it all had to have a lot of emotional intelligence to avoid being uh mean but still being usefully honest right right and it it worked out well for the most part but that's a long answer that could have been so let me ask you this

no i have not done this the classic 360 review have you ever suggested to management that someone should be fired because they're a jerk oh man i've gossiped a lot isn't that what a 360 review is like yeah it's codified gossip about you to your boss not solely because they were a jerk but i have talked i've talked about this on the show too i've talked to management about a a fellow engineer who i thought was not working out partially because of personality and and personal interaction issues i i didn't like hate them or and i also didn't even work with them i just saw that that the team was worse off after they joined because of the way they interacted with the team but it wasn't as directly it sounds like this

person might be like you're like sitting across from them in the open office and they have a spittoon that's like accidentally your keyboard or something i don't know it sounds like there's some there's some close yeah negative interactions there yeah i i think so too i how about you dave i have recommended to management that to fire people before uh only three times in 15 years but never because they're a jerk i have a really high jerk tolerance and i don't know what it is but i don't have a problem 64 that's that's proof you've made it this far with me 64 episodes of james improves a very high jerk tolerance but i do like i don't know what it is about me but like when someone's a jerk uh by the way every jerk every jerk engineer that i've known

is also very capable and competent and i seem to like be willing to look the other way on personality problems when they produce high quality engineering output i don't know what it is about me but i do i i know not everyone is like that thank goodness because um it's not like we need more jerks but yeah i've never gone to management and said on the grounds of this person's personality i think they should be let go well time to start yeah i don't know turn over a new leaf yeah there's there's not a lot of detail in here so i just have a lot of questions that will go unanswered because this is not a phone call but i'm wondering what does the person collecting the feedback know already do they know that you don't like this person and then

that could go one of two ways it could be like and it's because they're a jerk or it's because these two people just don't get along yeah and they do good work but they just butt heads a lot um which by the way that absolutely happens oh yeah just just because you don't get along with someone doesn't mean they don't get along with other people quite well yeah it feels awful when that happens by the way when you're part of the recipe for a personality conflict and and you realize well they get along with everyone else what's my problem yeah yeah are there performance problems or is it really just interpersonal conflict those are those are important questions And also, how much will this info that you deliver in the 360 review be filtered?

Right. Is the person who's collecting it going to say, and John Smith said, I hate you and you should be fired? Or are they going to like pull the useful feedback out of that? Because the goal of a 360 review, I believe, is to give feedback to the person being reviewed, not to get them fired. so if your feedback is i hate you and you should be fired um i don't know that there's a way that as a manager i could take that and say like here's a thing you could do differently because john smith hates you and thinks you should be fired here's a suggestion you should quit yeah before it's too late yeah that's a good point a 360 feedback session is supposed to be constructive

i mean the person the reason the company is investing in feedback is because they want to improve the person working and so it's like if your only feedback is they're a jerk uh that's where i think i would go with more specific examples of measurable if possible damage that the person does to the team this is really hard stuff to measure but um you could at least give a qualitative assessment of the outcomes of their actions and specifically identify the actions like when bob threw me through that plate glass window i was unable to type for two days and that hurt my ability to get jira tickets done yeah i mean if people leave the team if people uh have more conflict if if the team works less well together that's all

that's all stuff that is important yeah beyond just i don't like them yeah exactly so i would i would go it's very specific examples like if you can say this person's aggressive personality stymied a design conversation for three days when we could have been done in three hours you know that's that's lost output and that costs money and that's something that i think as a manager i could work with you know i could actually sit down with that person and coach them but uh if you just say they're mean that there's not much to go on there yeah as as part of some reviews i remember there's an engineer who i i really really really like and liked at the time we were working together but some of the feedback was uh sometimes they could be a little negative

and cynical and then we could see that affect other team members uh not not directly it's just like you just see it and then you're a little more cynical and negative too um and that's a different situation because it's not like we hated them and wanted them fired but but it was a a concrete thing to change instead of just the team hates this person you know yeah yeah so if there's some if there's some effect you can pull out of them being a jerk that and you can change the effect then that might be feedback you can give and then you need to decide if you should suck it up or quit because because the i don't get along with them thing is kind of more your problem than yeah the company's problem and and maybe in that case you should ask for some guidance or

coaching yeah but don't ask for them to be fired that sounds like uh you're looking for an easy out yep everyone i don't like gets fired yeah that's called the ceo that's right but even then i mean ceos don't just fire people they don't like good ones don't i think uh the other thing i would say is try to avoid recommending firing the person instead provide evidence that the person is doing significant damage to the team and then let the manager come to the conclusion of firing on their own and i suggest this for two reasons reason one is that when you suggest that a manager fires someone immediately the manager's response is sometimes going to be uh no i'm not going to do that that's hard like the

knee-jerk reaction will be no because that's that's a ton of work it's really hard to do and the person recommending the firing and the person doing the firing have like there's just a huge difference between what has to be done yeah yeah it's like you like drop a note off on someone's desk that's like hey construct this house for me by hand see ya yeah you wrote the note and then they build the house exactly one of these two people had an easy job so the second reason is if you give evidence and let your manager come to the conclusion on their own, it will actually be a more strongly held opinion that the manager comes to. Whereas if you try to force this idea of the action the manager should take into their head, but without

providing concrete substance and reasons, I think that the manager will be more likely to let it go. Does that make sense? Yeah, that does make sense. Anything else to say about this? Yeah, I would say that when you're sitting down providing feedback like this, rather than coming right out and saying, you know so and so did this bad thing i would actually frame it in the form of a question and i would say how harmful do you think it is if someone does x action and it's like the action that you've seen this person do and try to get a read from your manager because you know just because something is terribly frustrating to you doesn't necessarily mean it's harmful to the team and it would be interesting to learn what your manager's view is of it before you jump to the

conclusion of they should be fired and you might find that your manager is like oh wow that absolutely cannot stand you know or you might learn that your manager thinks that's no big deal you know like no i don't care if they deal drugs out of their desk you know i was thinking about that when you mentioned evidence yeah real evidence yeah unmarked bills i've watched a lot of csi what we need to do is set up a camera facing the shiny surface behind them and then zoom in and enhance and check out yeah the mean stuff they're writing on twitter about the boss enhance yeah and then the last thing i would recommend is to take what i call

the three-month challenge this is a cool down period where you you set it aside try not to think about it try not to take any action on it for three months and then look back and say do i still feel as strongly as i did at the beginning and if if three months has the effect of cooling you off then you probably can just let it go but if three months and you still feel just as upset about it then it's probably something significant that you need to take action on i found that when time passes sometimes it i don't know what it does but it like it like dampens the effect in some cases that turn out to be not that important sure that all makes sense i guess that was a long-winded way of saying that if i complained to my boss about everything that

bothers me i would do nothing but complain to my boss but if i put everything on a three-month timer then i will rarely complain to my boss yeah yeah that makes sense and i'll just be really angry inside bottle everything up and i'll blow up problem solved all right we have answered the question i really want to know what you end up doing yes me too very very very interested so please write in and let us know yep all right i'm gonna read our next question until now it's only been ops on call and app developers get to write whatever software they want that passes qa and gets into prod we are moving away from this model and in the next quarter or so need to convince 100 plus engineers

that they are now on call for reasons i can't quite articulate i care deeply about this i'm fighting to get it done right which brings me to my question how do you convince 100 plus engineers to take the pager oh this will be very interesting yeah um have you tried asking them to do more work for the same amount of money hey i would like you to work nights and weekends please and as a as a bonus for that i will continue paying you yeah into the future here's a list of bad things that won't happen to you if you do this um for reasons i can't quite articulate i care deeply about this i don't know what that means either i think it means that we have because you can't articulate them yeah i cared care deeply

about convincing developers that it's the right thing to do instead of just mandating it no i think i think what i took from that is that this listener cares deeply about having a good dev and ops culture some people would call it a devops culture yeah um which is becoming quite popular actually the the old days of developers sitting alone and throwing software over the fence and then leaving it up to some like second class team to operate that software it those days are coming to an end very quickly um so now now that devopsilers they take the software the developers build and then they work with the officers to deploy it to production yeah it's progress the devops people by the way talk to the developers and to the ops

if you have someone at your company with the title devops like devops engineer you probably don't have devops i thought that's concrete proof you do because you have someone whose job it is to do devops we have people with the job title anyways yeah huh huh so how do you convince 100 plus developers that they are going to be on call well first of all have you been on call are you asking me yeah yes i have not okay so i i think i started my first on-call rotation maybe three years ago four years ago and it's never ended yeah yeah that's a long rotation dave you're supposed to go off call i have yeah actually that's funny you would mention that because i was talking to one of

our ops engineers about i don't know four or five years ago and i said how's it going they had been at the company for about a year and they said honestly i'm thinking about changing jobs and i was like what why and they said well i've been on call for like nine months straight and i was like you have we're like yeah so i immediately went over to our cto and i said did you know that our ops engineer actually we i think at the time we called him a devops engineer which clearly demonstrated that we did not understand devops anyway i told my cto did you know he's been on call 24 7 for nine months my cto was like holy crap i had no idea so it was at that moment where we established a developer on call rotation that consisted of about i don't know four or five

developers and four or five uh infrastructure what we called infrastructure engineers and they would both be on call and and i joined the rotation at that time and so i was on call for two weeks out of eight or ten weeks uh ever since then and and even now at my new job i do about the same rotation i've never been part of a formal on-call rotation i think i've worked at startups small enough where it's just whoever cares the most is on call and most of the time that's been me there have been times where it hasn't been me but so so i've i've like kind of been on call for years at a time because i was just like the person who would answer the phone at 2 a.m or whatever but but it it started off like that i've never not been on call and then

had someone say hey jameson now you have to wake up in the middle of the night when stuff goes down where before you didn't before you chose to and now we choose for you to yeah it just feels like a different problem to to take people who aren't doing this and convince them to do it yeah so problem number one is convincing them that it's a good idea because if you can't convince them it's a good idea then no amount of process or other training or anything will fix that yeah as soon as you say that i my brain says a good idea for who it's fine my developer it sounds like a terrible idea yeah i would say that an ideal on-call rotation the customer benefits and not just because you

have more people keeping their software running but because having developers who write the software understand the full end-to-end life cycle and deployment and operational burden of their software will ultimately yield a better product for the customer it'll be more available more scalable and in theory that's the theory right and i i was just thinking to myself if i had to convince someone of this surely i could cite some research that shows that dev teams who on call who run on-call rotations for their software have fewer bugs and fewer outages but i wonder if that's if that exists i'm sure if it does it's not it's not uh it's junk science well i'm sure there are blog posts where someone asserts that and then maybe there's a couple studies that are off sample

sizes of like five people or something i don't know from the 80s yeah so that's that's like the state of the art in software engineering research for most stuff it feels like yeah so you'll probably have a hard time finding like solid research to support this but but somebody wants it to happen well and i i know haven't you i mean you you've run so you've basically been on call for a long time do you find that it influences your day-to-day development decisions oh yeah i sure spend a lot more time on monitoring and logging and things that affect my ability to diagnose and fix problems because it sucks so bad to get woken up in the middle of the night that's that's like the technical argument for it right it makes your software better because you

feel the pain of your software not exactly exactly it's not an externality that's what economists would call it yeah if you're not on call you write bugs and someone else feels the pain yeah yeah which is kind of sick and twisted when you think about it right it is i mean you're kind of arguing that developers need to be even more of generalists like they're already full stack developers and now they need the stack needs to extend further down into infrastructure which that feels a little weird to me i don't know like do front-end developers go on call right good good question i guess that's not the question they're asking though i'm which is like should people go on call should any developers go on call yeah yeah yeah i think you're right though

it's it's it's about the quality of your software and that's a thing that can motivate developers that they want to be craftsmen like they want to build high quality things that are robust and work well and it's a point of pride to build software that's solid and reliable and if if that's the kind of thing that can motivate your team then i think this can be enormously motivating that you get to see how it actually performs in the real world and you get to find the worst parts of it and make it better yeah exactly and if your developers like you were talking about specialization or the opposite of specialization where developers are being asked to learn more and more stuff to be able to be developers i think if you have to go to such lengths to deploy your software it

might be a symptom of having immature operational tools well it's not deploying it's fixing it when it's broken right deploys aren't part of on yeah my bad my bad yeah you're right i was thinking more like uh i use the word deploy but i should have used the word operate basically operations and if you're unable to operate your system diagnose problems and fix them it could be that your tools are immature right your operational stuff is immature i think the problems that you encounter in production will always be hard because as soon as you make the tools better you'll fix all the problems that are fixed by those tools and then the problems that show up now are not fixed by those tools i i don't think you can just say like i don't know i don't think

you can just say we'll use this sas provider and now now it's easy for us to operate our software right you replace all these easy problems with even harder problems yeah i i don't know now when things go wrong it's really you should invest in tooling and and it's not the the end result of that philosophy is nihilism where you say like we just carry floppy disks to someone's house don't deploy anything like invest in tooling yeah but the goal of making it easy to fix every operational outage feels un feels impossible to me yeah because you're weirder you're right and then that's where i'm like i don't know anything about like how the linux kernel handles tcp headers and i read all these blog posts about how that's the problem that caused an outage in

someone's software like i could i don't know i wouldn't be able to like tweak some some bootloader or something and you call yourself a real developer come on not anymore i've been okay let's back yeah what a what an imposter okay so let's let's back up a little bit so i think that if you want the developers to get on board with this idea they need to be part of or at least feel like they are part of the decision making process for going on call i would probably first try to gauge the level of opposition if any with this many developers we're talking 100 plus i'd probably send out a survey and say or maybe even just go in person and sample a few people and ask them point blank would you how do you feel about being on call

24 7 for one week at a time two weeks at a time whatever and just get get some data points on it maybe you have less of an uphill battle than you think and then if that's not surprised yeah that's true and then maybe send out a survey and just see how violently negative they are about it and then uh so then of course you just have to make the decision at some point and this has to come from leadership i think to say we're going to do this and leadership better darn well spell out why and why it's a good idea and who ultimately will benefit what's motivating it and then i would recommend letting developers have a lot of say in the schedule in the process for handing off on call let them own making that possible don't just take your existing process and assume that they

can just fit right into it let them have a seat at the table in designing the new on-call process that makes sense i like that idea of giving them some ownership over over this thing to make it feel less imposed it kind of fits it feels like it feels like it comes from a place of trust more right where you're saying this is the problem we're trying to solve yeah help us solve it instead of saying from monday to tuesday this person will be on call and right that's that's more like marching orders yes i i i think there's gonna i don't know maybe this is just me projecting but i feel like there's gonna be some pushback on this and i think you should prepare for it and be prepared to discuss it with people that feel particularly strongly and then also be

prepared to have to deal with the fact that you're going to make them do it anyways no matter how they feel about it you should probably invest in some good body armor and no not that but just like there there will be it'll make your software better people might be grumpy i mean somebody might quit over this if they hate it the most yeah that's possible you need to be be ready to deal with that i think one thing that i imagine will pop up right away is will ops used to do this what are they going to do now we're taking on all this work that used to be their job like are they just going to chill and relax and ping pong they're going to do what the developers used to do yeah play ping pong in their free time just go home and see their families uh and and i

think that question has a lot of good answers like they can now invest in infrastructure they can build tools that support the engineering team instead of just fight fires um they will be on call also the the load is now shared equally across the team where they were suffering disproportionately before lots of good answers to that question i think it is a question that will be asked though yeah i totally agree uh let's see oh training so uh at my last job we did two kinds of training that worked out really well for developers to come into the on-call rotation the first one was we had like a sit-down lecture style workshop style material where someone who had experience on call would sit down and walk them through common

scenarios things they needed to be aware of and that was pretty effective but what was even more effective was we did a weekly failure exercise on our dev environment where there were two people assigned one person was called the destroyer of worlds and the other and the other person was called uh i can't remember the fixer maybe and the destroyer of worlds would do some nefarious thing to the system like tweak some network config that would break something or like take out a host or just do something to break the system badly or maybe even write a bug that would like generate a a fork bomb or like you know overwhelm our servers or something and then the other person the fixer had to sit there and monitor our systems and try to figure out what

they did and then fix it and they did it uh in the inside of a google hangout when a bunch of people were watching so they could see their screen and watch what they were doing and it was really cool it was both stressful um but also very informative they the the fixer was screen sharing with the whole company while they're fixing stuff no just just the other people who were members of that on-call rotation oh okay interesting it wasn't like a sporting event i mean it kind of sounds like a sporting event well i mean in terms of attendees but yeah it was like that it was really fun and the destroyer of worlds always took it as a challenge to find some weird way that they had seen something break and then you know try to make it break again

and it was really fun huh that's very interesting i mean i've heard of the chaos monkey thing at netflix which is where they break stuff and fix it but i've never heard of that that specific tactic of like and look how the person who's fixing it will fix it that's cool yeah do you feel like it helped oh it absolutely did every single week someone would say i learned about a new tool that i didn't know or i learned about a new setting or i learned about something that uh could go wrong that i'd never seen before you know every week someone learned something yeah i'm not saying it was the most stress-free way to learn oh man so go ahead no you okay i'll go ahead uh did you want to talk about the training thing

because i have another subject uh briefly yeah this is an area where um operations people have a ton of experience around best practices for being on call and so if you don't know where to start there's great books and conference talks and uh google has a site reliability engineering book i think there's a book called effective devops um there's there's a lot of resources out there from people who have been on call for decades and have figured out ways to make that less painful beyond just like sharing the pain with developers i mean you can like part of it is you you need to have uh documentation of common problems and common solutions to problems so if you see this thing here's the steps you take to make sure that's

the problem and fix it and there's a bunch of stuff like that so you're not alone dropped in the middle of uh this scary world full of problems you don't understand yep totally agree and on that same note i will say that you should definitely look into having a shadow process where people can shadow an actual on-call rotation person while they're on call so that they can do that before getting thrown into the fire and you can think of it as like a crawl walk run kind of metaphor, where you should shadow someone for a rotation or two, then you should be on secondary on call for a little while to get used to it, and then go on to primary. And that's a good process. And that's, again, something that the developers and ops teams should work together

to build a process that people will feel comfortable with. Yeah. And the last thing I would say on that is, whatever process you define needs to have an easy and obvious escalation process so if i am on call and i cannot figure out what to do i need to have recourse and it can't just be oh i'll call the secondary it needs to be like i need to be able to call someone who really knows what's going on as a fallback in case i just get stuck so um i think that's a really good good process to have sure that makes sense and then to test that everything's working send your ops team on a two-week all paid all expenses paid vacation they've earned it and watch the world burn i've i've been in places that have accidentally done this and it's been interesting

to see other people step up and develop their skills yeah and i'm sure it could backfire horribly the places i've been at that have done similar things uh ended up maybe problems took a little bit longer to solve but people learned a lot and and the world didn't end don't uh isn't it true that the financial industry has people with like mandatory seven-day vacations that must span one weekend or something i have no idea so that if there's any like weird freaky manual stuff going on that it comes out i've heard of that i've heard that it's like mostly for detecting fraud you know like so they can't be in the office on sunday like yeah shuffling bills into their drawer yeah or like skimming money off of accounts or something and i don't know huh but in this case

you know send your ops team on vacation and let the dev team step up delete their slack accounts yeah cool cool this is a great question um i hope we have shed some light on it and i i would love to hear how it goes over with the team and what you decide to end up doing yeah definitely great question all right question answered question answered what if people want to ask their own question jameson what they can go to and i'm just gonna drag out this question as long as i can i would like to speak please um they can go to softskills.audio there is a link on that website where it takes them to a google form where they can give us as much or as little detail as they want um and that's the main place we take questions from we used to get a lot more over twitter and

we're getting more over the the google form which is good because it lets people give more more detail we can still take them over twitter if you want though our twitter account is at soft skills eng and that is where we also every once in a while tweet stuff yep usually it's just show announcements though so if you're already listening you'll you'll get a reminder um oh if you uh if speaking of tweeting if you have received a soft skills engineering sticker and you have placed it on something take a picture and tweet it to us and we will retweet you as long as the thing it's placed on is appropriate to retweet yeah like no lenovo laptops for example didn't know you were prejudiced against lenovo laptops

uh yeah i was just more thinking like i wrapped this piece of dog poop i found on the ground in a soft skills engineering sticker even though that might be appropriate we would not know we're better than that human poop or bust uh all right when i start making poop jokes that's the sign the episode is over thanks everybody thanks we'll catch you next week